A lesson in how to get a false positive result

Back to Rustum Roy and his ongoing attempt to show that water can have a 'healing' structure (whatever that means) imposed on it by the power of human intention. As Le Canard Noir said, in the comments to a previous post, this is car crash science. Although I know I should look away, I can't quite manage to do it.

Previously, Roy and his team tried to show that the Raman spectrum of a beaker of water could be changed merely by people all over the world thinking about it changing. Despite the horrible design of the experiment, the results were inconclusive. What Roy needs is a way of increasing the chances of getting a positive result. Another experiment is now planned, details of which are beginning to emerge at the Intention Experiment blog:

As you remember, our experiment isn’t conclusive – largely because we’re only looking at one parameter to see if it has changes.

This is a little like looking at an elephant from one side. If you look from the front, you will mainly see a trunk. Look from bottom, and you only see a giant mass hovering over you like a dark grey cloud.

Rainen’s new equipment consists of three separate devices that examine, respectively, the light scattering, the thermal expansion and any infrared changes in a sample of water. Once these measurements are taken, they are sent into a computer, and from this handful of data points, the computer can determine some 1000 parameters of the sample.

“This equipment represents a revolution in characterizing water,” says Roy.

From this, it sounds as though Roy and team are going to be comparing 1,000 variables. This raises the issue of multiple comparisons. If you compare 1,000 variables between two populations, using a hypothesis test at the 5% level, you would expect to get 'positive' results for 50 of the variables, even if there was no real difference between the two populations. Statisticians apply corrections to account for this effect. Since Roy's team have previously published a paper that purported to show differences between graphs without applying any statistical analysis at all, it's not certain that this will be done. Another issue is that these 1,000 parameters are derived from 'a handful' of measurements, so they presumably cannot be independent parameters. It seems that false positive results are much more likely from the new experiment. Result!

Science causes earthquakes...

Further evidence that the Independent will print any old rubbish. Richard Lapthorne, in a letter to the editor, writes:

"Given acceptance of the geologists' explanation of earthquakes as readjustments in the earth's crust, perhaps Bruce Homes (letters, 22 May) would see the sense of attributing them to acts of science rather than acts of God."

Scientific observation has revealed that "readjustments in the earth's crust", along fault zones, cause earthquakes. Ergo, science causes earthquakes. You just can't argue with that kind of logic. But why print these ravings in a national newspaper? I know the Independent is rapidly becoming a joke, but why accelerate the process?

42 days: get writing, folks

The report stage of the counter-terrorism bill is coming up. This bill would allow pre-charge detention of terrorist suspects for up to 42 days. The current limit is 28 days, already significantly higher than most comparable democracies. No good case has been made for this erosion of civil liberties, and it's time to get writing to your MP. The following went off to my MP today:

I am writing to express my concerns at the proposal in the counter-terrorism bill to extend pre-charge detention to 42 days in certain circumstances. This is a considerable extension to current powers, and for that reason a clear case needs to be made for the necessity of the measure. In my opinion, this has not been done. I note that the Director of Public Prosecutions, Sir Ken McDonald, and Lord Goldsmith, the former Attorney General, have stated that they see no need for an extension to pre-charge detention. I also note that Lord Dear, the former chief inspector of constabulary, has stated that most chief constables he has spoken to do not think a 42-day detention period is necessary. This raises the question of who exactly does think this provision is necessary.

As I understand them, the proposals would provide for the Home Secretary to make a statement to parliament, and for parliament to vote on an extension to pre-charge detention. However, it is difficult to see how the Home Secretary can come before parliament to discuss individual cases because of the risk of prejudicing them.

The proposals seem to be poorly thought out, unnecessary and unworkable, and I hope you will carefully consider the implications of the proposed legislation for civil liberties before the report stage of the bill.

Sincerely,

Paul Wilson

Nice to see...

I was happy to see this article in the Guardian on Saturday. Edzard Ernst holds the Laing chair in complementary medicine at the University of Exeter, and has published a book with Simon Singh on CAM, entitled "Trick or Treatment? Alternative Medicine on Trial". At the Hay literary festival, Ernst was talking about the book. He laid into pharmacists in general, and Boots in particular, for selling homeopathic remedies. There is no good evidence that such remedies work, and they have no plausible mode of action, as they are generally so diluted that the chance of them containing any of the purported active ingredient is vanishingly small.

Ernst is gratifyingly direct, saying that Boots "seems to be fast becoming the biggest seller of quack remedies in UK high streets". He also said that pharmacists have a responsibility to adhere to the code of ethics of the Royal Pharmaceutical Society: "This can only mean that pharmacists should tell their customers that a homeopathic remedy they are about to buy doesn't contain a single molecule of whatever it says on the label, and that there's no clinical evidence that it works beyond a placebo effect".

I couldn't agree more, and it's good to see someone come out and say these things clearly and publicly.

The intention experiment cargo cult: results are in!

You remember Rustum Roy and his attempts to measure changes in water structure caused by people all over the world really, really wanting the structure to change? Yes, the results are in! There's good news and bad news. First the good news: apparently "something happened", but the bad news is that "we're not sure what".

The intention experiment failed to change the structure of water. I'm quite surprised, actually. My prediction that they would find some changes in the water because it was such a poorly designed experiment has not come true. The sample "had enormous variation in light scattering to begin with", but there were no profound changes recorded. I would suggest that the variation is probably instrumental noise, but no actual results have been shown, so it would be impossible to tell even if I knew much about Raman spectroscopy.

Why did the best intentions of so many people fail? You might say "because trying to change water structure through the power of the mind doesn't work". And, to be fair, the intention experiment people have this eighth on their list of possibilities. Because "Although all of our other studies have produced a measurable effect, this hypothesis must always be considered". But not considered that seriously, you understand.

What could the other reasons be? The wildest one is the suggestion that "intention works out of time" (which I suppose is no dafter than imagining that intention works in the first place). Of course, here any recorded change in the water whatsoever, at any time, could be attributed to 'intention', so this isn't what you would call testable. Other suggestions include water purity (even the purest water is grossly contaminated, and having it sat on a lab bench with a probe in it is not going to help), environmental factors (a thunderstorm during the measurement), and problems with the experimental protocol. So the experiment will be repeated in a couple of months.

Some of the comments are fun. One commenter writes "This is what experimentation is all about. You just keep trying until you can verify your conclusions...". Another writes "I'm convinced that intention works. I don't see the need for further experiments". It seems that any negative result would have to be wrong, because we all know that it works. This is how homeopaths think, and is pretty much exactly what is meant by cargo-cult science.

It's worth remembering, of course, that all of this is based on Rustum Roy's evidence-free assertions that there is such a thing as 'healing water', that 'healing water' has a different structure from ordinary water, and that you can change the structure of water by thinking about it. Obviously it would be nice to really, really want a beaker of water to cure cancer, and have it work, but I think it might be worth persevering with proper biomedical research for now.

Andrew Lansley and Nadine Dorries, perhaps unsurprisingly, don't read this blog...

During the second reading debate of the Human Fertilisation and Embryology bill, there was an interesting exchange between Andrew Lansley (the shadow health secretary) and Nadine Dorries MP, on the recent research into survival rates of extremely premature infants.

Lansley talks about the Trent study, that showed no increase in survival rates for infants born before 24 weeks:

The authors also report that no babies born at 22 weeks survived. Like other studies, they suggest that this might represent the limit of viability. I cannot say whether they are right, but I note that, of 150 babies born alive at 22 weeks over the whole period of the study in Trent, only 24 were admitted to intensive care. In the latest period, 2000-05, only nine were admitted to intensive care. From my point of view—I am not a scientist, but I can read statistics—in order to draw any conclusion from such a small number of cases, one would need to assess why those babies were being born so prematurely. Were they failing to thrive? Were they twins or other multiple births? Were they suffering from a genetic defect?

Abortions at 22 weeks would be of a foetus that was otherwise healthy, unless they were carried out on the specific ground of a prospective abnormality or handicap of the child concerned. So while such a baby would require intensive care, and doubtless very large numbers might not survive, I do not regard that study as providing conclusive evidence that a baby cannot be sustained at 22 weeks.

I would personally be loth to move from the principle of linking the time limit for abortion to the viability of the foetus. That is where it was established in 1967; it has moved over time. It is arguable, but I would argue personally that the evidence would support a further reduction—in my view, to 22 weeks—in order to ensure a prospective legal framework that could accommodate improving medical science.

To which Dorries has this to say:

I am sure that my hon. Friend is aware of other studies—for example, in University College hospital, London, and in other countries—showing that if poorly babies who are born prematurely receive immediate good neonatal care from a specialist dedicated team of staff, the outcomes are very much better. University College hospital figures are much better than those emerging from the Trent study. One conclusion from the Trent study is that anyone going into premature labour should go to University College hospital and not to the Trent region.

This is exactly what you can't conclude from a comparison of the Trent and University College studies. It's difficult to compare the studies, because the Trent study includes all infants 'alive at the onset of labour', while the UCLH study included only infants that were born alive. Another issue with the UCLH study is that the numbers of infants admitted to the neonatal intensive care unit varies wildly between the different time periods of the study. This may be because the number of subjects is so small, leading to a large amount of statistical noise. Or it may be because those time periods are not directly comparable, perhaps due to differing policies on which infants should go to the neonatal intensive care unit through time.

And Lansley replies:

My hon. Friend may have gathered from what I said that I am effectively arguing that in circumstances where neonatal intensive care was provided to all such babies born at 22 weeks and where the foetus was otherwise healthy—or rather the baby was healthy at that stage—the prospects for survival at 22 weeks may well be far greater than suggested by the negative view expressed in the study from Nottingham and Leicester. It is not for me to say any more than that, but I think that it provides a basis for reducing the limit—albeit, in my view, by just two weeks.

In fact, the Trent study (Nottingham and Leicester) is more likely to give an accurate idea of the prospects for survival, because it is a multi-centre study (including 16 hospitals in the Trent health region) while the University College study concentrates on a single hospital. So the Trent study is less likely to be confounded by selection bias, and includes a much larger number of subjects, leading to statistically more robust results. It's difficult to see how Lansley can come to the conclusion that the study 'provides a basis for reducing the limit', given that it shows there has been no change in survival rates before 24 weeks since the early 1990s. Happily, the lead author of the Trent study, Prof. David Field of Leicester University, has pointed this out.

There may be other reasons for seeking a reduction in the limit for abortions, but the Trent study is not one of them. To suggest otherwise is simply to misrepresent the research.

Nadine Dorries makes an attempt to look at the evidence. She fails. [updated]

Nadine Dorries seems to have realised that simply dismissing as 'desperate tosh' a study in the British Medical Journal won't cut the mustard. The study looked at survival rates of premature babies in the Trent region of the UK, concluding that survival rates for births before 24 weeks had not increased since 1994-9. This undermines the argument that those survival rates are increasing, and the time limit for abortion should be brought down to 20 weeks from the current 24 weeks. Dorries has now at least attempted to address the evidence, on the 20 weeks campaign website. This is a marked improvement, but I think her effort still falls somewhat short.

First, some nonsense. Dorries writes:

"We were somewhat surprised to see that the article was not in this week's paper edition of the BMJ which many doctors did not receive until Saturday. Why then rush to get it onto the web and into the public domain? The BMJ editorial said that upper time limits were to be debated in the House of Commons 'this year'. What it didn't say is that they are actually to be debated and voted on in just ten days, on 20 May. So the rush to get this into the public domain was to influence that debate".

If you can't use the latest research to 'influence debate', then what is it for? Apart from that, it is perfectly normal for research articles to appear on journal websites some time before they appear in print. That is precisely so that articles can reach the public domain once they are ready for publication, without having to wait on production schedules.

What of the science, indeed. Dorries says that "the Trent study looks at results from 16 hospitals and has been running for years. It is not new and other studies have been published based on more recent data". She also claims that the results from Trent "have always been poor and well below those seen in top neonatal centres worldwide". As evidence for this, she cites a study from the University College London Hospital (UCLH). This study is a single centre cohort study comparing survival rates in the years 1981-85, 1986-90, 1991-95, and 1996-2000. [The first thing to note here is that the Trent study uses data up to 2005. So it is actually using much more recent data than the UCLH study.] The UCLH study found that "There was a progressive increase in survival at all gestational ages over the 20-year period".

This seems to be a classic case of how you have to be careful with research. The first thing to note is that the number of babies in the UCLH study is small. The total number of babies in the study was 12 for those born at 22 weeks gestation, and 56 for those born at 23 weeks gestation, over 20 years. This compares with 261 born at 22 weeks and 370 at 23 weeks, over 10 years, for the Trent study. The Trent study is likely to give more robust numbers. Dorries, on the other hand, writes that the UCLH study "showed no survivors at 22 or 23 weeks in 1981-85 but 71% (5/7) and 47% (8/17) respectively in 1996-2000". Note the tiny sample size this statement is based on. In the UCLH study, the authors don't give percentage survival rates for births at 22 weeks, because the sample sizes are simply too small to draw any conclusions. There is something odd about the admission figures here too. For births at 22 weeks gestation, there was 1 admission in 1981-85, 3 in 1986-90, 1 in 1991-95, and 7 in 1996-2000. The comparable figures for births at 23 weeks are 7, 23, 9 and 17. There are very large variations here that suggest that the different time periods may not be directly comparable.

Perhaps the most serious problem with comparing the Trent and UCLH studies, though, is that they don't appear to be measuring the same things. In the Trent study, the authors write "We included in the study infants alive at the onset of labour". In contrast, the UCLH authors write "All infants born between 22 and 25⁺⁶ weeks of gestation between 1981 and 2000, who were admitted to the tertiary neonatal intensive care unit of UCLH (UK), within 1 week of birth, were enrolled into our study. In addition, the UCLH labour records for the years 1991–2000 were scrutinised to identify all infants who were born alive between 22 and 25⁺⁶ weeks of gestation, but who died in the delivery room". Hence the Trent study includes still-births as deaths, but the UCLH study does not. According to the Trent study, of births at 22 weeks, 43% were stillborn in 1994-99, and 40% in 2000-5. Of births at 23 weeks, 28% were stillborn in 1994-99, and 20% in 2000-5. For the UCLH study we don't know what these rates are. The Ministry of Truth blog has noted an anomaly in that the UCLH study reports a massively lower proportion of deaths in the delivery room compared to the Trent study. So there is at least a possibility that the UCLH survival rates are inflated in comparison to the Trent study by selection bias: perhaps more infants are being admitted to neonatal intensive care in Trent than in UCLH.

It's also worth noting that the Trent study covered a whole area (the Trent health region in the UK), while the UCLH study covered a single hospital. As the authors of the UCLH study write, "Large population-based studies [such as the Trent study] have the advantage of avoiding problems of selection bias and enable the accumulation of large numbers to reduce statistical errors. However, they inevitably represent the outcome from a very heterogeneous range of perinatal facilities...It is therefore inevitable that these units encompassed a wide range of staffing, resources and expertise in the care of extremely preterm infants. In addition, it seems likely that the maternity units encompassed a range of management policies (both obstetric and paediatric) towards infants born at the limits of viability, ranging from ‘proactive’ to ‘noninterventionist’. Hence, it is not surprising that average survival figures obtained from large population based studies will be different from those observed from single centre cohorts". As the editorial in the BMJ that accompanies the Trent studies says, "Single centre studies are confounded by selection bias and tend to overestimate the likelihood of survival". In other words, to compare the Trent and UCLH studies in the way Dorries is trying to do, and draw the conclusion that survival rates in the Trent health region are 'poor', is not a sensible thing to do.

So Dorries has looked at the evidence, which is better than dismissing it as 'tosh'. However, she seems to be drawing erroneous conclusions from an inappropriate comparison of two different studies. By coincidence, this inappropriate comparison supports Dorrie's agenda. How convenient.

Yes, but what about the evidence? Nadine Dorries and the BMJ

This morning my Grauniad was carrying a story about new research, published in the BMJ, that suggests survival rates for premature babies born before 24 weeks are very low and have not improved, despite advances in medical care.

Now, I haven't actually read the whole study, but I've glanced at the abstract. It's a cohort study that looks at births of babies between 20 and 26 weeks in the Trent health region in the UK over two time periods: 1994-9 and 2000-5. The study found that although there was a significant increase in the number of surviving babies in 2000-5, this was attributable to an increase in survival rates for babies born at 24 and 25 weeks. There was no significant increase in survival rates for babies born before 24 weeks.

This is an important finding because the House of Commons will soon debate whether to lower the lower the legal cut-off point for most abortions from 24 weeks to 20 weeks. One of the arguments has been that advances in medical science mean that more and more babies born before 24 weeks are surviving. The research shows that this appears not to be the case.

What to do if you're on the side of the debate that wants a reduction to 20 weeks? If you're Nadine Dorries MP, you go in for some textbook fallacious reasoning, relying in particular on ad hominem and argument from incredulity.

No improvement in neo-natal care in twelve years? Really? So where has all the money that has been pumped into neo-natal services gone then? A baby born at 23 weeks today stands no better a chance of living than it did in 1996? This report is the most desperate piece of tosh produced by the pro-choice lobby and it smells of one thing, desperation.

Well, the evidence seems to suggest that a baby born at 23 weeks today really does stand no better chance of living than it did in 1996. It's no good just saying you don't believe it; you need to explain why that interpretation of the evidence is incorrect. Dorries also describes the British Medical Journal as the "trade magazine" of the British Medical Association. Because the BMA funds a pro-choice group, research published in the BMJ can be dismissed as emanating from the "pro-abortion lobby". In fact, the BMJ is a peer-reviewed medical journal, publishing "original scientific studies, review and educational articles, and papers commenting on the clinical, scientific, social, political, and economic factors affecting health." It is something more than a trade magazine, and for Dorries to describe it as such is disingenuous to say the least.

As I say, I have not read the whole paper. For all I know, there may be some problems with it. But Dorries is simply not contributing to a useful debate by dismissing the study out of hand. The paper gives us the evidence, and it is up to us, as a society, to decide what to do about it. We can't do that by falling back on fallacious reasoning that supports our pre-conceptions.

Update: Dorries continues to mis-represent the BMJ paper, writing that "For the BMA to argue that the survival rates of neo nates born 12 years ago should endorse the 24 week limit today, is, as I said yesterday, an act of desperation, but one that suits me and the media opportunities such a biased piece of 'research' provides." First of all, it is not the BMA that is doing the arguing. The research was conducted by a team working in the Department of Health Sciences at the University of Leicester. Second, the paper compares data from 1994-9 to data from 2000-5. Where Dorries gets '12 years ago' from is anyone's guess. Again, it is no good simply stating that the work is 'biased'. You need to explain why it is biased. I'm fairly sure that Dorries can't do this.

Creationism, language and structural geology

I wrote a little bit about this elsewhere, but I thought it was of interest to a 'bad science' audience. Dr. David Peacock, a consultant with Fugro-Robertson and well-known structural geologist, has just published a letter to the editor [extremely irritating paywall] in the May issue of the Journal of Structural Geology. The letter criticises the use of the term 'architecture' in published articles in the Journal of Structural Geology. In general, structural geologists use the term to describe the spatial arrangement of structural elements within larger structures, such as fault zones or geological basins.

So far, this is just an argument about arcane terminology in a fairly obscure field of scientific endeavour. However, one of Dr. Peacock's arguments against using 'architecture' is that it introduces a 'hint of creationism'. "Who is the architect?", he asks, going on to ask "Does use of the term imply divine construction or a belief in intelligent design?"

The answer to this latter question is clearly no. Geology has always borrowed terms from architecture (for a few examples, see here). This happens because some geological features are similar to features from the built environment (for example, here's a gratuitous picture of some nice folds). That doesn't imply that someone or something intelligent built them. More importantly, it seems to me that we would have to stop talking about structural geology entirely if we accepted Dr. Peacock's contention. The fact that the Earth has structure is what structural geologists study. So if you ask 'Who is the architect?', you also ask 'Who imparted the structure?' I make this point in a reply [paywall] to Dr. Peacock's letter, which at the time of writing is in press. The text can also be found here.

The wider point here is that the idea that any implication of 'structure' is an implication of 'design' is a classic creationist mis-direction. This is a point made eloquently in the context of evolution by Steve Novella. In structural geology, 'structure' and 'order' emerge from naturally occurring processes. There is no reason to think that saying a fault zone has an 'architecture' is also saying that some intelligence designed the fault zone.

Rustum Roy's intention experiment: we are all Qigong masters

Apparently, Manju Rao and Rustum Roy are busily working on the results from their Intention Experiment, where people all over the world concentrated on changing the Raman spectrum of a beaker of water. However, some preliminary results are filtering out for our entertainment.

According to the Intention Experiment blog, "Although the scientists are not finished examining their data, they have told me one thing: they’ve seen results they’ve never seen before with their equipment". This suggests to me that there was a problem with the experimental set-up, which of course has not been described in any detail.

Then "One reason it is taking so long is that our water had a great deal of variation an hour before the experiment was run. This could mean that our anticipation of the event began to affect the water. Or it could mean that our hypothesis is wrong. Or it could mean that with intention, we are emanating an energy like a Qigong master, which is being picked up by the spectroscopy before the event". Or could it mean that the impurities in the water are changing through time as it sits in an open beaker on a lab bench? Obviously, emanating energy like a Qigong master is a much more plausible explanation, but might it not be a good idea to control the experiment just in case? Also, looking at Rao and Roy's previous work on homeopathic remedies, it seems that 'stray light is eliminated by turning off all the room lights whenever data are being collected'. Unless they are in a proper darkroom, that isn't going to be enough to keep out all the ambient light. Raman is also noisy and highly sensitive to impurities.

There's also some weirdness going on here, as this page talks about taking a Raman spectrum with the 'laser turned off'. With the laser off, there is no incident beam to undergo Raman scattering, so I'm simply not clear what they could be talking about here at all. A 'member of the Penn State team' also says that "The laser light is absorbed by the water molecules, depending on how they are energetically configured or arranged, and then reradiated at a different wavelength". As I understand it, this isn't quite right: the incident beam is scattered, with a wavelength shift of the scattered beam. There is no absorption and re-radiation in Raman scattering. Do the Penn State team have any idea what they're doing?

I can't wait for the final results...