The intention experiment cargo cult: results are in!

You remember Rustum Roy and his attempts to measure changes in water structure caused by people all over the world really, really wanting the structure to change? Yes, the results are in! There's good news and bad news. First the good news: apparently "something happened", but the bad news is that "we're not sure what".

The intention experiment failed to change the structure of water. I'm quite surprised, actually. My prediction that they would find some changes in the water because it was such a poorly designed experiment has not come true. The sample "had enormous variation in light scattering to begin with", but there were no profound changes recorded. I would suggest that the variation is probably instrumental noise, but no actual results have been shown, so it would be impossible to tell even if I knew much about Raman spectroscopy.

Why did the best intentions of so many people fail? You might say "because trying to change water structure through the power of the mind doesn't work". And, to be fair, the intention experiment people have this eighth on their list of possibilities. Because "Although all of our other studies have produced a measurable effect, this hypothesis must always be considered". But not considered that seriously, you understand.

What could the other reasons be? The wildest one is the suggestion that "intention works out of time" (which I suppose is no dafter than imagining that intention works in the first place). Of course, here any recorded change in the water whatsoever, at any time, could be attributed to 'intention', so this isn't what you would call testable. Other suggestions include water purity (even the purest water is grossly contaminated, and having it sat on a lab bench with a probe in it is not going to help), environmental factors (a thunderstorm during the measurement), and problems with the experimental protocol. So the experiment will be repeated in a couple of months.

Some of the comments are fun. One commenter writes "This is what experimentation is all about. You just keep trying until you can verify your conclusions...". Another writes "I'm convinced that intention works. I don't see the need for further experiments". It seems that any negative result would have to be wrong, because we all know that it works. This is how homeopaths think, and is pretty much exactly what is meant by cargo-cult science.

It's worth remembering, of course, that all of this is based on Rustum Roy's evidence-free assertions that there is such a thing as 'healing water', that 'healing water' has a different structure from ordinary water, and that you can change the structure of water by thinking about it. Obviously it would be nice to really, really want a beaker of water to cure cancer, and have it work, but I think it might be worth persevering with proper biomedical research for now.

Andrew Lansley and Nadine Dorries, perhaps unsurprisingly, don't read this blog...

During the second reading debate of the Human Fertilisation and Embryology bill, there was an interesting exchange between Andrew Lansley (the shadow health secretary) and Nadine Dorries MP, on the recent research into survival rates of extremely premature infants.

Lansley talks about the Trent study, that showed no increase in survival rates for infants born before 24 weeks:

The authors also report that no babies born at 22 weeks survived. Like other studies, they suggest that this might represent the limit of viability. I cannot say whether they are right, but I note that, of 150 babies born alive at 22 weeks over the whole period of the study in Trent, only 24 were admitted to intensive care. In the latest period, 2000-05, only nine were admitted to intensive care. From my point of view—I am not a scientist, but I can read statistics—in order to draw any conclusion from such a small number of cases, one would need to assess why those babies were being born so prematurely. Were they failing to thrive? Were they twins or other multiple births? Were they suffering from a genetic defect?

Abortions at 22 weeks would be of a foetus that was otherwise healthy, unless they were carried out on the specific ground of a prospective abnormality or handicap of the child concerned. So while such a baby would require intensive care, and doubtless very large numbers might not survive, I do not regard that study as providing conclusive evidence that a baby cannot be sustained at 22 weeks.

I would personally be loth to move from the principle of linking the time limit for abortion to the viability of the foetus. That is where it was established in 1967; it has moved over time. It is arguable, but I would argue personally that the evidence would support a further reduction—in my view, to 22 weeks—in order to ensure a prospective legal framework that could accommodate improving medical science.

To which Dorries has this to say:

I am sure that my hon. Friend is aware of other studies—for example, in University College hospital, London, and in other countries—showing that if poorly babies who are born prematurely receive immediate good neonatal care from a specialist dedicated team of staff, the outcomes are very much better. University College hospital figures are much better than those emerging from the Trent study. One conclusion from the Trent study is that anyone going into premature labour should go to University College hospital and not to the Trent region.

This is exactly what you can't conclude from a comparison of the Trent and University College studies. It's difficult to compare the studies, because the Trent study includes all infants 'alive at the onset of labour', while the UCLH study included only infants that were born alive. Another issue with the UCLH study is that the numbers of infants admitted to the neonatal intensive care unit varies wildly between the different time periods of the study. This may be because the number of subjects is so small, leading to a large amount of statistical noise. Or it may be because those time periods are not directly comparable, perhaps due to differing policies on which infants should go to the neonatal intensive care unit through time.

And Lansley replies:

My hon. Friend may have gathered from what I said that I am effectively arguing that in circumstances where neonatal intensive care was provided to all such babies born at 22 weeks and where the foetus was otherwise healthy—or rather the baby was healthy at that stage—the prospects for survival at 22 weeks may well be far greater than suggested by the negative view expressed in the study from Nottingham and Leicester. It is not for me to say any more than that, but I think that it provides a basis for reducing the limit—albeit, in my view, by just two weeks.

In fact, the Trent study (Nottingham and Leicester) is more likely to give an accurate idea of the prospects for survival, because it is a multi-centre study (including 16 hospitals in the Trent health region) while the University College study concentrates on a single hospital. So the Trent study is less likely to be confounded by selection bias, and includes a much larger number of subjects, leading to statistically more robust results. It's difficult to see how Lansley can come to the conclusion that the study 'provides a basis for reducing the limit', given that it shows there has been no change in survival rates before 24 weeks since the early 1990s. Happily, the lead author of the Trent study, Prof. David Field of Leicester University, has pointed this out.

There may be other reasons for seeking a reduction in the limit for abortions, but the Trent study is not one of them. To suggest otherwise is simply to misrepresent the research.

Nadine Dorries makes an attempt to look at the evidence. She fails. [updated]

Nadine Dorries seems to have realised that simply dismissing as 'desperate tosh' a study in the British Medical Journal won't cut the mustard. The study looked at survival rates of premature babies in the Trent region of the UK, concluding that survival rates for births before 24 weeks had not increased since 1994-9. This undermines the argument that those survival rates are increasing, and the time limit for abortion should be brought down to 20 weeks from the current 24 weeks. Dorries has now at least attempted to address the evidence, on the 20 weeks campaign website. This is a marked improvement, but I think her effort still falls somewhat short.

First, some nonsense. Dorries writes:

"We were somewhat surprised to see that the article was not in this week's paper edition of the BMJ which many doctors did not receive until Saturday. Why then rush to get it onto the web and into the public domain? The BMJ editorial said that upper time limits were to be debated in the House of Commons 'this year'. What it didn't say is that they are actually to be debated and voted on in just ten days, on 20 May. So the rush to get this into the public domain was to influence that debate".

If you can't use the latest research to 'influence debate', then what is it for? Apart from that, it is perfectly normal for research articles to appear on journal websites some time before they appear in print. That is precisely so that articles can reach the public domain once they are ready for publication, without having to wait on production schedules.

What of the science, indeed. Dorries says that "the Trent study looks at results from 16 hospitals and has been running for years. It is not new and other studies have been published based on more recent data". She also claims that the results from Trent "have always been poor and well below those seen in top neonatal centres worldwide". As evidence for this, she cites a study from the University College London Hospital (UCLH). This study is a single centre cohort study comparing survival rates in the years 1981-85, 1986-90, 1991-95, and 1996-2000. [The first thing to note here is that the Trent study uses data up to 2005. So it is actually using much more recent data than the UCLH study.] The UCLH study found that "There was a progressive increase in survival at all gestational ages over the 20-year period".

This seems to be a classic case of how you have to be careful with research. The first thing to note is that the number of babies in the UCLH study is small. The total number of babies in the study was 12 for those born at 22 weeks gestation, and 56 for those born at 23 weeks gestation, over 20 years. This compares with 261 born at 22 weeks and 370 at 23 weeks, over 10 years, for the Trent study. The Trent study is likely to give more robust numbers. Dorries, on the other hand, writes that the UCLH study "showed no survivors at 22 or 23 weeks in 1981-85 but 71% (5/7) and 47% (8/17) respectively in 1996-2000". Note the tiny sample size this statement is based on. In the UCLH study, the authors don't give percentage survival rates for births at 22 weeks, because the sample sizes are simply too small to draw any conclusions. There is something odd about the admission figures here too. For births at 22 weeks gestation, there was 1 admission in 1981-85, 3 in 1986-90, 1 in 1991-95, and 7 in 1996-2000. The comparable figures for births at 23 weeks are 7, 23, 9 and 17. There are very large variations here that suggest that the different time periods may not be directly comparable.

Perhaps the most serious problem with comparing the Trent and UCLH studies, though, is that they don't appear to be measuring the same things. In the Trent study, the authors write "We included in the study infants alive at the onset of labour". In contrast, the UCLH authors write "All infants born between 22 and 25⁺⁶ weeks of gestation between 1981 and 2000, who were admitted to the tertiary neonatal intensive care unit of UCLH (UK), within 1 week of birth, were enrolled into our study. In addition, the UCLH labour records for the years 1991–2000 were scrutinised to identify all infants who were born alive between 22 and 25⁺⁶ weeks of gestation, but who died in the delivery room". Hence the Trent study includes still-births as deaths, but the UCLH study does not. According to the Trent study, of births at 22 weeks, 43% were stillborn in 1994-99, and 40% in 2000-5. Of births at 23 weeks, 28% were stillborn in 1994-99, and 20% in 2000-5. For the UCLH study we don't know what these rates are. The Ministry of Truth blog has noted an anomaly in that the UCLH study reports a massively lower proportion of deaths in the delivery room compared to the Trent study. So there is at least a possibility that the UCLH survival rates are inflated in comparison to the Trent study by selection bias: perhaps more infants are being admitted to neonatal intensive care in Trent than in UCLH.

It's also worth noting that the Trent study covered a whole area (the Trent health region in the UK), while the UCLH study covered a single hospital. As the authors of the UCLH study write, "Large population-based studies [such as the Trent study] have the advantage of avoiding problems of selection bias and enable the accumulation of large numbers to reduce statistical errors. However, they inevitably represent the outcome from a very heterogeneous range of perinatal facilities...It is therefore inevitable that these units encompassed a wide range of staffing, resources and expertise in the care of extremely preterm infants. In addition, it seems likely that the maternity units encompassed a range of management policies (both obstetric and paediatric) towards infants born at the limits of viability, ranging from ‘proactive’ to ‘noninterventionist’. Hence, it is not surprising that average survival figures obtained from large population based studies will be different from those observed from single centre cohorts". As the editorial in the BMJ that accompanies the Trent studies says, "Single centre studies are confounded by selection bias and tend to overestimate the likelihood of survival". In other words, to compare the Trent and UCLH studies in the way Dorries is trying to do, and draw the conclusion that survival rates in the Trent health region are 'poor', is not a sensible thing to do.

So Dorries has looked at the evidence, which is better than dismissing it as 'tosh'. However, she seems to be drawing erroneous conclusions from an inappropriate comparison of two different studies. By coincidence, this inappropriate comparison supports Dorrie's agenda. How convenient.

Yes, but what about the evidence? Nadine Dorries and the BMJ

This morning my Grauniad was carrying a story about new research, published in the BMJ, that suggests survival rates for premature babies born before 24 weeks are very low and have not improved, despite advances in medical care.

Now, I haven't actually read the whole study, but I've glanced at the abstract. It's a cohort study that looks at births of babies between 20 and 26 weeks in the Trent health region in the UK over two time periods: 1994-9 and 2000-5. The study found that although there was a significant increase in the number of surviving babies in 2000-5, this was attributable to an increase in survival rates for babies born at 24 and 25 weeks. There was no significant increase in survival rates for babies born before 24 weeks.

This is an important finding because the House of Commons will soon debate whether to lower the lower the legal cut-off point for most abortions from 24 weeks to 20 weeks. One of the arguments has been that advances in medical science mean that more and more babies born before 24 weeks are surviving. The research shows that this appears not to be the case.

What to do if you're on the side of the debate that wants a reduction to 20 weeks? If you're Nadine Dorries MP, you go in for some textbook fallacious reasoning, relying in particular on ad hominem and argument from incredulity.

No improvement in neo-natal care in twelve years? Really? So where has all the money that has been pumped into neo-natal services gone then? A baby born at 23 weeks today stands no better a chance of living than it did in 1996? This report is the most desperate piece of tosh produced by the pro-choice lobby and it smells of one thing, desperation.

Well, the evidence seems to suggest that a baby born at 23 weeks today really does stand no better chance of living than it did in 1996. It's no good just saying you don't believe it; you need to explain why that interpretation of the evidence is incorrect. Dorries also describes the British Medical Journal as the "trade magazine" of the British Medical Association. Because the BMA funds a pro-choice group, research published in the BMJ can be dismissed as emanating from the "pro-abortion lobby". In fact, the BMJ is a peer-reviewed medical journal, publishing "original scientific studies, review and educational articles, and papers commenting on the clinical, scientific, social, political, and economic factors affecting health." It is something more than a trade magazine, and for Dorries to describe it as such is disingenuous to say the least.

As I say, I have not read the whole paper. For all I know, there may be some problems with it. But Dorries is simply not contributing to a useful debate by dismissing the study out of hand. The paper gives us the evidence, and it is up to us, as a society, to decide what to do about it. We can't do that by falling back on fallacious reasoning that supports our pre-conceptions.

Update: Dorries continues to mis-represent the BMJ paper, writing that "For the BMA to argue that the survival rates of neo nates born 12 years ago should endorse the 24 week limit today, is, as I said yesterday, an act of desperation, but one that suits me and the media opportunities such a biased piece of 'research' provides." First of all, it is not the BMA that is doing the arguing. The research was conducted by a team working in the Department of Health Sciences at the University of Leicester. Second, the paper compares data from 1994-9 to data from 2000-5. Where Dorries gets '12 years ago' from is anyone's guess. Again, it is no good simply stating that the work is 'biased'. You need to explain why it is biased. I'm fairly sure that Dorries can't do this.

Creationism, language and structural geology

I wrote a little bit about this elsewhere, but I thought it was of interest to a 'bad science' audience. Dr. David Peacock, a consultant with Fugro-Robertson and well-known structural geologist, has just published a letter to the editor [extremely irritating paywall] in the May issue of the Journal of Structural Geology. The letter criticises the use of the term 'architecture' in published articles in the Journal of Structural Geology. In general, structural geologists use the term to describe the spatial arrangement of structural elements within larger structures, such as fault zones or geological basins.

So far, this is just an argument about arcane terminology in a fairly obscure field of scientific endeavour. However, one of Dr. Peacock's arguments against using 'architecture' is that it introduces a 'hint of creationism'. "Who is the architect?", he asks, going on to ask "Does use of the term imply divine construction or a belief in intelligent design?"

The answer to this latter question is clearly no. Geology has always borrowed terms from architecture (for a few examples, see here). This happens because some geological features are similar to features from the built environment (for example, here's a gratuitous picture of some nice folds). That doesn't imply that someone or something intelligent built them. More importantly, it seems to me that we would have to stop talking about structural geology entirely if we accepted Dr. Peacock's contention. The fact that the Earth has structure is what structural geologists study. So if you ask 'Who is the architect?', you also ask 'Who imparted the structure?' I make this point in a reply [paywall] to Dr. Peacock's letter, which at the time of writing is in press. The text can also be found here.

The wider point here is that the idea that any implication of 'structure' is an implication of 'design' is a classic creationist mis-direction. This is a point made eloquently in the context of evolution by Steve Novella. In structural geology, 'structure' and 'order' emerge from naturally occurring processes. There is no reason to think that saying a fault zone has an 'architecture' is also saying that some intelligence designed the fault zone.

Rustum Roy's intention experiment: we are all Qigong masters

Apparently, Manju Rao and Rustum Roy are busily working on the results from their Intention Experiment, where people all over the world concentrated on changing the Raman spectrum of a beaker of water. However, some preliminary results are filtering out for our entertainment.

According to the Intention Experiment blog, "Although the scientists are not finished examining their data, they have told me one thing: they’ve seen results they’ve never seen before with their equipment". This suggests to me that there was a problem with the experimental set-up, which of course has not been described in any detail.

Then "One reason it is taking so long is that our water had a great deal of variation an hour before the experiment was run. This could mean that our anticipation of the event began to affect the water. Or it could mean that our hypothesis is wrong. Or it could mean that with intention, we are emanating an energy like a Qigong master, which is being picked up by the spectroscopy before the event". Or could it mean that the impurities in the water are changing through time as it sits in an open beaker on a lab bench? Obviously, emanating energy like a Qigong master is a much more plausible explanation, but might it not be a good idea to control the experiment just in case? Also, looking at Rao and Roy's previous work on homeopathic remedies, it seems that 'stray light is eliminated by turning off all the room lights whenever data are being collected'. Unless they are in a proper darkroom, that isn't going to be enough to keep out all the ambient light. Raman is also noisy and highly sensitive to impurities.

There's also some weirdness going on here, as this page talks about taking a Raman spectrum with the 'laser turned off'. With the laser off, there is no incident beam to undergo Raman scattering, so I'm simply not clear what they could be talking about here at all. A 'member of the Penn State team' also says that "The laser light is absorbed by the water molecules, depending on how they are energetically configured or arranged, and then reradiated at a different wavelength". As I understand it, this isn't quite right: the incident beam is scattered, with a wavelength shift of the scattered beam. There is no absorption and re-radiation in Raman scattering. Do the Penn State team have any idea what they're doing?

I can't wait for the final results...

Homeopathy and wikipedia

Homeopathy-related articles on Wikipedia have been subject to probation for some time, because of edit wars between pro-homeopathy and anti-homeopathy contributors. The situation has become such a shambles that Wikipedia has opened an arbitration case on all its homeopathy-related pages. As a result of his editing behaviour, Dana Ullman has managed to get himself banned from editing any homeopathy-related pages for a period of three months.

The arbitration page is long and parts of it are somewhat tedious, but it is interesting in that it demonstrates some of the problems science has in presenting evidence to laypeople in a clear way. The literature on homeopathy is complex, consisting of a mixture of poorly conducted studies that show some (at best equivocal) evidence that homeopathy might work, and well-designed studies that show homeopathy has no effect. (A meta-analysis by Shang et al. that was published in the Lancet elegantly showed that the largest and least biased studies showed the least effect for homeopathy). It is then easy for advocates of homeopathy to emphasise the apparently positive studies, and claim that sceptics of homeopathy do not consider all the evidence. In other words, advocates of homeopathy take the studies at face value; they don't look at the studies and analyse them to see how they were conducted and whether the results make sense.

Because there are some people who think homeopathy works, and there are some peer-reviewed journals of homeopathy and other 'alternative' therapies that superficially appear to be scientific, it is possible to claim that there is some scientific controversy about the effectiveness of homeopathy. As Wikipedia is based around the idea of neutral point of view (NPOV), it seems superficially sensible to present a 'balanced' view whereby evidence for and against homeopathy is described. But, crucially, this is not a balanced view; it gives undue weight to poorly conducted studies. Looking at the evidence as a whole shows that homeopathy is nothing more than a placebo. Quite apart from that, it contains ideas (e.g. that the 'potency' of a remedy increases the more it is diluted) that are in conflict with well established science. A neutral point of view is surely that homeopathy is extremely implausible and that there is no evidence that it works.

This sort of thing is not just a problem for Wikipedia. There is a general problem of spurious 'balance' in scientific reporting. Another clear example is the BBC linking to anti-vaccine websites in stories about vaccines. Science is a notoriously complicated business, but there are times when the scientific evidence is clear, when it is evaluated properly. The challenge is to show that this is the case, without coming across as some kind of monolithic establishment that wishes to crush all dissent.