More fall-out from the Homeopathy memory of water issue.
One of the papers in the Homeopathy special issue was by Rao et al., involving work by Rustum Roy, and had been touted around various places by Dana Ullman as evidence that homeopathy could work. The paper purported to use spectroscopic techniques to differentiate between homeopathic remedies, showing that there was some difference between them. It was fairly comprehensively taken apart by people at the JREF forums (including a minor contribution from me), and eventually a letter was submitted to the editor of Homeopathy outlining our concerns. This letter has now been published, alongside a response by Rao.
Hopefully, the other authors will forgive me if I summarise the points in the letter as follows:
1. The presented spectrum of pure ethanol looks nothing like ethanol of any recognised degree of purity.
2. There is no way of knowing whether the 'differences' between different preparations could have been caused by them being prepared with solvent containing different levels of impurities.
3. No statistical information was presented, meaning that no conclusions could be drawn as to whether the remedies were actually different.
4. Spectra that were supposedly representative of the tested preparations were actually the more extreme examples.
5. One graph was reproduced twice, purporting to show different things each time.
You'd imagine that all this was fairly damning, so how did the authors respond? On all of these points, answer came there none. Regarding the possibility of different levels of contamination between different samples, Rao simply says that this "illustrates one of the values of our work to homeopathy producers and users, and other health researchers. We have provided them with potential quality control tools, and a refining of the arguments away from incorrect generalizations". On the critical issue of contamination potentially invalidating the results of the paper, astoundingly, there is no comment.
On statistical analysis, Rao writes "All the analytical data shown in the paper are the result of reproducible analyses, although we appreciate the suggestion of representing such data as an average with a standard deviation, we emphasize that our key identification by display of an envelope demonstrates that, there are indeed differences beyond the standard deviation range among individual homeopathic remedies, as used in practice". What exactly the 'envelope' actually is has never been explained. It is certainly not a standard technique for statistical analysis. Without knowing what the standard deviation range actually is, it is impossible to demonstrate differences beyond it. While it could perhaps be argued that there are spectroscopic differences between 'homeopathic remedies, as used in practice', it can't be argued on this evidence that these differences are specifically related to the process of producing a homeopathic remedy, when different levels of different impurities can explain the results just as well.
Rao also states that s/he and her co-authors are "neither champions nor detractors of homeopathy". I'm not sure how this squares with co-author Rustum Roy showing up in the Grauniad to defend homeopathy.
On the other points, there is not a word. The authors have entirely failed to defend their work. I'm amazed. I thought there would be a more than cursory attempt to invalidate our criticisms, or at least clarify some of the ambiguities in the original paper.
I think that Rao et al. is dead, and in a sane world it would be withdrawn by the journal.
Monday, 14 January 2008
Sunday, 13 January 2008
Memory of water: Martin Chaplin's response
As any regular readers will no doubt be aware, there was a recent issue of the journal Homeopathy devoted to the concept of memory of water. Ben Goldacre put together a journal club on the issue. I ended up submitting a letter to the editor in response to Martin F Chaplin's paper 'The memory of water: an overview'. Essentially, I made the point that Chaplin's paper was interesting, but didn't really describe anything of relevance to homeopathy; and that, in any case, there was no observed effect of homeopathy that needed explaining. The letter has now been published, alongside Chaplin's response.
Chaplin writes that my argument 'fails on two levels; firstly such an argument [that there is evidence of water memory of relevance to homeopathic treatment] was not the primary purpose of my review and secondly, in fairness to its inclusion in the journal ‘Homeopathy’, I did present such an argument (e.g. Table 1, p. 149).' Let's leave aside for now that this seems to be saying both that he did not make such an argument and that he did. The first point is partly fair enough: it may be true that Chaplin's primary purpose was not to provide a justification for homeopathy. However, the paper was published in Homeopathy, which suggests some relevance to homeopathy. As to the second point, Chaplin's Table 1 doesn't really provide any argument, but simply lists some mechanisms by which water might have something that Chaplin terms a 'memory'. How these could actually apply in practice to homeopathy is not discussed, and none of them show how a homeopathic remedy would be any different from plain old water. That was the point of my letter.
In his final paragraph, Chaplin states that "It is clear from the final paragraph of the letter that its author has his own preconceptions concerning homeopathy and that he was disappointed that I was not able to reinforce these." This refers to my contention that meta-analyses of placebo-controlled trials show that the best-conducted trials show no effect for homeopathy above placebo, referencing as an example the Shang et al. meta-analysis that was published in the Lancet. That being the case, there is nothing in homeopathy that requires any explanation, whether from 'memory of water' or anything else. It is perhaps true that I have some pre-conceptions with respect to homeopathy: it is, after all, extremely implausible given what we know about chemistry, physics and pharmacology. However, my reference to meta-analyses of placebo-controlled trials is clearly a reference to evidence, and as such has nothing to do with 'pre-conceptions'. Chaplin's comment is therefore no more than an ad hominem.
Chaplin goes on to criticise the Shang et al. paper, referencing his website. The Shang paper is one that has been widely misunderstood homeopaths, as I've written previously. Chaplin makes some of the same errors. In his website he writes that "The conclusion [of Shang et al.] was reached, however, in spite of the study apparently showing little evidence of differences between the two groups (homeopathy and conventional) when all the data was considered. There were differences when a tiny percentage of unmatched larger trials were cherry-picked for further analysis (that is, 102/110 of the homeopathy studies and 104/110 of the conventional studies were discarded). The remaining 6% of the studies, however, still showed positive (if not conclusive, possibly as the number of trials left in this final grouping was so small) evidence in favour of a homeopathic effect over placebo." This is just a misunderstanding of the study. Shang et al. did indeed show an effect for homeopathy above placebo when considering the 110 homeopathy trials. But the trials were not 'cherry-picked' for further analysis. They were selected based on stated criteria, and based on the fact the analysis had shown that the best predictor of bias in the studies was study size. Shang et al. showed that when studies of inadequate methodology and studies with small sample sizes were stripped out, there was no statistically significant effect for homeopathy (contrary to Chaplin's assertion that there was positive evidence for an effect greater than placebo). I stand by my interpretation of the study, that it shows that 'the best-conducted trials show no effect for homeopathic remedies beyond placebo'.
On his website, Chaplin also seems to misunderstand the placebo effect, when he writes "It should also be noted that placebo effects constitute real clinical effects, should be judged positively and probably account for a significant proportion of the success of prevailing established medicine". The point here is that the placebo effect is what you get when you give patients a sugar pill containing no active ingredients. If a remedy produces an effect no greater than placebo, then why should it be judged more positively than a blank sugar pill? That's the whole point of conducting placebo-controlled trials!
Having read Chaplin's response, I see no reason not to stand by my original points. Firstly, that while Chaplin has shown that water is interesting stuff, he hasn't shown any water memory effect that could provide a mechanism for homeopathy to work. Secondly, the best evidence shows that homeopathy doesn't work and needs no explanation.
Chaplin writes that my argument 'fails on two levels; firstly such an argument [that there is evidence of water memory of relevance to homeopathic treatment] was not the primary purpose of my review and secondly, in fairness to its inclusion in the journal ‘Homeopathy’, I did present such an argument (e.g. Table 1, p. 149).' Let's leave aside for now that this seems to be saying both that he did not make such an argument and that he did. The first point is partly fair enough: it may be true that Chaplin's primary purpose was not to provide a justification for homeopathy. However, the paper was published in Homeopathy, which suggests some relevance to homeopathy. As to the second point, Chaplin's Table 1 doesn't really provide any argument, but simply lists some mechanisms by which water might have something that Chaplin terms a 'memory'. How these could actually apply in practice to homeopathy is not discussed, and none of them show how a homeopathic remedy would be any different from plain old water. That was the point of my letter.
In his final paragraph, Chaplin states that "It is clear from the final paragraph of the letter that its author has his own preconceptions concerning homeopathy and that he was disappointed that I was not able to reinforce these." This refers to my contention that meta-analyses of placebo-controlled trials show that the best-conducted trials show no effect for homeopathy above placebo, referencing as an example the Shang et al. meta-analysis that was published in the Lancet. That being the case, there is nothing in homeopathy that requires any explanation, whether from 'memory of water' or anything else. It is perhaps true that I have some pre-conceptions with respect to homeopathy: it is, after all, extremely implausible given what we know about chemistry, physics and pharmacology. However, my reference to meta-analyses of placebo-controlled trials is clearly a reference to evidence, and as such has nothing to do with 'pre-conceptions'. Chaplin's comment is therefore no more than an ad hominem.
Chaplin goes on to criticise the Shang et al. paper, referencing his website. The Shang paper is one that has been widely misunderstood homeopaths, as I've written previously. Chaplin makes some of the same errors. In his website he writes that "The conclusion [of Shang et al.] was reached, however, in spite of the study apparently showing little evidence of differences between the two groups (homeopathy and conventional) when all the data was considered. There were differences when a tiny percentage of unmatched larger trials were cherry-picked for further analysis (that is, 102/110 of the homeopathy studies and 104/110 of the conventional studies were discarded). The remaining 6% of the studies, however, still showed positive (if not conclusive, possibly as the number of trials left in this final grouping was so small) evidence in favour of a homeopathic effect over placebo." This is just a misunderstanding of the study. Shang et al. did indeed show an effect for homeopathy above placebo when considering the 110 homeopathy trials. But the trials were not 'cherry-picked' for further analysis. They were selected based on stated criteria, and based on the fact the analysis had shown that the best predictor of bias in the studies was study size. Shang et al. showed that when studies of inadequate methodology and studies with small sample sizes were stripped out, there was no statistically significant effect for homeopathy (contrary to Chaplin's assertion that there was positive evidence for an effect greater than placebo). I stand by my interpretation of the study, that it shows that 'the best-conducted trials show no effect for homeopathic remedies beyond placebo'.
On his website, Chaplin also seems to misunderstand the placebo effect, when he writes "It should also be noted that placebo effects constitute real clinical effects, should be judged positively and probably account for a significant proportion of the success of prevailing established medicine". The point here is that the placebo effect is what you get when you give patients a sugar pill containing no active ingredients. If a remedy produces an effect no greater than placebo, then why should it be judged more positively than a blank sugar pill? That's the whole point of conducting placebo-controlled trials!
Having read Chaplin's response, I see no reason not to stand by my original points. Firstly, that while Chaplin has shown that water is interesting stuff, he hasn't shown any water memory effect that could provide a mechanism for homeopathy to work. Secondly, the best evidence shows that homeopathy doesn't work and needs no explanation.
Memory of water responses published
Following Ben Goldacre's journal club on the recent issue of Homeopathy that was devoted to the concept of 'memory of water', several responses to the issue have now been published in the latest issue of Homeopathy.
The things that I'm specifically involved in are this response to Martin F Chaplin's paper entitled 'Memory of water: an overview', and this response to Rao et al.'s attempt to distinguish between homeopathic remedies using materials science techniques.
I have a large piece of work due Tuesday, but I hope to be able to post in more detail on this later in the week.
The things that I'm specifically involved in are this response to Martin F Chaplin's paper entitled 'Memory of water: an overview', and this response to Rao et al.'s attempt to distinguish between homeopathic remedies using materials science techniques.
I have a large piece of work due Tuesday, but I hope to be able to post in more detail on this later in the week.
Subscribe to:
Posts (Atom)
