3. Neuro Psycho Pharmacology http://www.nature.com/npp/journal/v27/n2/abs/1395862a.html // Bacopa Monnieri for memory
I thought I'd have a quick look and explain why they're nonsense. Unfortunately, this hasn't proven to afford much in the way of intellectual exercise.
The first paper is by Graunke et al., and concerns, I kid you not, the treatment of tadpoles with homeopathic thyroxin. This is a well-known bad homeopathy paper. The tadpoles in the treatment group were more developed than those in the control group at the start of the experiment, so it wasn't much of a surprise that they were more developed at the end too. There is more discussion of this dreadful rubbish here.
The second paper is the famed Linde et al. meta-analysis, published in 1997. While this paper does say "The results of our meta-analysis are not compatible with the hypothesis that the clinical effects of homeopathy are completely due to placebo", there are some other things to bear in mind:
1. The paper also says "However, we found insufficient evidence from these studies that homeopathy is clearly efficacious for any single clinical condition".
2. A 1999 paper by the same authors, using improved methodology and including new trials, states that "It seems...likely that our meta-analysis at least overestimated the effects of homeopathic treatments".
3. A subsequent meta-analysis by Shang et al., published in the Lancet in 2005, using further improved methodology concluded that the results were compatible with homeopathy being a placebo.
Finally, the third study, by Roodenrys et al. in the journal Neuropsychopharmocology, is not about homeopathy at all, but rather about herbal medicine. In homeopathy, remedies are typically diluted such that it is very unlikely that they contain any of the original material: there is no active ingredient. In the Roodenrys study, what is being tested is brahmi, an Indian herb, of which the paper says:
Studies have shown that the herb contains many active constituents, including a number of alkaloids and saponins, however, the major constituents are the steroidal saponins, Bacosides A and B.
So it isn't entirely surprising that brahmi might have some effect.
From this fairly cursory glance at the studies provided by Nancy Malik, it's clear that she is from the Dana Ullman school of evaluating journal articles. This involves finding some papers that superficially appear to support your position, and then spamming them all over the internet. Luckily, for this approach there is no need to understand the articles, or even to read them. For people who think that magic water is medicine, that would be rather too much to expect.
Homeopathy is in the news once again, following a campaign set up by Merseyside Skeptics in which skeptics took an overdose of homeopathic pills, demonstrating that there's nothing in them other than lactose, and then went to the pub for some non-homeopathic beer. The campaign was mainly aimed at high-street pharmacist Boots, whose professional standards director admitted to the science and technology select committee of the House of Commons that there was no evidence that homeopathy worked, but they were very happy to continue charging people money for it. The campaign, and a forthcoming report of the select committee on whether the NHS should fund homeopathy, gave the Observer an excuse to publish a feature article, in which Anushka Asthana and Robin McKie examine the supposed controversy about whether homeopathy works or not.
Of course, scientifically speaking there is no controversy over homeopathy. It's perfectly clear that there is no scientific reason why it ought to work, and that when it is tested in properly conducted trials it works no better than placebo. I grant you that this probably wouldn't make for a very satisfying Sunday newspaper feature, and it would certainly be rather too short to fill up all of page 30 of the Observer. But you might expect there to be some discussion of the actual evidence. Not really, though. Edzard Ernst gets to say that "I have now published more than 100 papers on homeopathy and I am quite clear about its efficacy: you may as well take a glass of water than a homeopathic medicine". The evidence of Jane Lawrence of the Royal Pharmaceutical Society to the House of Commons select committee on science and technology that "There is no basis for [homeopathic remedies] being effective" is also quoted. But then Cristal Sumner of the British Homeopathic Association gets away with saying "Homeopathy helps patients and is not a placebo effect", despite all the evidence showing exactly the opposite.
This is typical of how the press treats scientific issues; they are presented in a superficially even-handed way, but crucially there is no attempt to weigh the evidence. Again, there is no scientific controversy over homeopathy, but by reading the Observer article you could be forgiven for thinking there was considerable room for doubt.
The worst things about the article, though, are the two inset boxes (these only appear in the printed version of the article, not in the online version, as far as I can tell). The first box is headed "Common Renedies" and lists "Popular homeopathic remedies for sale in Britain", including Arnica for clearing up bruises, and mixed pollen for treating hayfever. As these are homeopathic, they contain no Arnica or pollen, and there is no evidence they do anything except lighten your wallet, but this is not mentioned in the box. The second box is headed "Case Study", and recounts the story of one Helen Llewelyn, who claims that homeopathy helped control her endometriosis. This is an anecdote; it tells us that Llewelyn feels better, but it doesn't tell us anything about why, especially as we know nothing of what real medicines she might have been taking. From this website, it looks as though Llewelyn's case history is rather complex, and she is presently using homeopathy in conjunction with several kinds of real medicine. For medical treatments, you need much better evidence than this to show that they work. In the case of homeopathy, the evidence exists and it shows that homeopathy doesn't work. But there isn't an inset box anywhere that sums up the useful evidence: just one uncontrolled case report.
Given that there is genuinely a debate about whether the NHS should fund homeopathy, it would be good for this debate to be informed by the best currently available evidence. That is exactly what the Observer fails to do: all we get is "he said, she said" appeals to various forms of authority, and an anecdote. No wonder the newspapers are struggling.
You might remember me writing about the Intention Experiment in the past. The Intention Experiment hopes to harness the power of positive thinking to influence events. In one memorable example, they tried to influence the structure of water through the collected brainpower of a large collection of gullible numpties. Despite the experiment having a design likely to massively increase the chance of getting a false positive result, nothing astonishing happened.
Gimpy has now made my day by posting this story about homeopaths concerned that the British parliament's Science and Technology Committee will conclude that there is no evidence in favour of homeopathy. Instead of marshalling the evidence and trying to make a scientific case, the homeopaths are trying an intention experiment to influence the committee in favour of homeopathy.
There's just nothing to add to this: it's shear barking mad lunacy, and you have to think that maybe these people are their own worst enemies.
I'm not going to write a huge amount about this, because it is ably covered elsewhere. But the World Health Organisation finally came out today, and clearly stated that homeopathy should not be used for life-threatening conditions such as AIDS, malaria, TB and childhood diarrhoea.
The Society of Homeopaths (SoH) respond by saying that treating AIDS, malaria and TB would contravene their ethical guidelines. These guidelines are a joke, as you can see by perusing the Quackometer and Gimpy's blog. But apart from that, they say:
The Society of Homeopaths, the UK’s largest body of registered homeopaths, is concerned to learn, in an online article by the BBC (“WHO warns against homeopathy use’), that the World Health Organisation (WHO) has issued caution against the use of homeopathy for childhood diarrhoea following a letter by the charity Sense About Science.
They go on to cherry-pick and distort the research that has actually been conducted on homeopathy for childhood diarrhoea.
Well, who's surprised by that? Regular readers will be aware of just a few of the ways that homeopaths try to distort the evidence that homeopathy doesn't work. The point here is that the SoH, a supposedly professional organisation, is trying to defend a totally ineffective treatment for a disease that kills 2 million children every year. Homeopathy can't do anything to help, and using it instead of proper medical treatment could cost lives. I don't really mind if homeopaths sell ineffective sugar pills to the worried well in Alderley Edge, but this press release is delusional and irresponsible, and reasonable people should have no truck with this sort of dangerous nonsense.
This is somewhat old news, but I haven't had chance to write about it before. To add to the publications I have in Homeopathy, I now have one (as third author) in the Veterinary Record. This is starting to get silly; I'm supposed to be a geologist.
Perhaps unsurprisingly, this is related to a terrible homeopathy study [Hill et al., The Veterinary Record 164:364-370], this time on the treatment of skin conditions in dogs. It's another example of homeopaths continuing to do small, badly designed studies, when plenty of large and properly conducted studies, and systematic reviews and meta-analyses of those studies, show that homeopathy doesn't work. The letter I am involved in is one of three letters that were published criticising the study: they can be found, with the author's reply, at The Veterinary Record 164: 634-636 [apologies for the lack of links: there's no DOI for these that I can find]. There is also an excellent discussion of the paper, and some of the responses to it, over at JREF.
The design in this study is truly extraordinary. Initially, 20 dogs with skin problems were recruited to the study. All were treated with individualised remedies by a homeopath. In 15 cases, the dog owners reported no improvement. In 5 cases, the owners reported a significant improvement. Not looking good for homeopathy so far. Still, the five improved dogs were said to have responded well to homeopathy, and went on to phase 2, which was a proper randomised and blinded placebo-controlled trial. Unfortunately, one dog had to be euthanased before the trial could happen, and another dog's skin problems had resolved completely after the first stage, leaving only three dogs in phase 2. Supposedly, those dogs did better with homeopathy than with placebo, thus justifying, as ever, "further research".
This is possibly the easiest study to criticise that I've ever seen. Put simply, the first phase lacks a control group, so improvements cannot be attributed to homeopathy. There is simply no evidence that the five dogs recruited to phase 2 actually responded to homeopathy, rather than just improved spontaneously. Then the second phase of the trial includes only three dogs. There is no way to interpret the results of such a tiny, underpowered study. Those are the main problems, but there are others. For example, all the dogs were on some kind of conventional medication, so that cannot be ruled out as contributing to any improvement.
The only reasonable conclusion from the study is that there is no strong evidence that homeopathy did anything for the dogs in the trial. But the paper concludes that the improvement seen in the five dogs (which again cannot be attributed to homeopathy on the basis of this study) is enough to justify further research. No doubt the paper will also be spammed all over the internet by the likes of Dana Ullman, as proof positive that homeopathy works. Hopefully the letter I'm a co-author on, along with the two other letters critical of the study that were published, will go some way to addressing that. The signs are not good, though. The original Hill et al. paper included the statement that "Different homeopathic remedies and different dilutions of the same remedy have been distinguished from each other using Raman and infrared spectroscopy, even though all should contain nothing but water", with a reference to "Rao and others, 2007" [In fact, Rao et al. did not even claim that infrared spectroscopy showed any difference]. Regular readers will know that Rao and colleagues did nothing of the sort, and that to describe their paper as "discredited" would be something of an understatement. In the world of homeopathy, discredited papers never die. They are just recycled for use with audiences who don't know that they've been discredited. I suspect that this one will be no different.
As an aside, my favourite part of this study is that "constitutional signs" of each of the dogs, as used by the homeopath to pick a remedy, are listed [Table 2 of the paper]. For dog number 16, these are listed as:
So, this is the moment you’ve all been waiting for. A while ago I wrote a comment on an article that was published in Homeopathy. This article, among other things, purported to show that the authors of a Lancet meta-analysis (Aijin Shang and co-workers) that had negative results for homeopathy had engaged in post-hoc hypothesising and data dredging. That was an outrageous slur on what is a perfectly reasonable paper, if you understand it properly. My comment has now been published, along with a response from the authors. If anyone needs a copy of my comment and doesn’t fancy paying for it, drop me a line and I’ll bung you a PDF. In any case, the original version appears on my blog here.
Meanwhile, the reply by original authors Rutten and Stolper is an exercise in evasion and obfuscation, and doesn’t really address most of the points that I made. This seems to be fairly typical (and to be fair isn’t only restricted to non-science like homeopathy). In their original paper, Rutten and Stolper claimed that “Cut-off values for sample size [i.e. the number of subjects in a trial, above which the trial was defined as “large”] were not mentioned or explained in Shang el al's [sic] analysis”. This is simply not true. So what do Rutten and Stolper have to say about this embarrassing error?
“Wilson states that larger trials were defined by Shang as “Trials with SE [standard error] in the lowest quartile were defined as larger trials”. According to Wilson this was done to predefine 'larger trials'. We agree with Wilson that this is indeed a strange way of defining 'larger trials', but it is perfectly possible to simply define larger studies a priori according to sample size in terms like 'above median' as we suggested in our paper. Shang et al did not mention the sensitivity of the result to this choice of cut-off value: if median sample size (including 14 trials) is chosen homeopathy has the best (significantly positive) result, if 8 trials are selected homeopathy has the worst result. In the post-publication data they mentioned sample sizes but not Standard Errors. Isn't it odd that the authors did not mention the fact that homeopathy is effective based on a fully plausible definition of 'larger' trials, but stated that it is not effective based on a strange definition of 'larger', but that this was not apparent because of missing data?”
So, nothing there about how they failed to properly read the paper to check what Shang et al.’s definition of larger trials was, while essentially accusing them of research misconduct. Instead, they shift the goalposts and decide that they don’t like the definition that was provided. Now, it certainly would be possible to define larger studies as being “above median” sample size. By doing this you would be including studies of smaller size than would be included using Shang’s definition. As is well understood, and as Shang et al. clearly showed, including studies with smaller sample size will give you more positive but, crucially, less reliable results. So I don’t think it was particularly odd that Shang et al. failed to abandon their definition of larger trials in favour of someone else’s definition, published three years later, that would inevitably lead to less reliable results. Rutten and Stolper state that using 8 larger, high quality trials gives the worst results for homeopathy: but to get a positive result, you would have to include at least 14 trials, as Ludtke and Rutten show in another paper in the Journal of Clinical Epidemiology. And, again, it was perfectly apparent what definition Shang et al. used to define larger trials: it is clearly stated in their paper.
OK, so why use standard error rather than simply using sample size directly, as Rutten and Stolper want to do? In meta-analyses, a commonly used tool is a funnel plot. This plots, for each study included in the analysis, standard error against odds ratio. The odds ratio is a measure of the size of the effect of the intervention being studied. If the value is 1, there is no effect. If it is less than one, there is a positive effect (the intervention outperformed placebo), if greater than one there is a negative effect (placebo outperformed the intervention). The plot is typically used to identify publication bias (and other biases) in the set of trials: to simplify, if the plot is asymmetric, then biases exist. Using their funnel plot of 110 trials of homeopathy (Figure 2 in the Lancet paper), Shang et al. were able to show, (to a high degree of statistical significance, p<0.0001)that trials with higher standard error show more positive results. It then makes perfect sense to screen the trials by standard error rather than sample size, because it has been demonstrated that standard error correlates with odds ratio. Of course, you could plot sample size against odds ratio, but that is not the recommended approach.
Rutten and Stolper also claim to be "surprised" that one apparently positive trial of homeopathy was excluded from Shang's analysis. Since it was excluded based on the clearly stated exclusion criteria, I didn't find that surprising myself. How do Rutten and Stolper respond?
"We were indeed amazed that no matching trial could be found for a homeopathic trial on chronic polyarthritis by Wiesenauer. Shang did not specify criteria for matching of trials. We would expect the authors to explain this exclusion because Wiesenauer's trial would have made a difference in meta-regression analysis and possibly also in the selection of the eight larger good quality trials".
This routine is now wearily familiar. Someone makes a claim that Shang et al. didn’t do something, in this case specify criteria for matching of trials; I check the Lancet paper, and find that claim to be false. What did Shang have to say about matching of trials? On page 727, they say “For each homoeopathy trial, we identified matching trials of conventional medicine that enrolled patients with similar disorders and assessed similar outcomes. We used computer-generated random numbers to select one from several eligible trials of conventional medicine”. And, of course, the authors did explain why the trial was excluded; it met one of the pre-defined exclusion criteria. To me, that seems clear enough. As it stands, Rutten and Stolper’s point is nothing more than an argument from incredulity. They are amazed! Amazed that no matching trial could be found. But they haven’t actually found one to prove their point. It’s possible that this Weisenauer trial might have made a difference to the selection of 8 large, high quality trials. But I doubt it would have made any significant difference to the meta-regression analysis, which was based on 110 trials.
Having wrongly accused Shang et al. of doing a bad thing by defining sub-groups post-hoc, Rutten and Stolper applied all kinds of post-hoc rationalisations for excluding trials they don’t like. For example, they decided to throw out all the (resoundingly negative) trials of homeopathic arnica for muscle soreness in marathon runners, on the basis that homeopathy is not normally used to treat healthy people, and these trials therefore have low external validity. I argued that Shang et al. had to include those studies, since they met the inclusion criteria and did not meet the exclusion criteria. On what basis could they exclude them? From Rutten and Stolper, answer came there none:
"Wilson's remark about prominent homeopaths choosing muscle soreness as indication is not relevant. Using a marathon as starting point for a trial is understandable from a organisational point of view, although doubt is possible about external validity. Publishing negative trials in alternative medicine journals is correct behaviour. There is, however, strong evidence that homeopathic Arnica is not effective after long distance running and homeopathy as a method should not be judged by that outcome".
Yes, publish the negative trials. But why shouldn’t the negative trials be included in a meta-analysis? Because they’re negative, and that just can’t be right? I don’t see any rationale here for excluding these trials.
Rutten and Stolper also take the tine-honoured approach of arguing about statistics:
“…the asymmetry of funnel-plots is not necessarily a result of bias. It can also occur when smaller studies show larger effect just because they were done in a condition with high treatment effects, and thus requiring smaller patient numbers”.
I think this is nonsense, but anyone with more statistical knowledge should feel free to correct me. If the high treatment effects are real, then the larger studies will show them as well, and there will be no asymmetry in the funnel plot. The smaller studies are always going to be less reliable than the larger ones.
Finally, Rutten and Stolper conclude that:
"The conclusion that homeopathy is a placebo effect and that conventional medicine is not was not based on a comparative analysis of carefully matched trials, as stated by the authors".
Homeopaths do want this to be true, but no matter how many times they repeat it, it continues to be false. I think the problem is that they have become fixated on the analysis of the subgroup of larger, higher quality trials, which was only one part of the analysis. The meta-regression analysis for all 110 vs 110 trials gave the same results; the analysis of the “larger, higher quality” subgroup merely lends support to those results. So after all that palaver, there’s still no reason to think that there is anything particularly wrong with the Shang et al. Lancet paper, and there is certainly no excuse for accusing its authors of research misconduct.
Just a short post to say that the article I wrote pointing out some problems with a re-analysis (Rutten and Stolper 2008) of the Shang et al. Lancet meta-analysis has been accepted for publication by Homeopathy. I have also been sent the reply by the authors of the re-analysis. My comment and the reply will not actually appear in print until April, so I'd better not address the content of the reply at this point. I will say that I don't think it adequately addresses the points that I made. In particular, the authors don't have much to say in response to the point that information they claim is missing from the Shang paper is in fact clearly stated in that paper. More to come on this in April.
The Rutten and Stolper paper, and a companion paper in the Journal of Clinical Epidemiology by Ludtke and Rutten, were the subject of a press release titled "New Evidence for Homeopathy" claiming to cast doubt on the Shang meta-analysis. Perhaps I should issue a press release titled "New Evidence Against Homeopathy". Then again, maybe it would be better titled "New Evidence Against Homeopaths".
Now, despite apearances, I have to say that the subject of meta-analyses of homeopathy is not one that particularly fascinates me. It's just that a number of prominent homeopaths have made claims that the Shang study is flawed and/or fraudulent. In checking the claims that have been made, mainly be simply checking the Shang paper and its supplementary data, I have almost invariably found that they are false. Apgaylard has found similarly. I find it amazing that these false accusations have propagated across the internet and been accepted as truth, without anyone apparently doing the most basic of fact checking.
Generally, when the efficacy of a medical intervention is being assessed, you expect the quality of the trials and evidence available to increase through time. You might start out with uncontrolled case reports and observational studies, proceed through small-scale pilot studies, to large randomised placebo-controlled trials, and finally to systematic reviews and meta-analyses of the literature.
In CAM research this pattern is often not followed. Once systematic reviews and meta-analyses start to show that there is no evidence that the CAM treatment works, more small trials of poor methodology are conducted, many of which inevitably have (spurious) positive results. This allows CAM advocates to claim that there is lots of evidence in favour of their intervention, because they don't bother to account for study size and quality.
There is a fine example of this CAM tradition in the latest issue of Homeopathy. A systematic review, published in 2006, of homeopathy for treatment of allergic rhinitis concluded that "Some positive results were described in rhinitis with homeopathy in good-quality trials, but an equal number of negative studies counterbalance the positive ones. Therefore it is not possible to provide evidence-based recommendations for the use of homeopathy to treat allergic rhinitis, and further randomized controlled trials are needed". Well, perhaps: some would argue that the prior probability (close to nil) and currently existing evidence are enough to conclude that homeopathy does not work for allergic rhinitis (or, indeed, anything else). Be that as it may, it should be clear that the only useful new evidence would come from large and well-conducted RCTs. So what do Maria Goossens and a football team of colleagues do in the latest issue of Homeopathy? Why, publish a "prospective, open, non-comparative study" on homeopathy and allergic rhinitis, of course.
The methodology of the study consists of "treating" some patients suffering symptoms of allergic rhinitis with homeopathy, and getting them to fill in a quality of life questionaire at the start of the study, and after three weeks and four weeks. The physicians involved also assessed the severity of symptoms at baseline, three weeks, and four weeks. Unsurprisingly, the study found that people felt better with homeopathic treatment. But the methodological problems with this study are straightforward to point out. There is no control group. As a result, there can be no randomisation or blinding. Don't take my word for it; here's what the authors say in the discussion of their paper:
We did not distinguish between intermittent and persistent allergic rhinitis. All patients with intermittent allergic rhinitis (symptoms present less than four consecutive weeks a year) will be better after four weeks without any treatment. Patients who consult a homeopathic physician for allergic rhinitis usually have been suffering for a long time and from severe symptoms as the high level of the RQLQ score at baseline indicates. This study cannot be conclusive because there is no control group. Neither the physician, nor the patient was blinded. We cannot conclude that the degree of certainty of the physician about the appropriateness of the homeopathic prescription of a homeopathic remedy and the physician’s impression whether he had sufficient information about the patient’s condition influenced the outcome...it is not possible to draw a conclusion on the effect of the homeopathic treatment. This would require an RCT. To evaluate the effect of homeopathic treatment for allergic rhinitis an RCT should be performed.
So there you have it. The study cannot come to any useful conclusions. And, in the introduction to the paper, the authors write "This study was originally considered as a preliminary to a Randomized Clinical Trial (RCT) comparing standard conventional therapy with homeopathy (non-inferiority study). The RCT was never performed because sponsorship was withdrawn".
OK, that's life. Sometimes planned research funding fails to come off. These things happen. But why then publish the pilot study? Methodologically, it is useless, and it could never have added anything to the previously existing evidence from RCTs and systematic reviews. This study would never have been published anywhere other than a CAM journal, where scientific usefulness can take a back seat to an ideological desire to publish any evidence that looks as though it is in favour of homeopathy, no matter how methodologically weak it is, and in defiance of the higher level evidence that already exists.
Well, no-one is going to die from allergic rhinitis, so how much does it matter? The problem is that homeopaths don't stop at self-limiting conditions like hayfever. Some insist that homeopathy is a complete system of medicine and it can cure anything, including AIDS and malaria. Ben Goldacre's miniblog points to Jeremy Sherr's blog, for example, where Sherr is preparing to begin an unethical experiment on AIDS sufferers. This is a long road of madness, to be sure, but it begins where people believe they can cure hayfever through the use of magic sugar pills.
A while ago a submitted a comment to the journal Homeopathy, discussing a paper published in the journal by Rutten and Stolper, claiming to find a number of serious flaws in the Shang et al. meta-analysis that appeared in the Lancet in 2005. I had a slightly cryptic e-mail back from the journal last week:
Dear Dr Wilson,
Thank you for sending us this article. We have sent the article to the authors for comment.
We will be in touch again shortly.
I assume that means that the comment will be published, with a reply by the authors. I'll be interested in what they have to say, because at this point some of the errors in their paper seem to be indefensible.
While I was messing about at the University of Google, doing some research for the thing that I've just sent to homeopathy, I came across this [pdf]. Yes, the homeopaths (in the form of the International Homeopathic Medical League and the European Committee for Homeopathy) have put together a press release, based on the study in Homeopathy by Rutten and Stolper, and the study by Ludtke and Rutten in the Journal of Clinical Epidemiology. Both studies criticise (mostly wrongly) a perfectly good meta-analysis of homeopathy from the Lancet. The headline is "New evidence for homeopathy". No question mark, no caveats. You might guess that the International Homeopathic Medical League and the European Committee for Homeopathy would not be entirely unbiased sources of information. In fact, these studies are very far from being evidence for homeopathy, as I've tried to point out (and not just me). I suppose two studies sounds better than one, but the truth is that these are essentially the same study, with extra nonsense in the part that got published in Homeopathy. The studies apply a post-hoc analysis to the data from the Lancet article, and claim to be able to produce a positive result for homeopathy if you squint at the data and stand on one leg. Then they accuse the Lancet paper of data dredging. What a laugh riot.
Unfortunately, lots of scientific studies get reported based on their press releases. This means that the people who read the reports get no sense of the flaws in the study and the caveats that should be applied to its conclusions; these come from independent scientific scrutiny of the study once it has been peer-reviewed and published. Pushing press releases on your research may be a good way to get brownie points from your university, but it's not usually a good way of fostering an improved understanding of the scientific process. To be fair, it's by no means only homeopaths and their ilk who do this, as Ben Goldacre has pointed out.
Still, the good thing about this is that a search on Google News for this release shows that, at the time of writing, it has only been picked up by a few woo-ish magazines: mainstream western news seems to have ignored this more or less completely. Maybe the press has got fed up of this particular manufactured controversy, at least for now.
I wrote a little about a paper by Rutten and Stolper recently published in the amusing pseudo-journal Homeopathy. The paper performed the usual homeopath party trick of throwing incorrect allegations of research misconduct at the Shang et al. meta-analysis of homeopathy that was published in the Lancet, while also engaging in dubious statistical analysis. I've now had a little time to put together something a bit more meaty, with proper references and everything, and send it off as a letter to the editor of Homeopathy. I reproduce the text below.
Rutten and Stolper [1] have conducted a re-analysis of the data used in the landmark Lancet meta-analysis (Shang et al.) [2] of trials of homeopathy and conventional medicine. However, their approach to this work seems to have been influenced by a belief that the Shang analysis was deliberately skewed against homeopathy, and in favour of conventional medicine. I argue here that the evidence does not support that contention, and that the re-analysis by Rutten and Stolper does not show that the Shang et al. study was invalid.
Rationale for the re-analysis
In the abstract of their paper, Rutten and Stolper state “There is a discrepancy between the outcome of a meta-analysis published in 1997 of 89 trials of homeopathy by Linde et al and an analysis of 110 trials by Shang et al published in 2005, these reached opposite conclusions”, and on page 170 they write “The contradiction between Linde's conclusion based on 89 trials, and Shang et al's conclusion, based on 110 trials seems odd”. But there is nothing particularly surprising about this discrepancy. The Linde paper referred to was published in the Lancet in 1997 [3]. The same team re-analysed the data in a paper published in 1999 [4]. They concluded that because trials of higher methodological quality had smaller effect sizes, and that because a number of newly published high-quality trials showed negative results for homeopathy, their meta-analysis had over-estimated the effectiveness of homeopathy. Hence there is no reason to see to the discrepancy between Shang et al. and Linde et al. (1997) as being particularly “odd”.
Trial quality
Rutten and Stolper make statements about the “pre-specified hypotheses” of the Shang et al. study, but these are not consistent through the paper. In the introduction, they state:
“The hypotheses predefined mentioned in the introduction of Shang et al's paper were: ‘Bias in conduct and reporting of trials is a possible explanation for positive findings of placebo-controlled trials of both homeopathy and allopathy (conventional medicine)’; and: ‘These biases are more likely to affect small than large studies; the smaller a study, the larger the treatment effect necessary for the results to be statistically significant, whereas large studies are more likely to be of high methodological quality and published even if their results are negative’.”
Yet, in Rutter and Stolper’s section on “Pre-specified hypotheses” they include “quality in homeopathy is worse than in conventional medicine” as a hypothesis of Shang et al., and say that this hypothesis was falsified in the Shang et al. study. This is a straw man: it is not a hypothesis that was discussed in the Shang et al. paper, and Rutten and Stolper have missed the point of including a matched set of trials of conventional medicine. As Rutten and Stolper state (p. 170) “Pooling of results is…questionable if homeopathy works for some conditions and not for others”. This is a reasonable point. However, it is clear that some experimental conventional treatments work and some do not. The results of the analysis of conventional medicine were not consistent with the placebo hypothesis, showing that it is possible to obtain a positive result using the methods of Shang et al., even there is considerable heterogeneity in the results [5].
Post-hoc analysis?
Rutten and Stolper make the claim that the sub-sets of larger, higher quality studies were chosen post-hoc, presumably to make homeopathy appear less effective than it really is. In their paper, Rutten and Stolper state [p. 172-173]:
“Cut-off values for sample size were not mentioned or explained in Shang el al's [sic] analysis. Why were eight homeopathy trials compared with six conventional trials? Was this choice predefined or post-hoc? Post-publication data showed that cut-off values for larger higher quality studies differed between the two groups. In the homeopathy group the cut-off value was n = 98, including eight trials (38% of the higher quality trials). The cut-off value for larger conventional studies in this analysis was n = 146, including six trials (66% of the higher quality trials). These cut-off values were considerably above the median sample size of 65. There were 31 homeopathy trials larger than the homeopathy cut-off value and 24 conventional trials larger than the conventional cut-off value. We can think of no criterion that could be common to the two cut-off values. This suggests that this choice was post-hoc.”
The first thing to note is that it is not true that cut-off values for sample size were not mentioned or explained in the Shang et al. analysis. In the original Shang paper, on page 728, it is stated that “Trials with SE [standard error] in the lowest quartile were defined as larger trials”. In other words, the cut-off was not defined in terms of numbers of subjects, but in terms of standard error. It might be argued that this is a strange way of defining “larger” trials (and perhaps it should have been phrased as “lower standard error”). But it makes sense when criteria must be stated a priori. If a number of subjects were stated as a cut-off value, there would be no way of knowing how many studies would meet that criterion before looking at the data. You might find that a very large or very small number of studies met the criterion, making further analysis difficult. So, there is no mystery as to why the “cut-off values” were different between trials of homeopathy and trials of conventional medicine: it is because the distribution of standard errors is different between the two populations. This could be discovered simply by reading the original paper, and the conclusion that the groups were chosen post-hoc cannot be sustained.
A further point here is that the group of “larger” homeopathy trials contains smaller trials that would not have made the cut for “larger” trials in the conventional medicine group. Those smaller trials are more likely to show spurious positive results. It follows that had the authors engineered the groups to get the result they wanted, they had engineered them in favour of homeopathy.
Another paragraph in Rutten and Stolper states “We did not further investigate possible selection bias by excluding trials, but we were surprised by the exclusion of Wiesenauer's trial on chronic polyarthritis. This was a larger trial (n = 176), of good quality according to Linde, with positive results. This trial would have contributed positively to the outcome of the larger higher quality trials. Shang excluded this trial because no matching trial could be found” (page 171). Since the trial was excluded on the basis of the clearly stated, pre-specified exclusion criteria, what is surprising about it having been excluded? Including it would have made a nonsense of the design of the study and violated the pre-specified exclusion criteria, and would have been a gross error.
Another possible outcome?
Rutten and Stolper conduct a sensitivity analysis, but, as they note, the decisions they make in this analysis are highly subjective. They decide to exclude all trials of homeopathy for muscle soreness [6-9], on the grounds that “treatment of healthy individuals is very rare in homeopathic practice [and] this outcome has low external validity to judge the effect of homeopathy as a method” (page 173). Yet, the trials were conducted with the participation of prominent homeopaths, and some were published in homeopathic or alternative medicine journals [8, 9], so at least some homeopaths seem to be of the opinion that there is enough external validity for it to be worth conducting a trial. So how can the external validity of the trials be judged in a transparent way? In a meta-analysis based on clear, pre-specified criteria, there could be no justification for omitting the studies.
It is also notable that one of the authors was a co-author of another re-analysis published in the Journal of Clinical Epidemiology [10]. That analysis showed that if random-effects meta-analysis is used, it is possible to add smaller trials to Shang’s set of “larger, higher quality” trials of homeopathy, and get a statistically significant (although clinically unimpressive) benefit for homeopathy. All this really shows is that a finding in favour of homeopathy is not robust, and as Shang et al. showed, including smaller trials also decreases the reliability of the findings. The re-analysis also showed that the benefit for homeopathy was statistically insignificant when a meta-regression analysis was used: this negative finding was strangely not mentioned in the Homeopathy paper. Because the results differed between meta-regression and random-effects analyses, and because Shang et al. showed highly significant evidence of funnel-plot asymmetry in their complete dataset of 110 trials of homeopathy, it is arguable that meta-regression analysis is a more appropriate choice.
Overall, it is clear that “another outcome” (i.e. one favourable to homeopathy) is possible, as long as negative studies are excluded without good reason, smaller and less reliable studies are included, and a particular method of statistical analysis is used. In a paper that (wrongly) criticises a study for analysing data based on criteria established post-hoc, this seems like an odd point to make.
Conclusion
The analysis by Rutten and Stolper contains misconceptions of Shang et al., contains some important errors, and does not show that the Shang et al. study was an invalid analysis. In particular, there is no evidence that the Shang et al. study involved post-hoc choice of subgroups. The results of meta-analyses can be debated, but scientists should not be accused of research misconduct on the basis of no evidence, or on the basis of having failed to read their work properly.
References
1. Rutten ALB and Stolper CF. The 2005 meta-analysis of homeopathy: the importance of post-publication data. Homeopathy 2008; 97: 169-177.
2. Shang A, Huwiler-Müntener K, Nartey L et al. Are the clinical effects of homeopathy placebo effects? Comparative study of placebo-controlled trials of homeopathy and allopathy, Lancet 2005; 366: 726–732.
3. Linde K, Clausius N, Ramirez G et al. Are the clinical effects of homeopathy placebo effects? A meta-analysis of placebo-controlled trials, Lancet (1997); 350:834–843.
4. K. Linde K, Scholz M, Ramirez G, Clausius N, Melchart D, Jonas WB. Impact of study quality on outcome in placebo-controlled trials of homeopathy, J Clin Epidemiol 1999; 52: 631–36.
5. Shang A, Jüni P, Sterne JAC, Huwiler-Müntener K, Egger M. Are the clinical effects of homeopathy placebo effects? A meta-analysis of placebo-controlled trials: Author’s reply, Lancet 2005; 366: 2083-2084
6. Vickers AJ, Fisher P, Wyllie SE, Rees R. Homeopathic Arnica 30X is ineffective for muscle soreness after long-distance running – A randomized, double-blind, placebo-controlled trial. Clin J Pain 1998; 14: 227–231.
7. Vickers AJ, Fisher P, Smith C, Wyllie SE, Lewith GT. Homoeopathy for delayed onset muscle soreness - A randomised double blind placebo controlled trial. Brit J Sports Med 1997; 31: 304–307.
8. Jawara N, Lewith GT, Vickers AJ, Mullee MA, Smith C. Homoeopathic Arnica and Rhus toxicodendron for delayed onset muscle soreness - A pilot for a randomized, double-blind, placebo-controlled trial. Brit Hom J 1997; 86: 10–15.
9. Tveiten D, Bruset S, Borchgrevink CF, Norseth J. Effects of the homeopathic remedy Arnica D30 on marathon runners: A randomized, double-blind study during the 1995 Oslo Marathon. Complement Ther Med 1998; 6(2): 71–74.
10. Lüdtke R, Rutten ALB. The conclusions on the effectiveness of homeopathy highly depend on the set of analyzed trials. J Clin Epidemiol 2008; 61: 1197-1204
Having whined and bullshitted about the 2005 meta-analysis (Shang et al.) that was published in the Lancet (which showed that larger, high-quality trials demonstrated no effect for homeopathy beyond placebo), the homeopaths are now trying a bit of number-crunching. This is an improvement, although they don't seem to be any better at number crunching than they are at logic, as we saw with the recent Ludtke and Rutten paper in the Journal of Clinical Epidemiology.
In the Journal of Clinical Epidemiology, there is a proper review process, and the authors couldn't get away with writing just any arrant nonsense that came into their head. This led to a paper that broadly supported the original conclusions of the Shang study, which obviously wasn't that much use. Luckily, Homeopathy, the in-house fanzine of the Faculty of Homeopathy, has no such qualms about publishing more or less any old rubbish (and I should know, I've been published in it myself). Hence a paper by Rutten and Stolper in the October issue of Homeopathy criticising the Shang paper, entitled "The 2005 meta-analysis of homeopathy: the importance of post-publication data". I'm not going to analyse the whole thing-there comes a point when life really is too short-but I do want to pick up on one blatant error that seems to suggest they haven't carefully read the paper they are criticising.
One thing that Shang did was look at papers that were of good quality, based on how well they were randomised, how well they were blinded, and so on. They also sought to look at the "larger" trials, because their analysis showed that smaller trials with higher standard errors tended to be more positive. Rutten and Stolper claim that the cut-off size for larger trials was different between the homeopathy trials and conventional trials "without plausible reason". They write:
In the homeopathy group the cut-off value was n = 98, including eight trials (38% of the higher quality trials). The cut-off value for larger conventional studies in this analysis was n = 146, including six trials (66% of the higher quality trials). These cut-off values were considerably above the median sample size of 65. There were 31 homeopathy trials larger than the homeopathy cut-off value and 24 conventional trials larger than the conventional cutoff value. We can think of no criterion that could be common to the two cut-off values. This suggests that this choice was post-hoc.
What Rutten and Stolper are doing here is essentially accusing Shang and colleagues of fiddling the results so that homeopathy looked less effective than it really is. That is a serious accusation, and you might expect that someone making it would have carefully checked the original paper to see what the authors said their criterion for larger trials was. If you read the original Shang paper, on page 728 you will find this definition: "Trials with SE in the lowest quartile were defined as larger trials". There is nothing particularly opaque about this, and yet the idea that what was meant by "larger" trials was somehow unclear comes up time and time again.
Picking trials "with SE in the lowest quartile" may seem like an odd way of choosing the "larger" trials. Why not just pick a number, like 100, which Rutten and Stolper seem to want to do? Well, if you want to avoid post-hoc analysis it is a sensible thing to do, as you know you will have a reasonable number of "larger" trials to work with. If you just pick out a number, you might find that almost none, or almost all, of your trials qualify as being "larger", since you don't know what the distribution of trial sizes is until you do your analysis.
Of course, another thing is that in a comparison between homeopathy and conventional medicine, the discrepancy in trial size numbers actually gives homeopathy an advantage. This is because smaller trials of homeopathy that would have missed the cut for conventional trials are included, and these are more likely to show spurious positive results. So, if the difference in trial sizes really were due to the authors' bias, then they were biased in favour of homeopathy.
So, Rutten and Stolper have made an erroneous accusation of biased post-hoc analysis based on not bothering to properly read, or carefully think about, the paper they are criticising. In my opinion they have made themselves look rather silly. Yet, even sillier still, their entire paper is an exercise in post-hoc analysis, as they try to find ways of torturing the data to get the result they want, i.e. that homeopathy works. This is how things go in the wonderful world of homeopathy, where all kinds of pseudoscience can be justified as long as they give you the right answer.
In the comments to a post at Respectful Insolence, my favourite homeopath Dana Ullman weighs in with the suggestion that the Shang et al. meta-analysis of trials of homeopathy and conventional medicine (which has been written about extensively by me and apgaylard), had been "blown out of the water". Ullman makes this assertion based on a new paper by Ludtke and Rutten, entitled "The conclusions on the effectiveness of homeopathy highly depend on the set of analyzed trials", that has been accepted by the Journal of Clinical Epidemiology. It's nice to see that this paper does exist after all. So does the article really blow Shang out of the water? A quick look at the conclusions tells us that the answer is no:
Our results do neither prove that homeopathic medicines are superior to placebo nor do they prove the opposite...it occurs that Shang’s conclusions are not so definite as they have been reported and discussed.
What Ludtke and Rutten have done is actually quite interesting, though, so I'd like to explore it in a bit more detail. What they've done is taken the 21 trials of homeopathy that Shang et al. considered to be of "higher quality". They have then performed various analyses on this subset of trials to see what happens if you exclude trials based on their size or other parameters.
The authors plotted a funnel plot of odds ratio (a measure of the size of the effect of the intervention: values below 1 indicate a benefit over placebo) versus standard error (which is dependent on trial size). For all of the 21 trials, they found that there was substantial (but not statistically significant) asymmetry in the plot (if the funnel plot is asymmetrical, then biases are present in the data: these might be publication bias, small study effects, or a whole host of other effects). They also note that no evidence for asymmetry was found for the 8 largest trials of the 21. This actually re-iterates one of the main points of the Shang analysis: that a large number of trials is needed to identify asymmetry, and therefore to get an idea of bias in the dataset. That is why Shang et al. looked at all 110 trials that matched their inclusion criteria; that enabled them to identify a highly significant asymmetry in their funnel plot. This was important because it showed that the smaller the study size, the larger the apparent effect.
The thing in the paper that homeopaths will probably emphasise is that for the set of 21 higher quality trials, the pooled odds ratio (from random effects meta-analysis) was 0.76, suggesting a benefit over placebo. But wait! What are the 95% confidence intervals? 0.59-0.99. This indicates anything from a unimpressive benefit to a clinically negligible one. In other words, it's a deeply uninspiring result, but homeopaths will be trying to tell you that it shows that homeopathic remedies do something.
The interesting thing that the authors then did was to take the 2 largest trials, and look at what happens to the results when you add trials in descending order of patient numbers (Figure 2 of the paper). Once you get to the point where you've included the 14 largest trials, the resulting odds ratio is always less than 1 (except for the case of 17 trials). This is interesting in a way, but all it really does is demonstrate what Shang et al. said: that smaller trials are more likely to give positive results. So the more trials you add, the more positive but less reliable the results are; with 14 or more trials you might just about scrape a statistically significant benefit, but that result is not as reliable as the analysis restricted to the eight largest trials. It's also worth noting that the upper limits of the confidence intervals in Figure 2 are always close to 1, showing that any benefit is likely to be clinically insignificant. They perform a similar analysis in their Figure 3, except they use a meta-regression analysis rather than a random effects meta-analysis, and for that they show no statistically significant benefit no matter how many studies they include.
Another thing that homeopaths will probably jump on is that if one very negative trial (on Arnica for muscle soreness) is omitted from the set of 21 trials, the results appear more positive (odds ratio 0.73, 95% confidence interval 0.56-0.93) when a random effects meta-analysis is used. There are a number of other trials that can be removed from the dataset to give apparently positive results, but only when random effects analysis is used: a meta-regression analysis shows that there is no statistically significant benefit no matter which study you remove. Also, when performing a similar analysis on the 8 large, higher quality trials originally identified by Shang et al., no statistically significant benefit is found whichever trial you decide to remove. Again, note that the 8 largest trials are intrinsically more reliable than the smaller ones.
All the way through the paper, it is noticeable that meta-regression analysis shows more negative results than a random effects meta-analysis. In fact, the authors point out that in their meta-regression analysis "no single predicted OR [odds ratio] could be shown to differ significantly from unity". So which should be used? The authors write "... there is no guideline which tells a researcher when to prefer meta-regression to random effects metaanalysis or vice versa. As the statistical test for asymmetry only has a small power, Egger suggests to perform metaregressions when the respective P-value falls below 0.10. Applying this criterion there seemed to be no need to perform a meta-regression in most of the subsets we analyzed". But this conclusion is based on the restricted analysis of 21 higher quality trials. Shang et al.'s original analysis of 110 trials of homeopathy showed asymmetry with p<0.0001, suggesting that a meta-regression analysis would be more appropriate.
So, the upshot is that the paper's title is misleading. The conclusions on the effectiveness of homeopathy do not highly depend on the set of analyzed trials, if an appropriate test is used. Asymmetry is not adequately identified in the dataset because too few trials are used. And, even if you can convince yourself that you can get a statistically significant benefit by playing around with the numbers, the actual clinical benefit is negligible. In some ways, the paper actually reinforces the conclusions of Shang et al., and it certainly doesn't show that homeopathic medicines work.
Just a little bit more on the interview with Dana Ullman that I wrote about here.
Ullman claims that a re-analysis of Shang et al. has been accepted for publication in the Journal of Clinical Epidemiology. The only reference I can find to this study is this, where a study dated 2007, entitled "The conclusions on the effectiveness of homeopathy highly depend on the set of analysed trials" by R Ludtke and ALB Rutten is listed as being 'in press' in the Journal of Clinical Epidemiology.
Here's the list of articles in press in the Journal of Clinical Epidemiology. There is no sign of any such paper. Various searches fail to find any similar papers published anywhere else, or in earlier issues of the Journal of Clinical Epidemiology. The only thing I can find is a paper in Homeopathy called "‘Proof’ against homeopathy in fact supports Homeopathy", in which one Lex Rutten is credited as the first author. Whether this is the same Rutten I cannot say. The main point of the paper seems to be that if you add four positive trials to the Shang dataset, the result would be more positive. And they accuse Shang of cherry-picking. Two of the trials complained about were excluded [PDF] from the Shang meta-analysis: the Fisher et al. paper because it had an ineligible study design, and the Weisenauer and Gaus paper because no matching conventional trial could be found. Of the other two, one by Arnal-Laserre appears to be a French thesis of some description [EDIT: This is a French thesis: it was mentioned in the Cochrane review of "Homoeopathy for the induction of labour". Apparently, the reviewers could not obtain a copy of the thesis, which perhaps explains why Shang et al. did not include it], and the other by Maiwald et al. was not a placebo-controlled trial.
So, does this re-analysis exist, or is it just another figment of the collective homeopathic imagination? And if it ever does get published, is it likely that it will have anything useful to say?
Here's Ullman, a US base homeopath, in an interview published on the website of Sue Young, a London-based homeopath. There's all kinds of drivel here, but there is an exchange on the Shang et al. meta-analysis of homeopathy (published in the Lancet in 2005) that particularly caught my eye, because it's full of absolute nonsense. Not only that, but Ullman has had his misconceptions about this paper explained to him in numerous places on the internet, including on this very blog. Here's what he and his interviewer, one Louise Mclean of the Zeus information service, had to say:
DANA: In fact there is a new study that is coming out shortly which is a re-analysis of the 2005 Lancet review of Shang. The researchers got it accepted in a major international journal of research. What they have finally done is what Shang didn’t do. He didn’t review ALL of the high calibre research but only a small part of it. He ignored comprehensive analysis entirely. I think he knew exactly what it was but he didn’t want to report on it, as it was too positive. Instead he only reported on trials with very large numbers of subjects because when you do that, most of those studies use one remedy for everybody without any degree of individuality.
LOUISE: We individualise.
DANA: We do individualise but sometimes the single remedy or the formulas will work for a broad number of people.
DANA: In fact they had a remark in the Shang article published in the Lancet, where they specifically made reference to trials on respiratory ailments and that the results were robust, but they said they couldn’t trust them because there were only 8 studies. But then again they based their entire analysis on 8 homeopathic studies and 6 conventional ones. So they can’t have it both ways and this new journal article in the Journal of Clinical Epidemiology which is ranked as one of the top international journals of reviews of research, has accepted the new studies.
Sigh. Why is this nonsense? Let me count the ways.
1. Shang et al. did in fact analyse ALL of the trials of homeopathy that met their inclusion criteria. This allowed them to establish, using statistical methods, that smaller trials and those of less robust methodology showed better results for homeopathy, because of bias. The good quality, large studies showed that homeopathy had no effect. This is the pattern you would expect to see if homeopathy is a placebo.
2. Ah, individualisation. In fact, a number of the trials in the Shang study were of individualised homeopathy (including two of those that were considered large and of high quality). There was no evidence that individualised homeopathy was better than any other type of homeopathy (p=0.636). In any case, individualisation is only important when it suits Ullman, as seen when he says "We do individualise but sometimes the single remedy or the formulas will work for a broad number of people".
3. The meta-analysis not only included the Reilly paper in the BMJ that is linked to, but two other Reilly papers, as can be seen from the additional material [PDF] to the paper that is available online. This is contrary to Ullman's assertion that "for some reason they did not include any of David Reilly’s research".
4. The point that Shang et al. make about the 8 studies of respiratory ailments is that 8 studies was too few for a meta-analysis restricted to those studies to detect the bias that is revealed by an analysis of the complete dataset. The eight studies of homeopathy that Ullman wrongly claims Shang et al. "based their entire analysis on" were identified as the studies most likely to be free of bias, based on an analysis of the entire dataset. So the authors are not trying to have it both ways at all, and Ullman is comparing apples with oranges.
What I find particularly annoying about this is that Ullman and Mclean are essentially accusing Shang and his co-workers of research misconduct. What do they base this very serious accusation on? On a total misunderstanding of their paper, and a flat-out lie that they omitted research that was 'too positive', when that research was in fact included in the analysis. I am not a statistician, but the paper is not that difficult to understand, if you read it. Followers of Dana Ullman's career will not be surprised by his disingenuousness on this, I'm sure.
It seems that no matter how often I (and others, notably apgaylard) write about the persistent mis-representation of the Shang paper, the homeopaths carry on regardless.
More from the new issue of comedy journal Homeopathy. Homeopaths continue to pursue research on 'high dilutions' (i.e. dilutions such that there is highly unlikely to be any of the original solute remaining) in search of a vaguely plausible mechanism for homeopathy to work. Since the evidence is that homeopathy doesn't work, this is unlikely to be a fruitful endeavour, but it does provide entertainment for connoiseurs of pseudoscience.
The latest offering is a paper by one R. Assumpção , entitled "Electrical impedance and HV plasma images of high dilutions of sodium chloride" and an acccompanying editorial by Cyril Smith. Two sets of observations are presented: a set of Kirlian photographs and a set of measurements of the impedance of various homeopathic dilutions of NaCl.
What is Kirlian photography? When an object resting on a photographic plate is connected to a source of high voltage, an image is formed on the plate. Semyon Kirlian, whom the technique is named after, thought that such images might be compared to a human 'aura'. You can see how this would be attractive to various kinds of energy medicine fruitcakes. In the paper, Assumpção provides Kirlian photographs of homeopathic dilutions of an NaCl solution (6c, 12c, 24c and 30c) and of succussed (shaken) water without dilution. The images are certainly pretty, but what do they show? Not a lot, as far as I can tell. It isn't clear how such images should be interpreted, and all Assumpção really concludes is that different dilutions look different. This, in itself, is not a particularly exciting finding, in the absence of any hypothesis as to why that is and what it might mean.
What of the impedance data? Assumpção finds that impedance of the samples increases with dilution, up to about 12c (which is the point where Avogadro's constant suggests that there is unlikely to be any of the original NaCl remaining), and then decreases again slightly with further dilution. "This phenomenon is inexplicable in terms of conventional chemical theory" concludes Assumpção, a statement that really should be followed by "OMG!!!!11!1!1!!1!!!!!1".
Well, I think I have a reasonable explanation for the results that causes no problems for conventional chemical theory. As expected, dilution of the NaCl solution causes an increase in impedance as ions are removed from the solution. But there is also a competing increase in the concentration of dissolved ions caused by the succussion (agitation) process, as atmospheric gases and other impurities are dissolved in the water. So the small decrease in impedance beyond 12c could result from this process. I'm more puzzled as to why the impedance of the distilled water increases when it is shaken, suggesting a decrease in the concentration of dissolved ions. However, I suspect that there is a conventional explanation, but Assumpção doesn't seem to be particularly curious about this. Is there a change in the ionic species in the water as it is shaken? Does shaking facilitate outgassing of the dissolved gases that remain in the distilled water? We don't know. This continues the homeopathic water research tradition of publishing things without any thought as to what factors (other than the magic of water) might have led to the results.
Of course, the other thing to note is that the paper only looks at one solute, NaCl. For homeopathy, what would be really interesting is if you could show that there was a difference between two different homeopathic substances at dilutions greater than 12c. Previous attempts to do this have ended in some embarassment, as Smith acknowledges in the accompanying editorial, when he states "in the light of the controversy which has attended previous claims in this field, caution, and independent repetition of these results is required". There seems to be little sign of any such caution so far.
Finally, in Figure 5 of the paper, Assumpção had omitted to label to the impedance curves, showing which curve belongs to which dilution. The solution? Simply write the labels in, with a pen, afterwards. It is truly amateur hour.
The new issue of Homeopathy (the in-house shitrag of the Faculty of Homeopathy) is out, which is always a joy. To my surprise, there's actually a paper in there that seems to be groping towards a vague sort of sense. The authors (Clare Relton and Alicia O' Cathain of the University of Sheffield, and Kate J. Thomas of the University of Leeds) seem to be on the cusp of imagining what homeopathy could actually be useful for. They don't say outright that the principles of homeopathy make no sense and that it doesn't work, but they do seem to be getting close to an idea of what might make homeopathy useful in the absence of specific effects from the wee sugar pills they give you. As we know, the evidence from the largest and best quality trials tells us that those pills don't do anything over and above the placebo effect. But what can we do with the placebo effect? In some ways, the paper is a missed opportunity, but there are at least signs of movement.
In the paper, the authors distinguish between several meanings of "homeopathy". Does it refer to the medicines? To consultations with a homeopath? To the principles of homeopathy? Or to the homeopathic therapeutic system as a whole? To be honest, I don't really care. But then the authors look at the definitions of 'efficacy' and 'effectiveness':
In clinical epidemiology ‘efficacy’ refers to ‘the extent to which a specific intervention, procedure, drug, regimen, or service produces a beneficial result under ideal conditions’. Evidence of efficacy is usually interpreted as requiring evidence from placebo-controlled randomised controlled trials (RCTs). Whereas ‘effectiveness’ is a ‘measure of the extent to which a specific intervention, procedure, drug, regimen or service, when deployed in the field in routine circumstances does what it is intended to do for a specific population’. Evidence of effectiveness requires pragmatic RCTs and well conducted observational studies of routine clinical practice.
The authors then argue that evidence of cost effectiveness, based on the clinical effectiveness of the intervention is required by NICE for interventions that are provided by the NHS, not evidence of efficacy. Then they go on to look at the evidence.
The authors divide the evidence into experimental evidence, by which they mean randomised, double-blind, placebo-controlled trials. This is where things go off the rails a bit. The authors suggest that looking at the 'active ingredient' of homeopathy as being the medicine given is far too reductionist; perhaps the active ingredient should be thought of as including "Six putative active ingredients... : patient's openness to the mind body connection, consultational empathy, in depth enquiry into bodily complaints, disclosure, the remedy matching process, homeopathic remedies". Not only that, but "Other authors have discussed the difficulties of separating out the effects of the homeopathic medicine from the consultation effects".
Now, this is interesting. It seems that what the authors are saying here is that much of the benefit that patients get from homeopathy is unrelated to the medicine they are given; it is related to the relationship between the homeopath and the patient, and to the consultation process. In medical science, there's a term for this: it's called the placebo effect. That's why you conduct placebo-controlled trials; because those effects exist, and you therefore need to separate them from the effect of the actual medicine that is given to the patients. That placebo-controlled trials are specifically designed to address this problem seems to have passed the authors by. Placebo-controlled trials of individualised homeopathy (i.e. the whole homeopathic package, including consulation) have been carried out, and show no better results than trials that did not include the consultation. But at least there is a recognition here that non-specific effects play a very large role in homeopathy.
The authors go on to look at what they call 'observational evidence', consisting of observational studies and case reports. The authors admit that this sort of evidence is "prone to many types of bias, it is regarded as weaker than experimental evidence, and is generally disregarded in systematic reviews of evidence". But they say "However, this type of evidence does acknowledge the homeopath i.e., focuses on treatment by a homeopath in contrast to the experimental evidence which focuses only on the homeopathic medicine".
This seems to be just the usual special pleading that 'my woo can't be tested by your cold-hearted science'. Then again, the authors do seem to be suggesting that the pills containing nothing are not as important as the context in which those pills are delivered. So where next? Unfortunately, the authors miss this opportunity by simply making some weak recommendations about terminology. But if the homeopathic community could admit that their drugs don't work, and sit down to figure out how they could strip out all the pseudo-science about energy medicine, water memory, quantum entanglement and so on, and how they could harness the power of the placebo effect, then what remained could conceivably be of some use.
Of course, that the homeopaths haven't managed to figure this out for themselves yet is a little surprising, given that this is what Ben Goldacre has been arguing for some time. It's also interesting to reproduce the last paragraph of the famous Shang et al. meta-analysis which was so damning of homeopathy:
"We emphasise that our study, and the trials we examined, exclusively addressed the narrow question of whether homoeopathic remedies have specific effects. Context effects can influence the effects of interventions, and the relationship between patient and carer might be an important pathway mediating such effects. Practitioners of homoeopathy can form powerful alliances with their patients, because patients and carers commonly share strong beliefs about the treatment’s effectiveness, and other cultural beliefs, which might be both empowering and restorative. For some people, therefore, homoeopathy could be another tool that complements conventional medicine, whereas others might see it as purposeful and antiscientific deception of patients, which has no place in modern health care. Clearly, rather than doing further placebo-controlled trials of homoeopathy, future research efforts should focus on the nature of context effects and on the place of homoeopathy in health-care systems."
Could the homeopathic and conventional medicine communities be moving towards common ground?
According to this article in Pulse, the number of prescriptions written for homeopathic 'remedies' in the UK fell significantly between 2005 and 2007. 83,000 prescriptions were written in 2005, 62,700 in 2006 and 49,300 in 2007.
Naturally, if you're a homeopath, this is the result of a 'hate campaign' conducted against homeopathy by Prof Edzard Ernst, the Laing Professor of Complementary Medicine at the Peninsula Medical School (Universities of Exeter and Plymouth). At least, this is what one Dr Tim Robinson says in the Pulse article. This is fairly typical of the homeopathic community, who inevitably confuse a desire to rigorously assess the evidence base with a 'hate campaign'. Dr Robinson writes "Patients are not asking for [homeopathy] because of what has been written in the press..." This ignores woefully uncritical articles like this one that appear in the press, and columns for homeopaths in local papers that amount to free advertising, like this in the Manchester Evening News. If it was left to the press, people could be forgiven for thinking that there was decent evidence that homeopathy works.
Perhaps the message is getting out: that in an NHS with scarce funding, there's no room for funding things that don't work.
A recent conference on the current state of research in homeopathy once again exposes the field as an intellectual shambles.
The bloggers have already written a good deal about some of the presentations at the conference. AP Gaylard points out that homeopaths are still "saying the thing that is not" about a meta-analysis (Shang et al.) that was published in 2005. The Quackometer goes into great detail about why Lionel Milgrom is talking rubbish; something that is also ably demonstrated in several excellent posts by shpalman. And Gimpy explains why Dr Alex Tournier PhD's assessment of the evidence is fundamentally flawed.
What I find difficult to understand is how the homeopaths can justify their selective quoting of evidence. The repeated instances of saying the thing that is not about the Shang paper are only one example. A materials science paper by Rao et al. is constantly wheeled out as evidence that homeopathy could work, for all the world as if it hadn't been demolished by a critical letter to the editor in a subsequent issue of the journal (I should know, I was a co-author of the letter). This letter has gone down the homeopathic memory hole, never to be mentioned again, while the original (execrably bad) paper refuses to die. There is also an identifiable pattern to how homeopaths discuss meta-analyses. Apparently positive ones that take no account of trial quality are always mentioned, but crucial caveats are always omitted (for example, Linde et al. 1997 said they could find no evidence that homeopathy worked for any particular condition: this is never mentioned). A later paper by Linde et al. looked at trial quality, and concluded that the results weakened the positive conclusions of their earlier paper: this is never mentioned. The Shang et al. paper is only mentioned to make erroneous criticisms of its methodology, to incorrectly state that critical information was missing from the paper, or to accuse the authors of scientific misconduct based on a total misunderstanding of the paper.
A bit of googling the other day turned up this. Back in 1861, a former editor of the North American Journal of Homeopathy, one Dr Peters, publicly renounced homeopathy. I recommend having a quick read of it. The points made are just as valid today as they were in 1861. And yet since then, homeopathy has persisted in heading into an intellectual cul-de-sac. Perhaps this is because homeopaths only retain the bits of evidence that superficially seem to be in favour of homeopathy: everthing else goes down the memory hole, even when it is published in homeopathic journals.
For some time, I've been considering putting together a summary of the evidence on homeopathy, and sending it off to Homeopathy (the in-house comic of the Faculty of Homeopathy) for publication. It seems clear that there's no point, even if the journal would print it. Firstly, homeopaths will pretend it never existed. And secondly, there are plenty of decent books on the subject for those who are genuinely curious about the evidence. The homeopaths will continue to be deluded.
Occasionally, people sympathetic to homeopathy try to discuss scientific studies. It's usually fun when this happens, because I get chance to do some paper deconstruction. What can I say, I'm a geek. I usually learn something from doing it, even if only that I'm talking rubbish.
In the comments to this post, a lady (I assume) named Vicky posted three studies claimed to support homeopathy. Of these, the Graunke et al. paper on tadpoles has some glaring problems which are discussed here, and the Oberbaum et al. paper simply contains too few subjects (15 in each group) to come to any reliable conclusions. The third paper, by Bernstein et al., was a bit of an oddity, and I'd like to take a slightly closer look at it, in honour of homeopathy awareness week.
Firstly, the paper is published in the American Journal of Therapeutics. This journal does seem to be peer-reviewed, but is not listed in the ISI Journal Citation Reports, which suggests that it is not a top-of-the-line journal. But that in itself doesn't necessarily mean the findings are wrong.
The paper studies the effect of a proprietary homeopathic remedy, Relieva, on psoriasis. First red flag: psoriasis is a condition where patients experience flare-ups and remissions. So any improvement could be a result of this natural cyclicity rather than any treatment effect.
Relieva is an unusually concentrated substance in homeopathic terms. The study describes the active ingredient as "a highly concentrated, proprietary extract of M. aquifolium 10% known as Psorberine". This remedy actually contains some active ingredient, unlike many homeopathic remedies, so there is at least a chance of it working without needing to overturn practically everything we know about pharmacology.
The study is described as a randomised, double-blind, placebo-controlled study, and in general it seems to be well conducted. There were a total of 200 subjects, 100 in each group, suggesting the trial is large enough to get decent results. The authors conducted an 'intention to treat' analysis. In this analysis, all subjects enrolled in the trial are considered, rather than just those who completed it. The authors 'impute' a worst-case score (12 points) for Psoriasis Area Severity Index (PASI), a measure of psoriasis severity, to all the patients who dropped out of the trial. I'm not sure why they did this: they might have a sound reason for doing that, but I don't see it explained in the paper. Notably, 3 subjects dropped out of the treatment group, compared to 26 in the placebo group. This is a large difference, and it would be expected to introduce a bias into the results of the intention to treat analysis. Also, since a whopping 26% of the placebo arm dropped out of the trial, the results are at least of questionable reliability. The authors suggest that the high drop-out rate in the placebo group compared to the treatment group might be an indication of the efficacy of the treatment. However, their analysis of the subjects that completed the trial suggests that the placebo group experienced some improvement in their psoriasis (a PASI point reduction of 2.22 compared to 3.58 in the treatment group), so I'm not sure that explanation stacks up. The maximum increase was +3 for the treatment group and +4 for the placebo group. It's also worth noting that this big difference in drop-out rates has the potential to break the blinding of the trial.
The acknowledgments of the paper are very odd, at least to me. They say "The authors thank Global Clinicals, Inc (www.GlobalClinicals.com ) and its staff for performing this study, Missy Magill for the statistical evaluation, and Andrea Gwosdow (PhD) for preparing this manuscript". It's possible I'm being naive here, but this raises the question of what the authors actually did to earn the right to be credited as authors.
Who then are Global Clinicals, Inc? Here's their webpage. They undertake clinical research on behalf of their clients, so there is at least the appearance of a conflict of interest here; clients are not going to be happy if their treatment is shown to be no good. This is the sort of the thing that Big Pharma are (rightly) criticised for.
So overall, I would suggest that the evidence in the trial does not unequivocally show that Relieva works. There are also some shenanigans surrounding the authorship of the paper that would be highly criticised by homeopaths if they occurred in a trial of conventional medicine. Finally, even if you accept that the trial provides evidence in favour of Relieva, it provides no general evidence in favour of the basic tenets of homeopathy ('like cures like' and 'potentisation'). For that, you would need to show that Relieva causes psoriasis in healthy volunteers in a 'proving', and show that its effects increased with increasing dilution. If it does work, it's not because it's homeopathic.
This is the blog of Paul Wilson. You can find science- and pseudoscience-related things here, as well as occasional posts on my current research. There is also some stuff about what I do in my spare time (mostly cycling and complaining about politics).
I am a research associate in the School of Earth, Atmospheric and Environmental sciences at the University of Manchester, where I work on the structural geology and tectonics of sedimentary basins.
