Friday 20 June 2008

Fun with trials, for homeopathy awareness week...

Occasionally, people sympathetic to homeopathy try to discuss scientific studies. It's usually fun when this happens, because I get chance to do some paper deconstruction. What can I say, I'm a geek. I usually learn something from doing it, even if only that I'm talking rubbish.

In the comments to this post, a lady (I assume) named Vicky posted three studies claimed to support homeopathy. Of these, the Graunke et al. paper on tadpoles has some glaring problems which are discussed here, and the Oberbaum et al. paper simply contains too few subjects (15 in each group) to come to any reliable conclusions. The third paper, by Bernstein et al., was a bit of an oddity, and I'd like to take a slightly closer look at it, in honour of homeopathy awareness week.

Firstly, the paper is published in the American Journal of Therapeutics. This journal does seem to be peer-reviewed, but is not listed in the ISI Journal Citation Reports, which suggests that it is not a top-of-the-line journal. But that in itself doesn't necessarily mean the findings are wrong.

The paper studies the effect of a proprietary homeopathic remedy, Relieva, on psoriasis. First red flag: psoriasis is a condition where patients experience flare-ups and remissions. So any improvement could be a result of this natural cyclicity rather than any treatment effect.

Relieva is an unusually concentrated substance in homeopathic terms. The study describes the active ingredient as "a highly concentrated, proprietary extract of M. aquifolium 10% known as Psorberine". This remedy actually contains some active ingredient, unlike many homeopathic remedies, so there is at least a chance of it working without needing to overturn practically everything we know about pharmacology.

The study is described as a randomised, double-blind, placebo-controlled study, and in general it seems to be well conducted. There were a total of 200 subjects, 100 in each group, suggesting the trial is large enough to get decent results. The authors conducted an 'intention to treat' analysis. In this analysis, all subjects enrolled in the trial are considered, rather than just those who completed it. The authors 'impute' a worst-case score (12 points) for Psoriasis Area Severity Index (PASI), a measure of psoriasis severity, to all the patients who dropped out of the trial. I'm not sure why they did this: they might have a sound reason for doing that, but I don't see it explained in the paper. Notably, 3 subjects dropped out of the treatment group, compared to 26 in the placebo group. This is a large difference, and it would be expected to introduce a bias into the results of the intention to treat analysis. Also, since a whopping 26% of the placebo arm dropped out of the trial, the results are at least of questionable reliability. The authors suggest that the high drop-out rate in the placebo group compared to the treatment group might be an indication of the efficacy of the treatment. However, their analysis of the subjects that completed the trial suggests that the placebo group experienced some improvement in their psoriasis (a PASI point reduction of 2.22 compared to 3.58 in the treatment group), so I'm not sure that explanation stacks up. The maximum increase was +3 for the treatment group and +4 for the placebo group. It's also worth noting that this big difference in drop-out rates has the potential to break the blinding of the trial.

The acknowledgments of the paper are very odd, at least to me. They say "The authors thank Global Clinicals, Inc ( ) and its staff for performing this study, Missy Magill for the statistical evaluation, and Andrea Gwosdow (PhD) for preparing this manuscript". It's possible I'm being naive here, but this raises the question of what the authors actually did to earn the right to be credited as authors.

Who then are Global Clinicals, Inc? Here's their webpage. They undertake clinical research on behalf of their clients, so there is at least the appearance of a conflict of interest here; clients are not going to be happy if their treatment is shown to be no good. This is the sort of the thing that Big Pharma are (rightly) criticised for.

So overall, I would suggest that the evidence in the trial does not unequivocally show that Relieva works. There are also some shenanigans surrounding the authorship of the paper that would be highly criticised by homeopaths if they occurred in a trial of conventional medicine. Finally, even if you accept that the trial provides evidence in favour of Relieva, it provides no general evidence in favour of the basic tenets of homeopathy ('like cures like' and 'potentisation'). For that, you would need to show that Relieva causes psoriasis in healthy volunteers in a 'proving', and show that its effects increased with increasing dilution. If it does work, it's not because it's homeopathic.


Anonymous said...

Nice deconstruction. The drop-out rate in the control (placebo) group is very large. At the risk of seeming like a Bausell Booster, "Did 25% or more of the participants drop out of the study before it was over? (If so this will invalidate the study regardless of its results)."

Looks like the drop-out rate (at 26%) is large enough to get an experienced research methodolgist to call the results invalid.

It may also indicate a problem with the placebo or randomisation. Was the placebo indistinguishable - or could the control group tell that they wern't getting the 'treatment'? Were the subjects in the control group disproportionately approaching a flare-up when the study started?

Anyway, good work as ever.

Paul Wilson said...

Cheers, apgaylard.

The randomisation looks to be OK: the PASI scores were similar for both groups at the start of the experiment. But it's possible that subjects in the control group were disproportionately approaching a flare-up, as you say.

Just to clarify that the total drop-out rate was 14.5%, the 26% figure is for the control group. That still seems high though.

Paul Wilson said...

Another point worth making is that while the trial appears to show statistically significant results, it's not clear how clinically significant they are. The analysis is based only on the change in PASI and Quality of Life indexes. Here's an article discussing end-points for psoriasis trials. According to the article, the FDA standard for clinical effectiveness is a 75% reduction in PASI. According to the Bernstein et al. study, mean PASI in the treatment group was initially 6.93, with a mean change of -3.58 at the end of the study: a reduction of 51.7%. In the control group, there was a mean 2.22 point reduction in PASI, from an initial value of 6.85: a reduction of 32.4%. Of course, we don't know how many in each group acheived a clinically significant reduction in PASI, and the article doesn't define a clinically significant endpoint.

Anonymous said...

I see that The Cochrane Collaboration is now offering a regular column on CAM Cochrane Reports in Explore. Would it be a Frightfully Good Idea if some homeopathy journals were to implement a similar scheme?

Unknown said...
This comment has been removed by a blog administrator.
Anonymous said...

The whole article shows you started with a bias and miserably and pitiably concluded with a bias. People like you can not understand anything beyond the dictates of MNC pharma companies! You views are a fine example of how people analyse without knowing a subject nor willing to know it lest it will call your bluff!
Never mind - Homoeopathy will stay. In India it is common seen as this- when elephants walk majestically on the streets a number of street dogs bark, but the elephnats move on majestically!!!

Paul Wilson said...


It's always possible that my analysis of this paper is incorrect, but it doesn't get us very far if you don't bother to explain why you think it is incorrect.

Paul Wilson said...

People like you can not understand anything beyond the dictates of MNC pharma companies!

Well, I'm still waiting for my cheque.

In all seriousness, though, Relieva is not being handed out for free. According to this, it starts at $9.95 for the "lotion, cream and shampoo". It's just as much about making money as any of the big pharmaceutical companies are.