A mystery paper...

Just a little bit more on the interview with Dana Ullman that I wrote about here.

Ullman claims that a re-analysis of Shang et al. has been accepted for publication in the Journal of Clinical Epidemiology. The only reference I can find to this study is this, where a study dated 2007, entitled "The conclusions on the effectiveness of homeopathy highly depend on the set of analysed trials" by R Ludtke and ALB Rutten is listed as being 'in press' in the Journal of Clinical Epidemiology.

Here's the list of articles in press in the Journal of Clinical Epidemiology. There is no sign of any such paper. Various searches fail to find any similar papers published anywhere else, or in earlier issues of the Journal of Clinical Epidemiology. The only thing I can find is a paper in Homeopathy called "‘Proof’ against homeopathy in fact supports Homeopathy", in which one Lex Rutten is credited as the first author. Whether this is the same Rutten I cannot say. The main point of the paper seems to be that if you add four positive trials to the Shang dataset, the result would be more positive. And they accuse Shang of cherry-picking. Two of the trials complained about were excluded [PDF] from the Shang meta-analysis: the Fisher et al. paper because it had an ineligible study design, and the Weisenauer and Gaus paper because no matching conventional trial could be found. Of the other two, one by Arnal-Laserre appears to be a French thesis of some description [EDIT: This is a French thesis: it was mentioned in the Cochrane review of "Homoeopathy for the induction of labour". Apparently, the reviewers could not obtain a copy of the thesis, which perhaps explains why Shang et al. did not include it], and the other by Maiwald et al. was not a placebo-controlled trial.

So, does this re-analysis exist, or is it just another figment of the collective homeopathic imagination? And if it ever does get published, is it likely that it will have anything useful to say?

Dana Ullman says the thing that is not...

...no surprise there, then.

Here's Ullman, a US base homeopath, in an interview published on the website of Sue Young, a London-based homeopath. There's all kinds of drivel here, but there is an exchange on the Shang et al. meta-analysis of homeopathy (published in the Lancet in 2005) that particularly caught my eye, because it's full of absolute nonsense. Not only that, but Ullman has had his misconceptions about this paper explained to him in numerous places on the internet, including on this very blog. Here's what he and his interviewer, one Louise Mclean of the Zeus information service, had to say:

DANA: In fact there is a new study that is coming out shortly which is a re-analysis of the 2005 Lancet review of Shang. The researchers got it accepted in a major international journal of research. What they have finally done is what Shang didn’t do. He didn’t review ALL of the high calibre research but only a small part of it. He ignored comprehensive analysis entirely. I think he knew exactly what it was but he didn’t want to report on it, as it was too positive. Instead he only reported on trials with very large numbers of subjects because when you do that, most of those studies use one remedy for everybody without any degree of individuality.

LOUISE: We individualise.

DANA: We do individualise but sometimes the single remedy or the formulas will work for a broad number of people.

LOUISE: Like Mixed Pollen for hayfever.

DANA: That’s right or Oscillococcinum. But for some reason they did not include any of David Reilly’s research. http://www.bmj.com/cgi/content/abstract/321/7259/471

I don’t know why they ignored it.

LOUISE: It was too positive.

DANA: In fact they had a remark in the Shang article published in the Lancet, where they specifically made reference to trials on respiratory ailments and that the results were robust, but they said they couldn’t trust them because there were only 8 studies. But then again they based their entire analysis on 8 homeopathic studies and 6 conventional ones. So they can’t have it both ways and this new journal article in the Journal of Clinical Epidemiology which is ranked as one of the top international journals of reviews of research, has accepted the new studies.

Sigh. Why is this nonsense? Let me count the ways.

1. Shang et al. did in fact analyse ALL of the trials of homeopathy that met their inclusion criteria. This allowed them to establish, using statistical methods, that smaller trials and those of less robust methodology showed better results for homeopathy, because of bias. The good quality, large studies showed that homeopathy had no effect. This is the pattern you would expect to see if homeopathy is a placebo.

2. Ah, individualisation. In fact, a number of the trials in the Shang study were of individualised homeopathy (including two of those that were considered large and of high quality). There was no evidence that individualised homeopathy was better than any other type of homeopathy (p=0.636). In any case, individualisation is only important when it suits Ullman, as seen when he says "We do individualise but sometimes the single remedy or the formulas will work for a broad number of people".

3. The meta-analysis not only included the Reilly paper in the BMJ that is linked to, but two other Reilly papers, as can be seen from the additional material [PDF] to the paper that is available online. This is contrary to Ullman's assertion that "for some reason they did not include any of David Reilly’s research".

4. The point that Shang et al. make about the 8 studies of respiratory ailments is that 8 studies was too few for a meta-analysis restricted to those studies to detect the bias that is revealed by an analysis of the complete dataset. The eight studies of homeopathy that Ullman wrongly claims Shang et al. "based their entire analysis on" were identified as the studies most likely to be free of bias, based on an analysis of the entire dataset. So the authors are not trying to have it both ways at all, and Ullman is comparing apples with oranges.

What I find particularly annoying about this is that Ullman and Mclean are essentially accusing Shang and his co-workers of research misconduct. What do they base this very serious accusation on? On a total misunderstanding of their paper, and a flat-out lie that they omitted research that was 'too positive', when that research was in fact included in the analysis. I am not a statistician, but the paper is not that difficult to understand, if you read it. Followers of Dana Ullman's career will not be surprised by his disingenuousness on this, I'm sure.

It seems that no matter how often I (and others, notably apgaylard) write about the persistent mis-representation of the Shang paper, the homeopaths carry on regardless.

Dana Ullman MPH gives me some homework

Dana Ullman MPH (the only person I've ever seen put their qualifications on their blogger username) commented on my piece on Alan Bennett's description of alternative therapies here, suggested that I'm confused about what homeopathy is, and gave me some homework to do. I've read Ullman's webpage, and I can tell you that it's a farrago of nonsense, written in superficially scientific language.

I was aware of some of Dana Ullman's work previously. For example, there's an entire thread dedicated to him at JREF, which you can find here. He posted there (under the username James Gully) to write some utter nonsense about famous scientific figures who supposedly used or supported homeopathy. Having had his arse handed to him by several posters on the forum, he accused them of 'intellectual dishonesty' and disappeared. But I hadn't seen this webpage before. It's entitled 'Why homeopathy makes sense and works', but fails to demonstrate either. You can find it here.

Ullman starts off by talking about side effects. He writes "It should be noted that people often incorrectly assume that conventional drugs have 'side effects.' Actually, in purely pharmacological terms, drugs do NOT have side effects; drugs only have 'effects,' and physicians arbitrarily differentiate between those effects that they like as the effects of the drug, while they call those symptoms that they don’t like 'side effects.' This is akin to saying that the effects of a bomb are that it destroys buildings, but its side effects are that it kills people. Needless to say, one cannot truly separate out one effect from the other. The reason that drugs create 'side effects' that are often worse than the original disease is that these drugs tend to suppress the symptoms the sick person is experiencing and push them deeper into the person’s body."

In fact, the distinction between effects and side effects is simply that the effects are desired, and the 'side effects' are not. It's not an arbitrary distinction, and no-one is trying to claim that side-effects are not effects caused by the drug. That's why medicines are tested for safety before they are licensed for sale. The reason why drugs have side effects is that they contain biologically active substances. A good explanation for the lack of side effects with homeopathic remedies is that they do not contain biologically active substances, that is, they don't work.

Ullman describes the principles behind homeopathy. In a section headed 'Determining what a medicine can cure', he writes about homeopathic provings. In a proving, subjects are given a dose of a substance, and their 'symptoms' are recorded. It is then assumed that a small (or nonexistent) dose of the substance will cure the same symptoms. Ullmann calls these provings 'toxicological studies', but it's easy to see that as scientific studies they leave a lot to be desired. It's an exercise in the post hoc ergo propter hoc fallacy, in that it's impossible to know whether the 'symptoms' were caused by the dose because there are no adequate controls.

Ullman then goes on to describe how homeopathic remedies are made, and talks about the power of 'nanodoses'. Firstly, this is a little disingenuous, as it suggests that homeopathic remedies contain very small amounts of an active ingredient, whereas in many cases they contain no active ingredient at all. Perhaps 'nonodose' would be a better term. Ullman describes the process of dilution and 'succussion' (shaking) well enough, but then bizarrely states that "It is inaccurate to say that homeopathic medicines are extremely diluted; they are extremely 'potentized'". It is not inaccurate to say that the medicines are extremely diluted: they are extremely diluted, as Ullman shows when he talks about serial 1:10 or 1:100 dilutions being conducted up to 1,000,000 times (as an aside, how can this easily be done? If I assume one 'potentisation' step can be done in one minute, it would take nearly two years to do 1,000,000 times, assuming I work 24 hours a day). The homeopathic theory is that this dilution and shaking makes the remedy more 'potent'. Apart from being against common sense (which after all can be wrong), this also goes against the dose-response effect well known from pharmacology, i.e. that a greater dose causes a greater effect. In a section headed "Other evidence on the power of nanodoses" Ullmann writes about certain compounds that have biological effects at very low concentrations, or that have very different effects at low concentrations than they do at high concentrations. This is not relevant to homeopathy, where substances are supposed to be biologically active at zero concentration.

Ullman also writes about clinical evidence for homeopathy. One thing about this section is that Ullman seems to misunderstand p values, when he writes that p=0.008 "means that there was a 99.2% chance that this treatment was effective". It means that if you conducted the experiment 1,000 times, you would expect to get a positive result 8 times through chance. It doesn't tell you about biases, poor experimental design, or other problems with the study (there's a useful discussion of some of these things here, here and here). For any of the examples Ullman gives of studies showing benefits for homeopathy, there are several that show the opposite. A recent well-conducted meta-analysis in the Lancet looked at homeopathy versus 'conventional' treatments. It found that the best conducted studies showed no benefit for homeopathic remedies beyond placebo, whereas the conventional treatments did show a benefit beyond placebo. Tellingly, Ullman mentions New Scientist, a popular science magazine that does not publish original research.

In the last paragraph of the 'clinical evidence' section, Ullman mentions 'water memory'. As regular readers of hawk/handsaw will know (hi to all two of you!), a recent issue of the journal Homeopathy was dedicated to this concept. None of the papers in it showed any 'memory' effect relevant to homeopathy, as discussed here, here, here, here, here and here. It's wishful thinking, at best.

For me, there's nothing on Dana Ullman's page that makes me think that homeopathy 'makes sense and works'. Not only that, but I didn't learn anything I didn't already know about homeopathy.

Edit: Here's a link to the Respectful Insolence blog on the COPD study mentioned by Dana Ullman in the comments to this piece.