Muddying the water: Rustum Roy in the Grauniad

For me, Christmas came early this year, in the shape of a Guardian ‘Response’ piece by Rustum Roy, of all people. Roy is an eminent professor of materials science at Pennsylvania State University, but he also appears periodically to try and give scientific justification for homeopathy. I’m sure others will write about this, but I feel vaguely involved because of Roy’s connections to the recent ‘memory of water’ fiasco in the journal Homeopathy. Roy’s piece is entitled ‘“Homeophobia’ must not be tolerated”, and it seeks to rebut Ben Goldacre’s excellent and celebrated piece on homeopathy that was published in the Guardian’s G2 supplement a little while ago. There’s a lot of drivel for such a short piece, but I’ll try to keep it concise.

Roy begins by stating “For the record, I have never studied or held a position for, or against, the clinical effectiveness of homeopathy”. This seems a strange statement, given that Roy is responding to Ben Goldacre’s piece which mainly discussed that clinical evidence. Avoiding discussion of clinical evidence allows Roy to erect a classic straw man. He suggests that scientists main objection to homeopathy is the very high dilutions, such that there is very little chance of any of the original substance remaining, that are routinely used in homeopathic remedies. Indeed, many scientists do quite reasonably object to this, but the main objection to homeopathy is simply that the clinical evidence shows that it doesn’t work.

Unfortunately, Roy is not on firmer ground when discussing his own field of materials science. He attempts to create a theoretical justification for homeopathy, based on the idea of water being able to structurally ‘remember’ substances that are no longer present. Roy states that it is the structure of water that is important, not what it contains, and references Prof Eugene Stanley of Boston University in saying that there must be at least 64 different structures that occur in liquid water. He compares this to graphite and diamond, which are chemically of the same composition but have radically different properties because they have radically different structures. In this way he seeks to get around the problem of homeopathic remedies typically being diluted so much that it is very unlikely that any of the original substance can possibly remain. It’s the structure of the water that has an effect, not the substances dissolved or suspended in the water.

The fact that water can have so many different structures is interesting, but in terms of homeopathy it doesn’t really get us any further forward. To demonstrate even a theoretical justification for homeopathy, Roy would still need to show that the hundreds of different homeopathic remedies could all create different structures in water; that only the specific remedy, and not any other impurities in the water, causes this structuring; that water with different structures has different biological effects, and that the strength of that effect could increase as a solution is further diluted. So far Roy has only shown that water can contain interesting structures. One thing you could do to test Roy’s hypothesis is try to distinguish homeopathic remedies on the basis of structural differences between them. In fact, Roy and co-workers claim to have done this using spectrophotometric techniques, in a paper recently published in Homeopathy. This paper is so flawed that it’s almost a shame Roy doesn’t mention it in his piece. It doesn’t contain any usable observations, and all it shows is that the alcohol used to prepare the tested remedies likely came from multiple sources.

Roy also states that Prof Martin Chaplin of South Bank University has, in the journal Homeopathy, “discussed how water could retain a ‘memory’”. In fact, Chaplin has done no such thing. He has simply demonstrated that water has a lot of unique and interesting properties, something which is not in dispute. He doesn’t show how the structure of water in a homeopathic remedy could be influenced by substances that are no longer present. I will have a critical letter to the editor of Homeopathy published shortly outlining this point in more detail.

Roy says that the placebo effect is ‘without doubt present in every homeopathic intervention’, but goes on to say that ‘it is far more powerfully present in orthodox medical pills because they are advertised so widely in billion-dollar campaigns’. I’m not aware of any evidence that this is the case, although I’m certainly not an expert in this field. It seems unlikely because orthodox medicines are usually trialled for efficacy and safety before they come on the market. But in any case it seems like a confused argument. If the placebo effect ‘works’ and it is more powerful in ‘orthodox medical pills’, isn’t this evidence that orthodox medical pills are better? The central point is that the clinical evidence shows that homeopathic remedies have no more benefit than a sugar pill, while orthodox medicines do. But Roy conveniently doesn’t take a position on the clinical evidence.

Roy closes by asking whether Goldacre ‘seriously believes that a homeopathy paper with a positive outcome would be treated fairly in any mainstream journal’. In fact, a number of such papers have been published, although they have generally turned out to be flawed. One example is a paper on homeopathic treatment of critically ill patients that appeared in Chest, and has been thoroughly trashed here. The problem is not with the bias in mainstream journals, it is with the quality of research into homeopathy. Hence the Shang et al. meta-analysis found only 165 trials of homeopathy that matched the inclusion criteria, compared to 353,809 conventional trials. Trials of homeopathy are usually published in specialised journals whose standards of scrutineering leave a lot to be desired.

Essentially, Roy has failed to show that what he calls ‘homeophobia’ is anything other than a sensible response to the evidence that homeopathy doesn’t work, especially given that homeopaths are giving irresponsible and dangerous advice about such things as malaria.

My own bad science

Well, that's science for you. A paper that I worked long and hard on was sent out into the world, and I've got it back torn to shreds and covered in red ink. Yes, my paper was rejected. These things happen.

One of the reviewers thought the paper was OK, while the other one thought it had some serious flaws. The editor agreed with the second reviewer. I can't say that I'm happy about it, but I have to admit that the criticisms are not unfair. Still, I think the observations in the paper are useful and interesting, so I'm going to mount a salvage operation, and attempt to re-submit the paper.

It's not nice to have your work criticised, but of course, that's how science works. You submit your work for rigorous examination by other experts in your field, and only the best stuff makes it through. If the paper does eventually get published, it will be much stronger as a result.

At least, that's what I'm telling myself, but I can't pretend that it doesn't feel like a kick in the teeth...

Labour 'flexibility' in our universities

I went to a meeting of the Universities and Colleges Union the other day, about fixed-contract staff at the university. I'm on a fixed-term contract myself which will shortly be ending, so this is of more than academic interest for me. I was astounded by the statistics presented. Apparently, in 2005/6 the proportion of research staff at UK universities who were on fixed-term contracts was 85%! And at Manchester, it was 94%! Although the University of Manchester is supposed to be committed to reducing the number of fixed-term contracts, the proportion has only decreased to 94%, from 96% in 2004/5. Meanwhile, the proportion of teaching staff on fixed-term contracts has soared from 46% to 63%. The proportion of staff responsible for both teaching and research who are on fixed term contracts has declined significantly, from 24% in 2004/5 to 12% in 2005/6.

This raises a number of problems. Staff who are both teachers and researchers (i.e. they are lecturers, readers or professors) are generally responsible for heading up research groups, chasing up funding, and supervising research. They're the brains of the operation, if you like. But the backbone, the people who are responsible for most of the actual research, are post-docs on short-term contracts and graduate students. Research is essentially being done by temps.

Since they have no long-term job security, post-docs tend to leave as soon as they have an opportunity to do so. Especially as only around 20% of post-docs will ever get a permanent position as a lecturer. Essentially you have the option of sticking around on short-term contracts until you become unemployable, or you go and get a proper job. People tend to stay longer than maybe they should, because research is what they want to do, and they'll put up with a lot to keep doing it. There are post-docs at Manchester who have been here for over 10 years, all on short-term contracts.

Obviously, this is stressful and demoralising for individual post-docs. But it also can't be good for research. Good people leave, and those who stay are looking over their shoulder. And after training for many years to become a post-doc, at the bottom rung of the academic ladder, you can't blame people for thinking that they deserve better.

The answer is 42

The government has decided that it wants to extend the limit for detention without charge to 42 days. Why 42? Who knows? The government certainly doesn't seem to have any particular rationale. 90 days has recently been floated again, as has 56 days. Apparently 42 days is thought to be a reasonable compromise.

The proposal follows on from Tony Blair's defeat in the Commons over raising the limit to 90 days. An amendment was passed reducing the limit to 28 days, which is the current limit. This is still significantly longer than any comparable democracy, as this report from Liberty shows. Apparently we are facing a new and different threat, but why this requires such illiberal measures has never been made clear. The UK has faced terrorism before from the IRA, at a time when the maximum length of pre-charge detention was just three days. Why is the current threat so much more serious? And why is the threat so much more serious here than it is anywhere else?

Who wants this extension? According to an article in today's Guardian, no-one except the police and Lord Carlile, the independent reviewer of terror legislation. No-one else seems convinced, which seems reasonable enough as the government has provided no evidence to show that 42-day detention is necessary or desirable. In a democracy, we shouldn't just reflexively give the police what they want. We should ask why they want it, and what the costs and benefits to our democracy might be if we do give them what they want. This debate is happening, but there doesn't seem to be any evidence that the government is listening to it.

There was a debate on this in the Lords yesterday, which makes interesting reading if you're into that sort of thing. Lord West is grilled on why the proposals have been made, even though the process of consultation is not complete. Lord Thomas asks "despite all the consultation about which the Minister has talked, there has been not a shred of consensus on the further extension. Why has the process stopped now? Why do we not further seek consensus on such a divisive issue?". Answer came there none. It seems that the government has decided to pick a random number between the current 28 days and the mooted 56 days, and call this a 'consensus'.

In his evidence to the Lords, Lord West talks about detailed 'trend analysis' that shows that some cases will need more than 28 days. Without seeing this data, it's difficult to say much about it. However, an alternative interpretation would be that the trend analysis is showing detention periods increasing towards the limit, not because longer detention is needed but because it is available. This could be an argument against an extension, as extended powers are likely to be used whether they are needed or not.

Anway, it seems as though I'll be writing to my MP again, although it didn't do me a lot of good last time.

Edit to add: You can get some of the documentation associated with this here. The Summary of Responses to the Counter Terrorism Bill Consultation clearly says that "The responses to the consultation clearly indicate that pre-charge detention is a controversial issue and the majority of the responses which we received did not support an outright extension to the current 28 day limit". So what was the point in the consultation again?

The way the proposal works is that the Home Secretary could increase the limit to 42 days immediately, if a chief constable and the director of public prosecutions back the move. It must then be approved by the Commons and Lords within thirty days. The independent reviewer of terrorist legislation, Lord Carlile of Berriew QC, specifically recommended against this proposal. To quote from Lord Carlile's report:

"This might involve the recall of Parliament during a recess to deal with a single case which in itself was not the cause of a national emergency. This could prove extremely impractical. Further, the potential unfairness to the uncharged suspect of a Parliamentary debate on his/her case is self-evident."

So what is the point in the independent reviewer again?

It's also worth noting Lord Carlile's point that "There has been no case so far in which it has been shown clearly that more than 28 days’ detention would have been likely to result in charges which in fact were not made, although there remain many terrorist cases which are yet to be tried".

Homeopathy in the Manchester Evening News

I had no idea, but it turns out that the Manchester Evening News employs a homeopath, Zoe Meachem, as a 'Health and Beauty' columnist. Here's her recent column on prevention of cystitis. In the column, she recommends a number of homeopathic remedies at a 'potency' of 30c. In a 30c remedy, the 'mother tincture' is diluted by a factor of 100, thirty times. This is a dilution factor of 1x10⁶⁰, or a 1 followed by sixty zeroes. The chance of their being any of the 'mother tincture' left after this dilution is vanishingly small. The best available evidence, i.e. meta-analyses of placebo-controlled trials, shows that homeopathy has no effect beyond placebo. In short, these remedies won't do a single thing to prevent or cure cystitis. Pity, because the other advice in the column is sensible. Drink plenty of water, and seek medical attention if you have a fever, lower back pain, blood or sediment in the urine.

Ms Meachem is registered with the Society of Homeopaths, whose code of conduct says that "No advertising may be used which expressly or implicitly claims to cure named diseases". While a comment piece in the MEN probably doesn't count as advertising, perhaps it should, especially given that the piece ends with the words "To make an appointment with Zoe call xxxx xxx xxxx".

Ms Meachem also has a BSc (hons) in Homeopathic Medicine, to which the only possible response is "You've got a degree in baloney!"

What's wrong with Shang et al.?

Shang et al. recently published a meta-analysis comparing homeopathy with 'conventional' treatments in the Lancet (Lancet 366; 726-732). This is the most recent of the meta-analyses cited by Ben Goldacre in his excellent Guardian piece on homeopathy. Homeopaths have not been amused by this paper, and have tried very hard to discredit it. This is perhaps predictable since the paper concludes that homeopathic treatments have no effect beyond placebo, once the poorly-designed and biased studies have been stripped out. The consensus amongst homeopaths seems to be that the study is outrageously flawed, and scientists show themselves up by referring to it. Some recent discussions of it involving homeopaths can be found here and here. Despite the large amount of noise surrounding the discussion of this paper, I haven't seen a lot of signal, and so far I haven't seen a critique that stands up to scrutiny.

What did the paper do? The authors set out to test the hypothesis that homeopathic treatment effects can be attributed to the placebo effect. If that were the case, then any positive trial results for homeopathy would have to result from poor study design and/or bias. The authors tested this proposition by identifying 105 papers reporting 110 trials of homeopathy, and matching them with 110 trials of 'allopathy', or conventional treatments, on the basis of disorder treated and type of outcome measured. The authors then assessed the methodological quality of the papers, based on factors such as whether the trial was adequately blinded and whether it was adquately randomised. The paper found that for all the homeopathic trials, there was an effect beyond placebo. However, when the trials that were of low methodological quality and/or had sample sizes that were small were stripped out of the analysis, the remaining 8 trials showed no effect beyond placebo. On the other hand, when the same procedure was followed for the conventional medicine trials, the six remaining trials did show an effect beyond placebo.

So far, so good. What have been the criticisms of the paper?

One criticism has been that the trials deemed to be large and of higher quality were not identified, and that the reporting of the meta-analysis was inadequate. This criticism does carry some weight, and the reporting in the original paper was not good enough. However, the authors recognised the problem, and rectified it by identifying the trials in a reply to published criticisms that appeared in the Lancet (Lancet 366: 2083). You can find all the details of the study via apgaylard's blog here. So, this criticism is no longer valid.

Another criticism has been that the meta-analysis only uses 8 papers out of 105 to conclude that homeopathic remedies are no better than placebo. This seems to totally miss the point of the study. For one thing, it's a meta-analysis, so it pools studies in order to get more statistically significant results than single studies. The eight studies of homeopathy have a total n of 1,923, which is quite respectable. Also, Shang et al. have not employed some sort of sleight of hand to dismiss the other 97 papers. They have filtered them out because they are of inadequate methodological quality and/or size, based on clearly stated criteria. This allows the authors to compare the results from all the studies with the results from the best studies. When you use only the best studies, there is no longer any benefit for homeopathy beyond placebo. In contrast, using the best studies of conventional treatments, there is an effect beyond placebo. Again, this is the whole point of the study, and criticising it on the basis that it seeks to use the best-quality studies seems somewhat misguided.

Another common criticism from homeopaths is that the study doesn't test 'real' homeopathy. Shang et al. split studies of homeopathy into four types:

1. Classical homeopathy: individualised treatment based on homeopathic history-taking
2. Clinical homeopathy: no history-taking involved, each patient gets the same remedy
3. Complex homeopathy: patients take a mixture of several different remedies
4. Isopathy: the agent judged to be the cause of the disorder was used

For example, here's a website where they state flat out that there is no such thing as clinical homeopathy. This would be news to anyone who has wandered into Boots and seen the homeopathic remedies on sale there. More commonly, the criticism is that only 'classical homeopathy' is really homeopathy, and the other types don't count. Even if we allow this criticism, the fact is that 18 of the included trials were of 'classical homeopathy', as defined by the authors, and two of those made it into the group of eight large, high quality trials. The statistical analysis also showed that there was little evidence that effects differed between different types of homeopathy. So, not only did the study include trials of individualised homeopathy, it showed that these were no more effective than the other forms of homeopathy.

So, on the whole, it seems to me that the methodology of Shang et al. is reasonable, and the conclusions justified. I think it's probably true that no study is entirely without flaws, and I'm willing to be corrected on this. But so far I've seen no good criticism of the Shang et al. study that invalidates its conclusions.

Edit: Just as an aside it's interesting to read the second last paragraph of Shang et al., where they discuss the place of homeopathy in treatment systems. I take the liberty of reproducing the paragraph below:

"We emphasise that our study, and the trials we examined, exclusively addressed the narrow question of whether homoeopathic remedies have specific effects. Context effects can influence the effects of interventions, and the relationship between patient and carer might be an important pathway mediating such effects. Practitioners of homoeopathy can form powerful alliances with their patients, because patients and carers commonly share strong beliefs about the treatment’s effectiveness, and other cultural beliefs, which might be both empowering and restorative. For some people, therefore, homoeopathy could be another tool that complements conventional medicine, whereas others might see it as purposeful and antiscientific deception of patients, which has no place in modern health care. Clearly, rather than doing further placebo-controlled trials of homoeopathy, future research efforts should focus on the nature of context effects and on the place of homoeopathy in health-care systems."

This seems to be entirely reasonable, and suggests that Shang et al. have no particular bias against homeopathy.

Something to prove

There's an interesting debate currently taking place in the pages of the journal Homeopathy. It's interesting in that there is actually an attempt to have a debate about one of the fundamentals of homeopathic practice. It's also interesting to see some of the anti-scientific responses to the debate. The original papers discussed can be found through the website of Homeopathy. Unfortunately, you probably won't be able to access them unless you or your institution subscribes to the journal.

In the January issue of Homeopathy, Dantas et al. published a review of homeopathic provings, or Homeopathic Pathogenetic Trials (HPTs), as the authors prefer to call them. The authors defined HPTs as being "clinical trials designed to investigate the effects of the exposure of human volunteers, in good health, to potentially toxic or pathogenetic substances, diluted and serially agitated according to homeopathic pharmacopoeial methods, with a view to providing data to inform their use as homeopathic medicines". The idea is that symptoms caused by the homeopathic preparations can be cured by the same preparations, under the principle of 'like cures like'. There is no good evidence that this principle can be applied as a general rule, but even so it has become one of the foundation stones of homeopathy. One problem is that the symptoms in an HPT are recorded by the volunteers who take part in the proving. No quantitative data is collected about symptoms, and there are well-known problems with such self-reporting studies. Also, in many cases there is no way of telling whether the symptoms occurred as a result of the homeopathic preparation, or for some other reason, because such trials are not always placebo-controlled (Dantas et al. claim that 58% of the trials in their review were placebo-controlled).

The review by Dantas et al. concluded that "Most studies had design flaws, particularly absence of proper randomization, blinding, placebo control and criteria for analysis of outcomes", and went so far as to state that "The central question of whether homeopathic medicines in high dilutions can provoke effects in healthy volunteers has not yet been definitively answered, because of methodological weaknesses of the reports". Their central point is that while provings often turn up all kinds of symptoms, methodological flaws mean that you can't tell whether the symptoms were caused by the homeopathic preparation or not. The authors recommend that improved methodology should be adopted for future HPTs.

This is interesting stuff, and suggests that there are at least some homeopaths who question the value of HPTs, and on perfectly reasonable scientific grounds. It all starts to go a bit wrong in the responses to the article, which were published in the current issue of Homeopathy.

Sherr and Quirk's response is probably the most fun, and I suggest you track it down for yourself (but only if you've got time for such nonsense). Their point of view can be summarised by a paragraph towards the end of the paper, where they state "Eliminating the majority of symptoms or characteristic single symptoms due to over scientific vigour or a concern about statistical significance or background noise, risks throwing out the baby with the bathwater. It is important to remember the proof of provings is first and foremost their clinical usability and efficiency". Over-scientific vigour or a concern over statistical significance, indeed. This is pre-enlightenment thinking if ever I saw it.

They also say that "A good proving is not about producing every possible symptom. It is about producing enough symptoms of quality so that the intelligent homoeopath can perceive a meaningful totality". I take this to mean that you don't have to worry about using the best possible methodology, because the homeopath has some magic way of 'perceiving a meaningful totality'. Also, the object of the proving is to produce 'enough symptoms', not the ones actually caused by the preparation. (Here I gloss over the fact that homeopathic preparations tend to contain no active ingredient, so will in all likelihood produce no symptoms at all). This is illustrated by a proving of hydrogen mentioned in Dantas et al., where the original trial produced 50 times more symptoms than a subsequent trial with improved methodology. According to Sherr and Quirk, the problem here is not with the original trial, but with the improved one, which produced too few symptoms to constitute a usable proving.

Dantas et al. respond with a paper entitled 'We must distinguish symptoms caused by the medicine from other symptoms'. In this case, the title is probably an adequate response on its own.

Then Harald Walach has a paper in response to Sherr and Quirk, entitled "Potential nonlocal mechanisms make placebo controls in pathogenetic trials difficult". This, once again, is quantum gibberish being used to claim that placebo-controlled trials can't work for homeopathy, because of 'entanglement' between patient, practitioner and remedy. Not to put too fine a point on it, this is bollocks, because entanglement has not been observed for systems containing more than a few particles. This is just homeopaths trying to find a way out of all the negative placebo-controlled trials of homeopathy. The attempts by those sympathetic to homeopathy to explain it via quantum mechanics are taken apart in some detail on shpalman's blog here and also here. My favourite bit of Walach's response is this: "It is a well-known lore of homeopathic proving that those in control groups, relatives, or even the pet dog may develop proving symptoms although they have not taken the remedy. This lore, although anecdotal and not scientific evidence at all, is valuable since it suggests that placebo controls might not be adequate". So, although this 'lore' is 'not scientific evidence at all' it is still valuable as evidence that placebo controls may be inadequate. Hm. Perhaps another explanation is that the proving symptoms recorded in the trial had nothing whatever to do with the homeopathic preparation being trialled, and so could be expected to be found in people (or dogs) not taking the preparation? That's why you do a placebo-controlled trial in the first place, and that's why quantitative data (as opposed to self-reporting) on symptoms are so important.

At the end of it all, you have to wonder what would happen if relatively sceptical authors such as those responsible for Dantas et al. started to address the results from meta-analyses that persistently show that homeopathic preparations have no benefit beyond placebo. Unfortunately, there seems to be no sign of this happening, as the authors conclude their paper by saying "As evidence accumulates for the efficacy and safety of homeopathy from rigorous clinical trials, there is an increasing need to investigate and develop valid methodologies for the experimental pillar of homeopathy—the homeopathic pathogenetic trial". Still, perhaps this drive towards better methodology may have unintended consequences. As we know from the Shang et al. meta-analysis in the Lancet, the better the methodology of your study, the more likely it is to show no effect beyond placebo for homeopathy.

References

Dantas, F., Fisher, P., Walach, H., Wieland, F., Rastogi, D.P., Teixeira, H., Koster, D., Jansen, J.P., Eizayaga, J., Alvarez, M.E.P., Marim, M., Belon, P. and Weckx, L.L.M. 2007. A systematic review of the quality of homeopathic pathogenetic trials published from 1945 to 1995. Homeopathy, 96: 4-16.

Dantas, F., Fisher, P., Rastogi, D.P., Teixeira, H., Eizayaga, J., Alvarez, M.E.P., Belon, P. and Weckx, L.L.M. 2007. Authors' response: we must distinguish symptoms caused by the medicine from other symptoms. Homeopathy, 96: 275-276

Shang, A., Huwiler-Müntener, K., Nartey, L., Jüni, P., Dörig, S., Sterne, J.A.C., Pewsner, D., Egger, M. 2005. Are the clinical effects of homoeopathy placebo effects? Comparitive study of placebo-controlled trials of homoeopathy and allopathy. Lancet, 366: 726-732.

Sherr, J. and Quirk, T. 2007. Systematic review of homeopathic pathogenetic trials: an excess of rigour? Homeopathy, 96: 273-275

Walach, H. 2007. Response: potential nonlocal mechanisms make placebo controls in pathogenetic trials difficult. Homeopathy, 96: 278.