Hawk/Handsaw

Infinite costs wouldn't be Nice

2010-08-25T14:47:00.004+01:00

There's been a bit of a furore over the last couple of days about a decision not to approve a drug for treating advanced bowel cancer, Avastin, on the grounds that it would not be cost-effective. This has led to a huge amount of criticism, and headlines ranging from the relatively sober to the Mail's hysterical "Betrayal of the cancer patients: Rationing watchdog accused of talking down wonder drug's power to save lives".

The decision (or guidance: in fact, the final decision has not yet been made, with the guidance subject to consultation and appeal) was made by the National Institute of Health and Clinical Excellence (Nice), which decides on the cost-effectiveness of treatments and whether they should be available to patients on the NHS. According to trial data submitted by the drug manufacturer, Roche, the drug can extend the mean lifespan of people diagnosed with advanced bowel cancer from 19.9 months on chemotherapy alone, to 21.3 months with chemotherapy and avastin. This is by no means a miracle cure or wonder drug. This is a six-week gain, but it comes at a cost: the drug costs up to £21,000 per patient.

Barbara Moss, a survivor of bowel cancer, appears in both the BBC and Guardian articles. The Guardian says:

Mrs Moss said the National Institute for Health and Clinical Excellence (Nice) had put a "value on life" after it said the price was too high for the extra benefit it gives patients

While Mrs Moss is quoted in both pieces as saying:

It seems immoral to me that, as a result of negative NICE decisions like this one, people's choice of living or dying depends on whether they can afford a drug, because it isn't available to them on the NHS.

It's difficult not to have some sympathy for this point of view. But someone has to put a "value on life" (or in fact, a thing called a Quality Adjusted Life Year, QALY). Otherwise Roche could come up with a life-saving drug that cost £1 trillion per patient, and the NHS would have to fund it because you "can't put a value on life". Ultimately, the question is not just about avastin. There is only a limited amount of money for the NHS to spend on drugs, so if avastin gets funded something else (with probably greater benefit) doesn't get funded . These are clearly not easy decisions to make, which is why we have Nice to do it in the first place.

My point here is not that Nice have necessarily got this right (I'm not qualified to judge), or even that the QALY approach is the best possible one. The point is that Nice is doing a difficult, unpopular and necessary job. Accusing them of betraying cancer patients, when what they're actually trying to do is get the greatest benefit from a limited budget, is not really helpful to anyone.

Bad cycling: I'm the champion (but I'm not very fast)

2010-08-24T11:28:00.003+01:00

Last night was the Stockport Clarion club hill-climb, on the infamous Cat and Fiddle road between Macclesfield and Buxton. I rode in this event last year, and had a good ride on a clear and dry day with a good tailwind. I had high hopes of bettering the 26:00 I did last year, but things didn't go exactly to plan.

There's a webcam attached to the side of the Cat and Fiddle pub, which looks over the finish of the course at the road summit. I had a look at it yesterday afternoon, and there wasn't much to see except cloud and massive wagons heading over to Buxton with their fog lights on. It didn't look like a lot of fun. Once I got home, I was having serious second thoughts about the whole enterprise as I got ready for the 18-mile ride to Macclesfield, with the rain rattling the windows in my flat. But I made myself head out, leaving in absolutely filthy weather and gnarly traffic for a fun rush-hour jaunt through Stockport. The weather gradually cleared up as I got towards Macclesfield, and I arrived with about ten minutes to spare before the 7pm start.

Unfortunately, the start had been put back to 7:30, so I ended up waiting around in the cold for 40 minutes. Will, the timekeeper, didn't make it until just before 7:30 because he had to pick someone up from the airport. So, dispensing with numbers, the four riders who had shown up got going as soon as possible. The weather was now dry, but cool and windy. I decided to ride with lights in case it was cloudy at the summit. I went off number 2, and it was one of those days when you know it's going to be rough as soon as you set off. I hadn't had enough of a warm-up, and my legs just didn't want to have any of it. I didn't really get settled in until I was past Walker Barn and had done the most difficult bit. By then, the two guys behind me had already gone past, but I had caught up with my minute man. Emerging into the open country beyond Walker Barn, the gale-force tailwind came into play, and the middle section of the course was very fast. I even had to touch the brakes on some of the corners. Then the last climb onto the top: I almost got blown off the road on a cross-wind section, and struggled past the timekeeper in 26:36, 36 seconds slower than last year. But, with the only other Stockport Clarion rider being my minute man, who I had passed about two and a half miles earlier, I was the club hill-climb champion. This is nice, especially as I was several times hill-climb champion at my former club, Birdwell Wheelers, but there's no pretending that I was particularly quick. Several riders in the club would have put minutes into me had they ridden, but like the lottery, you have to be in it to win it. Jolan managed to capture a webcam image of us at the top: I'm the guy in the red shirt at left...

My reward was a long, windy ride home, mainly in darkness, but I got to see the sun go down over Fiddlers Ferry power station as I headed back down into Macclesfield.

Bad cycling: I get round in 27:45

2010-08-17T11:47:00.003+01:00

Following my New Year's Day adventures, I haven't done as much bike racing as I hoped I would this year. I was away In Norway all of June for work, and then the Stockport Clarion evening time-trials took a break during July. However, the last three Monday evenings I've been out on the Chelford 10-mile course, in an attempt to get some race fitness for next week's hill climb on the Cat and Fiddle. My aim was to get under 28 minutes for 10 miles, which is pretty modest. After a horrible day in the first event, where I was recovering from a migraine and should probably have stayed at home, I clocked 29:30 and wasn't optimistic. But the following week was better with 28:24, and last night I cracked it, just about, with 27:45. For the first time in these 10s, there were actually a couple of people slower than me. It was a good night, dry and sunny with a light headwind on the slightly longer outward leg, and times were generally fast-ish. There was a 21-minute ride, which is pretty quick for such a slow course.

Coming into Chelford on the homeward leg, there was an electronic sign that gives a read-out of your speed. I was grovelling up the slight incline at a bare 21 mph, so the days when I could get under 25 minutes seem long ago. One of the other riders went through at 27 mph. Still, I felt good, and I'm looking forward to the Cat and Fiddle next week.

"Dr" Nancy Malik is spamming my blog again...

2010-08-17T11:09:00.002+01:00

In the comments to a couple of old posts, homeopathic apologist and internet numpty extraordinaire Nancy Malik says the following:

Studies in support of Homeopathy published in reputed journals

1. Scientific World Journal
http://www.ncbi.nlm.nih.gov/pubmed/17982565

2. Lancet
http://www.ncbi.nlm.nih.gov/pubmed/9310601

3. Neuro Psycho Pharmacology
http://www.nature.com/npp/journal/v27/n2/abs/1395862a.html // Bacopa Monnieri for memory

I thought I'd have a quick look and explain why they're nonsense. Unfortunately, this hasn't proven to afford much in the way of intellectual exercise.

The first paper is by Graunke et al., and concerns, I kid you not, the treatment of tadpoles with homeopathic thyroxin. This is a well-known bad homeopathy paper. The tadpoles in the treatment group were more developed than those in the control group at the start of the experiment, so it wasn't much of a surprise that they were more developed at the end too. There is more discussion of this dreadful rubbish here.

The second paper is the famed Linde et al. meta-analysis, published in 1997. While this paper does say "The results of our meta-analysis are not compatible with the hypothesis that the clinical effects of homeopathy are completely due to placebo", there are some other things to bear in mind:

1. The paper also says "However, we found insufficient evidence from these studies that homeopathy is clearly efficacious for any single clinical condition".

2. A 1999 paper by the same authors, using improved methodology and including new trials, states that "It seems...likely that our meta-analysis at least overestimated the effects of
homeopathic treatments".

3. A subsequent meta-analysis by Shang et al., published in the Lancet in 2005, using further improved methodology concluded that the results were compatible with homeopathy being a placebo.

Finally, the third study, by Roodenrys et al. in the journal Neuropsychopharmocology, is not about homeopathy at all, but rather about herbal medicine. In homeopathy, remedies are typically diluted such that it is very unlikely that they contain any of the original material: there is no active ingredient. In the Roodenrys study, what is being tested is brahmi, an Indian herb, of which the paper says:

Studies have shown that the herb contains many active constituents, including a number of alkaloids and saponins, however, the major constituents are the steroidal saponins, Bacosides A and B.

So it isn't entirely surprising that brahmi might have some effect.

From this fairly cursory glance at the studies provided by Nancy Malik, it's clear that she is from the Dana Ullman school of evaluating journal articles. This involves finding some papers that superficially appear to support your position, and then spamming them all over the internet. Luckily, for this approach there is no need to understand the articles, or even to read them. For people who think that magic water is medicine, that would be rather too much to expect.

Private Eye still believes in Wakefield...

2010-02-03T21:18:00.002Z

As discussed in my previous post, and many other places all over the internet, the GMC ruling against Andrew Wakefield was damning. But not so much if you write for Private Eye, who have been defending Wakefield for years. So, in the light of the GMC findings that Wakefield behaved unethically, with "callous disregard" for the wellbeing of the children involved, and that parts of his research were fraudulent, is it time for the Eye to admit it was wrong?

Apparently not. In an "In the Back" piece, the Eye had the following to say:

Although the GMC said the hearing was not about vaccination and autism, it is fairly clear that the two and a half years of disciplinary proceedings were to bring the MMR debate to a conclusion. The three doctors were to some degree being accused (and found guilty) of causing a public health scare which led to a fall in vaccination rates, so damaging "herd immunity", particularly with regard to measles.

Nope. They were found guilty of conducting invasive diagnostic procedures, against the children's interests and without ethical approval; of not disclosing serious conflicts of interest; and of presenting false information in the publication of their research (which finally led to the Lancet retracting the paper).

Several parents of the children who featured in the team's original research paper, which was at the centre of the GMC case, stormed out of the hearing in angry protest at the findings - particularly the suggestion that their children's tests were not clinically necessary...They say they would have told the GMC that the treatment they received at the hospital helped their children's symptoms - but they were never called to testify.

Eh? We're not talking about treatment here, we're talking about invasive diagnostic tests. These were done without ethical approval, without the required expertise, and against the interests of the children. Perhaps we should just get rid of ethical approval, as it gets in the way of heroic doctors doing whatever they need to vunerable patients.

And finally:

None of this debate about conduct, however, changes the fact that no subsequent research has supported Wakefield's thesis of a possible link between the MMR vaccine and autism. Nor does it change the fact that despite previous attempts to justify the Eye's coverage at the time, some readers are still deeply critical of the magazine's reporting of the MMR debate between 2001 and 2007, when we wrote about the concerns of Wakefield, the families and their lawyers, and endorsed calls for more research (see Letters).

Well, indeed. So why is the Eye still apparently defending the indefensible? I still buy the Eye, for all the other good that they do, but they've made a mistake over MMR and they ought to be big enough to admit it. They would surely ask the same of any of the politicians and industry figures they regularly lampoon.

It's literature integrity week...

2010-02-03T20:19:00.003Z

The integrity of the scientific literature is at issue in not one but two news stories this week. Firstly, the 1998 Lancet paper by Andrew Wakefield and several co-authors, which purported to show a link between autism and the MMR vaccination, was retracted [PDF] by the journal. Meanwhile, there is an ongoing brouhaha about a 1990 Nature paper that has some implications for climate change research.

I'm only going to deal with Wakefield here because, you know, I have other things to do. The retraction by the Lancet is really the final nail in the coffin of Wakefield's paper, which has been heavily criticised for numerous reasons. Of the thirteen authors, ten had previously retracted the interpretation that there was any link between autism associated with gastro-intestinal problems and the MMR vaccine. Only Wakefield himself and one P Harvey refused to sign up to this: the remaining author could not be contacted. It has also been known for some time that the methodology of the paper was flawed. So what suddenly changed this week?

For some time the General Medical Council (GMC) has been investigating Wakefield and two colleagues, Simon Murch and John Walker-Smith, over allegations relating to their research on autism. The GMC findings were released on January 28th, and can only be described as devastating. You can find the whole thing here [PDF], or there's a good summary at Respectful Insolence.

The GMC did not look at whether the research findings were right or wrong: rather they looked at the research methodology. Essentially, the major problems were these:

1. Wakefield had an undisclosed conflict of interest, because he was being paid by lawyers whose clients believed their children had been harmed by the MMR vaccine.

2. Wakefield ordered invasive diagnostic tests, including colonoscopies and lumbar punctures, that were unnecessary and not in the children's interests, and he had neither the required ethical approval nor the requisite expertise to order those tests.

3. Wakefield obtained blood samples at his son's birthday party by paying children £5 each. The GMC panel described Wakefield's actions as comprising "callous disregard for the distress and pain the children might suffer".

4. The 12 children were described as having been "consecutively referred", but this was not true.

Heres what the Lancet had to say:

Following the judgment of the UK General Medical Council’s Fitness to Practise Panel on Jan 28, 2010, it has become clear that several elements of the 1998 paper by Wakefield et al1 are incorrect, contrary to the findings of an earlier investigation.2 In particular, the claims in the original paper that children were “consecutively referred” and that investigations were “approved” by the local ethics committee have been proven to be false. Therefore we fully retract this paper from the published record.

So that's that. The problem with the paper is not that it was wrong; that has been known for some time. In fact, simply being wrong would not be a reason to retract the paper. Science often progresses by building on papers that were not quite right. No, the problem with the Wakefield Lancet paper was that it was fraudulent, unethical and incompetent, as well as being wrong. As a result, the takeup of MMR has fallen below the ~95% level at which herd immunity is maintained, and measels has once again been declared endemic in the UK. That's why you should try to avoid publishing fraudulent and unethical research. We'll let you off if your research is merely wrong, as long as it was honestly and competently wrong.

Wakefield eh, what a hero? Unfortunately, as we'll see in my next post, some people in the most unexpected places still believe in the Cult of Andy...

The Observer ignores the evidence on homeopathy

2010-01-31T15:42:00.003Z

Homeopathy is in the news once again, following a campaign set up by Merseyside Skeptics in which skeptics took an overdose of homeopathic pills, demonstrating that there's nothing in them other than lactose, and then went to the pub for some non-homeopathic beer. The campaign was mainly aimed at high-street pharmacist Boots, whose professional standards director admitted to the science and technology select committee of the House of Commons that there was no evidence that homeopathy worked, but they were very happy to continue charging people money for it. The campaign, and a forthcoming report of the select committee on whether the NHS should fund homeopathy, gave the Observer an excuse to publish a feature article, in which Anushka Asthana and Robin McKie examine the supposed controversy about whether homeopathy works or not.

Of course, scientifically speaking there is no controversy over homeopathy. It's perfectly clear that there is no scientific reason why it ought to work, and that when it is tested in properly conducted trials it works no better than placebo. I grant you that this probably wouldn't make for a very satisfying Sunday newspaper feature, and it would certainly be rather too short to fill up all of page 30 of the Observer. But you might expect there to be some discussion of the actual evidence. Not really, though. Edzard Ernst gets to say that "I have now published more than 100 papers on homeopathy and I am quite clear about its efficacy: you may as well take a glass of water than a homeopathic medicine". The evidence of Jane Lawrence of the Royal Pharmaceutical Society to the House of Commons select committee on science and technology that "There is no basis for [homeopathic remedies] being effective" is also quoted. But then Cristal Sumner of the British Homeopathic Association gets away with saying "Homeopathy helps patients and is not a placebo effect", despite all the evidence showing exactly the opposite.

This is typical of how the press treats scientific issues; they are presented in a superficially even-handed way, but crucially there is no attempt to weigh the evidence. Again, there is no scientific controversy over homeopathy, but by reading the Observer article you could be forgiven for thinking there was considerable room for doubt.

The worst things about the article, though, are the two inset boxes (these only appear in the printed version of the article, not in the online version, as far as I can tell). The first box is headed "Common Renedies" and lists "Popular homeopathic remedies for sale in Britain", including Arnica for clearing up bruises, and mixed pollen for treating hayfever. As these are homeopathic, they contain no Arnica or pollen, and there is no evidence they do anything except lighten your wallet, but this is not mentioned in the box. The second box is headed "Case Study", and recounts the story of one Helen Llewelyn, who claims that homeopathy helped control her endometriosis. This is an anecdote; it tells us that Llewelyn feels better, but it doesn't tell us anything about why, especially as we know nothing of what real medicines she might have been taking. From this website, it looks as though Llewelyn's case history is rather complex, and she is presently using homeopathy in conjunction with several kinds of real medicine. For medical treatments, you need much better evidence than this to show that they work. In the case of homeopathy, the evidence exists and it shows that homeopathy doesn't work. But there isn't an inset box anywhere that sums up the useful evidence: just one uncontrolled case report.

Given that there is genuinely a debate about whether the NHS should fund homeopathy, it would be good for this debate to be informed by the best currently available evidence. That is exactly what the Observer fails to do: all we get is "he said, she said" appeals to various forms of authority, and an anecdote. No wonder the newspapers are struggling.

Medical Hypotheses row resurfaces

2010-01-16T14:57:00.005Z

Last year, publishers Elsevier got into trouble with HIV-AIDS researchers, after Medical Hypotheses (an Elsevier journal) published two papers on the subject of AIDS: one by Peter Duesberg claiming that the AIDS epidemic in South Africa was overhyped, and another by Marco Ruggiero suggesting that the Italian health ministry did not believe that HIV was the sole cause of AIDS (blog posts at Bad Science and Respectful Insolence). The papers were deeply flawed, and were retracted by Elsevier pending an investigation into how they were published. The story has now resurfaced in the Times Higher Education Supplement (THES), because:

Prominent Aids researchers contacted Elsevier to object to the article and wrote to the US National Library of Medicine requesting that Medical Hypotheses be removed from the Medline citation database - an act that would exclude it from the mainstream scientific-communication network.

Elsevier have now convened an expert panel to decide on the future of Medical Hypotheses, with conclusions due by the end of 2010.

In fact, there is no great mystery as to how these flawed papers came to be published. Medical Hypotheses is not peer reviewed: instead, decisions on publication are taken solely by the journal's editor, Prof Bruce Charlton. Articles are often accepted within days, or even hours, of being submitted, suggesting there is little or no quality control on what gets published. Prof Charlton defends this process on two grounds: firstly, that there ought to be some outlet for speculative and bizarre ideas that will not be published by mainstream journals. Secondly, that Medical Hypotheses is a successful and influential journal. Here's what he has to say on the comments following the THES article:

The basic facts are that Medical Hypotheses - www.elsevier.com/locate/mehy - is explicitly and proudly editorially-reviewed (i.e. by me - not peer reviewed); aims to publish radical and revolutionary scientific ideas; and it is objectively a successful journal. It makes a profit, the Thomson ISI Impact Factor is 1.416 (much better than average, and rising), and I know from internal sources that there are half a million papers downloaded per year - which is equivalent download usage to the prestigious Journal of Theoretical Biology. Clearly, in spite or because of our policy to publish bold and sometimes bizarre ideas, Medical Hypotheses plays a significant role in medical science. Fact; not opinion. The editorial advisory board currently includes such respected figures as Nobelist Arvid Carlsson http://en.wikipedia.org/wiki/Arvid_Carlsson; Sir Roy Calne http://en.wikipedia.org/wiki/Roy_Calne; Antonio Damasio http://en.wikipedia.org/wiki/Antonio_Damasio and V.S. Ramachandran http://en.wikipedia.org/wiki/Vilayanur_S._Ramachandran . Past editorial advisors have included Sir Karl Popper and Nobelist Sir James Black. *** There are only two possible legitimate outcomes to the current process. Either: 1. Medical Hypotheses could continue as an influential, profitable and well-known editorially-reviewed journal with a radical mission. Or else: 2. The journal could be closed-down altogether, and the title abolished. But it would obviously not be ethically acceptable to launch a new ‘imposter’ journal - with utterly different editorial aims, procedures and personnel; yet retaining the 34 year established title of Medical Hypotheses.

As I keep saying, the impact factor of a journal tells you nothing about its quality. For example, here are three peer-reviewed pseudojournals that repeatedly publish abject nonsense and pdeudoscience, with their impact factors according to Journal Citation Reports:

Homeopathy: 1.041

Evidence-based Complementary and Alternative Medicine: 1.954

Journal of Alternative and Complementary Medicine: 1.628

The articles in these journals are typically written by quacks, and are cited by other quacks writing in quack journals, giving a high-ish but meaningless impact factor. Perhaps Medical Hypotheses is also highly influential among pseudoscientists?

But the main point here is about radical and controversial hypotheses. I think most people would agree that these have their place in scientific discourse, and there ought to be somewhere to publish them. However, this isn't really what the argument is about. In this case, two fatally flawed papers were published with little or no scrutiny: these papers have potential global health implications. In the case of the Duesberg paper, reviews posted on the Denying AIDS blog show the major problems with the paper. There's a difference between publishing provocative ideas that might inspire new research, and ones that are just demonstrably wrong. While the likes of Peter Duesberg have the right to say what they like, they don't have the right to say it in a MEDLINE-indexed journal. This is not an argument about free speech, it's an argument about the integrity of the scientific literature. There may be a place for journals such as Medical Hypotheses, but there has to be some level of quality control. Otherwise, why should anyone take them seriously?

Bad cycling: New Years Day '10'

2010-01-04T11:17:00.004Z

Not so much bad science, as bad cycling. Last year I returned to bike racing after more than 10 years away, riding a few club events organised by my club, Stockport Clarion. The triumph of my season was my not-too-bad performance in the club hill climb up the Cat and Fiddle. Since then, I've not really been on the bike too much, because it's dark and cold and I'm a bit of a wuss like that. I did have an ill-fated ride before Christmas, where I fell on a patch of ice just before the Great Stone Road roundabout in Stretford, and then almost got run over by the gritter that was following me. But Stockport Clarion traditionally organises a 10-mile time trial on New Years Day, and I thought it would be a good way to kick-start 2010.

It was an 11am start at Chelford, so I reckoned I had to leave about 9:30. This seemed less and less like a good idea as I headed off into the -4 weather, wearing pretty much every piece of bike kit I own. At least the roads were quiet. There was virtually no wind, but riding at about 15 mph was enough to freeze my hands within a few miles, despite my winter gloves. There was a bloke staggering about and yelling as I went through the edge of Didsbury. Hopefully he was on his way home. There was hardly anyone else about. I rattled through a deserted Alderley Edge and made it to the sign-on about 10:45, just as light snow began to fall. Dunc, the timekeeper, had a good story about the previous year's race, when it had also snowed. Apparently a rider behind him noticed that he was leaving two tyre tracks instead of one: he'd twisted his forks in a crash a few days before.

By the time 11:00 came around, there were still only two riders. It was now -3, and the news was that both Snake Pass and the Cat and Fiddle were closed because of drifting snow. One rider from Poynton had taken one look at the thermometer (-6) and decided to go back to bed. Having come this far, I thought we might as well have a ride anyway. Then the perenially late Will arrived, on a full time-trial machine, to make it three riders.

The Chelford course is a bit of a horror, as time trial courses go. But there are just too many traffic lights around Manchester for a selection of courses, so Chelford is what you get. It's hardly a dragstrip, but not really a sporting course either. It's flattish, narrow and typically busy with fast traffic, but being New Years Day it was relatively quiet. The road surface is pretty heavy and there's a couple of lumps to keep you honest, the main one being the railway bridge in Chelford. That probably doesn't sound too bad, but it's amazing how much effort it costs to maintain speed over the damned thing, with only about a mile to go to the finish.

I started number 2, and within a few hundred yards my feet were frozen. There's a roundabout after about a mile, where Ric the marshal was shivering and pacing about to keep warm, and then the turn is just outside Knutsford. By then I'd already been passed by Will at number 3, and was clearly losing time to Dan at number 1. By the time I got to the turn, I was starting to warm up, but my legs just wouldn't have it over the last 4 miles and I finished in 32:53. That's the slowest '10' I've ever done, a minute slower than the first one I ever did riding as a schoolboy under the banner of Penistone Grammar School. I rode back to the timekeeper's car, and took a drink from my bottle: there was ice in it.

Still, the winning time was 28:39, from a rider who would be doing 24 minute rides in summer, so I couldn't feel too bad. In any case, I didn't have time: to avoid hypothermia, I jumped back on the bike and set off home ASAP.

Hopefully there'll be more bad science in the year ahead (there's certainly no shortage of it about), and perhaps even some good science of my own. Whatever else happens, though, I'm certain that there'll be more bad cycling.

Fighting parliamentary committees...with your mind

2009-11-21T17:58:00.002Z

You might remember me writing about the Intention Experiment in the past. The Intention Experiment hopes to harness the power of positive thinking to influence events. In one memorable example, they tried to influence the structure of water through the collected brainpower of a large collection of gullible numpties. Despite the experiment having a design likely to massively increase the chance of getting a false positive result, nothing astonishing happened.

Gimpy has now made my day by posting this story about homeopaths concerned that the British parliament's Science and Technology Committee will conclude that there is no evidence in favour of homeopathy. Instead of marshalling the evidence and trying to make a scientific case, the homeopaths are trying an intention experiment to influence the committee in favour of homeopathy.

There's just nothing to add to this: it's shear barking mad lunacy, and you have to think that maybe these people are their own worst enemies.

Tuesday is the wettest day of the week...?

2009-10-26T16:11:00.003Z

It was a bit disappointing to get back from a week in the field in Sinai, to find my employers at the University of Manchester advertising the Manchester Science Festival with this story. Apparently, Tuesday is the wettest day of the week in Manchester.

Well, every day is rainy in Manchester. Perhaps predictably, if you look at the data this conclusion is based on, you can see that there isn't exactly an astounding difference in rainfall between different days of the week. Certainly, statistically significant differences have not been demonstrated. The research has some interesting things to say about how rainfall patterns seem to have changed over the last 30 years or so: Manchester is somehow managing to get wetter. This is consistent with warmer temperatures, as more water vapour can be moved around when temperatures are higher. This is interesting stuff, and excellent for illustrating local changes in climate for a science festival. So why emphasise that Tuesday is supposedly the wettest day of the week, when the data surely don't convincingly support that? I suppose the university press office thinks that wet Tuesdays are a more interesting story than local climate changes, but I don't think I would agree.

A quick google shows that the story has been picked up by the Express (Why Tuesday is the day you'll most likely need your umbrella) and the Telegraph (Tuesday is the rainiest day), which are clearly based around the press release. I think a better press release might have been headlined "Manchester getting rainier", which is interesting and also has the benefit of being supported by the data.

REF consultation document published

2009-09-23T10:51:00.001+01:00

For anyone interested in how research funding is allocated (fascinating stuff, I know), a consultation document on the Research Excellence Framework (REF) is now available here. REF is the mooted replacement for the old Research Assessment Exercise (RAE), the last one of which was conducted in 2008. Enjoy...

Playing the game: the impact of research assessment

2009-09-18T13:32:00.005+01:00

Yesterday I was sent this report, produced by the Research Information Network, in conjunction with the Joint Information Systems Committee, and entitled "Communicating knowledge: How and why UK researchers publish and disseminate their findings". The report used a literature review, bibliometric analysis, an online survey of UK researchers, and focus groups or interviews with researchers to look at how and why researchers put information into the public domain. Being an early-career researcher, I'm interested in this sort of thing: I know why I'm publishing and disseminating information, but it's interesting to see why everyone else is doing it. It's also interesting to see the extent to which research assessment in the UK - until recently the Research Assessment Exercise (RAE) and in future the mysterious Research Excellence Framework (REF) - influence the decisions that researchers make. What particularly struck me about the report was the number of times researchers talked about "playing games": the framework of research assessment is seen as a game to be played, with the needs of research being subordinated to the need to put in a good performance. This has important implications for the REF, in which bibliometric indicators are likely to play an important role.

The key point of the report is that there is some confusion among researchers about what exactly it is they're supposed to be doing. There are conflicting and unclear messages form different bodies about what sort of research contributions are valued. The perception is that the only thing that really counts in terms of research assessment is peer-reviewed journal articles. Other contributions, such as conference proceedings, books, book chapters, monographs, government reports and so on are not valued. As a result, the proportion journal articles compared to other outputs increased significantly between 2003 and 2008. A couple of comments by researchers quoted in the report (p.15):

[There is] much more emphasis on peer reviewed journals …Conferences, working papers and book chapters are pretty much a waste of time … Books and monographs are worth concentrating on if they help one demarcate a particular piece of intellectual territory.

There is a strong disincentive to publish edited works and chapters in edited works, even though these are actually widely used by researchers and educators in my field, and by our students.

This is certainly the impression I get from my own field. In fact, I have been advised by senior colleagues to target high-impact journals, rather than, for example, special publications. I have never received any formal guidance on what research outputs are expected of me, but the prevailing atmosphere gives the impression that it's all about journal articles. After publishing a couple of things from my PhD, it took another three years to publish anything from my first post-doc. I worried about that: it seemed that the numerous conferences and internal company reports and presentations I produced over that time counted for nothing career-wise.

The report makes it clear that, in the case of the RAE, it is more perceptios than the reality causing the problem: the RAE rules meant that most outputs were admissible, and all would be treated equally. But it's perceptions that drive the way researchers respond to research assessment. Clearer guidance is needed.

An interesting point brought up by the report is how, when there is more than one author for a journal article, the list of authors is arranged. In my field, authors are typically listed in order of contribution, so I was surprised to find that this is by no means always the case. In some fields, especially in the humanities and social sciences, authors are commonly listed alphabetically. In some cases, the leader of the research group is listed first, in other cases last. And there are various mixtures of listing by contribution, grant-holding and alphabetic order. There is even a significant minority where papers based on work done by students have the student's supervisor as first author! This means that there is no straightforward way of apportioning credit to multiple authors of a paper, something that David Colquhoun has already pointed out. This is a huge problem for any system of assessment based on bibliometrics.

The report also examines how researchers cite the work of other people. Other researcher's work should be cited because it forms part of the background of the new research, because it supports a statement made in the new paper, or as part of a discussion of how the new paper fits into the context of previous research. Crucially, this includes citing work with which the authors disagree, or that is refuted or cast into doubt in the light of the new work (p.30):

Citing somebody often indicates opposition / disagreement, rather than esteem and I am as likely to cite and critique work that I do not rate highly as work I value.

So any system that relies on bibliometric indicators is likely to reward controversial science as much as good science (not that those categories are mutually exclusive, but they don't completely overlap either).

Researchers are perfectly clear that a system based on bibliometrics will cause them to change their publication behaviour: 22% will try to produce more publications, 33% will submit more work to high-status journals, 38% will cite their collaborators work more often, while 6% will cite their competitors work less often. This will lead to more journal articles of poorer quality, a the decline of perfectly good journals that have low "impact", and corruption in citation behaviour. In general, researchers aren't daft, and they've clearly identified the incentives that would be created by such a system.

The report presents a worrying picture of research, and scientific literature, distorted by the perverse incentives created by poorly thought-out and opaque forms of research assessment. It can be argued that scientists who allow their behaviour to be distorted by these incentives are acting unprofessionally: I wouldn't disagree. But for individuals playing the game, the stakes are high. Perhaps we ought to be thinking about whether research is the place for playing games. It surely can't lead to good science.

I get e-mail

2009-09-16T16:23:00.002+01:00

Got this today, sent out to academic and academic-related staff in my department:

Dear All,

Please find attached NSS results by Faculty, School and JACS Level 3 subjects. Also included is a mapping document to accompany the JACS report to assist you in understanding which programmes of study are included under each heading. The Word document, 'APPENDIX 06-Surveys - NSS Table EPS.doc' shows the data that will be included in the OPR documentation.

Please note that the data is FOR INTERNAL USE ONLY.

I have no idea what NSS, JACS or OPR mean, so this e-mail makes no sense to me whatsoever. I seem to be getting an increasing number of these things, all with acronyms I've never heard of.

What happens when you don't have peer review

2009-09-16T14:43:00.005+01:00

Normally, when a scientific paper is submitted, it is subjected to scrutiny by two or more scientists working in a similar field. Only if the paper gets through this peer review process, and if corrections required by the reviewers have been made, does the paper actually get published. This process is by no means perfect: bad papers slip through, and good papers get blocked by over-zealous reviewers. But there are two examples this week of what can go wrong when papers are not peer reviewed.

Firstly, Ben Goldacre and Respectful Insolence discuss the case of two papers, recently published in Medical Hypotheses, that were so bad they were withdrawn by publishers Elsevier. Given that Elsevier happily publishes Homeopathy, the fanzine of the Faculty of Homeopathy, this should give pause for thought. Medical Hypotheses is a bit of an oddity: it does not send papers out for peer review. Rather, they are approved solely by the editor of the journal, one Bruce Charlton. It appears that many papers are approved within days, sometimes hours, of being submitted, suggesting that there is very little scrutiny of the papers.

The two papers are one by Duesberg et al., and one by Ruggiero et al., both of which seek to deny the magnitude of the AIDS crisis. Seth Kalichman of the Denying Aids blog did an experiment by sending the manuscript out for blind peer review. All three "reviewers" rejected the manuscript on the basis that it was filled with logical flaws and mis-representations of the published literature.

Elsevier says:

This Article-in-Press has been withdrawn pending the results of an investigation. The editorial policy of Medical Hypotheses makes it clear that the journal considers "radical, speculative, and non-mainstream scientific ideas", and articles will only be acceptable if they are "coherent and clearly expressed." However, we have received serious expressions of concern about the quality of this article, which contains highly controversial opinions about the causes of AIDS, opinions that could potentially be damaging to global public health. Concern has also been expressed that the article contains potentially libelous material. Given these important signals of concern, we judge it correct to investigate the circumstances in which this article came to be published online. When the investigation and review have been completed we will issue a further statement. Until that time, the article has been removed from all Elsevier databases. The Publisher apologizes for any inconvenience this may cause. The full Elsevier Policy on Article Withdrawal can be found at http://www.elsevier.com/locate/withdrawalpolicy.

The second example is a paper published in Proceedings of the National Academy of Sciences, amusingly known as PNAS. This is a venerable and respected journal, but it has a little-known wrinkle: members of the National Academy of Sciences are allowed to bypass formal peer review by "communicating" papers for other researchers. This is how the PNAS "Information for Authors" page describes the process:

An Academy member may “communicate” for others up to 2 manuscripts per year that are within the member's area of expertise. Before submission to PNAS, the member obtains reviews of the paper from at least 2 qualified referees, each from a different institution and not from the authors' or member's institutions. Referees should be asked to evaluate revised manuscripts to ensure that their concerns have been adequately addressed. The names and contact information, including e-mails, of referees who reviewed the paper, along with the reviews and the authors' response, must be included. Reviews must be submitted on the PNAS review form, and the identity of the referees must not be revealed to the authors. The member must include a brief statement endorsing publication in PNAS along with all of the referee reports received for each round of review. Members should follow National Science Foundation (NSF) guidelines to avoid conflict of interest between referees and authors (see Section iii). Members must verify that referees are free of conflicts of interest, or must disclose any conflicts and explain their choice of referees. These papers are published as “Communicated by" the responsible editor.

The paper in question is was submitted via this communication process. It was written by Donald Williamson, a retired academic from the University of Liverpool, and suggests that butterflies and caterpillars orginated as different species:

I reject the Darwinian assumption that larvae and their adults evolved from a single common ancestor. Rather I posit that, in animals that metamorphose, the basic types of larvae originated as adults of different lineages, i.e., larvae were transferred when, through hybridization, their genomes were acquired by distantly related animals.

The paper has been criticised on the basis that it contains no supporting data for what is, after all, a fairly extraordinary hypothesis. Not only that, but it turns out that it had previously been rejected by seven different journals.

In both Medical Hypotheses and PNAS, the defence seems to be that there needs to be some mechanism by which speculative ideas that go against current mainstream opinion can be presented and discussed. This seems fair enough, but is anything gained by publishing hypotheses that are not supported by any data, or papers that are logically flawed and contain mis-representations? In both these cases, it seems that the papers would not have been published had they been reviewed properly.

Soviet Manchester

2009-08-26T11:35:00.005+01:00

There's an exhibition of maps of Manchester at the John Rylands Library just now, and I'm hoping to have a look at it next week when I take a much-needed break. One of the exhibits in particular is attracting a lot of attention: a map of the city prepared by the Soviets, for use in the event of an invasion of the UK.

It shows the roads - familiar to many Mancunians - which the Soviets felt were wide enough to carry tanks including Washway Road, the Mancunian Way, and Princess Road.

I biked down Washway Road last night on my way home from a training ride, and it's quite hard to imagine columns of Soviet tanks clanking along it towards Stretford. Though I suppose if there ever had been a Soviet invasion, Washway Road would be even more of a post-nuclear wasteland than it currently appears. Apparently, this map is only 35 years old, and it's very strange to think that so recently there were plans for a Soviet Manchester.

Washway Road, yesterday

Society of Homeopaths defends ineffective treatment for condition that kills 2 million children a year...

2009-08-21T21:23:00.006+01:00

...surprise!

I'm not going to write a huge amount about this, because it is ably covered elsewhere. But the World Health Organisation finally came out today, and clearly stated that homeopathy should not be used for life-threatening conditions such as AIDS, malaria, TB and childhood diarrhoea.

The Society of Homeopaths (SoH) respond by saying that treating AIDS, malaria and TB would contravene their ethical guidelines. These guidelines are a joke, as you can see by perusing the Quackometer and Gimpy's blog. But apart from that, they say:

The Society of Homeopaths, the UK’s largest body of registered homeopaths, is concerned to learn, in an online article by the BBC (“WHO warns against homeopathy use’), that the World Health Organisation (WHO) has issued caution against the use of homeopathy for childhood diarrhoea following a letter by the charity Sense About Science.

They go on to cherry-pick and distort the research that has actually been conducted on homeopathy for childhood diarrhoea.

Well, who's surprised by that? Regular readers will be aware of just a few of the ways that homeopaths try to distort the evidence that homeopathy doesn't work. The point here is that the SoH, a supposedly professional organisation, is trying to defend a totally ineffective treatment for a disease that kills 2 million children every year. Homeopathy can't do anything to help, and using it instead of proper medical treatment could cost lives. I don't really mind if homeopaths sell ineffective sugar pills to the worried well in Alderley Edge, but this press release is delusional and irresponsible, and reasonable people should have no truck with this sort of dangerous nonsense.

A long trip to a closed pub

2009-08-18T11:09:00.003+01:00

Several years ago, I used to do a lot of time trialling. Given that, at my peak fitness, I weighed 8 and a half stone and looked like a bundle of pipe cleaners tied together, my favourite races were always hill climbs. Hill climbs are essentially short time-trials up a hill, and they usually happen at the end of the season in September and October. In a typical season I would ride several around Sheffield, Monsal Head and Curbar Edge being particular favourites. Then in 1999 I went to Canada to study for my PhD, and I didn't race again until this year.

This year I joined Stockport Clarion CC, and I've been riding the odd weekday 10-mile time-trial with no great success. But the event I was really looking forward to was the club hill climb championships. Hill climb courses are typically between several hundred yards and a couple of miles long, and they're usually steep, with gradients of 10-20%. Just to be different, ours is on the Cat and Fiddle road, between Macclesfield and the eponymous pub that stands, hemmed in by peat bog, at the road summit. That's about 6.5 miles of climbing, but at an average gradient of less than 4%. There are steeper railway bridges around here, but there's still about 335 m of height gain involved. The course starts in the outskirts of Macclesfield, opposite a bus stop, and the first 3 miles or so is a fairly steady 4.8%. The next mile and a half is very bendy, contains some short downhill sections, and is nearly flat on average. The final mile and a half takes you up onto Axe Edge Moor proper, climbing at 3.5% or so.

I had never ridden up the climb before, but I had a gameplan of sorts. The first section was the bit where my light weight would give me an advantage, so I would ride hard, but not flat out: the key is not to overdo it, what with there still being 3 miles to go. I would then use the fast section to take a bit of a breather, riding at slightly less than 10-mile pace, since you don't gain much time there anyway. Then it would be eyeballs out over the last mile and a half to the summit. What gears would I use? Hell if I knew. I would figure it out as I went along.

The race was last night, and I actually really enjoyed it. The conditions could scarcely have been better. It was clear and dry, about 18 C, and there was a moderate tailwind. There were only 8 riders, and I was off Number 2, with a slower rider starting one minute before me. I started well, settling in very quickly, and once I had emerged from the trees at the bottom of the climb I could see my minute man almost all of the time. I was clearly gaining on him, and that gave me a psychological boost, as I finally reeled him in on a sharp right-hand bend about a mile and a half in, just before Walker Barn. I was first on the road now, a nice feeling, and my legs were holding up just fine. After Walker Barn, into the fast part of the course, I switched into the big ring, and tried to maintain a slightly-slower-than-10-mile-time-trial pace. My biggest problem here was gear selection. The gradient changes so often and there are so many sharp curves that I felt like I was using nearly every gear on the bike, and I was changing front rings too often: I should have picked one and stuck to it. But I still felt good, and I knew I had something left for the last mile and a half. No strategy involved here; just eyeballs out until the summit. There was a guy out for a training ride in front, and that gave me something to chase. Round the last bend, and there was the welcome sight of the Cat and Fiddle Inn. One last leg-breaking effort, and I was past the timekeeper in 26 minutes dead, gasping like a freshly landed fish. It was about a minute faster than I'd hoped for, so I couldn't have been happier.

It was glorious at the summit, with views across the Cheshire Plain and the Peak District. Unfortunately, the pub is closed on Monday nights, so there was no chance of a swift half before the 25 mile ride back home. At least the first seven or so was downhill.

Publication bias: medical science is looking for it. Are you?

2009-08-07T20:31:00.006+01:00

Via the mighty Ben Goldacre's miniblog, here's a study that's just been published in the BMJ on the subject of publication bias. Publication bias typically occurs when studies with positive results are preferentially published over studies with negative results. There are many reasons why this might happen; drug companies don't want to publish negative trials of their drugs for obvious reasons, but there is also a tendency for journals to be uninterested in publishing negative trials, and for busy authors to be reluctant to write them up. There's nothing sexy about a negative trial.

In the BMJ study, Santiago Moreno and colleagues look at anti-depressants. This is a good area to look at, because of the availability of data that was submitted to the Federal Drug Administration in the United States. Legal requirements enforce submission of ALL data to the FDA, so the authors consider the FDA dataset to be unbiased (although not necessarily complete). This unbiased dataset can then be compared to the data that is available in published journal articles.

The comparison is done with our old friend the funnel plot. This plots the standard error for a trial against the size of the effect that the trial found. The authors of the BMJ study added a new twist by contouring the funnel plot for statistical significance: at a glance it can be seen where studies fall in terms of statistical significance.

The results are dramatic. Of the 74 trials registered with the FDA, 23 were not published. In the FDA data, there is a wide spread of results across the contours marking 1%, 5% and 10% levels of significance. When only the published data are considered, there is a clear cut-off at the contour for the 5% significance level, which is typically used in clinical trials to establish statistical significance. That strongly suggests that publication bias is a serious problem in the set of published trials: trials with statistically non-significant results have been systematically excluded.

What this means is that the published literature over-estimates the benefit of anti-depressants. It doesn't show that anti-depressants don't work: meta-analysis of the FDA data still shows a beneficial effect. The point is that the real benefit (as shown by the FDA data) is less than the benefit you would expect if you looked only at the published literature. Anti-depressants work less well than you might think, but they still work.

The problem is, of course, that most of the time we only have the published literature to work with. So the BMJ paper adds a useful visual technique for identifying publication bias as a likely problem, even if we don't have access a bias-free dataset for comparison. There is no doubt that biases exist in published data; the response from medical science, as seen here, is to try to identify and account for these biases. Regular readers will know that the response of CAM research is to manipulate the data in order to pretend that the problem doesn't exist.

I have a bad feeling about this...

2009-08-05T13:34:00.003+01:00

I was idly browsing the front page of the Guardian's website, when I really ought to be doing increasingly tedious corrections to a paper, and I saw this. Eczema sufferer Peter Beech, after many years of decreasingly effective steroid treatments, is trying homeopathy. Beech also claims to be a sceptic of alternative medicine.

First things first. Is there any evidence that homeopathy is effective for treatment of eczema? A quick search on Pubmed showed this to be the most recent (2003) relevant review. Incidentally, Pubmed is a freely accessible service, unlike many of the journal databases used by academics, and anyone with an internet connection can do the same thing as me and come up with the presently available evidence in a couple of minutes. Here's what the summary says:

Alternative methods are commonly used in patients with dermatologic diseases, with homeopathy being one of the most common. Homeopathy was developed by Samuel Hahnemann (1755–1843) and is based on the law of similars and the law of infinitesimals. It is a regulatory therapy where high dilutions of particular compounds are thought to induce a counterreaction in the organism. In dermatology, homeopathy is often used in atopic dermatitis, other forms of eczema, psoriasis, and many other conditions. To date, however, there is no convincing evidence for a therapeutic effect. There are only a few controlled trials, most of them with negative results. The few studies with positive results have not been reproduced. Acceptance by the patient seems largely based on counseling and emotional care rather than on objective responses to the homeopathic drugs.

Not particularly convincing, is it?

Beech is at least unlikely to do himself any harm by trying homeopathy. However, real harm can result when people use homeopathy and avoid real medicine, as this tragic case in Australia shows. Nine-month old Gloria Thomas died after her eczema allowed an infection to get out of control. Her parents had been treating her with homeopathy instead of real medicine: they were later convicted of manslaughter by gross criminal negligence.

But the main reason that I have a bad feeling about this, is that even if Beech gets better, it won't prove anything about homeopathy (and the same goes if he doesn't see any improvement, to be fair). This is essentially an uncontrolled case report, and there would be no way of showing that any improvement resulted from the homeopathy, rather than, say, the natural cyclicity of eczema, or the placebo effect. Whatever happens to Beech, it isn't going to trump the evidence from the most recent systematic review of the evidence from clinical trials. So Beech's experiment cannot add anything to what we laughingly call the "debate" over the efficacy of homeopathy.

Still, good luck to him. I hope he does get some relief from his symptoms, but if he does the chances of it having anything to do with homeopathy are vanishingly small.

Simon Singh: "Beware the spinal trap"

2009-07-29T10:53:00.003+01:00

As many of you will no doubt be aware, science writer Simon Singh is being sued by the British Chiropractic Association (BCA). The BCA objected to a piece that Singh published in the Guardian, pointing out that chiropractic has little or no evidence base, particularly when used to treat anything other than back problems, and that it can be dangerous. After Singh was sued, the Guardian took down the article pending the legal proceedings. As part of their campaign to "Keep the libel laws out of science", Sense About Science is asking bloggers to re-publish an edited version of the article, because scientific arguments should not be settled in the courtroom. I'm glad to do my bit.

Beware the spinal trap

Some practitioners claim it is a cure-all, but the research suggests chiropractic therapy has mixed results - and can even be lethal, says Simon Singh.

You might be surprised to know that the founder of chiropractic therapy, Daniel David Palmer, wrote that '99% of all diseases are caused by displaced vertebrae'. In the 1860s, Palmer began to develop his theory that the spine was involved in almost every illness because the spinal cord connects the brain to the rest of the body. Therefore any misalignment could cause a problem in distant parts of the body.

In fact, Palmer's first chiropractic intervention supposedly cured a man who had been profoundly deaf for 17 years. His second treatment was equally strange, because he claimed that he treated a patient with heart trouble by correcting a displaced vertebra.

You might think that modern chiropractors restrict themselves to treating back problems, but in fact some still possess quite wacky ideas. The fundamentalists argue that they can cure anything, including helping treat children with colic, sleeping and feeding problems, frequent ear infections, asthma and prolonged crying - even though there is not a jot of evidence.

I can confidently label these assertions as utter nonsense because I have co-authored a book about alternative medicine with the world's first professor of complementary medicine, Edzard Ernst. He learned chiropractic techniques himself and used them as a doctor. This is when he began to see the need for some critical evaluation. Among other projects, he examined the evidence from 70 trials exploring the benefits of chiropractic therapy in conditions unrelated to the back. He found no evidence to suggest that chiropractors could treat any such conditions.

But what about chiropractic in the context of treating back problems? Manipulating the spine can cure some problems, but results are mixed. To be fair, conventional approaches, such as physiotherapy, also struggle to treat back problems with any consistency. Nevertheless, conventional therapy is still preferable because of the serious dangers associated with chiropractic.

In 2001, a systematic review of five studies revealed that roughly half of all chiropractic patients experience temporary adverse effects, such as pain, numbness, stiffness, dizziness and headaches. These are relatively minor effects, but the frequency is very high, and this has to be weighed against the limited benefit offered by chiropractors.

More worryingly, the hallmark technique of the chiropractor, known as high-velocity, low-amplitude thrust, carries much more significant risks. This involves pushing joints beyond their natural range of motion by applying a short, sharp force. Although this is a safe procedure for most patients, others can suffer dislocations and fractures.

Worse still, manipulation of the neck can damage the vertebral arteries, which supply blood to the brain. So-called vertebral dissection can ultimately cut off the blood supply, which in turn can lead to a stroke and even death. Because there is usually a delay between the vertebral dissection and the blockage of blood to the brain, the link between chiropractic and strokes went unnoticed for many years. Recently, however, it has been possible to identify cases where spinal manipulation has certainly been the cause of vertebral dissection.

Laurie Mathiason was a 20-year-old Canadian waitress who visited a chiropractor 21 times between 1997 and 1998 to relieve her low-back pain. On her penultimate visit she complained of stiffness in her neck. That evening she began dropping plates at the restaurant, so she returned to the chiropractor. As the chiropractor manipulated her neck, Mathiason began to cry, her eyes started to roll, she foamed at the mouth and her body began to convulse. She was rushed to hospital, slipped into a coma and died three days later. At the inquest, the coroner declared: 'Laurie died of a ruptured vertebral artery, which occurred in association with a chiropractic manipulation of the neck.'

This case is not unique. In Canada alone there have been several other women who have died after receiving chiropractic therapy, and Edzard Ernst has identified about 700 cases of serious complications among the medical literature. This should be a major concern for health officials, particularly as under-reporting will mean that the actual number of cases is much higher. If spinal manipulation were a drug with such serious adverse effects and so little demonstrable benefit, then it would almost certainly have been taken off the market.

Simon Singh is a science writer in London and the co-author, with Edzard Ernst, of Trick or Treatment? Alternative Medicine on Trial. This is an edited version of an article published in The Guardian for which Singh is being personally sued for libel by the British Chiropractic Association.

On peer review

2009-07-24T09:28:00.003+01:00

If you spend a lot of time hanging around at sceptical websites, you'll be aware that when evidence is demanded in support of a claim, it is usually demanded in the form of a peer-reviewed journal article. The suggestion is that peer review provides some sort of check on quality: any peer reviewed paper should meet a certain minimum standard. But is this really the case?

First of all, what does it mean to say that an article is peer reviewed? If an article is peer reviewed, it has been checked over by scientists who work in a similar field to the submitted article. A submitted article will usually be sent to two or three reviewers, who will each read the paper and submit a report to the journal editor. The editor will then decide whether the article merits publication.

On the face of it, this would seem to imply that certain standards are being met. But there is some evidence that this isn't necessarily the case. For example, this article (found via Ben Golacre's miniblog) suggests that in Chinese journals, only 5.9-7.7% of supposed randomised controlled trials reported in peer reviewed articles had adequate procedures for randomisation. A lack of adequate randomisation means that there is a good chance of introducing bias into your trial, and it ought to be one of the first things a reviewer would check. While the article specifically addresses trials published in Chinese journals, I don't think there's any compelling reason to think that things are dramatically different in what we laughingly call the west. Anecdotally, anyone who spends time wading through journals as part of their day job will be able to come up with several examples of utterly dreadful papers that should never have been published. This is without looking at pseudojournals, such as those that concentrate on complementary and alternative medicine, where articles on quackery are peer reviewed by other quacks.

So, if peer review can't tell you whether a trial described as randomised is in fact randomised, what can it tell you? Does it really act as any kind of guarantee of minimum quality? I would suggest not.

That is not to say that peer review is useless as it stands. In my fairly limited experience, papers that I have submitted have always been significantly improved by peer review. But surely there's a way of making peer review "fit for purpose", to use the current jargon?

This post was prompted by a discussion at the Bad Science forum, where the idea of applying industrial-style quality assurance to journal articles was raised. This would mean that there would be some sort of checklist that a reviewer would have to go through, and this would be checked to make sure it had been done. It would not be much use to do this informally; there would need to be some formal way of doing it.

In fact, this is not too far from what already happens, in many cases. I've just got a review back in which the reviewers have answered a series of yes/no questions (in addition to their more detailed written comments). For example, "Are the interpretations and conclusions sound and supported by the interpretation provided?", and "Is the paper logically organised?". For the example of trials published in Chinese journals above, there could be a question like "Is the methodology appropriate for testing the specified hypotheses?". Again, there would have to be some checks that this had been adequately done; this is really what the journal editor should be doing. At present, I think the role of the editor is often too weak. They do little more than send out and receive reviews. This is probably not surprising, given that editors are usually working more or less voluntarily and tend to have plenty of other things that they need to do. And it is not always the case: there are many excellent editors who make a lot of effort to engage with the papers they are handling, and the reviewer's comments on them. But if the role of editors were beefed up, such that they spend time formally checking that reviews have been carried out adequately, then peer review might actually provide the quality guarantee that we seem to think it should.

That might require actually paying editors and reviewers for their time. This would be a fairly radical step, but if it led to a professionalisation of the journal reviewing and editing process it would probably be a good thing. And if it led to a reduction in the number, and an increase in the quality, of papers published, that would not be a bad thing either.

I get my name in the Veterinary Record

2009-07-01T14:46:00.004+01:00

This is somewhat old news, but I haven't had chance to write about it before. To add to the publications I have in Homeopathy, I now have one (as third author) in the Veterinary Record. This is starting to get silly; I'm supposed to be a geologist.

Perhaps unsurprisingly, this is related to a terrible homeopathy study [Hill et al., The Veterinary Record 164:364-370], this time on the treatment of skin conditions in dogs. It's another example of homeopaths continuing to do small, badly designed studies, when plenty of large and properly conducted studies, and systematic reviews and meta-analyses of those studies, show that homeopathy doesn't work. The letter I am involved in is one of three letters that were published criticising the study: they can be found, with the author's reply, at The Veterinary Record 164: 634-636 [apologies for the lack of links: there's no DOI for these that I can find]. There is also an excellent discussion of the paper, and some of the responses to it, over at JREF.

The design in this study is truly extraordinary. Initially, 20 dogs with skin problems were recruited to the study. All were treated with individualised remedies by a homeopath. In 15 cases, the dog owners reported no improvement. In 5 cases, the owners reported a significant improvement. Not looking good for homeopathy so far. Still, the five improved dogs were said to have responded well to homeopathy, and went on to phase 2, which was a proper randomised and blinded placebo-controlled trial. Unfortunately, one dog had to be euthanased before the trial could happen, and another dog's skin problems had resolved completely after the first stage, leaving only three dogs in phase 2. Supposedly, those dogs did better with homeopathy than with placebo, thus justifying, as ever, "further research".

This is possibly the easiest study to criticise that I've ever seen. Put simply, the first phase lacks a control group, so improvements cannot be attributed to homeopathy. There is simply no evidence that the five dogs recruited to phase 2 actually responded to homeopathy, rather than just improved spontaneously. Then the second phase of the trial includes only three dogs. There is no way to interpret the results of such a tiny, underpowered study. Those are the main problems, but there are others. For example, all the dogs were on some kind of conventional medication, so that cannot be ruled out as contributing to any improvement.

The only reasonable conclusion from the study is that there is no strong evidence that homeopathy did anything for the dogs in the trial. But the paper concludes that the improvement seen in the five dogs (which again cannot be attributed to homeopathy on the basis of this study) is enough to justify further research. No doubt the paper will also be spammed all over the internet by the likes of Dana Ullman, as proof positive that homeopathy works. Hopefully the letter I'm a co-author on, along with the two other letters critical of the study that were published, will go some way to addressing that. The signs are not good, though. The original Hill et al. paper included the statement that "Different homeopathic remedies and different
dilutions of the same remedy have been distinguished from each other using Raman and infrared spectroscopy, even though all should contain nothing but water", with a reference to "Rao and others, 2007" [In fact, Rao et al. did not even claim that infrared spectroscopy showed any difference]. Regular readers will know that Rao and colleagues did nothing of the sort, and that to describe their paper as "discredited" would be something of an understatement. In the world of homeopathy, discredited papers never die. They are just recycled for use with audiences who don't know that they've been discredited. I suspect that this one will be no different.

As an aside, my favourite part of this study is that "constitutional signs" of each of the dogs, as used by the homeopath to pick a remedy, are listed [Table 2 of the paper]. For dog number 16, these are listed as:

Affectionate
Fears thunderstorms
Clairvoyant
Grief
Desires chicken; oranges aggravate

A clairvoyant dog! And this was published in a respected veterinary journal.

What do bibliometrics actually add to research evaluation?

2009-06-24T11:24:00.009+01:00

Firstly, the reason that I haven't posted in an age is that I've been in Norway, interpreting seismic data for the new project I'm working on. Hopefully I can now post a bit more regularly, as I should actually be in Manchester for a few consecutive weeks, for the first time this year.

Regular readers will know that I like to whinge about the increasing use of statistical indicators (bibliometrics) to evaluate research performance. Previously in England, research performance has been evaluated by the Research Assessment Exercise, a cumbersome and involved system based around expert peer review of research. Currently, HEFCE (the body that decides how scarce research funding is allocated to English universities) is looking into replacing this with a cumbersome and involved system based around bibliometrics and "light-touch" peer review. To this end, a pilot exercise using bibliometrics and including 22 universities has been underway. An interim report on the pilot is now available.

Essentially, three approaches have been evaluated:

i) Based on institutional addresses: here papers are assigned to a university based on the addresses of the the authors, as stated in the paper. This would be cheap to do, as it would need no input from the universities.

ii) Based on all papers published by authors. In this approach, all papers written by staff selected for the 2008 RAE were identified. This requires a lot of data to be collected.

iii) Based on selected papers published by authors. Again, this approach used all staff selected for the 2008 RAE, but only used the most cited papers.

For each approach, the exercise was conducted twice: once using the Web Of Science (WoS) database, and once using Scopus. The results were then compared with those from the 2008 RAE.

Well, the results are interesting, if you like this sort of thing. It is clear that the results can be very different from those provided by the RAE, whichever method was used, although the "selected papers" method tends to give the closest results. It is also notable that the two different databases give different results, sometimes radically so; Scopus seems to consistently give higher values than WoS. Workers in some fields complained that they made more use of other databases, such as the arXiv or Google Scholar (it's worth noting that the favoured databases are proprietary, while the arXiv and Google Scholar are publically accessible).

In general, the institutions involved in the pilot preferred the "selected papers" method, but it seems that none of the methods produced particularly convincing results. According to the report (paras 66 and 67):

In many disciplines (particularly in medicine, biological and physical sciences and psychology), members reported that the ‘top 6’ model (which looked at the most highly cited papers only) generally produced reasonable results, but with a number of significant discrepancies. In other disciplines (particularly in the social sciences and mathematics) the results were less credible, and in some disciplines (such as health sciences, engineering and computer science) there was a more mixed picture. Members generally reported that the other two models (which looked at ‘all papers’) did not generally produce credible results or provide sufficient differentiation.

One of the questions here is what is meant by "reasonable" or "credible" results? The institutions involved in the pilot seem to assume that the best results are the ones that most closely match those of the RAE. I suspect this is because the large universities that currently receive the lion's share of research funding are not going to support any system that significantly changes the status quo.

The institutions involved in the pilot seem to think that bibliometrics would be most useful when used in conjunction with expert peer review. From the report:

Members discussed whether the benefits of using bibliometrics would outweigh the costs. Some found this difficult to answer given limited knowledge about the costs. Nevertheless there was broad agreement that overall the benefits would outweigh the costs – assuming a selective approach. For institutions this would involve a similar level of burden to the RAE and any additional cost of using bibliometrics would be largely absorbed by internal management within institutions. For panels, some members felt that bibliometrics might involve additional work (for example in resolving differences between panel judgements and citation scores); others felt that they could be used to increase sampling and reduce panels’ workloads.

According to the interim report, the "best" results (i.e. those most closely matching the results of the RAE) were obtained using a methodology that will have a similar administrative burden as the RAE. Even then the results had "significant discrepancies". So, if the aim of the pilot was to get similar results to the RAE with a lesser administrative burden, it seems that the pilot exercise has failed on both counts. So if bibliometrics don't seem to add much to the process, it's worth considering what they might take away. For which, see my previous post...

The usual excuse for not posting

2009-05-05T12:45:00.004+01:00

Yes, I've been hanging about in Egypt again, looking at rocks for my day job. In the absence of any bad science related stuff, here are some pretty pictures.

El Tor, the town where we stayed, at sunset.

Downtown El Tor.

Fossilised burrows in Miocene syn-rift rocks. There's a lot of this in the study area, which usually means that structures that would help to understand the depositional environment are obscured.

Part of the field area. To the right are rocks of the Precambrian basement. In the foreground, a major normal fault separates those Precambrian rocks from Nubian sandstone, Eocene carbonate units, and Miocene syn-rift calc-arenites.

Homeopathy paper published

2009-04-08T11:15:00.006+01:00

So, this is the moment you’ve all been waiting for. A while ago I wrote a comment on an article that was published in Homeopathy. This article, among other things, purported to show that the authors of a Lancet meta-analysis (Aijin Shang and co-workers) that had negative results for homeopathy had engaged in post-hoc hypothesising and data dredging. That was an outrageous slur on what is a perfectly reasonable paper, if you understand it properly. My comment has now been published, along with a response from the authors. If anyone needs a copy of my comment and doesn’t fancy paying for it, drop me a line and I’ll bung you a PDF. In any case, the original version appears on my blog here.

Meanwhile, the reply by original authors Rutten and Stolper is an exercise in evasion and obfuscation, and doesn’t really address most of the points that I made. This seems to be fairly typical (and to be fair isn’t only restricted to non-science like homeopathy). In their original paper, Rutten and Stolper claimed that “Cut-off values for sample size [i.e. the number of subjects in a trial, above which the trial was defined as “large”] were not mentioned or explained in Shang el al's [sic] analysis”. This is simply not true. So what do Rutten and Stolper have to say about this embarrassing error?

“Wilson states that larger trials were defined by Shang as “Trials with SE [standard error] in the lowest quartile were defined as larger trials”. According to Wilson this was done to predefine 'larger trials'. We agree with Wilson that this is indeed a strange way of defining 'larger trials', but it is perfectly possible to simply define larger studies a priori according to sample size in terms like 'above median' as we suggested in our paper. Shang et al did not mention the sensitivity of the result to this choice of cut-off value: if median sample size (including 14 trials) is chosen homeopathy has the best (significantly positive) result, if 8 trials are selected homeopathy has the worst result. In the post-publication data they mentioned sample sizes but not Standard Errors. Isn't it odd that the authors did not mention the fact that homeopathy is effective based on a fully plausible definition of 'larger' trials, but stated that it is not effective based on a strange definition of 'larger', but that this was not apparent because of missing data?”

So, nothing there about how they failed to properly read the paper to check what Shang et al.’s definition of larger trials was, while essentially accusing them of research misconduct. Instead, they shift the goalposts and decide that they don’t like the definition that was provided. Now, it certainly would be possible to define larger studies as being “above median” sample size. By doing this you would be including studies of smaller size than would be included using Shang’s definition. As is well understood, and as Shang et al. clearly showed, including studies with smaller sample size will give you more positive but, crucially, less reliable results. So I don’t think it was particularly odd that Shang et al. failed to abandon their definition of larger trials in favour of someone else’s definition, published three years later, that would inevitably lead to less reliable results. Rutten and Stolper state that using 8 larger, high quality trials gives the worst results for homeopathy: but to get a positive result, you would have to include at least 14 trials, as Ludtke and Rutten show in another paper in the Journal of Clinical Epidemiology. And, again, it was perfectly apparent what definition Shang et al. used to define larger trials: it is clearly stated in their paper.

OK, so why use standard error rather than simply using sample size directly, as Rutten and Stolper want to do? In meta-analyses, a commonly used tool is a funnel plot. This plots, for each study included in the analysis, standard error against odds ratio. The odds ratio is a measure of the size of the effect of the intervention being studied. If the value is 1, there is no effect. If it is less than one, there is a positive effect (the intervention outperformed placebo), if greater than one there is a negative effect (placebo outperformed the intervention). The plot is typically used to identify publication bias (and other biases) in the set of trials: to simplify, if the plot is asymmetric, then biases exist. Using their funnel plot of 110 trials of homeopathy (Figure 2 in the Lancet paper), Shang et al. were able to show, (to a high degree of statistical significance, p<0.0001)that trials with higher standard error show more positive results. It then makes perfect sense to screen the trials by standard error rather than sample size, because it has been demonstrated that standard error correlates with odds ratio. Of course, you could plot sample size against odds ratio, but that is not the recommended approach.

Rutten and Stolper also claim to be "surprised" that one apparently positive trial of homeopathy was excluded from Shang's analysis. Since it was excluded based on the clearly stated exclusion criteria, I didn't find that surprising myself. How do Rutten and Stolper respond?

"We were indeed amazed that no matching trial could be found for a homeopathic trial on chronic polyarthritis by Wiesenauer. Shang did not specify criteria for matching of trials. We would expect the authors to explain this exclusion because Wiesenauer's trial would have made a difference in meta-regression analysis and possibly also in the selection of the eight larger good quality trials".

This routine is now wearily familiar. Someone makes a claim that Shang et al. didn’t do something, in this case specify criteria for matching of trials; I check the Lancet paper, and find that claim to be false. What did Shang have to say about matching of trials? On page 727, they say “For each homoeopathy trial, we identified matching trials of conventional medicine that enrolled patients with similar disorders and assessed similar outcomes. We used computer-generated random numbers to select one from several eligible trials of conventional medicine”. And, of course, the authors did explain why the trial was excluded; it met one of the pre-defined exclusion criteria. To me, that seems clear enough. As it stands, Rutten and Stolper’s point is nothing more than an argument from incredulity. They are amazed! Amazed that no matching trial could be found. But they haven’t actually found one to prove their point. It’s possible that this Weisenauer trial might have made a difference to the selection of 8 large, high quality trials. But I doubt it would have made any significant difference to the meta-regression analysis, which was based on 110 trials.

Having wrongly accused Shang et al. of doing a bad thing by defining sub-groups post-hoc, Rutten and Stolper applied all kinds of post-hoc rationalisations for excluding trials they don’t like. For example, they decided to throw out all the (resoundingly negative) trials of homeopathic arnica for muscle soreness in marathon runners, on the basis that homeopathy is not normally used to treat healthy people, and these trials therefore have low external validity. I argued that Shang et al. had to include those studies, since they met the inclusion criteria and did not meet the exclusion criteria. On what basis could they exclude them? From Rutten and Stolper, answer came there none:

"Wilson's remark about prominent homeopaths choosing muscle soreness as indication is not relevant. Using a marathon as starting point for a trial is understandable from a organisational point of view, although doubt is possible about external validity. Publishing negative trials in alternative medicine journals is correct behaviour. There is, however, strong evidence that homeopathic Arnica is not effective after long distance running and homeopathy as a method should not be judged by that outcome".

Yes, publish the negative trials. But why shouldn’t the negative trials be included in a meta-analysis? Because they’re negative, and that just can’t be right? I don’t see any rationale here for excluding these trials.

Rutten and Stolper also take the tine-honoured approach of arguing about statistics:

“…the asymmetry of funnel-plots is not necessarily a result of bias. It can also occur when smaller studies show larger effect just because they were done in a condition with high treatment effects, and thus requiring smaller patient numbers”.

I think this is nonsense, but anyone with more statistical knowledge should feel free to correct me. If the high treatment effects are real, then the larger studies will show them as well, and there will be no asymmetry in the funnel plot. The smaller studies are always going to be less reliable than the larger ones.

Finally, Rutten and Stolper conclude that:

"The conclusion that homeopathy is a placebo effect and that conventional medicine is not was not based on a comparative analysis of carefully matched trials, as stated by the authors".

Homeopaths do want this to be true, but no matter how many times they repeat it, it continues to be false. I think the problem is that they have become fixated on the analysis of the subgroup of larger, higher quality trials, which was only one part of the analysis. The meta-regression analysis for all 110 vs 110 trials gave the same results; the analysis of the “larger, higher quality” subgroup merely lends support to those results. So after all that palaver, there’s still no reason to think that there is anything particularly wrong with the Shang et al. Lancet paper, and there is certainly no excuse for accusing its authors of research misconduct.

Bloody Elsevier

2009-03-20T09:04:00.003Z

Some time ago, I had a paper on normal fault evolution in the Gulf of Suez accepted for publication in the Journal of Structural Geology. This is an Elsevier journal, and the paper duly went off to the Elsevier production people to be published. Now, one of the figures in the paper is a large and spectacularly detailed geological map of the study area, done in the late 1990s by my co-author and former University of Manchester post-doc Ian Sharp. This is an excellent piece of work in itself, and it had never been published; we decided that this paper would be a good place to finally publish it. The level of detail on this map is such that we wanted to reproduce it in colour, at A3 size. We knew that this would cost money, but the industrial sponsors of the work were happy to cover the costs.

After a long and generally fruitless attempt at corresponding with the Elsevier production department (which has been outsourced to India, incidentally), I finally received a PDF proof of the paper in which the geological map was reproduced at A3 size. All well and good. Until the final version of the paper was published [paywall: for God's sake, don't pay $31.50 for this...if you really want a copy, e-mail me and I'll send you a PDF], and the map was back to A4 size, with much of the fine detail lost as a result.

Gah!

Now, surely it isn't on for Elsevier to unilaterally make changes to an article without consulting the authors about it. I know some people who have been involved in editing this journal, and it seems they are unhappy with how it is being run by Elsevier. As Dr Aust points out, companies like Elsevier charge large amounts of money for papers, in just about the only example of publishing in which the authors don't want to be paid for producing all the content. Elsevier makes massive profits out of journal publishing, gets to hide all of the content behind ridiculous paywalls, and doesn't even make a particularly good job of the journal production. There must be a better way.

What is the Russell Group for?

2009-03-19T11:04:00.004Z

The Russell Group contains the 20 major research-intensive universities in the UK. The University of Liverpool is a member of the group, and has recently made the news by earmarking its departments of Politics and Communication, Statistics, and Philosophy for closure. The reason is that those departments are seen as having underperformed in the 2008 RAE (Research Assessment Exercise).

In the RAE, departments are ranked by the proportion of research they have in five different categories, as follows:

4*: Quality that is world-leading in terms of originality, significance and rigour.

3*: Quality that is internationally excellent in terms of originality, significance and rigour but which nonetheless falls short of the highest standards of excellence.

2*: Quality that is recognised internationally in terms of originality, significance and rigour.

1*: Quality that is recognised nationally in terms of originality, significance and rigour.

Unclassified: Quality that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purposes of this assessment.

The three departments faced with closure had no research ranked in category 4*. According to Times Higher Education, "The university has questioned whether this is “acceptable” for a member of the Russell Group of 20 research-led institutions".

So, how did the threatened departments do overall? Here's their breakdown from the 2008 RAE (source):

Statistics: 4*, 0%; 3*, 35%; 2*, 50%; 1*, 15%; UC, 0%.

Politics and Communication: 4*, 0%; 3*, 15%; 2*, 55%; 1*, 25%; UC, 5%.

Philosophy: 4*, 0%; 3*, 25%; 2*, 60%; 1*, 15%; UC, 0%.

These results are surely not disastrously bad. In all cases, the vast majority of research is ranked at 3* and 2* levels: that is, it is considered to be internationally excellent or internationally recognised. Is this really such a poor performance that it requires the closure of the departments?

The threat of closure of these departments raises the question of what a university is actually for. If it only exists to receive as much research funding as possible, then closure is a perfectly sensible action. But if you consider the university as a community of scholars, with everyone (from undergraduates to professors) learning from each other, then closing these departments is going to contribute to the narrowing of the university experience for everyone. Is that really what the University of Liverpool wants to acheive? And is that what the Russell Group is supposed to be about?

Another example of bad science with serious real world consequences

2009-03-13T13:06:00.002Z

Via Respectful Insolence, I came across this story: massive research fraud has been uncovered in the field of anaesthesiology. It appears that one Scott Reuben, MD, is accused of fabricating results in at least 21 studies he conducted in the field of multi-modal analgesia; this is discussed in more detail at Science Based Medicine. The studies are now in the process of being formally retracted by the journals that published them.

It's difficult to over-emphasise the seriousness of this. Recommendations about best practice for pain management have been made on the strength of these studies. It is now not clear that those recommendations are appropriate. Until further studies are done to sort this mess out, people are going to be denied the best possible standard of care. Bad evidence has consequences.

What is particularly galling about this case is that it was not uncovered through the scientific method. Peer review didn't uncover it, and neither did a failure to independently replicate Reuben's results. In fact, it was eventually uncovered because it was noticed that Reuben did not have approval to conduct research on human subjects for two abstracts he had submitted for presentation. The scientific community has nothing to be proud of here. Fair enough, it's largely impossible for peer review to spot fraud: there has to be a degree of trust that the data presented is not simply fabricated. But fraudulent research has entered the literature, and had recommendations based on it. Make no mistake about it, this is a massive failure. It's no good saying that the scientific method ensures that such fraud will eventually be discovered: it didn't ensure it in this case, and by now the damage is done. The science based medicine community needs to urgently consider how this sort of thing can be prevented in future.

Cheers for the measles, antivaccine campaigners...

2009-03-13T10:40:00.006Z

The Manchester Evening News reports that there is a measles outbreak in northeast Manchester and Oldham. Three children have been hospitalised so far.

Here's an example of bad science having consequences in the real world. Take-up of the MMR (measles, mumps, rubella) vaccine fell in the 1990s, following research by Andrew Wakefield that suggested a connection between autism, bowel problems and the MMR vaccine. To say that this research has now been discredited is really an understatement. Not only was the research incompetent, but there is strong evidence that it was fraudulent too. All the authors of the article, apart from Wakefield and another who could not be contacted, retracted the interpretation that MMR was linked to autism and bowel problems. This piece of bad science has led, indirectly but inexorably, to three children in Manchester having to be hospitalised with measles (which is not a harmless childhood illness). That is why this stuff matters.

There is no evidence that the MMR vaccine causes autism. If you don't get your child vaccinated, not only are you putting them at risk, you're putting other children at risk. MMR is safe: tell your friends.

In the absence of other news...

2009-03-02T14:54:00.006Z

...here are some pictures from Harstad, north Norway, where I've been working with StatoilHydro on a new project for the last week. Harstad is in the arctic circle, and it's snowy and cold just now. But the scenery is breathtaking.

Here I am with my new friend, the stuffed polar bear that has pride of place in the entrance lobby of the StatoilHydro office. Apparently the bear doesn't have a name yet. That can be a project for my next visit (which will probably be in June). This polar bear isn't that big, but I reckon it could still rip your face off pretty good.

The view from the window of the StatoilHydro office in Harstad.

Mountains, fjords and beach huts

Mountains, fjords and stunted Norwegian trees

I win the peer review lottery

2009-02-21T21:55:00.005Z

It's been a good week. On top of the article I had accepted for publication in Homeopathy (and let's face it, they publish any old rubbish), I've just had a paper accepted by a proper scientific journal, the Journal of Structural Geology. I wrote about this paper before, because it initially got rejected. In fact, this is the third time it has been through review, and it's taken about a year and a half, but we've finally got there.

This is how I looked when I found out the good news. I confess that I poured myself a generous Scotch in celebration.

I'm going to be in Norway for the next week for work reasons, so I won't be posting anything over that time, but then posting has been pretty patchy this year. Thanks to all those who have continued to read anyway.

Homeopathy meta-analysis comment accepted for publication

2009-02-17T16:13:00.004Z

Just a short post to say that the article I wrote pointing out some problems with a re-analysis (Rutten and Stolper 2008) of the Shang et al. Lancet meta-analysis has been accepted for publication by Homeopathy. I have also been sent the reply by the authors of the re-analysis. My comment and the reply will not actually appear in print until April, so I'd better not address the content of the reply at this point. I will say that I don't think it adequately addresses the points that I made. In particular, the authors don't have much to say in response to the point that information they claim is missing from the Shang paper is in fact clearly stated in that paper. More to come on this in April.

The Rutten and Stolper paper, and a companion paper in the Journal of Clinical Epidemiology by Ludtke and Rutten, were the subject of a press release titled "New Evidence for Homeopathy" claiming to cast doubt on the Shang meta-analysis. Perhaps I should issue a press release titled "New Evidence Against Homeopathy". Then again, maybe it would be better titled "New Evidence Against Homeopaths".

Now, despite apearances, I have to say that the subject of meta-analyses of homeopathy is not one that particularly fascinates me. It's just that a number of prominent homeopaths have made claims that the Shang study is flawed and/or fraudulent. In checking the claims that have been made, mainly be simply checking the Shang paper and its supplementary data, I have almost invariably found that they are false. Apgaylard has found similarly. I find it amazing that these false accusations have propagated across the internet and been accepted as truth, without anyone apparently doing the most basic of fact checking.

Jeremy Sherr and the ethics of CAM

2009-01-18T15:36:00.008Z

As a geologist, I don't tend to have to worry about the ethics of research involving humans; I usually only deal with rocks. But in medical research, ethics are crucially important. In CAM research, the ethical dimension often seems to be lacking, which brings me to the case of Jeremy Sherr.

Jeremy Sherr is a prominent homeopath who claims to treat AIDS patients in Tanzania with homeopathy, based on the usual poor quality anecdotal evidence and wishful thinking. He recently caused a stir in the badscience blogosphere, with a blog entry in which he mused about conducting a trial of homeopathy in AIDS patients. The mooted trial is transparently unethical, as pointed out by a number of sceptical bloggers (notably in the comments on Sherr's blog, and at Gimpy's blog, here and here, and The Lay Scientist, here and here), and as we'll see below. Now, Sherr likes to edit his blogposts and delete comments, but let's have a look at what Sherr had to say about the trial.

I am happy to go for a simple trial initially, treating AIDS patients who are not taking ARVs. There is no shortage of patinets who, although they have been offered ARVs, have chosen not to take them, usually because of the serious and debilitating side effects. There are plenty of statistics on ARV treatment and patients with no treatment at all that we can compare to.

Why is this unethical? I would say for three reasons. Firstly, it is a general principle of medical ethics that patients in a clinical trial should not be denied proven treatments for their condition. Clearly, in this trial AIDS patients would be denied ARVs. Sherr seems to think that this is OK because his patients have decided not to take ARVs themselves, but this is, I think, irrelevant. You would still be running a trial in which the subjects are not receiving the best possible standard of care. The issue of informed consent is again critical here; patients would need to be informed that not taking ARVs could be severely damaging to their health.

Secondly, the trial as mooted will not provide any useful information, because there is no control group. Whatever happens in the trial, it is impossible to say that it happened because of homeopathy, rather than sources of bias in the trial design. Since the trial could not provide any usable information, it would be unethical.

Thirdly, I would say we have enough evidence and knowledge about homeopathy to say that it is not going to cure AIDS. Given that there is no likelihood of a true positve result, it is unethical to involve patients in a clinical trial. Informed consent comes into play again here: patients in the trial ought to be told that the current evidence shows that there's essentially no chance of homeopathy having any beneficial effects beyond placebo.

Sherr says, in the recent blog post in which he calls the waaaaambulance over the criticism he has received, that "Any research I may undertake will be subject to rigorous ethical review of the highest standard". Hopefully that will in fact happen, in which case the mooted trial will surely not go ahead. What is disturbing is that Sherr has stated in the past, referring to research protocols and ethics review, that "You have to find willing partners and get a protocol through an ethics committee, and you need to talk their language. I hope it will work but if not, I will just go and do it on a small scale myself - I am determined to do that". This is the most telling comment, I think: it makes it clear that Sherr is not really interested in medical ethics, except as a hoop he must reluctantly jump through in order to experiment on terminally ill patients. And if he can't get ethics approval, he'll just do it anyway.

So much for Sherr. Disturbingly, however, he seems not to be anomalous in CAM circles in his total lack of any sense of ethics. A review of a book "Complementary and Alternative Medicine: Ethics, the Patient, and the Physician” has just been posted on the Science Based Medicine blog. The reviewer comments that "We do not read a word about how to approach a patient who has suffered damage due to CAM, or how to approach those who have stopped their regular treatment" [emphasis mine]. One would have thought that this would be a key issue for any book purporting to address ethics in CAM.

As one of the commenters to the Science Based Medicine piece astutely points out, CAM is a "deprofessionalization phenomenon". Researchers in the field of CAM seem to have no idea about research ethics, and no idea about the linked issue of how to conduct good research. If a text on ethics in CAM is so careless of these important ethical questions, how can we expect CAM practitioners to be any more careful?

UPDATE: There is now an excellent and comprehensive post on the Sherr saga at Respectful Insolence...

Yet one more radically pointless homeopathy study

2009-01-10T22:10:00.006Z

Generally, when the efficacy of a medical intervention is being assessed, you expect the quality of the trials and evidence available to increase through time. You might start out with uncontrolled case reports and observational studies, proceed through small-scale pilot studies, to large randomised placebo-controlled trials, and finally to systematic reviews and meta-analyses of the literature.

In CAM research this pattern is often not followed. Once systematic reviews and meta-analyses start to show that there is no evidence that the CAM treatment works, more small trials of poor methodology are conducted, many of which inevitably have (spurious) positive results. This allows CAM advocates to claim that there is lots of evidence in favour of their intervention, because they don't bother to account for study size and quality.

There is a fine example of this CAM tradition in the latest issue of Homeopathy. A systematic review, published in 2006, of homeopathy for treatment of allergic rhinitis concluded that "Some positive results were described in rhinitis with homeopathy in good-quality trials, but an equal number of negative studies counterbalance the positive ones. Therefore it is not possible to provide evidence-based recommendations for the use of homeopathy to treat allergic rhinitis, and further randomized controlled trials are needed". Well, perhaps: some would argue that the prior probability (close to nil) and currently existing evidence are enough to conclude that homeopathy does not work for allergic rhinitis (or, indeed, anything else). Be that as it may, it should be clear that the only useful new evidence would come from large and well-conducted RCTs. So what do Maria Goossens and a football team of colleagues do in the latest issue of Homeopathy? Why, publish a "prospective, open, non-comparative study" on homeopathy and allergic rhinitis, of course.

The methodology of the study consists of "treating" some patients suffering symptoms of allergic rhinitis with homeopathy, and getting them to fill in a quality of life questionaire at the start of the study, and after three weeks and four weeks. The physicians involved also assessed the severity of symptoms at baseline, three weeks, and four weeks. Unsurprisingly, the study found that people felt better with homeopathic treatment. But the methodological problems with this study are straightforward to point out. There is no control group. As a result, there can be no randomisation or blinding. Don't take my word for it; here's what the authors say in the discussion of their paper:

We did not distinguish between intermittent and persistent allergic rhinitis. All patients with intermittent allergic rhinitis (symptoms present less than four consecutive weeks a year) will be better after four weeks without any treatment. Patients who consult a homeopathic physician for allergic rhinitis usually have been suffering for a long time and from severe symptoms as the high level of the RQLQ score at baseline indicates. This study cannot be conclusive because there is no control group. Neither the physician, nor the patient was blinded. We cannot conclude that the degree of certainty of the physician about the appropriateness of the homeopathic prescription of a homeopathic remedy and the physician’s impression whether he had sufficient information about the patient’s condition influenced the outcome...it is not possible to draw a conclusion on the effect of the homeopathic treatment. This would require an RCT. To evaluate the effect of homeopathic treatment for allergic rhinitis an RCT should be performed.

So there you have it. The study cannot come to any useful conclusions. And, in the introduction to the paper, the authors write "This study was originally considered as a preliminary to a Randomized Clinical Trial (RCT) comparing standard conventional therapy with homeopathy (non-inferiority study). The RCT was never performed because sponsorship was withdrawn".

OK, that's life. Sometimes planned research funding fails to come off. These things happen. But why then publish the pilot study? Methodologically, it is useless, and it could never have added anything to the previously existing evidence from RCTs and systematic reviews. This study would never have been published anywhere other than a CAM journal, where scientific usefulness can take a back seat to an ideological desire to publish any evidence that looks as though it is in favour of homeopathy, no matter how methodologically weak it is, and in defiance of the higher level evidence that already exists.

Well, no-one is going to die from allergic rhinitis, so how much does it matter? The problem is that homeopaths don't stop at self-limiting conditions like hayfever. Some insist that homeopathy is a complete system of medicine and it can cure anything, including AIDS and malaria. Ben Goldacre's miniblog points to Jeremy Sherr's blog, for example, where Sherr is preparing to begin an unethical experiment on AIDS sufferers. This is a long road of madness, to be sure, but it begins where people believe they can cure hayfever through the use of magic sugar pills.

Does the REF add up to good science?

2009-01-09T16:17:00.003Z

The RAE (Research Assessment Exercise) results from the 2008 were published back in December. You might have noticed this from the number of university websites that could be found frantically spinning the results. My very own University of Manchester, for example, is claiming that Manchester had broken into the “golden triangle” of UK research, that is, Oxford, Cambridge and institutions based in London. It seems that depending on the measure you pick, we’re anywhere between third and sixth place in the UK. Clearly these are excellent results, but whether we’re really up there with the Oxfords, Cambridges, Imperials and UCLs of the world I’m not sure.

In any case, that was the last ever RAE. It has been a fairly cumbersome process, involving expert peer review of the research contribution of research institutions, that has been a real burden on the academics who have had to administer it. I’m sure there are few who will mourn its passing. Now the world of English academia is waiting, like so many rats in an experimental maze, to find out what will replace the RAE. The replacement will be a thing called the Research Excellence Framework, or REF, and at this stage exactly what it will involve is fairly sketchy. However, it will be based on the use of bibliometrics (statistical indicators that are usually based on how much published work is cited in other publications) and “light-touch peer review”.

What kind of bibliometric indicators are we talking about? Last year HEFCE (the Higher Education Funding Council for England, the body that evaluates research and decides who gets scarce research funding) published a “Scoping study on the use of bibliometric analysis to measure the quality of research in UK higher education institutions” produced by the Centre for Science and Technology Studies at the University of Leiden, Netherlands. I’ve spent a fair amount of time reading through this, and in some ways I was encouraged. It’s clear that some thought has gone into creating bibliometric indicators that are as sensible as possible: I was dreading a crude approach based around impact factors, which have already done so much damage to the pursuit of good science. The authors of the “scoping study” came up with an “internationally standardised impact indicator”: I will abbreviate this as ISII for concision. The ISII takes the average number of citations for publications for the academic unit you are interested in (this might be a research group, an academic department or an entire university), and divides it by a weighted, field-specific international reference level. The reference level is calculated by taking the average number of citations for all publications in a specific field: if the publication falls under more than one field (as many will in practice), the reference level can be calculated as a weighted average of the number of citations generated by publications in all the fields in question. So, if the ISII for your research group comes out as 1, you’re average, if above 1, better than the average, and if below 1, worse than the average. The authors of the scoping study say that they regard the ISII as being “the most appropriate research performance indicator”, and suggest that a value of >1.5 indicates a scientifically strong institution. They also suggest a threshold of 3.0 to identify research excellence. It seems that the HEFCE is expecting to adopt the ISII as the main research performance indicator, according to their FAQs, where they say “We propose to measure the number of citations received by each paper in a defined period, relative to worldwide norms. The number of citations received by a paper will be 'normalised' for the particular field in which it was published, for the year in which it was published, and for the type of output”. However, they are still deciding what thresholds they will use to decide which institutions are producing high-quality research.

All well and good. If you insist that bibliometric indicators are necessary, this is probably as good a way as any of generating those data. However, there are some problems here, as well as philosophical difficulties with the entire approach.

Firstly, what is it we are trying to measure? In theory, what HEFCE wants to do is evaluate research quality. But the ISII does not directly measure research quality. Like any indicator based on citation rates, it is measuring the “impact” of the research: how many other researchers published papers that cited the research. It ought to be clear that while this should reflect quality to some degree, there are significant confounding factors. For example, research that is done in a highly active topic is likely to be cited more than research in which fewer groups are working. This does not mean that work in less active topics is of intrinsically lower quality, or even that it is less useful.

Secondly, there is an assumption that the be-all and end-all of scientific research is publication in peer-reviewed journals that are indexed in the Web of Science citation database published by Thomson Scientific. This a proprietary database that lists articles in the journals that it indexes, and also tracks citations. Criteria for journals to be included are not in the public domain (although the scoping report suggests these are picked based on their citation impact, p. 43). A number of journals that I would not consider to be scientifically reputable are included. For example, under the heading of Integrative and Complementary Medicine, the 2007 Journal Citation Reports (a database that compiles bibliometric statistics for journals in the citation database) includes 12 journals, including Evidence Based Complementary and Alternative Medicine (impact factor 2.535!) and the Journal of Alternative and Complementary Medicine (impact factor 1.526). This reinforces the point made above: it would be possible to publish outright quackery in either of these journals, have it cited by other quacks in the quackery that they publish, and get a respectable rating on the ISII. The ISII can’t tell you that this is a vortex of nonsense: it only sees that other authors have cited the work. It is also true that not all journals are included in the citation index: for example, in my own field the Bulletin of Canadian Petroleum Geology fails to make the cut, although it has always published good quality research. Although the authors of the scoping report make clear that it is possible to expand bibliometrics beyond the citation database, this will take much more effort and it seems that HEFCE will not take this route. So we will be relying on a proprietary and opaque database to make decisions on future research funding. A further point is that it is not clear how open access publications will be incorporated in the citation index: in principle there is no reason that this can’t happen, but can we be sure it will?

Thirdly, there is the assumption that research output can only be evaluated in terms of published articles in peer-reviewed journals. I’m not sure that this accurately reflects the actual research output of many scientists. For example, most of us put a lot of effort into presentations at scientific conferences, chapters in books, or government reports that will never make it into a citation database. This has become a problem for things like, in my own field, the special publications of the Geological Society of London. These are volumes that collect recent research on specific topics, and they generally contain excellent research. But they aren’t included in citation databases and they have no impact factor. This has led to a lack of interest in publishing results in these special publications, because they don’t tick the right boxes in terms of publication metrics. This is surely a bad thing. A similar problem occurs with things like government open-file reports. These are not, in general, pieces of world-class, cutting edge research. But that does not mean that they are useless or that they have no value. For example, good regional geological work can allow mineral exploration to be better targeted, benefiting the local economy. Yet that kind of work is ignored in a framework that only considers journal articles: HEFCE says only that “We accept that citation impact provides only a limited reflection of the quality of applied research, or its value to users. We invite proposals for additional indicators that could capture this”. To me, research quality and value cannot be measured by bibliometric indicators. It can only be evaluated by reading the research, understanding its context within the totality of pre-existing research, and understanding how it contributes to new understanding. That is, it can only be evaluated through peer review.

Which brings me to my fourth point; there are some questions about the role of peer review within the REF. HEFCE says that “the scoping study recommends that experts with subject knowledge should be involved in interpreting the data. It does not recommend that primary peer review (reading papers) is needed in order to produce robust indicators that are suitable for the purposes of the REF”. However, I’m not convinced that this accurately summarises what is written in the scoping report, which says “In the application of indicators, no matter how advanced, it remains of the utmost importance to know the limitations of the method and to guard against misuse, exaggerated expectations of non-expert users, and undesired manipulations by scientists themselves…Therefore, as a general principle we state that optimal research evaluation is realised through a combination of metrics and peer review. Metrics, particularly advanced analysis, provides the tools to keep the peer review process objective and transparent. Metrics and peer review both have their strengths and limits. The challenge is to combine the two methodologies in such a way that the strengths of one compensates for the limitations of the other”.

Finally, there is a hint of conflict of interest in the preparation of the scoping report by the Centre for Science and Technological Studies: according to their website, the centre is involved in selling "products" based on its research and development in the area of bibliometric indicators. Their report in favour of bibliometric indicators might allow them to drum up significant business from HEFCE.

At present, the proposals for the REF are at a fairly early stage, but the use of bibliometric indicators seems to be entrenched, and there will be a pilot exercise on bibliometric indicators this year. However, this is based on “expert advice” that consists of a single report from an organisation that makes money by creating bibliometric indicators. While academia in general might welcome the proposals on the grounds that they will be less burdensome than the RAE and give everyone more time to do research, I don’t think many academics will be kidding themselves that the bibliometric indicators involved actually tell us much about research quality and usefullness.

Pediatrics publishes a critique of an "anti-vaccine" book

2008-12-31T16:36:00.003Z

I've tended not to write anything about vaccines, partly because it's a long way from my field of expertise, and partly because it brings out the internet loons in greater numbers than almost anything else. Still, I ran across this via Ben Goldacre's miniblog, and it overlaps with my interest in scientific publishing: Pediatrics, the official journal of the American Academy of Pediatrics, has published an article by Paul Offit and Charlotte Moser. The article is highly critical of "The Vaccine Book: Making the Right Decision for Your Child", by Dr. Robert Sears, also known in media-friendly mode as Dr. Bob in his role as one part of "America's family of pediatricians". This is something that I haven't seen before: the journal of a learned society publishing a rebuttal of a popular health book. The article goes through Dr. Sears's book, and picks out misleading statements, flawed logic and questionable assumptions, particularly concentrating on the alternative vaccine schedules suggested in the book.

It seems unlikely that worried parents will pick up this article; they are much more likely to read the book. But perhaps by putting these arguments in one place in Pediatrics, it makes it easier for pediatricians themselves to counter the arguments against vaccines that they hear from parents. It might even be possible to give parents the article to look through. And many of the arguments in Dr. Bob's book ARE arguments against vaccines, in spite of the protestations of Dr. Bob himself. The paper is not written in overly technical language, although it does contain a lot of references that are unlikely to be easily available to the average concerned parent. It gives pediatricians a chance to say that actually mainstream scientific opinion does not agree with Dr. Bob. This is unlikely to do much to sway those who've drunk the anti-vaccine kool-aid and think that Dr. Offit is an unscrupulous shill of big pharma, willing to cause lasting harm to untold numbers of children in order to make money (just do a google search to see what I mean). But it might sway sensible people who have inevitably picked up doubts from the unbelievably incompetent reporting of the science surrounding vaccines in the media.

Perhaps it could be worth putting a similar article debunking the myths on MMR into something like the BMJ or the Lancet? Or perhaps something similar already exists and I've missed it?

News on latest Homeopathy submission

2008-12-12T16:47:00.002Z

A while ago a submitted a comment to the journal Homeopathy, discussing a paper published in the journal by Rutten and Stolper, claiming to find a number of serious flaws in the Shang et al. meta-analysis that appeared in the Lancet in 2005. I had a slightly cryptic e-mail back from the journal last week:

Dear Dr Wilson,

Thank you for sending us this article. We have sent the article to the authors for comment.

We will be in touch again shortly.

I assume that means that the comment will be published, with a reply by the authors. I'll be interested in what they have to say, because at this point some of the errors in their paper seem to be indefensible.

Post #100, and what Wilson is up to

2008-12-12T16:22:00.002Z

The previous post on Manchester's congestion charge was the hundredth post on Hawk/Handsaw. I'd celebrate with a beer, if I hadn't come back from fieldwork in Egypt with some friendly bacteria that are currently restricting me to soup and toast.

For those how are interested in what I'm doing, I've just completed my first post-doctoral project at the University of Manchester. The first paper from that work now has a DOI number and is available as an "accepted manuscript" at the Journal of Structural Geology website (behind a paywall, I'm afraid: if anyone is desperately interested and can't access it, e-mail me and I can bung you a PDF).

I will be starting a new project in January, still based at Manchester, working on sub-surface data from the North Sea. I'll be sponsored by StatoilHydro, so I'll be doing a fair bit of work at various StatoilHydro offices in Norway, starting in Harstad in late January. It's a three year contract, so I have a bit of stability. The plan is to get more papers out, and try for lectureships towards the end of the project.

Manchester says No!

2008-12-12T15:42:00.003Z

The results of Manchester's referendum on congestion charging and improvements to public transport are in, and they could hardly be clearer. According to the Guardian, 79% of respondents said no, on the back of a 53% turnout. That's close to a general election level of turnout around here. Although the Yes campaign claimed that 9 out of 10 people would not pay the charge, around 4 out of 10 people decided that they didn't want it.

I have distinctly mixed feelings about this. Mostly, I'm disappointed; after all, I did vote yes myself. That's easy for me: I don't own a car, walk to work, and rely on public transport to get around Greater Manchester. The congestion charge would cost me precisely nothing. According to the No campaign, it would work out at £1200 a year for those who would have to pay the maximum amount. It's clearly a regressive tax, and that is not a small amount of money for people who are already struggling to make ends meet. Of course, that's the point: the charge wouldn't work if everyone could easily afford it.

For me, the most important thing about the proposals was the improvements to public transport. There is no doubt that these are needed. Some of them, such as extensions of the Metrolink tram system to Oldham and Rochdale, will happen anyway. But I suspect we will be stuck with an unreliable and expensive (deregulated) bus service for a long time to come.

The problem is that, in the words of the Yes campaign, "there is no plan B". That being the case, it seems crazy to put the question to a referendum. Saying that you either agree with these proposals or you get nothing is hardly a democratic way of posing a referendum question; it's a pretense of democracy. It would have been more sensible, and democratic, to either have our local representatives on the Manchester Council decide to implement the proposals come what may, or have a referendum in which there was a genuine choice of competing proposals.

Improvements in public transport are a good thing, but on their own they will not reduce car use. Driving will always be more convenient than taking the bus, unless there are clear disincentives for car use. It seems that congestion charging is a political impossibility in Manchester, at least right now. An alternative might be to make driving in the city less convenient, by restricting parking, restricting access to certain streets, etc. This would have the advantage of not stinging the poor with another regressive tax. What is clear is that doing nothing is not an option in the long term: it's back to the drawing board for Manchester's transport policy.

Science by press release (epic fail)

2008-11-19T20:03:00.006Z

While I was messing about at the University of Google, doing some research for the thing that I've just sent to homeopathy, I came across this [pdf]. Yes, the homeopaths (in the form of the International Homeopathic Medical League and the European Committee for Homeopathy) have put together a press release, based on the study in Homeopathy by Rutten and Stolper, and the study by Ludtke and Rutten in the Journal of Clinical Epidemiology. Both studies criticise (mostly wrongly) a perfectly good meta-analysis of homeopathy from the Lancet. The headline is "New evidence for homeopathy". No question mark, no caveats. You might guess that the International Homeopathic Medical League and the European Committee for Homeopathy would not be entirely unbiased sources of information. In fact, these studies are very far from being evidence for homeopathy, as I've tried to point out (and not just me). I suppose two studies sounds better than one, but the truth is that these are essentially the same study, with extra nonsense in the part that got published in Homeopathy. The studies apply a post-hoc analysis to the data from the Lancet article, and claim to be able to produce a positive result for homeopathy if you squint at the data and stand on one leg. Then they accuse the Lancet paper of data dredging. What a laugh riot.

Unfortunately, lots of scientific studies get reported based on their press releases. This means that the people who read the reports get no sense of the flaws in the study and the caveats that should be applied to its conclusions; these come from independent scientific scrutiny of the study once it has been peer-reviewed and published. Pushing press releases on your research may be a good way to get brownie points from your university, but it's not usually a good way of fostering an improved understanding of the scientific process. To be fair, it's by no means only homeopaths and their ilk who do this, as Ben Goldacre has pointed out.

Still, the good thing about this is that a search on Google News for this release shows that, at the time of writing, it has only been picked up by a few woo-ish magazines: mainstream western news seems to have ignored this more or less completely. Maybe the press has got fed up of this particular manufactured controversy, at least for now.

I know I said life was too short...

2008-11-19T11:15:00.004Z

I wrote a little about a paper by Rutten and Stolper recently published in the amusing pseudo-journal Homeopathy. The paper performed the usual homeopath party trick of throwing incorrect allegations of research misconduct at the Shang et al. meta-analysis of homeopathy that was published in the Lancet, while also engaging in dubious statistical analysis. I've now had a little time to put together something a bit more meaty, with proper references and everything, and send it off as a letter to the editor of Homeopathy. I reproduce the text below.

Rutten and Stolper [1] have conducted a re-analysis of the data used in the landmark Lancet meta-analysis (Shang et al.) [2] of trials of homeopathy and conventional medicine. However, their approach to this work seems to have been influenced by a belief that the Shang analysis was deliberately skewed against homeopathy, and in favour of conventional medicine. I argue here that the evidence does not support that contention, and that the re-analysis by Rutten and Stolper does not show that the Shang et al. study was invalid.

Rationale for the re-analysis

In the abstract of their paper, Rutten and Stolper state “There is a discrepancy between the outcome of a meta-analysis published in 1997 of 89 trials of homeopathy by Linde et al and an analysis of 110 trials by Shang et al published in 2005, these reached opposite conclusions”, and on page 170 they write “The contradiction between Linde's conclusion based on 89 trials, and Shang et al's conclusion, based on 110 trials seems odd”. But there is nothing particularly surprising about this discrepancy. The Linde paper referred to was published in the Lancet in 1997 [3]. The same team re-analysed the data in a paper published in 1999 [4]. They concluded that because trials of higher methodological quality had smaller effect sizes, and that because a number of newly published high-quality trials showed negative results for homeopathy, their meta-analysis had over-estimated the effectiveness of homeopathy. Hence there is no reason to see to the discrepancy between Shang et al. and Linde et al. (1997) as being particularly “odd”.

Trial quality

Rutten and Stolper make statements about the “pre-specified hypotheses” of the Shang et al. study, but these are not consistent through the paper. In the introduction, they state:

“The hypotheses predefined mentioned in the introduction of Shang et al's paper were: ‘Bias in conduct and reporting of trials is a possible explanation for positive findings of placebo-controlled trials of both homeopathy and allopathy (conventional medicine)’; and: ‘These biases are more likely to affect small than large studies; the smaller a study, the larger the treatment effect necessary for the results to be statistically significant, whereas large studies are more likely to be of high methodological quality and published even if their results are negative’.”

Yet, in Rutter and Stolper’s section on “Pre-specified hypotheses” they include “quality in homeopathy is worse than in conventional medicine” as a hypothesis of Shang et al., and say that this hypothesis was falsified in the Shang et al. study. This is a straw man: it is not a hypothesis that was discussed in the Shang et al. paper, and Rutten and Stolper have missed the point of including a matched set of trials of conventional medicine. As Rutten and Stolper state (p. 170) “Pooling of results is…questionable if homeopathy works for some conditions and not for others”. This is a reasonable point. However, it is clear that some experimental conventional treatments work and some do not. The results of the analysis of conventional medicine were not consistent with the placebo hypothesis, showing that it is possible to obtain a positive result using the methods of Shang et al., even there is considerable heterogeneity in the results [5].

Post-hoc analysis?

Rutten and Stolper make the claim that the sub-sets of larger, higher quality studies were chosen post-hoc, presumably to make homeopathy appear less effective than it really is. In their paper, Rutten and Stolper state [p. 172-173]:

“Cut-off values for sample size were not mentioned or explained in Shang el al's [sic] analysis. Why were eight homeopathy trials compared with six conventional trials? Was this choice predefined or post-hoc? Post-publication data showed that cut-off values for larger higher quality studies differed between the two groups. In the homeopathy group the cut-off value was n = 98, including eight trials (38% of the higher quality trials). The cut-off value for larger conventional studies in this analysis was n = 146, including six trials (66% of the higher quality trials). These cut-off values were considerably above the median sample size of 65. There were 31 homeopathy trials larger than the homeopathy cut-off value and 24 conventional trials larger than the conventional cut-off value. We can think of no criterion that could be common to the two cut-off values. This suggests that this choice was post-hoc.”

The first thing to note is that it is not true that cut-off values for sample size were not mentioned or explained in the Shang et al. analysis. In the original Shang paper, on page 728, it is stated that “Trials with SE [standard error] in the lowest quartile were defined as larger trials”. In other words, the cut-off was not defined in terms of numbers of subjects, but in terms of standard error. It might be argued that this is a strange way of defining “larger” trials (and perhaps it should have been phrased as “lower standard error”). But it makes sense when criteria must be stated a priori. If a number of subjects were stated as a cut-off value, there would be no way of knowing how many studies would meet that criterion before looking at the data. You might find that a very large or very small number of studies met the criterion, making further analysis difficult. So, there is no mystery as to why the “cut-off values” were different between trials of homeopathy and trials of conventional medicine: it is because the distribution of standard errors is different between the two populations. This could be discovered simply by reading the original paper, and the conclusion that the groups were chosen post-hoc cannot be sustained.

A further point here is that the group of “larger” homeopathy trials contains smaller trials that would not have made the cut for “larger” trials in the conventional medicine group. Those smaller trials are more likely to show spurious positive results. It follows that had the authors engineered the groups to get the result they wanted, they had engineered them in favour of homeopathy.

Another paragraph in Rutten and Stolper states “We did not further investigate possible selection bias by excluding trials, but we were surprised by the exclusion of Wiesenauer's trial on chronic polyarthritis. This was a larger trial (n = 176), of good quality according to Linde, with positive results. This trial would have contributed positively to the outcome of the larger higher quality trials. Shang excluded this trial because no matching trial could be found” (page 171). Since the trial was excluded on the basis of the clearly stated, pre-specified exclusion criteria, what is surprising about it having been excluded? Including it would have made a nonsense of the design of the study and violated the pre-specified exclusion criteria, and would have been a gross error.

Another possible outcome?

Rutten and Stolper conduct a sensitivity analysis, but, as they note, the decisions they make in this analysis are highly subjective. They decide to exclude all trials of homeopathy for muscle soreness [6-9], on the grounds that “treatment of healthy individuals is very rare in homeopathic practice [and] this outcome has low external validity to judge the effect of homeopathy as a method” (page 173). Yet, the trials were conducted with the participation of prominent homeopaths, and some were published in homeopathic or alternative medicine journals [8, 9], so at least some homeopaths seem to be of the opinion that there is enough external validity for it to be worth conducting a trial. So how can the external validity of the trials be judged in a transparent way? In a meta-analysis based on clear, pre-specified criteria, there could be no justification for omitting the studies.

It is also notable that one of the authors was a co-author of another re-analysis published in the Journal of Clinical Epidemiology [10]. That analysis showed that if random-effects meta-analysis is used, it is possible to add smaller trials to Shang’s set of “larger, higher quality” trials of homeopathy, and get a statistically significant (although clinically unimpressive) benefit for homeopathy. All this really shows is that a finding in favour of homeopathy is not robust, and as Shang et al. showed, including smaller trials also decreases the reliability of the findings. The re-analysis also showed that the benefit for homeopathy was statistically insignificant when a meta-regression analysis was used: this negative finding was strangely not mentioned in the Homeopathy paper. Because the results differed between meta-regression and random-effects analyses, and because Shang et al. showed highly significant evidence of funnel-plot asymmetry in their complete dataset of 110 trials of homeopathy, it is arguable that meta-regression analysis is a more appropriate choice.

Overall, it is clear that “another outcome” (i.e. one favourable to homeopathy) is possible, as long as negative studies are excluded without good reason, smaller and less reliable studies are included, and a particular method of statistical analysis is used. In a paper that (wrongly) criticises a study for analysing data based on criteria established post-hoc, this seems like an odd point to make.

Conclusion

The analysis by Rutten and Stolper contains misconceptions of Shang et al., contains some important errors, and does not show that the Shang et al. study was an invalid analysis. In particular, there is no evidence that the Shang et al. study involved post-hoc choice of subgroups. The results of meta-analyses can be debated, but scientists should not be accused of research misconduct on the basis of no evidence, or on the basis of having failed to read their work properly.

References

1. Rutten ALB and Stolper CF. The 2005 meta-analysis of homeopathy: the importance of post-publication data. Homeopathy 2008; 97: 169-177.

2. Shang A, Huwiler-Müntener K, Nartey L et al. Are the clinical effects of homeopathy placebo effects? Comparative study of placebo-controlled trials of homeopathy and allopathy, Lancet 2005; 366: 726–732.

3. Linde K, Clausius N, Ramirez G et al. Are the clinical effects of homeopathy placebo effects? A meta-analysis of placebo-controlled trials, Lancet (1997); 350: 834–843.

4. K. Linde K, Scholz M, Ramirez G, Clausius N, Melchart D, Jonas WB. Impact of study quality on outcome in placebo-controlled trials of homeopathy, J Clin Epidemiol 1999; 52: 631–36.

5. Shang A, Jüni P, Sterne JAC, Huwiler-Müntener K, Egger M. Are the clinical effects of homeopathy placebo effects? A meta-analysis of placebo-controlled trials: Author’s reply, Lancet 2005; 366: 2083-2084

6. Vickers AJ, Fisher P, Wyllie SE, Rees R. Homeopathic Arnica 30X is ineffective for muscle soreness after long-distance running – A randomized, double-blind, placebo-controlled trial. Clin J Pain 1998; 14: 227–231.

7. Vickers AJ, Fisher P, Smith C, Wyllie SE, Lewith GT. Homoeopathy for delayed onset muscle soreness - A randomised double blind placebo controlled trial. Brit J Sports Med 1997; 31: 304–307.

8. Jawara N, Lewith GT, Vickers AJ, Mullee MA, Smith C. Homoeopathic Arnica and Rhus toxicodendron for delayed onset muscle soreness - A pilot for a randomized, double-blind, placebo-controlled trial. Brit Hom J 1997; 86: 10–15.

9. Tveiten D, Bruset S, Borchgrevink CF, Norseth J. Effects of the homeopathic remedy Arnica D30 on marathon runners: A randomized, double-blind study during the 1995 Oslo Marathon. Complement Ther Med 1998; 6(2): 71–74.

10. Lüdtke R, Rutten ALB. The conclusions on the effectiveness of homeopathy highly depend on the set of analyzed trials. J Clin Epidemiol 2008; 61: 1197-1204

Journal of Structural Geology paper

2008-11-06T13:10:00.002Z

If anyone is interested in the paper I recently got accepted in the Journal of Structural Geology, there is a short summary of it here...

Hazel Blears talks rubbish about the blogosphere

2008-11-06T10:03:00.005Z

I was struck by this article in yesterday's Grauniad. Hazel Blears, the MP for Salford and communities minister (who I incidentally saw having a swift half in Manchester's City Arms the other day), weighs in on the culture of "corrosive cynicism" which is supposedly damaging political discourse in the UK. This, of course, is all the fault of the media, and in particular the blogosphere. The following quote is from an address that Blears is giving today at a Hansard Society conference on growing political disengagement in Britain:

"Perhaps because of the nature of the technology, there is a tendency for political blogs to have a 'Samizdat' style. The most popular blogs are rightwing, ranging from the considered Tory views of Iain Dale, to the vicious nihilism of Guido Fawkes. Perhaps this is simply anti-establishment. Blogs have only existed under a Labour government. Perhaps if there was a Tory government, all the leading blogs would be left-of-centre? But mostly, political blogs are written by people with disdain for the political system and politicians, who see their function as unearthing scandals, conspiracies and perceived hypocrisy. Until political blogging 'adds value' to our political culture, by allowing new voices, ideas and legitimate protest and challenge, and until the mainstream media reports politics in a calmer, more responsible manner, it will continue to fuel a culture of cynicism and despair."

Now, I don't want to claim that all of what Blears says here is total nonsense. Clearly the media in general does have a lot to answer for. But perhaps there could be other reasons for political disengagement in the UK? I can think of a handful off the top of my head.

1. The growing reliance of the Labour party on rich donors, which has led to a number of scandals, including the "cash for honours" affair. The corollary of this is a decrease in party membership, an erosion of internal party democracy, a lack of connection between the party grassroots and the government, and a perception amongst the electorate that the government is corrupt.

2. The Iraq war, opposed by a large proportion of the population, and launched on the basis of statements that were not true (whether or not they were strictly "lies"), has now led to the deaths of more than 175 British soldiers and serious injuries to many more. This is aside from the civilian death toll in Iraq, the best that can be said of which is that we don't know what it is, but it is a hell of a lot.

3. A failure to apply appropriate regulation to financial markets allowed the inflation of a credit and asset bubble which has now burst, and will lead to perhaps hundreds of thousands of people losing their jobs and their homes.

4. A massive proliferation in anti-terror laws, which despite re-assurances when they were brought in, are now being used to suppress legitimate protest and freeze the assets of Icelandic banks.

No doubt you can think of a few more. So perhaps this corrosive cynicism has as much to do with the cynicism and incompetence of the government as that of the media and the blogosphere?

Quite apart from that, it is interesting to look at what Blears says about the blogosphere. She says that political bloggers "see their function as unearthing scandals, conspiracies and perceived hypocrisy", and then adds that "until political blogging 'adds value' to our political culture...it will continue to fuel a culture of cynicism and despair." To me, unearthing scandals does add value to our political culture. Or should government wrong-doing just be hidden? An important aspect of democratic government is that those who govern us can be held to account. That can't happen if no-one knows what they're up to.

It's obviously true that a large number of blogs are dreadful and useless. But it's equally true that many are valuable. By indulging in a rant against the media and the blogosphere, without addressing the contribution of government corruption and incompetence, Blears is not doing much to further the debate on political disengagement.

More meta-analysis delight

2008-10-20T14:57:00.007+01:00

Having whined and bullshitted about the 2005 meta-analysis (Shang et al.) that was published in the Lancet (which showed that larger, high-quality trials demonstrated no effect for homeopathy beyond placebo), the homeopaths are now trying a bit of number-crunching. This is an improvement, although they don't seem to be any better at number crunching than they are at logic, as we saw with the recent Ludtke and Rutten paper in the Journal of Clinical Epidemiology.

In the Journal of Clinical Epidemiology, there is a proper review process, and the authors couldn't get away with writing just any arrant nonsense that came into their head. This led to a paper that broadly supported the original conclusions of the Shang study, which obviously wasn't that much use. Luckily, Homeopathy, the in-house fanzine of the Faculty of Homeopathy, has no such qualms about publishing more or less any old rubbish (and I should know, I've been published in it myself). Hence a paper by Rutten and Stolper in the October issue of Homeopathy criticising the Shang paper, entitled "The 2005 meta-analysis of homeopathy: the importance of post-publication data". I'm not going to analyse the whole thing-there comes a point when life really is too short-but I do want to pick up on one blatant error that seems to suggest they haven't carefully read the paper they are criticising.

One thing that Shang did was look at papers that were of good quality, based on how well they were randomised, how well they were blinded, and so on. They also sought to look at the "larger" trials, because their analysis showed that smaller trials with higher standard errors tended to be more positive. Rutten and Stolper claim that the cut-off size for larger trials was different between the homeopathy trials and conventional trials "without plausible reason". They write:

In the homeopathy group the cut-off value was n = 98, including eight trials (38% of the higher quality trials). The cut-off value for larger conventional studies in this analysis was n = 146, including six trials (66% of the higher quality trials). These cut-off values were considerably above the median sample size of 65. There were 31 homeopathy trials larger than the homeopathy cut-off value and 24 conventional trials larger than the conventional cutoff value. We can think of no criterion that could be common to the two cut-off values. This suggests that this choice was post-hoc.

What Rutten and Stolper are doing here is essentially accusing Shang and colleagues of fiddling the results so that homeopathy looked less effective than it really is. That is a serious accusation, and you might expect that someone making it would have carefully checked the original paper to see what the authors said their criterion for larger trials was. If you read the original Shang paper, on page 728 you will find this definition: "Trials with SE in the lowest quartile were defined as larger trials". There is nothing particularly opaque about this, and yet the idea that what was meant by "larger" trials was somehow unclear comes up time and time again.

Picking trials "with SE in the lowest quartile" may seem like an odd way of choosing the "larger" trials. Why not just pick a number, like 100, which Rutten and Stolper seem to want to do? Well, if you want to avoid post-hoc analysis it is a sensible thing to do, as you know you will have a reasonable number of "larger" trials to work with. If you just pick out a number, you might find that almost none, or almost all, of your trials qualify as being "larger", since you don't know what the distribution of trial sizes is until you do your analysis.

Of course, another thing is that in a comparison between homeopathy and conventional medicine, the discrepancy in trial size numbers actually gives homeopathy an advantage. This is because smaller trials of homeopathy that would have missed the cut for conventional trials are included, and these are more likely to show spurious positive results. So, if the difference in trial sizes really were due to the authors' bias, then they were biased in favour of homeopathy.

So, Rutten and Stolper have made an erroneous accusation of biased post-hoc analysis based on not bothering to properly read, or carefully think about, the paper they are criticising. In my opinion they have made themselves look rather silly. Yet, even sillier still, their entire paper is an exercise in post-hoc analysis, as they try to find ways of torturing the data to get the result they want, i.e. that homeopathy works. This is how things go in the wonderful world of homeopathy, where all kinds of pseudoscience can be justified as long as they give you the right answer.

Result!

2008-10-10T12:01:00.001+01:00

I've just had the first paper from my post-doctoral work at Manchester accepted by the Journal of Structural Geology, pending minor revisions.

Beeeeerrrrrssssss!

Shang study remains firmly in the water

2008-10-08T12:04:00.007+01:00

In the comments to a post at Respectful Insolence, my favourite homeopath Dana Ullman weighs in with the suggestion that the Shang et al. meta-analysis of trials of homeopathy and conventional medicine (which has been written about extensively by me and apgaylard), had been "blown out of the water". Ullman makes this assertion based on a new paper by Ludtke and Rutten, entitled "The conclusions on the effectiveness of homeopathy highly depend on the set of analyzed trials", that has been accepted by the Journal of Clinical Epidemiology. It's nice to see that this paper does exist after all. So does the article really blow Shang out of the water? A quick look at the conclusions tells us that the answer is no:

Our results do neither prove that homeopathic medicines are superior to placebo nor do they prove the opposite...it occurs that Shang’s conclusions are not so definite as they have been reported and discussed.

What Ludtke and Rutten have done is actually quite interesting, though, so I'd like to explore it in a bit more detail. What they've done is taken the 21 trials of homeopathy that Shang et al. considered to be of "higher quality". They have then performed various analyses on this subset of trials to see what happens if you exclude trials based on their size or other parameters.

The authors plotted a funnel plot of odds ratio (a measure of the size of the effect of the intervention: values below 1 indicate a benefit over placebo) versus standard error (which is dependent on trial size). For all of the 21 trials, they found that there was substantial (but not statistically significant) asymmetry in the plot (if the funnel plot is asymmetrical, then biases are present in the data: these might be publication bias, small study effects, or a whole host of other effects). They also note that no evidence for asymmetry was found for the 8 largest trials of the 21. This actually re-iterates one of the main points of the Shang analysis: that a large number of trials is needed to identify asymmetry, and therefore to get an idea of bias in the dataset. That is why Shang et al. looked at all 110 trials that matched their inclusion criteria; that enabled them to identify a highly significant asymmetry in their funnel plot. This was important because it showed that the smaller the study size, the larger the apparent effect.

The thing in the paper that homeopaths will probably emphasise is that for the set of 21 higher quality trials, the pooled odds ratio (from random effects meta-analysis) was 0.76, suggesting a benefit over placebo. But wait! What are the 95% confidence intervals? 0.59-0.99. This indicates anything from a unimpressive benefit to a clinically negligible one. In other words, it's a deeply uninspiring result, but homeopaths will be trying to tell you that it shows that homeopathic remedies do something.

The interesting thing that the authors then did was to take the 2 largest trials, and look at what happens to the results when you add trials in descending order of patient numbers (Figure 2 of the paper). Once you get to the point where you've included the 14 largest trials, the resulting odds ratio is always less than 1 (except for the case of 17 trials). This is interesting in a way, but all it really does is demonstrate what Shang et al. said: that smaller trials are more likely to give positive results. So the more trials you add, the more positive but less reliable the results are; with 14 or more trials you might just about scrape a statistically significant benefit, but that result is not as reliable as the analysis restricted to the eight largest trials. It's also worth noting that the upper limits of the confidence intervals in Figure 2 are always close to 1, showing that any benefit is likely to be clinically insignificant. They perform a similar analysis in their Figure 3, except they use a meta-regression analysis rather than a random effects meta-analysis, and for that they show no statistically significant benefit no matter how many studies they include.

Another thing that homeopaths will probably jump on is that if one very negative trial (on Arnica for muscle soreness) is omitted from the set of 21 trials, the results appear more positive (odds ratio 0.73, 95% confidence interval 0.56-0.93) when a random effects meta-analysis is used. There are a number of other trials that can be removed from the dataset to give apparently positive results, but only when random effects analysis is used: a meta-regression analysis shows that there is no statistically significant benefit no matter which study you remove. Also, when performing a similar analysis on the 8 large, higher quality trials originally identified by Shang et al., no statistically significant benefit is found whichever trial you decide to remove. Again, note that the 8 largest trials are intrinsically more reliable than the smaller ones.

All the way through the paper, it is noticeable that meta-regression analysis shows more negative results than a random effects meta-analysis. In fact, the authors point out that in their meta-regression analysis "no single predicted OR [odds ratio] could be shown to differ significantly from unity". So which should be used? The authors write "... there is no guideline which tells a researcher when to prefer meta-regression to random effects metaanalysis or vice versa. As the statistical test for asymmetry only has a small power, Egger suggests to perform metaregressions when the respective P-value falls below 0.10. Applying this criterion there seemed to be no need to perform a meta-regression in most of the subsets we analyzed". But this conclusion is based on the restricted analysis of 21 higher quality trials. Shang et al.'s original analysis of 110 trials of homeopathy showed asymmetry with p<0.0001, suggesting that a meta-regression analysis would be more appropriate.

So, the upshot is that the paper's title is misleading. The conclusions on the effectiveness of homeopathy do not highly depend on the set of analyzed trials, if an appropriate test is used. Asymmetry is not adequately identified in the dataset because too few trials are used. And, even if you can convince yourself that you can get a statistically significant benefit by playing around with the numbers, the actual clinical benefit is negligible. In some ways, the paper actually reinforces the conclusions of Shang et al., and it certainly doesn't show that homeopathic medicines work.

Sayed Gooda: a tribute

2008-10-03T10:51:00.003+01:00

There was sad news for me this week, and for more or less everyone who has done geological fieldwork in Egypt. Sayed Gooda, our driver on our trips to Sinai, has passed away.

Without Sayed, our work would have been a lot more difficult. Sayed didn't just drive us, over bad roads and difficult terrain. He looked after us, kept us out of trouble, made us delicious sandwiches for lunch (the highlight of the Sinai field day), and tried valiantly to teach us Arabic . Practically everyone who has worked on the Gulf of Suez owes something to Sayed. I will always remember arriving in a hot, noisy and fume-choked Cairo for the first time, nerves jangling from an overnight flight from Amsterdam, and being put at ease right away by the big friendly man in the big friendly green jeep.

Sayed was in Saudi Arabia when he passed away, and has been buried there. In the e-mail telling us of his death, his friend Tarek Moustafa wrote "I think he died Happy as this would have been a dream come true to him to die in the Holy lands". We can only hope so. Thanks for everything, Sayed.

If I was Prof Michael Reiss, I'd be annoyed...

2008-09-19T10:55:00.005+01:00

Poor Prof Reiss: not only has he been forced to resign for making a generally sensible speech that was leapt on and distorted by anti-religious Fellows of the Royal Society and the media, but the media (in the shape of the Guardian) has actually admitted that it had a part in his downfall. In a column titled "In praise of Prof Michael Reiss": nice touch.

Quotes:

The subtlety of Prof Reiss's position was lost in some media reports, while the headlines in many newspapers- including this one - did not convey the nuance of his message. This appears to have cost him his job.

and:

But Prof Reiss's shabby treatment smacks of an organisation that is frightened of a debate about how creationist views should be tackled by teachers.

You can bet that this is the line that will be taken by those who would wish to pollute science with creationist nonsense. This seems like a spectacular own goal to me.

Creationism in schools: a manufactured controversy

2008-09-15T11:36:00.007+01:00

I wrote briefly about the comments of Prof. Michael Reiss, director of education at the Royal Society, that creationism should be taught in schools in certain circumstances. The reaction to Reiss's comments has been somewhat alarming, with calls for him to be sacked [UPDATE: Professor Reiss has now resigned].

Here's Sir Harry Kroto:

I warned the president of the Royal Society that his [Reiss] was a dangerous appointment a year ago. I did not realise just how dangerous it would turn out to be.

And Sir Richard Roberts:

I think it is outrageous that this man is suggesting that creationism should be discussed in a science classroom. It is an incredible idea and I am drafting a letter to other Nobel laureates - which would be sent to the Royal Society - to ask that Reiss be made to stand down.

And, inevitably, Richard Dawkins:

A clergyman in charge of education for the country's leading scientific organisation - it's a Monty Python sketch.

To me these comments seem to taking things too far, especially in the light of a letter to today's Guardian from Prof. Reiss himself:

Your headline (Teach creationism, says top scientist, September 12) misrepresents the views of myself and the Royal Society. The society believes that if a young person raises the issue of creationism in a science class, a teacher should be in a position to examine why it does not stand up to scientific investigation. This does not put it on a par with evolution, which is recognised as the best explanation for the history of life on Earth from its beginnings and for the diversity of species.

Evolution is rightly taught as an essential part of biology and science courses in schools, colleges and universities across the world. Creationism, which has no scientific validity, can be discussed in a science class if it is raised by a pupil, but should in no way be seen as comparable to evolution or any other scientific theory which is backed up with evidence.

If that is what Reiss is saying, it seems to be fairly difficult to argue, and it's essentially what I was getting at in my previous post. But it's a bit of tangle, and there are several interlocking issues. A number of people seem to believe that Reiss's position as an ordained church minister makes him ineligible for his post, as is made most clear by Dawkins. I would have to disagree with that. Just because Reiss is a committed Christian, that clearly doesn't mean he has to believe in the literal truth of the Genesis account of creation. Plenty of perfectly sensible people take it as an allegory or a fable. There are creationists and creationists; if Reiss were a young Earth creationist, arguing that the Earth was 6,000 years old and was created in seven days, his position would be untenable. But that is clearly not what he is saying, as a reading of what he actually said will show. The controversy seems to have stemmed from the way the Guardian quoted him, when they wrote that Reiss thought that "science teachers should not see creationism as a 'misconception' but as an alternative 'world view'. What he actually said was:

Creationism can profitably be seen not as a simple misconception that careful science teaching can correct. Rather, a student who believes in creationism has a non-scientific way of seeing the world, and one very rarely changes one's world view as a result of a 50-minute lesson, however well taught.

That should be seen in the context of what Reiss said about discussion of creationism:

If questions or issues about creationism and intelligent design arise during science lessons they can be used to illustrate a number of aspects of how science works.

This is exactly my view on the matter, and I would say that it is a more nuanced and sensible point than the Guardian's precis would suggest. I can't be too scathing about that, as I wrote my previous post based on the Guardian's construction of Reiss's comments. Which just goes to show that you should always go back to original sources.

For me the grave difficulty here is that scientists are seen to be saying that there is no place for the religious in science. A religious man is being attacked for religious views that he doesn't actually hold on closer inspection. In some cases he is being attacked simply for being religious. That is wrong. There have always been religious people in science, and many of them have been exceptionally capable. What Reiss has said is actually reasonably sensible, and a lot of the opprobrium seems to have come from reading the press, rather than what Reiss has actually said.

You lose, quack

2008-09-15T10:45:00.005+01:00

Another fantastic result for the mighty Ben Goldacre: a nutritionist by the name of Matthias Rath was suing Dr Goldacre, and the Guardian newspaper, which publishes his Saturday column. Dr Goldacre had criticised Rath for his advice that nutritional supplements could reverse the course of AIDS, and that patients should stop taking anti-retroviral drugs (ARVDs). Rath has now dropped the case, and been ordered to pay costs. Hopefully this will generate enough publicity for people to realise just how dangerous bad, unevidenced health advice can be.

We know that ARVDs work, and we know that nutritional supplements do not work, for treating people with AIDS. So what Rath was doing was advising people not to take drugs that work, and instead to take nutritional supplements which could have had no effect on the course of their illness. It is difficult to describe this as anything other than murderous quackery. There seems to be little doubt that people have died because of this advice.

You may think it is fair to say that Rath probably doesn't fall into the 'mainstream' of nutritionists. But I am not so sure. Patrick Holford, for example, a prominent UK nutritionist, has said that "AZT, the first prescribable anti-HIV drug, is potentially harmful and proving less effective than vitamin C”. This is superficially based on an actual scientific study. But it goes far, far beyond a reasonable interpretation of the study, which looked at what happens to some cells in a dish on a lab bench when you put some vitamin C on them. It is simply impossible to take that kind of bench research and apply it to what happens in something hugely complex like the human body. Holford is not a marginal figure: he is at the top of his profession. At least he doesn't suggest that ARVDs don't save lives.

Rath is clearly barking. But he thrives in the environment of denigration of mainstream science, and misunderstanding of science, that obtains in the world of nutritionism. The difference is that Rath, in taking his supplement quackery to somewhere that desperately needed drugs, not vitamins, he was able to cause a hell of a lot of damage.

As Ben Goldacre points out, the title "Nutritionist" is not protected, so anyone can call themselves a nutritionist. It follows that I am a nutritionist too. As your nutritionist, I would suggest that if you feel that you need nutritional advice, you should seek out a registered dietitian, who will actually be a qualified health professional.

Creationism in schools redux

2008-09-12T11:45:00.003+01:00

I suppose the debate as to whether you should teach creationism in science class will always be with us. Here's Prof Michael Reiss, director of education at the Royal Society, wading into the morass in today's Guardian. My opinion on this, for what it's worth, is that what needs to be taught is what science is, how it is done, and what it's useful for. Without that context, students don't have the tools to evaluate the arguments, and you're basically engaged in indoctrination, whether you're teaching evolution or creationism. In my view, the evolution versus creationism debate is a perfect opportunity to provide that context; simply pretending that there's no discussion is not helpful.

Unfortunately, this does not seem to be the position that Reiss is taking. From the article:

Reiss said he used to be an "evangelist" for evolution in the classroom, but that the approach had backfired. "I realised that simply banging on about evolution and natural selection didn't lead some pupils to change their minds at all. Now I would be more content simply for them to understand it as one way of understanding the universe," he said.
Reiss, who is an ordained Church of England minister, told the British Association Festival of Science in Liverpool that science teachers should not see creationism as a "misconception" but as an alternative "world view".

This seems to be taking the rather wishy-washy view that all world views have equal explanatory power. To me, creationism is not a way of 'understanding' the universe: it's a way of refusing to understand it. For all I know, there could be a creator of some kind. But that is not a scientific hypothesis that enables me to understand anything I observe. The place of creationism in science class is as an example of what is not science, and where the limits of scientific enquiry might lie.

Heavy metals in "safe", natural, alternative medicines...why we need evidence based medicine

2008-08-27T13:32:00.007+01:00

I saw this in today's Guardian. A team led by Robert Saper of the Boston Medical Centre ordered 230 different ayurvedic medicines from websites, and analysed the 193 they actually received for metal content [interesting that they never received 37 of them...]. For those interested, the paper is in JAMA 300 (8): 915-923, but it's behind a paywall...

What is Ayurvedic medicine? It's not something I know a lot about, but it's a traditional form of Indian medicine. Medicines are made up of herbs or mixtures of herbs, which can be combined with metals, minerals or animal products. The medicines that include metals, minerals or gems are known as rasa shastra medicines. As is typical for any form of alternative medicine, Ayurveda is described as being safe and natural by the various people who sell it: try a simple Google search to see what I mean. But how safe are they really?

Well, according to Saper and colleagues, a good proportion of them are stuffed full of heavy metals. Of the 193 medicines they analysed, 20.7% contained detectable levels of lead, mercury or arsenic. Unsurprisingly, metals were more prevalent in the rasa shastra medicines, of which 40.6% had detectable levels of nasty metals. The authors then took the recommended doses for each of the metal-containing medicines, and calculated what the daily intake of heavy metals would be for someone taking the medicine according to the stated dose. They found that for ALL of the medicines that contained detectable metal levels, a person taking the recommended dose would exceed standards for the ingestion of heavy metals. For some of the medicines manufactured in India, ingestion of heavy metals exceeded the limits by up to 10,000 times.

Pretty scary, eh? There are some caveats here. Firstly, these medicines were bought online: the authors concede that the results may not be similar for medicines given in consultation with an Ayurveda practitioner, or over the counter in western pharmacies. Even so, a previous study of medicines sold in Boston by Saper and colleagues showed that 20% contained lead, mercury or arsenic. So be careful: it is clear that while Ayurveda may be a 'natural' alternative to conventional medicine, it is not necessarily 'safe', as these case reports also testify. This is why conventional medicines are tested for safety before they can be sold. While there is no doubt that the evidence-based process of efficacy and safety testing can sometimes go wrong, it should be supported; the alternative is taking medicine of no proven benefit and with no understanding of the risks. This is too often what happens with inadequately regulated alternative treatments.

Renaissance for t'other blog

2008-08-26T17:05:00.004+01:00

As some of you may know, I have another blog that I use to write about my research, and other things of interest in the geology of sedimentary basins and petroleum geology. Or at least, that was what I was supposed to do with it. Of course, its far too much fun to go homeopath baiting, and so I've tended to neglect it.

I've now got a plan to try and write something for that blog once a week. The first of these posts is up now, and addresses the question of abiogenic oil. I hope at least some of you will be vaguely interested enough to have a look. Future stuff will include some neat bits and pieces from my own research.

Pride and Prejudice

2008-08-26T13:06:00.003+01:00

I was in Manchester city centre for the Pride parade on Saturday. It was a wonderful sunny day, and everyone had a smile on their face; you don't see that in Manchester every day. You've never so many loons in one place in your life. Cracking stuff.

The parade is a pretty big event these days. It takes about an hour and a half to go past. Half the city seems to be in it, with the other half watching. My better half was sprinting around the parade route with her running club, the Manchester Frontrunners, so I was in town to watch.

As you might imagine, there was a small group of Christian fruitcakes just by the Town Hall, holding placards about how God destroyed Sodom and Gomorrah, and so on. This was never going to be popular, and they were surrounded by a phalanx of Manchester's finest. There was occasionally a chorus of boos as the more militant sections of the parade went past, but they were mainly ignored or just treated as part of the entertainment. Once the parade had gone past, the Christian loon-in-chief was shouting about how further Pride events would bring destruction upon Manchester. After all, God had punished New Orleans for its Mardi Gras celebrations. A small crowd of people were stood around, roaring with laughter at every new piece of nonsense, before gradually filtering away to various of Manchester's hostelries.

It seems like poetic justice that of all the people in Manchester on Saturday, the Christian loons were the only ones not having a good time.

A mystery paper...

2008-08-19T11:48:00.004+01:00

Just a little bit more on the interview with Dana Ullman that I wrote about here.

Ullman claims that a re-analysis of Shang et al. has been accepted for publication in the Journal of Clinical Epidemiology. The only reference I can find to this study is this, where a study dated 2007, entitled "The conclusions on the effectiveness of homeopathy highly depend on the set of analysed trials" by R Ludtke and ALB Rutten is listed as being 'in press' in the Journal of Clinical Epidemiology.

Here's the list of articles in press in the Journal of Clinical Epidemiology. There is no sign of any such paper. Various searches fail to find any similar papers published anywhere else, or in earlier issues of the Journal of Clinical Epidemiology. The only thing I can find is a paper in Homeopathy called "‘Proof’ against homeopathy in fact supports Homeopathy", in which one Lex Rutten is credited as the first author. Whether this is the same Rutten I cannot say. The main point of the paper seems to be that if you add four positive trials to the Shang dataset, the result would be more positive. And they accuse Shang of cherry-picking. Two of the trials complained about were excluded [PDF] from the Shang meta-analysis: the Fisher et al. paper because it had an ineligible study design, and the Weisenauer and Gaus paper because no matching conventional trial could be found. Of the other two, one by Arnal-Laserre appears to be a French thesis of some description [EDIT: This is a French thesis: it was mentioned in the Cochrane review of "Homoeopathy for the induction of labour". Apparently, the reviewers could not obtain a copy of the thesis, which perhaps explains why Shang et al. did not include it], and the other by Maiwald et al. was not a placebo-controlled trial.

So, does this re-analysis exist, or is it just another figment of the collective homeopathic imagination? And if it ever does get published, is it likely that it will have anything useful to say?

Dana Ullman says the thing that is not...

2008-08-18T13:20:00.004+01:00

...no surprise there, then.

Here's Ullman, a US base homeopath, in an interview published on the website of Sue Young, a London-based homeopath. There's all kinds of drivel here, but there is an exchange on the Shang et al. meta-analysis of homeopathy (published in the Lancet in 2005) that particularly caught my eye, because it's full of absolute nonsense. Not only that, but Ullman has had his misconceptions about this paper explained to him in numerous places on the internet, including on this very blog. Here's what he and his interviewer, one Louise Mclean of the Zeus information service, had to say:

DANA: In fact there is a new study that is coming out shortly which is a re-analysis of the 2005 Lancet review of Shang. The researchers got it accepted in a major international journal of research. What they have finally done is what Shang didn’t do. He didn’t review ALL of the high calibre research but only a small part of it. He ignored comprehensive analysis entirely. I think he knew exactly what it was but he didn’t want to report on it, as it was too positive. Instead he only reported on trials with very large numbers of subjects because when you do that, most of those studies use one remedy for everybody without any degree of individuality.
LOUISE: We individualise.

DANA: We do individualise but sometimes the single remedy or the formulas will work for a broad number of people.

LOUISE: Like Mixed Pollen for hayfever.

DANA: That’s right or Oscillococcinum. But for some reason they did not include any of David Reilly’s research. http://www.bmj.com/cgi/content/abstract/321/7259/471

I don’t know why they ignored it.

LOUISE: It was too positive.

DANA: In fact they had a remark in the Shang article published in the Lancet, where they specifically made reference to trials on respiratory ailments and that the results were robust, but they said they couldn’t trust them because there were only 8 studies. But then again they based their entire analysis on 8 homeopathic studies and 6 conventional ones. So they can’t have it both ways and this new journal article in the Journal of Clinical Epidemiology which is ranked as one of the top international journals of reviews of research, has accepted the new studies.

Sigh. Why is this nonsense? Let me count the ways.

1. Shang et al. did in fact analyse ALL of the trials of homeopathy that met their inclusion criteria. This allowed them to establish, using statistical methods, that smaller trials and those of less robust methodology showed better results for homeopathy, because of bias. The good quality, large studies showed that homeopathy had no effect. This is the pattern you would expect to see if homeopathy is a placebo.

2. Ah, individualisation. In fact, a number of the trials in the Shang study were of individualised homeopathy (including two of those that were considered large and of high quality). There was no evidence that individualised homeopathy was better than any other type of homeopathy (p=0.636). In any case, individualisation is only important when it suits Ullman, as seen when he says "We do individualise but sometimes the single remedy or the formulas will work for a broad number of people".

3. The meta-analysis not only included the Reilly paper in the BMJ that is linked to, but two other Reilly papers, as can be seen from the additional material [PDF] to the paper that is available online. This is contrary to Ullman's assertion that "for some reason they did not include any of David Reilly’s research".

4. The point that Shang et al. make about the 8 studies of respiratory ailments is that 8 studies was too few for a meta-analysis restricted to those studies to detect the bias that is revealed by an analysis of the complete dataset. The eight studies of homeopathy that Ullman wrongly claims Shang et al. "based their entire analysis on" were identified as the studies most likely to be free of bias, based on an analysis of the entire dataset. So the authors are not trying to have it both ways at all, and Ullman is comparing apples with oranges.

What I find particularly annoying about this is that Ullman and Mclean are essentially accusing Shang and his co-workers of research misconduct. What do they base this very serious accusation on? On a total misunderstanding of their paper, and a flat-out lie that they omitted research that was 'too positive', when that research was in fact included in the analysis. I am not a statistician, but the paper is not that difficult to understand, if you read it. Followers of Dana Ullman's career will not be surprised by his disingenuousness on this, I'm sure.

It seems that no matter how often I (and others, notably apgaylard) write about the persistent mis-representation of the Shang paper, the homeopaths carry on regardless.

More bad homeopathic water science

2008-08-15T11:43:00.004+01:00

More from the new issue of comedy journal Homeopathy. Homeopaths continue to pursue research on 'high dilutions' (i.e. dilutions such that there is highly unlikely to be any of the original solute remaining) in search of a vaguely plausible mechanism for homeopathy to work. Since the evidence is that homeopathy doesn't work, this is unlikely to be a fruitful endeavour, but it does provide entertainment for connoiseurs of pseudoscience.

The latest offering is a paper by one R. Assumpção , entitled "Electrical impedance and HV plasma images of high dilutions of sodium chloride" and an acccompanying editorial by Cyril Smith. Two sets of observations are presented: a set of Kirlian photographs and a set of measurements of the impedance of various homeopathic dilutions of NaCl.

What is Kirlian photography? When an object resting on a photographic plate is connected to a source of high voltage, an image is formed on the plate. Semyon Kirlian, whom the technique is named after, thought that such images might be compared to a human 'aura'. You can see how this would be attractive to various kinds of energy medicine fruitcakes. In the paper, Assumpção provides Kirlian photographs of homeopathic dilutions of an NaCl solution (6c, 12c, 24c and 30c) and of succussed (shaken) water without dilution. The images are certainly pretty, but what do they show? Not a lot, as far as I can tell. It isn't clear how such images should be interpreted, and all Assumpção really concludes is that different dilutions look different. This, in itself, is not a particularly exciting finding, in the absence of any hypothesis as to why that is and what it might mean.

What of the impedance data? Assumpção finds that impedance of the samples increases with dilution, up to about 12c (which is the point where Avogadro's constant suggests that there is unlikely to be any of the original NaCl remaining), and then decreases again slightly with further dilution. "This phenomenon is inexplicable in terms of conventional chemical theory" concludes Assumpção, a statement that really should be followed by "OMG!!!!11!1!1!!1!!!!!1".

Well, I think I have a reasonable explanation for the results that causes no problems for conventional chemical theory. As expected, dilution of the NaCl solution causes an increase in impedance as ions are removed from the solution. But there is also a competing increase in the concentration of dissolved ions caused by the succussion (agitation) process, as atmospheric gases and other impurities are dissolved in the water. So the small decrease in impedance beyond 12c could result from this process. I'm more puzzled as to why the impedance of the distilled water increases when it is shaken, suggesting a decrease in the concentration of dissolved ions. However, I suspect that there is a conventional explanation, but Assumpção doesn't seem to be particularly curious about this. Is there a change in the ionic species in the water as it is shaken? Does shaking facilitate outgassing of the dissolved gases that remain in the distilled water? We don't know. This continues the homeopathic water research tradition of publishing things without any thought as to what factors (other than the magic of water) might have led to the results.

Of course, the other thing to note is that the paper only looks at one solute, NaCl. For homeopathy, what would be really interesting is if you could show that there was a difference between two different homeopathic substances at dilutions greater than 12c. Previous attempts to do this have ended in some embarassment, as Smith acknowledges in the accompanying editorial, when he states "in the light of the controversy which has attended previous claims in this field, caution, and independent repetition of these results is required". There seems to be little sign of any such caution so far.

Finally, in Figure 5 of the paper, Assumpção had omitted to label to the impedance curves, showing which curve belongs to which dilution. The solution? Simply write the labels in, with a pen, afterwards. It is truly amateur hour.

Are the homeopaths getting there?

2008-08-14T13:13:00.006+01:00

The new issue of Homeopathy (the in-house shitrag of the Faculty of Homeopathy) is out, which is always a joy. To my surprise, there's actually a paper in there that seems to be groping towards a vague sort of sense. The authors (Clare Relton and Alicia O' Cathain of the University of Sheffield, and Kate J. Thomas of the University of Leeds) seem to be on the cusp of imagining what homeopathy could actually be useful for. They don't say outright that the principles of homeopathy make no sense and that it doesn't work, but they do seem to be getting close to an idea of what might make homeopathy useful in the absence of specific effects from the wee sugar pills they give you. As we know, the evidence from the largest and best quality trials tells us that those pills don't do anything over and above the placebo effect. But what can we do with the placebo effect? In some ways, the paper is a missed opportunity, but there are at least signs of movement.

In the paper, the authors distinguish between several meanings of "homeopathy". Does it refer to the medicines? To consultations with a homeopath? To the principles of homeopathy? Or to the homeopathic therapeutic system as a whole? To be honest, I don't really care. But then the authors look at the definitions of 'efficacy' and 'effectiveness':

In clinical epidemiology ‘efficacy’ refers to ‘the extent to which a specific intervention, procedure, drug, regimen, or service produces a beneficial result under ideal conditions’. Evidence of efficacy is usually interpreted as requiring evidence from placebo-controlled randomised controlled trials (RCTs). Whereas ‘effectiveness’ is a ‘measure of the extent to which a specific intervention, procedure, drug, regimen or service, when deployed in the field in routine circumstances does what it is intended to do for a specific population’. Evidence of effectiveness requires pragmatic RCTs and well conducted observational studies of routine clinical practice.

The authors then argue that evidence of cost effectiveness, based on the clinical effectiveness of the intervention is required by NICE for interventions that are provided by the NHS, not evidence of efficacy. Then they go on to look at the evidence.

The authors divide the evidence into experimental evidence, by which they mean randomised, double-blind, placebo-controlled trials. This is where things go off the rails a bit. The authors suggest that looking at the 'active ingredient' of homeopathy as being the medicine given is far too reductionist; perhaps the active ingredient should be thought of as including "Six putative active ingredients... : patient's openness to the mind body connection, consultational empathy, in depth enquiry into bodily complaints, disclosure, the remedy matching process, homeopathic remedies". Not only that, but "Other authors have discussed the difficulties of separating out the effects of the homeopathic medicine from the consultation effects".

Now, this is interesting. It seems that what the authors are saying here is that much of the benefit that patients get from homeopathy is unrelated to the medicine they are given; it is related to the relationship between the homeopath and the patient, and to the consultation process. In medical science, there's a term for this: it's called the placebo effect. That's why you conduct placebo-controlled trials; because those effects exist, and you therefore need to separate them from the effect of the actual medicine that is given to the patients. That placebo-controlled trials are specifically designed to address this problem seems to have passed the authors by. Placebo-controlled trials of individualised homeopathy (i.e. the whole homeopathic package, including consulation) have been carried out, and show no better results than trials that did not include the consultation. But at least there is a recognition here that non-specific effects play a very large role in homeopathy.

The authors go on to look at what they call 'observational evidence', consisting of observational studies and case reports. The authors admit that this sort of evidence is "prone to many types of bias, it is regarded as weaker than experimental evidence, and is generally disregarded in systematic reviews of evidence". But they say "However, this type of evidence does acknowledge the homeopath i.e., focuses on treatment by a homeopath in contrast to the experimental evidence which focuses only on the homeopathic medicine".

This seems to be just the usual special pleading that 'my woo can't be tested by your cold-hearted science'. Then again, the authors do seem to be suggesting that the pills containing nothing are not as important as the context in which those pills are delivered. So where next? Unfortunately, the authors miss this opportunity by simply making some weak recommendations about terminology. But if the homeopathic community could admit that their drugs don't work, and sit down to figure out how they could strip out all the pseudo-science about energy medicine, water memory, quantum entanglement and so on, and how they could harness the power of the placebo effect, then what remained could conceivably be of some use.

Of course, that the homeopaths haven't managed to figure this out for themselves yet is a little surprising, given that this is what Ben Goldacre has been arguing for some time. It's also interesting to reproduce the last paragraph of the famous Shang et al. meta-analysis which was so damning of homeopathy:

"We emphasise that our study, and the trials we examined, exclusively addressed the narrow question of whether homoeopathic remedies have specific effects. Context effects can influence the effects of interventions, and the relationship between patient and carer might be an important pathway mediating such effects. Practitioners of homoeopathy can form powerful alliances with their patients, because patients and carers commonly share strong beliefs about the treatment’s effectiveness, and other cultural beliefs, which might be both empowering and restorative. For some people, therefore, homoeopathy could be another tool that complements conventional medicine, whereas others might see it as purposeful and antiscientific deception of patients, which has no place in modern health care. Clearly, rather than doing further placebo-controlled trials of homoeopathy, future research efforts should focus on the nature of context effects and on the place of homoeopathy in health-care systems."

Could the homeopathic and conventional medicine communities be moving towards common ground?

Some good news...

2008-07-18T14:40:00.003+01:00

According to this article in Pulse, the number of prescriptions written for homeopathic 'remedies' in the UK fell significantly between 2005 and 2007. 83,000 prescriptions were written in 2005, 62,700 in 2006 and 49,300 in 2007.

Naturally, if you're a homeopath, this is the result of a 'hate campaign' conducted against homeopathy by Prof Edzard Ernst, the Laing Professor of Complementary Medicine at the Peninsula Medical School (Universities of Exeter and Plymouth). At least, this is what one Dr Tim Robinson says in the Pulse article. This is fairly typical of the homeopathic community, who inevitably confuse a desire to rigorously assess the evidence base with a 'hate campaign'. Dr Robinson writes "Patients are not asking for [homeopathy] because of what has been written in the press..." This ignores woefully uncritical articles like this one that appear in the press, and columns for homeopaths in local papers that amount to free advertising, like this in the Manchester Evening News. If it was left to the press, people could be forgiven for thinking that there was decent evidence that homeopathy works.

Perhaps the message is getting out: that in an NHS with scarce funding, there's no room for funding things that don't work.

The homeopathic memory hole: not much progress since 1861...

2008-07-07T13:16:00.004+01:00

A recent conference on the current state of research in homeopathy once again exposes the field as an intellectual shambles.

The bloggers have already written a good deal about some of the presentations at the conference. AP Gaylard points out that homeopaths are still "saying the thing that is not" about a meta-analysis (Shang et al.) that was published in 2005. The Quackometer goes into great detail about why Lionel Milgrom is talking rubbish; something that is also ably demonstrated in several excellent posts by shpalman. And Gimpy explains why Dr Alex Tournier PhD's assessment of the evidence is fundamentally flawed.

What I find difficult to understand is how the homeopaths can justify their selective quoting of evidence. The repeated instances of saying the thing that is not about the Shang paper are only one example. A materials science paper by Rao et al. is constantly wheeled out as evidence that homeopathy could work, for all the world as if it hadn't been demolished by a critical letter to the editor in a subsequent issue of the journal (I should know, I was a co-author of the letter). This letter has gone down the homeopathic memory hole, never to be mentioned again, while the original (execrably bad) paper refuses to die. There is also an identifiable pattern to how homeopaths discuss meta-analyses. Apparently positive ones that take no account of trial quality are always mentioned, but crucial caveats are always omitted (for example, Linde et al. 1997 said they could find no evidence that homeopathy worked for any particular condition: this is never mentioned). A later paper by Linde et al. looked at trial quality, and concluded that the results weakened the positive conclusions of their earlier paper: this is never mentioned. The Shang et al. paper is only mentioned to make erroneous criticisms of its methodology, to incorrectly state that critical information was missing from the paper, or to accuse the authors of scientific misconduct based on a total misunderstanding of the paper.

A bit of googling the other day turned up this. Back in 1861, a former editor of the North American Journal of Homeopathy, one Dr Peters, publicly renounced homeopathy. I recommend having a quick read of it. The points made are just as valid today as they were in 1861. And yet since then, homeopathy has persisted in heading into an intellectual cul-de-sac. Perhaps this is because homeopaths only retain the bits of evidence that superficially seem to be in favour of homeopathy: everthing else goes down the memory hole, even when it is published in homeopathic journals.

For some time, I've been considering putting together a summary of the evidence on homeopathy, and sending it off to Homeopathy (the in-house comic of the Faculty of Homeopathy) for publication. It seems clear that there's no point, even if the journal would print it. Firstly, homeopaths will pretend it never existed. And secondly, there are plenty of decent books on the subject for those who are genuinely curious about the evidence. The homeopaths will continue to be deluded.

Blogging is a transferable skill...

2008-06-25T16:23:00.002+01:00

I've just returned from Aberdeen, where I was being interviewed for a job at the university (fingers crossed). In my limited experience of these things, it's always a bit nerve-wracking, and you can never quite predict what questions you'll be asked (although there are some that always come up: where do you see yourself in five years?). One of the five-member panel asked me if, apart from the research-related stuff on my CV, there were any other skills or things that I would like the panel to know about. I was unprepared for that, and I figured that saying 'no' would sound like 'no, I have nothing of interest to say about myself whatsoever', and couldn't be a good thing. Although I hadn't planned to, I talked a little bit about this blog, and explained how I used it to try and explain scientific issues in an accessible way. To my surprise, it went down very well with the panel, and I was asked how my blogging experience might help in university teaching. Given my relative lack of teaching experience, having the opportunity to talk about that can't have done my chances any harm.

So, if you're a blogger doing interviews, it's probably worth bringing up your blog. After all, you use it to distill complex issues down to their essentials, don't you? That's definitely a transferable skill.

More intention experiment: the ghost of Benveniste

2008-06-25T14:50:00.005+01:00

I should probably avoid the Intention Experiment website, but it's such a fabulous compendium of nonsense and loony ideas that it's hard to leave alone. Now they're channeling the ghost of Jaques Benveniste. As many will know, Benveniste caused a storm of controversy when his lab published a paper in Nature that appeared to show that homeopathic concentrations of a certain type of antibody could have a biological effect, even though the chances of the solution containing any actual molecules of the antibody were tiny. This is the basis of the idea of 'memory of water' in homeopathy. The Nature paper was published with an unprecedented "editorial reservation", and a team assembled by Nature visited the Benveniste lab to look into the results. The results of the investigation were damning, showing that the design of the experiments was poor, and an article outlining the problems was published in a later issue of Nature. Ever since, homeopaths and other brands of quacks have been convinced that Benveniste's results were suppressed by the scientific establishment on behalf of Big Pharma, and so on.

Benveniste, however, was not put off by this setback, and continued in his work, going on to suggest that not only did the 'memory of water' effect exist, but that it could be transmitted digitally, down phone lines or over the internet. Brilliantly, this only gave 'positive' results when the equipment was being run by a particular researcher. Benveniste doesn't seem to have reflected too hard on why that might have been the case. Benveniste called this "digital biology", which would have "immense consequences on medical diagnostic procedures and the agro-food industry, with huge technological and commercial impact", and was only being held back because scientists are "opposed to the evolution of science". You recognise the narrative here, don't you?

Why do I mention all this? Because the Intention Experiment blog carries news of an experiment into "healing by e-mail". Apparently, a "Francesca McCarney, Ph.D., teacher of professional intuitives at the Academy of Intuitive Studies and Intuition Medicine" conducted the experiment, where 88 people were each sent 2 e-mails. One of these e-mails had "healing energy" "encapsulated" into it, and the other one did not. Apparently, the e-mails were indentified correctly 31.9% of the time, against a 25% probablity of getting them right by chance. This seems like a deeply unimpressive result, with no confidence interval to give us an idea of how likely it would be for such a result to occur by chance, but we're told that "scientists would consider it highly significant result" [sic]. We also don't know whether there might inadvertently have been clues in the text of the e-mails.

It seems that loony ideas never die; they just re-appear periodically in a slightly different form.

Another study for homeopathy awareness week...

2008-06-20T15:06:00.003+01:00

It's homeopathy awareness week (until tomorrow), with a focus on homeopathic treatment for allergies.

In the spirit of awareness, here's a systematic review of complementary therapies in the treatment of allergic rhinitis and asthma in the Journal of Allergy and Clinical Immunology, with a summary at the NHS National Library for Health here.

For homeopathy, the authors found that "Some positive results were described with homeopathy in good-quality trials in rhinitis, but a number of negative studies were also found. Therefore it is not possible to provide evidence-based recommendations for homeopathy in the treatment of allergic rhinitis, and further trials are needed".

Fun with trials, for homeopathy awareness week...

2008-06-20T10:36:00.007+01:00

Occasionally, people sympathetic to homeopathy try to discuss scientific studies. It's usually fun when this happens, because I get chance to do some paper deconstruction. What can I say, I'm a geek. I usually learn something from doing it, even if only that I'm talking rubbish.

In the comments to this post, a lady (I assume) named Vicky posted three studies claimed to support homeopathy. Of these, the Graunke et al. paper on tadpoles has some glaring problems which are discussed here, and the Oberbaum et al. paper simply contains too few subjects (15 in each group) to come to any reliable conclusions. The third paper, by Bernstein et al., was a bit of an oddity, and I'd like to take a slightly closer look at it, in honour of homeopathy awareness week.

Firstly, the paper is published in the American Journal of Therapeutics. This journal does seem to be peer-reviewed, but is not listed in the ISI Journal Citation Reports, which suggests that it is not a top-of-the-line journal. But that in itself doesn't necessarily mean the findings are wrong.

The paper studies the effect of a proprietary homeopathic remedy, Relieva, on psoriasis. First red flag: psoriasis is a condition where patients experience flare-ups and remissions. So any improvement could be a result of this natural cyclicity rather than any treatment effect.

Relieva is an unusually concentrated substance in homeopathic terms. The study describes the active ingredient as "a highly concentrated, proprietary extract of M. aquifolium 10% known as Psorberine". This remedy actually contains some active ingredient, unlike many homeopathic remedies, so there is at least a chance of it working without needing to overturn practically everything we know about pharmacology.

The study is described as a randomised, double-blind, placebo-controlled study, and in general it seems to be well conducted. There were a total of 200 subjects, 100 in each group, suggesting the trial is large enough to get decent results. The authors conducted an 'intention to treat' analysis. In this analysis, all subjects enrolled in the trial are considered, rather than just those who completed it. The authors 'impute' a worst-case score (12 points) for Psoriasis Area Severity Index (PASI), a measure of psoriasis severity, to all the patients who dropped out of the trial. I'm not sure why they did this: they might have a sound reason for doing that, but I don't see it explained in the paper. Notably, 3 subjects dropped out of the treatment group, compared to 26 in the placebo group. This is a large difference, and it would be expected to introduce a bias into the results of the intention to treat analysis. Also, since a whopping 26% of the placebo arm dropped out of the trial, the results are at least of questionable reliability. The authors suggest that the high drop-out rate in the placebo group compared to the treatment group might be an indication of the efficacy of the treatment. However, their analysis of the subjects that completed the trial suggests that the placebo group experienced some improvement in their psoriasis (a PASI point reduction of 2.22 compared to 3.58 in the treatment group), so I'm not sure that explanation stacks up. The maximum increase was +3 for the treatment group and +4 for the placebo group. It's also worth noting that this big difference in drop-out rates has the potential to break the blinding of the trial.

The acknowledgments of the paper are very odd, at least to me. They say "The authors thank Global Clinicals, Inc (www.GlobalClinicals.com ) and its staff for performing this study, Missy Magill for the statistical evaluation, and Andrea Gwosdow (PhD) for preparing this manuscript". It's possible I'm being naive here, but this raises the question of what the authors actually did to earn the right to be credited as authors.

Who then are Global Clinicals, Inc? Here's their webpage. They undertake clinical research on behalf of their clients, so there is at least the appearance of a conflict of interest here; clients are not going to be happy if their treatment is shown to be no good. This is the sort of the thing that Big Pharma are (rightly) criticised for.

So overall, I would suggest that the evidence in the trial does not unequivocally show that Relieva works. There are also some shenanigans surrounding the authorship of the paper that would be highly criticised by homeopaths if they occurred in a trial of conventional medicine. Finally, even if you accept that the trial provides evidence in favour of Relieva, it provides no general evidence in favour of the basic tenets of homeopathy ('like cures like' and 'potentisation'). For that, you would need to show that Relieva causes psoriasis in healthy volunteers in a 'proving', and show that its effects increased with increasing dilution. If it does work, it's not because it's homeopathic.

Homeopathy works! At least in Mailworld...

2008-06-13T10:20:00.008+01:00

There's a terrible, terrible article on homeopathy in the Daily Mail online (cheers to Dr. T on the Bad Science forum for spotting it). The title is "Homeopathy works!", and it really is just about the least critical article I can imagine.

For a start, the fact that homeopathic remedies are generally so diluted that they contain no active ingredient is glossed over with the phrase "Critics argue that the active substance is so diluted that homeopathic remedies have no more effect than placebo or dummy treatment".

The article reports on a study by Dr. David Reilly at, God help us, the Glasgow Homoeopathic Hospital. What do we know about the study? Virtually nothing. According to the article, the study looked at 50 patients suffering from nasal allergies. Some were given homeopathy and some a placebo. I tried to find the study, so I could understand what they did. A search on the pubmed database doesn't find it. There is no mention of the status of the study in the Mail article. Has it been submitted to a journal? Presented at a conference? Written up on the back of a fag packet? We don't know. So there is no way the claims made in the article can be evaluated in any sensible way. Was the study blinded? Did it have adequate randomisation? What was the placebo used? We don't know. The only thing I can say about it is that the study size is small, and so more likely to produce a false positive result.

So, the reporting of the particular study that the article discusses is lamentable. But perhaps even worse is the total failure to place the study in the context of the rest of the scientific evidence. Dr Reilly is allowed to get away with saying that the study replicates some of his previous studies, and that "there were positive findings in 70 per cent of a further 180 clinical trials" of homeopathy.

Perhaps 70% of trials of homeopathy did apparently show a positive result. But this could be a result of poor study designs or insufficient sample sizes. Luckily, this can be tested using a technique called a meta-analysis. Essentially, you get the data from all the published trials, pool it together, and apply statistical techniques to the pooled data to evaluate bias. Such an analysis was conducted by Aijin Shang and co-workers, and published in the Lancet in 2005. This is a study much misunderstood by homeopaths, but what it shows is that large and well-designed trials show no effect for homeopathy, and that positive results of some trials can be explained by poor experimental design, bias and small sample sizes. A previous meta-analysis by Linde et al., published in the Journal of Clinical Epidemiology, came to similar conclusions. Dr. Reillys' previous work on homeopathic treatment for allergies was included in the Shang study, and was not deemed to be of high quality.

So, the current state of the scientific evidence is that there is no plausible mechanism for homeopathy to work, and that the results of large, well-conducted studies are consistent with homeopathic remedies being no more than placeboes. Going by the Mail article, you would get the impression that homeopathy was dismissed by blinkered scientists, but that evidence was gradually stacking up in favour of it. That is simply not the case.

But at least the Mail didn't claim that homeopathy can cure cancer...

EDIT: It has come to my attention that the Mail story is actually reporting the results of a trial that was published in the BMJ...in 2000. So why it has appeared 8 years later is anyone's guess. This trial was also included in the Shang meta-analysis, and not deemed to be of high quality. This is perhaps because the trial was not double-blind, even though it was described as such in the BMJ [Edit: my mistake. The wash-in period of the trial was single blind. The intervention was double-blind]. The authors also calculated that they needed to recruit 60 subjects in each group (a total of 120) for the study to have the required statistical power. There was a total of 50 subjects. So the only sensible conclusion would be that there was insufficient evidence either way.

Edit 2: Here's the author of the piece, Jenny Hope, giving evidence to the House of Commons Select Committee on Health regarding the influence of drug companies on research and research reporting. Note this bit:

When a story is in the public domain—or being placed with the specific intention of generating publicity—someone will write about it. If it's me than I know where the information is coming from, the background, and carried out the interviews—there is some quality control.

The quality control on this particular story seems to be negligible. Perhaps Ms. Hope was having a bad day?

Watch your units!

2008-06-06T14:49:00.003+01:00

When I was doing GCSE science and A level chemistry, I was always taught to be careful about units of measurement, especially when I was doing calculations. This is a lesson that would not be lost on the odd science journalist. Today's Guardian has an article on a photovoltaic solar farm in Portugal. According to the article, the farm is "expected to supply 45MW of electricity each year, enough to power 30,000 homes”.

A megawatt (MW) is a unit of power, equivalent to 1,000,000 watts. 1 watt is equivalent to one joule per second. So it doesn't make sense to say that the farm will supply 45MW "each year".

Pedantic, I know, but you'd expect a national newspaper to get that sort of thing right.

Lorenzo's Oil: not quite a miracle cure

2008-06-05T11:05:00.005+01:00

I've seen a couple of stories in the press about the death of Lorenzo Odone, who had adrenoleukodystrophy (ALD). ALD occurs when an essential protein that removes very long chain fatty acids (VLCFAs) is missing, resulting in a build-up of VLCFAs in the body which damage the brain. Most sufferers die within 1 to 10 years of showing symptoms.

Lorenzo Odone lived until the age of 30, despite doctors giving him little chance of surving beyond the age of 8 or so. His relative longevity has been ascribed to 'Lorenzo's Oil', a mixture of glycerol trioleate and glycerol trierucate. The oil was initially used as a treatment by Lorenzo Odone's parents, who were understandably unwilling to simply watch their son die. They undertook a considerable amount of research, despite being laypeople with no medical expertise. They thought that the oil would help inhibit the creation of VLCFAs. Lorenzo's condition apparently stabilised after the oil became part of his diet. Since two laypeople creating a therapy that had eluded the finest minds of medicine is quite a story, a film was made, titled 'Lorenzo's Oil'. The money the Odone's made from the film went into the Myelin project, which funds research into diseases like ALD and MS.

However, the Odones found the scientific community sceptical. In the Guardian obituary, Lorenzo's sister Cristina Odone writes "Not all members of the scientific community welcomed this approach; or, indeed, their portrayal on film as hard-hearted and arrogant. A backlash against the Odones saw various researchers from around the world denying the efficacy of Lorenzo's oil, and most doctors refused to prescribe it. In 2005, however, the world's top authority on ALD, the late Dr Hugo Moser, published the findings of a 10-year study which showed that a statistically lower incidence of ALD occurred in those boys whose diet included Lorenzo's oil". This is what interested me in the story, as it fits into a neat narrative about laypeople taking on the might of the scientific establishment, and being treated in a cold-hearted and callous manner as a result.

Firstly, I would say that an initial sceptical approach was sensible. The only evidence that Lorenzo's Oil might work was from the apparent stabilisation in the condition of Lorenzo himself. There could be a huge number of other reasons why this happened that had nothing to do with Lorenzo's Oil. The only way to find out whether it really worked or not was to conduct scientific trials. Until this had been done, it was sensible to be sceptical of a therapy for which there was essentially no evidence of efficacy.

So, what of Moser's study? This is certainly not my area of expertise, but I can at least have a look at the methodology of the paper. The trial followed 89 boys who were thought to be at risk of ALD based on an assay of VLCFAs in their plasma. All of the boys were given Lorenzo's Oil and 'moderate fat restriction' in their diet. The study concluded that treatment with Lorenzo's Oil caused a reduction in plasma hexacosanoic acid, which was associated with a reduced risk of developing abnormalities on an MRI scan. However, this was a single arm trial: there was no placebo group. I'm not criticising the study for this; if the treatment is likely to help, then it is probably unethical to withhold it from the subjects of the study. But it does make it more difficult to know whether the results are because of the therapy, or because of something else. What would be more interesting, I think, would be to look at whether there is a difference between subjects given Lorenzo's Oil and a reduced VLCFA diet, and those just given the diet. As far as I can tell, this has not been done. So it is difficult to know if the oil has any effect on its own. A more recent study in the journal Metabolic Brain Disease followed 11 subjects, all of whom were taking Lorenzo's Oil and were on a diet restricted in VLCFAs. Again, this doesn't allow us to separate the effects of the oil from the effects of the diet.

While these studies might show some evidence for an effect of Lorenzo's Oil before symptoms appear, it seems that the treatment doesn't work well in symptomatic subjects. This may be because the compounds that make up Lorenzo's Oil do not cross the blood/brain barrier.

Now, I'm not saying that Lorenzo's Oil doesn't work. This isn't homeopathy, after all, and there does seem to be a plausible mechanism by which it might work. I'm not even saying that what the Odones have acheived is not impressive. What I am saying is that the evidence that it does work is fairly weak, it is far from being a miracle cure, and the story is a little bit more complicated than the narrative that is being presented.

A lesson in how to get a false positive result

2008-05-29T11:46:00.003+01:00

Back to Rustum Roy and his ongoing attempt to show that water can have a 'healing' structure (whatever that means) imposed on it by the power of human intention. As Le Canard Noir said, in the comments to a previous post, this is car crash science. Although I know I should look away, I can't quite manage to do it.

Previously, Roy and his team tried to show that the Raman spectrum of a beaker of water could be changed merely by people all over the world thinking about it changing. Despite the horrible design of the experiment, the results were inconclusive. What Roy needs is a way of increasing the chances of getting a positive result. Another experiment is now planned, details of which are beginning to emerge at the Intention Experiment blog:

As you remember, our experiment isn’t conclusive – largely because we’re only looking at one parameter to see if it has changes.

This is a little like looking at an elephant from one side. If you look from the front, you will mainly see a trunk. Look from bottom, and you only see a giant mass hovering over you like a dark grey cloud.

Rainen’s new equipment consists of three separate devices that examine, respectively, the light scattering, the thermal expansion and any infrared changes in a sample of water. Once these measurements are taken, they are sent into a computer, and from this handful of data points, the computer can determine some 1000 parameters of the sample.

“This equipment represents a revolution in characterizing water,” says Roy.

From this, it sounds as though Roy and team are going to be comparing 1,000 variables. This raises the issue of multiple comparisons. If you compare 1,000 variables between two populations, using a hypothesis test at the 5% level, you would expect to get 'positive' results for 50 of the variables, even if there was no real difference between the two populations. Statisticians apply corrections to account for this effect. Since Roy's team have previously published a paper that purported to show differences between graphs without applying any statistical analysis at all, it's not certain that this will be done. Another issue is that these 1,000 parameters are derived from 'a handful' of measurements, so they presumably cannot be independent parameters. It seems that false positive results are much more likely from the new experiment. Result!

Science causes earthquakes...

2008-05-28T14:35:00.003+01:00

Further evidence that the Independent will print any old rubbish. Richard Lapthorne, in a letter to the editor, writes:

"Given acceptance of the geologists' explanation of earthquakes as readjustments in the earth's crust, perhaps Bruce Homes (letters, 22 May) would see the sense of attributing them to acts of science rather than acts of God."

Scientific observation has revealed that "readjustments in the earth's crust", along fault zones, cause earthquakes. Ergo, science causes earthquakes. You just can't argue with that kind of logic. But why print these ravings in a national newspaper? I know the Independent is rapidly becoming a joke, but why accelerate the process?

42 days: get writing, folks

2008-05-27T13:09:00.002+01:00

The report stage of the counter-terrorism bill is coming up. This bill would allow pre-charge detention of terrorist suspects for up to 42 days. The current limit is 28 days, already significantly higher than most comparable democracies. No good case has been made for this erosion of civil liberties, and it's time to get writing to your MP. The following went off to my MP today:

I am writing to express my concerns at the proposal in the counter-terrorism bill to extend pre-charge detention to 42 days in certain circumstances. This is a considerable extension to current powers, and for that reason a clear case needs to be made for the necessity of the measure. In my opinion, this has not been done. I note that the Director of Public Prosecutions, Sir Ken McDonald, and Lord Goldsmith, the former Attorney General, have stated that they see no need for an extension to pre-charge detention. I also note that Lord Dear, the former chief inspector of constabulary, has stated that most chief constables he has spoken to do not think a 42-day detention period is necessary. This raises the question of who exactly does think this provision is necessary.

As I understand them, the proposals would provide for the Home Secretary to make a statement to parliament, and for parliament to vote on an extension to pre-charge detention. However, it is difficult to see how the Home Secretary can come before parliament to discuss individual cases because of the risk of prejudicing them.

The proposals seem to be poorly thought out, unnecessary and unworkable, and I hope you will carefully consider the implications of the proposed legislation for civil liberties before the report stage of the bill.

Sincerely,

Paul Wilson

Nice to see...

2008-05-27T11:16:00.002+01:00

I was happy to see this article in the Guardian on Saturday. Edzard Ernst holds the Laing chair in complementary medicine at the University of Exeter, and has published a book with Simon Singh on CAM, entitled "Trick or Treatment? Alternative Medicine on Trial". At the Hay literary festival, Ernst was talking about the book. He laid into pharmacists in general, and Boots in particular, for selling homeopathic remedies. There is no good evidence that such remedies work, and they have no plausible mode of action, as they are generally so diluted that the chance of them containing any of the purported active ingredient is vanishingly small.

Ernst is gratifyingly direct, saying that Boots "seems to be fast becoming the biggest seller of quack remedies in UK high streets". He also said that pharmacists have a responsibility to adhere to the code of ethics of the Royal Pharmaceutical Society: "This can only mean that pharmacists should tell their customers that a homeopathic remedy they are about to buy doesn't contain a single molecule of whatever it says on the label, and that there's no clinical evidence that it works beyond a placebo effect".

I couldn't agree more, and it's good to see someone come out and say these things clearly and publicly.

The intention experiment cargo cult: results are in!

2008-05-22T17:29:00.004+01:00

You remember Rustum Roy and his attempts to measure changes in water structure caused by people all over the world really, really wanting the structure to change? Yes, the results are in! There's good news and bad news. First the good news: apparently "something happened", but the bad news is that "we're not sure what".

The intention experiment failed to change the structure of water. I'm quite surprised, actually. My prediction that they would find some changes in the water because it was such a poorly designed experiment has not come true. The sample "had enormous variation in light scattering to begin with", but there were no profound changes recorded. I would suggest that the variation is probably instrumental noise, but no actual results have been shown, so it would be impossible to tell even if I knew much about Raman spectroscopy.

Why did the best intentions of so many people fail? You might say "because trying to change water structure through the power of the mind doesn't work". And, to be fair, the intention experiment people have this eighth on their list of possibilities. Because "Although all of our other studies have produced a measurable effect, this hypothesis must always be considered". But not considered that seriously, you understand.

What could the other reasons be? The wildest one is the suggestion that "intention works out of time" (which I suppose is no dafter than imagining that intention works in the first place). Of course, here any recorded change in the water whatsoever, at any time, could be attributed to 'intention', so this isn't what you would call testable. Other suggestions include water purity (even the purest water is grossly contaminated, and having it sat on a lab bench with a probe in it is not going to help), environmental factors (a thunderstorm during the measurement), and problems with the experimental protocol. So the experiment will be repeated in a couple of months.

Some of the comments are fun. One commenter writes "This is what experimentation is all about. You just keep trying until you can verify your conclusions...". Another writes "I'm convinced that intention works. I don't see the need for further experiments". It seems that any negative result would have to be wrong, because we all know that it works. This is how homeopaths think, and is pretty much exactly what is meant by cargo-cult science.

It's worth remembering, of course, that all of this is based on Rustum Roy's evidence-free assertions that there is such a thing as 'healing water', that 'healing water' has a different structure from ordinary water, and that you can change the structure of water by thinking about it. Obviously it would be nice to really, really want a beaker of water to cure cancer, and have it work, but I think it might be worth persevering with proper biomedical research for now.

Andrew Lansley and Nadine Dorries, perhaps unsurprisingly, don't read this blog...

2008-05-19T10:39:00.004+01:00

During the second reading debate of the Human Fertilisation and Embryology bill, there was an interesting exchange between Andrew Lansley (the shadow health secretary) and Nadine Dorries MP, on the recent research into survival rates of extremely premature infants.

Lansley talks about the Trent study, that showed no increase in survival rates for infants born before 24 weeks:

The authors also report that no babies born at 22 weeks survived. Like other studies, they suggest that this might represent the limit of viability. I cannot say whether they are right, but I note that, of 150 babies born alive at 22 weeks over the whole period of the study in Trent, only 24 were admitted to intensive care. In the latest period, 2000-05, only nine were admitted to intensive care. From my point of view—I am not a scientist, but I can read statistics—in order to draw any conclusion from such a small number of cases, one would need to assess why those babies were being born so prematurely. Were they failing to thrive? Were they twins or other multiple births? Were they suffering from a genetic defect?
Abortions at 22 weeks would be of a foetus that was otherwise healthy, unless they were carried out on the specific ground of a prospective abnormality or handicap of the child concerned. So while such a baby would require intensive care, and doubtless very large numbers might not survive, I do not regard that study as providing conclusive evidence that a baby cannot be sustained at 22 weeks.

I would personally be loth to move from the principle of linking the time limit for abortion to the viability of the foetus. That is where it was established in 1967; it has moved over time. It is arguable, but I would argue personally that the evidence would support a further reduction—in my view, to 22 weeks—in order to ensure a prospective legal framework that could accommodate improving medical science.

To which Dorries has this to say:

I am sure that my hon. Friend is aware of other studies—for example, in University College hospital, London, and in other countries—showing that if poorly babies who are born prematurely receive immediate good neonatal care from a specialist dedicated team of staff, the outcomes are very much better. University College hospital figures are much better than those emerging from the Trent study. One conclusion from the Trent study is that anyone going into premature labour should go to University College hospital and not to the Trent region.

This is exactly what you can't conclude from a comparison of the Trent and University College studies. It's difficult to compare the studies, because the Trent study includes all infants 'alive at the onset of labour', while the UCLH study included only infants that were born alive. Another issue with the UCLH study is that the numbers of infants admitted to the neonatal intensive care unit varies wildly between the different time periods of the study. This may be because the number of subjects is so small, leading to a large amount of statistical noise. Or it may be because those time periods are not directly comparable, perhaps due to differing policies on which infants should go to the neonatal intensive care unit through time.

And Lansley replies:

My hon. Friend may have gathered from what I said that I am effectively arguing that in circumstances where neonatal intensive care was provided to all such babies born at 22 weeks and where the foetus was otherwise healthy—or rather the baby was healthy at that stage—the prospects for survival at 22 weeks may well be far greater than suggested by the negative view expressed in the study from Nottingham and Leicester. It is not for me to say any more than that, but I think that it provides a basis for reducing the limit—albeit, in my view, by just two weeks.

In fact, the Trent study (Nottingham and Leicester) is more likely to give an accurate idea of the prospects for survival, because it is a multi-centre study (including 16 hospitals in the Trent health region) while the University College study concentrates on a single hospital. So the Trent study is less likely to be confounded by selection bias, and includes a much larger number of subjects, leading to statistically more robust results. It's difficult to see how Lansley can come to the conclusion that the study 'provides a basis for reducing the limit', given that it shows there has been no change in survival rates before 24 weeks since the early 1990s. Happily, the lead author of the Trent study, Prof. David Field of Leicester University, has pointed this out.

There may be other reasons for seeking a reduction in the limit for abortions, but the Trent study is not one of them. To suggest otherwise is simply to misrepresent the research.

Nadine Dorries makes an attempt to look at the evidence. She fails. [updated]

2008-05-15T15:30:00.006+01:00

Nadine Dorries seems to have realised that simply dismissing as 'desperate tosh' a study in the British Medical Journal won't cut the mustard. The study looked at survival rates of premature babies in the Trent region of the UK, concluding that survival rates for births before 24 weeks had not increased since 1994-9. This undermines the argument that those survival rates are increasing, and the time limit for abortion should be brought down to 20 weeks from the current 24 weeks. Dorries has now at least attempted to address the evidence, on the 20 weeks campaign website. This is a marked improvement, but I think her effort still falls somewhat short.

First, some nonsense. Dorries writes:

"We were somewhat surprised to see that the article was not in this week's paper edition of the BMJ which many doctors did not receive until Saturday. Why then rush to get it onto the web and into the public domain? The BMJ editorial said that upper time limits were to be debated in the House of Commons 'this year'. What it didn't say is that they are actually to be debated and voted on in just ten days, on 20 May. So the rush to get this into the public domain was to influence that debate".

If you can't use the latest research to 'influence debate', then what is it for? Apart from that, it is perfectly normal for research articles to appear on journal websites some time before they appear in print. That is precisely so that articles can reach the public domain once they are ready for publication, without having to wait on production schedules.

What of the science, indeed. Dorries says that "the Trent study looks at results from 16 hospitals and has been running for years. It is not new and other studies have been published based on more recent data". She also claims that the results from Trent "have always been poor and well below those seen in top neonatal centres worldwide". As evidence for this, she cites a study from the University College London Hospital (UCLH). This study is a single centre cohort study comparing survival rates in the years 1981-85, 1986-90, 1991-95, and 1996-2000. [The first thing to note here is that the Trent study uses data up to 2005. So it is actually using much more recent data than the UCLH study.] The UCLH study found that "There was a progressive increase in survival at all gestational ages over the 20-year period".

This seems to be a classic case of how you have to be careful with research. The first thing to note is that the number of babies in the UCLH study is small. The total number of babies in the study was 12 for those born at 22 weeks gestation, and 56 for those born at 23 weeks gestation, over 20 years. This compares with 261 born at 22 weeks and 370 at 23 weeks, over 10 years, for the Trent study. The Trent study is likely to give more robust numbers. Dorries, on the other hand, writes that the UCLH study "showed no survivors at 22 or 23 weeks in 1981-85 but 71% (5/7) and 47% (8/17) respectively in 1996-2000". Note the tiny sample size this statement is based on. In the UCLH study, the authors don't give percentage survival rates for births at 22 weeks, because the sample sizes are simply too small to draw any conclusions. There is something odd about the admission figures here too. For births at 22 weeks gestation, there was 1 admission in 1981-85, 3 in 1986-90, 1 in 1991-95, and 7 in 1996-2000. The comparable figures for births at 23 weeks are 7, 23, 9 and 17. There are very large variations here that suggest that the different time periods may not be directly comparable.

Perhaps the most serious problem with comparing the Trent and UCLH studies, though, is that they don't appear to be measuring the same things. In the Trent study, the authors write "We included in the study infants alive at the onset of labour". In contrast, the UCLH authors write "All infants born between 22 and 25⁺⁶ weeks of gestation between 1981 and 2000, who were admitted to the tertiary neonatal intensive care unit of UCLH (UK), within 1 week of birth, were enrolled into our study. In addition, the UCLH labour records for the years 1991–2000 were scrutinised to identify all infants who were born alive between 22 and 25⁺⁶ weeks of gestation, but who died in the delivery room". Hence the Trent study includes still-births as deaths, but the UCLH study does not. According to the Trent study, of births at 22 weeks, 43% were stillborn in 1994-99, and 40% in 2000-5. Of births at 23 weeks, 28% were stillborn in 1994-99, and 20% in 2000-5. For the UCLH study we don't know what these rates are. The Ministry of Truth blog has noted an anomaly in that the UCLH study reports a massively lower proportion of deaths in the delivery room compared to the Trent study. So there is at least a possibility that the UCLH survival rates are inflated in comparison to the Trent study by selection bias: perhaps more infants are being admitted to neonatal intensive care in Trent than in UCLH.

It's also worth noting that the Trent study covered a whole area (the Trent health region in the UK), while the UCLH study covered a single hospital. As the authors of the UCLH study write, "Large population-based studies [such as the Trent study] have the advantage of avoiding problems of selection bias and enable the accumulation of large numbers to reduce statistical errors. However, they inevitably represent the outcome from a very heterogeneous range of perinatal facilities...It is therefore inevitable that these units encompassed a wide range of staffing, resources and expertise in the care of extremely preterm infants. In addition, it seems likely that the maternity units encompassed a range of management policies (both obstetric and paediatric) towards infants born at the limits of viability, ranging from ‘proactive’ to ‘noninterventionist’. Hence, it is not surprising that average survival figures obtained from large population based studies will be different from those observed from single centre cohorts". As the editorial in the BMJ that accompanies the Trent studies says, "Single centre studies areconfounded by selection bias and tend to overestimate the likelihoodof survival". In other words, to compare the Trent and UCLH studies in the way Dorries is trying to do, and draw the conclusion that survival rates in the Trent health region are 'poor', is not a sensible thing to do.

So Dorries has looked at the evidence, which is better than dismissing it as 'tosh'. However, she seems to be drawing erroneous conclusions from an inappropriate comparison of two different studies. By coincidence, this inappropriate comparison supports Dorrie's agenda. How convenient.

Yes, but what about the evidence? Nadine Dorries and the BMJ

2008-05-09T11:03:00.007+01:00

This morning my Grauniad was carrying a story about new research, published in the BMJ, that suggests survival rates for premature babies born before 24 weeks are very low and have not improved, despite advances in medical care.

Now, I haven't actually read the whole study, but I've glanced at the abstract. It's a cohort study that looks at births of babies between 20 and 26 weeks in the Trent health region in the UK over two time periods: 1994-9 and 2000-5. The study found that although there was a significant increase in the number of surviving babies in 2000-5, this was attributable to an increase in survival rates for babies born at 24 and 25 weeks. There was no significant increase in survival rates for babies born before 24 weeks.

This is an important finding because the House of Commons will soon debate whether to lower the lower the legal cut-off point for most abortions from 24 weeks to 20 weeks. One of the arguments has been that advances in medical science mean that more and more babies born before 24 weeks are surviving. The research shows that this appears not to be the case.

What to do if you're on the side of the debate that wants a reduction to 20 weeks? If you're Nadine Dorries MP, you go in for some textbook fallacious reasoning, relying in particular on ad hominem and argument from incredulity.

No improvement in neo-natal care in twelve years? Really? So where has all the money that has been pumped into neo-natal services gone then? A baby born at 23 weeks today stands no better a chance of living than it did in 1996? This report is the most desperate piece of tosh produced by the pro-choice lobby and it smells of one thing, desperation.

Well, the evidence seems to suggest that a baby born at 23 weeks today really does stand no better chance of living than it did in 1996. It's no good just saying you don't believe it; you need to explain why that interpretation of the evidence is incorrect. Dorries also describes the British Medical Journal as the "trade magazine" of the British Medical Association. Because the BMA funds a pro-choice group, research published in the BMJ can be dismissed as emanating from the "pro-abortion lobby". In fact, the BMJ is a peer-reviewed medical journal, publishing "original scientific studies, review and educational articles, and papers commenting on the clinical, scientific, social, political, and economic factors affecting health." It is something more than a trade magazine, and for Dorries to describe it as such is disingenuous to say the least.

As I say, I have not read the whole paper. For all I know, there may be some problems with it. But Dorries is simply not contributing to a useful debate by dismissing the study out of hand. The paper gives us the evidence, and it is up to us, as a society, to decide what to do about it. We can't do that by falling back on fallacious reasoning that supports our pre-conceptions.

Update: Dorries continues to mis-represent the BMJ paper, writing that "For the BMA to argue that the survival rates of neo nates born 12 years ago should endorse the 24 week limit today, is, as I said yesterday, an act of desperation, but one that suits me and the media opportunities such a biased piece of 'research' provides." First of all, it is not the BMA that is doing the arguing. The research was conducted by a team working in the Department of Health Sciences at the University of Leicester. Second, the paper compares data from 1994-9 to data from 2000-5. Where Dorries gets '12 years ago' from is anyone's guess. Again, it is no good simply stating that the work is 'biased'. You need to explain why it is biased. I'm fairly sure that Dorries can't do this.

Creationism, language and structural geology

2008-05-02T12:59:00.004+01:00

I wrote a little bit about this elsewhere, but I thought it was of interest to a 'bad science' audience. Dr. David Peacock, a consultant with Fugro-Robertson and well-known structural geologist, has just published a letter to the editor [extremely irritating paywall] in the May issue of the Journal of Structural Geology. The letter criticises the use of the term 'architecture' in published articles in the Journal of Structural Geology. In general, structural geologists use the term to describe the spatial arrangement of structural elements within larger structures, such as fault zones or geological basins.

So far, this is just an argument about arcane terminology in a fairly obscure field of scientific endeavour. However, one of Dr. Peacock's arguments against using 'architecture' is that it introduces a 'hint of creationism'. "Who is the architect?", he asks, going on to ask "Does use of the term imply divine construction or a belief in intelligent design?"

The answer to this latter question is clearly no. Geology has always borrowed terms from architecture (for a few examples, see here). This happens because some geological features are similar to features from the built environment (for example, here's a gratuitous picture of some nice folds). That doesn't imply that someone or something intelligent built them. More importantly, it seems to me that we would have to stop talking about structural geology entirely if we accepted Dr. Peacock's contention. The fact that the Earth has structure is what structural geologists study. So if you ask 'Who is the architect?', you also ask 'Who imparted the structure?' I make this point in a reply [paywall] to Dr. Peacock's letter, which at the time of writing is in press. The text can also be found here.

The wider point here is that the idea that any implication of 'structure' is an implication of 'design' is a classic creationist mis-direction. This is a point made eloquently in the context of evolution by Steve Novella. In structural geology, 'structure' and 'order' emerge from naturally occurring processes. There is no reason to think that saying a fault zone has an 'architecture' is also saying that some intelligence designed the fault zone.

Rustum Roy's intention experiment: we are all Qigong masters

2008-05-01T10:07:00.004+01:00

Apparently, Manju Rao and Rustum Roy are busily working on the results from their Intention Experiment, where people all over the world concentrated on changing the Raman spectrum of a beaker of water. However, some preliminary results are filtering out for our entertainment.

According to the Intention Experiment blog, "Although the scientists are not finished examining their data, they have told me one thing: they’ve seen results they’ve never seen before with their equipment". This suggests to me that there was a problem with the experimental set-up, which of course has not been described in any detail.

Then "One reason it is taking so long is that our water had a great deal of variation an hour before the experiment was run. This could mean that our anticipation of the event began to affect the water. Or it could mean that our hypothesis is wrong. Or it could mean that with intention, we are emanating an energy like a Qigong master, which is being picked up by the spectroscopy before the event". Or could it mean that the impurities in the water are changing through time as it sits in an open beaker on a lab bench? Obviously, emanating energy like a Qigong master is a much more plausible explanation, but might it not be a good idea to control the experiment just in case? Also, looking at Rao and Roy's previous work on homeopathic remedies, it seems that 'stray light is eliminated by turning off all the room lights whenever data are being collected'. Unless they are in a proper darkroom, that isn't going to be enough to keep out all the ambient light. Raman is also noisy and highly sensitive to impurities.

There's also some weirdness going on here, as this page talks about taking a Raman spectrum with the 'laser turned off'. With the laser off, there is no incident beam to undergo Raman scattering, so I'm simply not clear what they could be talking about here at all. A 'member of the Penn State team' also says that "The laser light is absorbed by the water molecules, depending on how they are energetically configured or arranged, and then reradiated at a different wavelength". As I understand it, this isn't quite right: the incident beam is scattered, with a wavelength shift of the scattered beam. There is no absorption and re-radiation in Raman scattering. Do the Penn State team have any idea what they're doing?

I can't wait for the final results...

Homeopathy and wikipedia

2008-04-20T16:53:00.004+01:00

Homeopathy-related articles on Wikipedia have been subject to probation for some time, because of edit wars between pro-homeopathy and anti-homeopathy contributors. The situation has become such a shambles that Wikipedia has opened an arbitration case on all its homeopathy-related pages. As a result of his editing behaviour, Dana Ullman has managed to get himself banned from editing any homeopathy-related pages for a period of three months.

The arbitration page is long and parts of it are somewhat tedious, but it is interesting in that it demonstrates some of the problems science has in presenting evidence to laypeople in a clear way. The literature on homeopathy is complex, consisting of a mixture of poorly conducted studies that show some (at best equivocal) evidence that homeopathy might work, and well-designed studies that show homeopathy has no effect. (A meta-analysis by Shang et al. that was published in the Lancet elegantly showed that the largest and least biased studies showed the least effect for homeopathy). It is then easy for advocates of homeopathy to emphasise the apparently positive studies, and claim that sceptics of homeopathy do not consider all the evidence. In other words, advocates of homeopathy take the studies at face value; they don't look at the studies and analyse them to see how they were conducted and whether the results make sense.

Because there are some people who think homeopathy works, and there are some peer-reviewed journals of homeopathy and other 'alternative' therapies that superficially appear to be scientific, it is possible to claim that there is some scientific controversy about the effectiveness of homeopathy. As Wikipedia is based around the idea of neutral point of view (NPOV), it seems superficially sensible to present a 'balanced' view whereby evidence for and against homeopathy is described. But, crucially, this is not a balanced view; it gives undue weight to poorly conducted studies. Looking at the evidence as a whole shows that homeopathy is nothing more than a placebo. Quite apart from that, it contains ideas (e.g. that the 'potency' of a remedy increases the more it is diluted) that are in conflict with well established science. A neutral point of view is surely that homeopathy is extremely implausible and that there is no evidence that it works.

This sort of thing is not just a problem for Wikipedia. There is a general problem of spurious 'balance' in scientific reporting. Another clear example is the BBC linking to anti-vaccine websites in stories about vaccines. Science is a notoriously complicated business, but there are times when the scientific evidence is clear, when it is evaluated properly. The challenge is to show that this is the case, without coming across as some kind of monolithic establishment that wishes to crush all dissent.

More water-related nonsense from Rustum Roy

2008-04-19T18:18:00.005+01:00

Regular readers will remember a recent paper involving Rustum Roy, a materials science professor at Penn State, that purported to show that you could use UV-VIS and Raman spectroscopy to distinguish different homeopathic remedies. Given that the homeopathic remedies were at a 30c 'potency' (that is, a dilution factor of 1x10^60, a one followed by sixty zeroes), the chance of any molecules of the active ingredient remaining are vanishingly small, and this seemed an extremely unlikely claim. Sure enough, on looking at the paper, it's riddled with basic errors, including duplicated graphs, and doesn't support the conclusions of the authors. A group of sceptical scientists, including me, wrote a letter to the journal's editor about the paper. The authors offered a totally inadequate response that failed to address any of the serious problems with the paper.

Fortunately, Roy has not been discouraged by this setback. He is now involved in an experiment that attempts to show that we can change the structure of water with our minds. There are some truly astounding statements by Roy here.

Structured water is found in the cytoplasm of healthy tissues and it is characterized by having a high solubility for body minerals. It is also found in healing waters. This appears to be the structure shared by very different healing waters from some healing spas to silver aquasols used worldwide.

Structured waters have been produced using various forms of energy, such as light, sound, heat, pressure and radiation.

In our proposed experiment, we aim to examine whether we can structure water with intention alone. We’ll be monitoring any change against control using analytical tools such as spectroscopy.

As far as I can tell, there is no evidence that 'healing waters' actually do any healing, and there is no evidence that different structures of water have different biological effects. But why worry about that? Why not set up a badly designed experiment anyway?

What Roy is proposing to do is get a beaker of water, and collect a Raman spectrum of it. Raman spectroscopy is a technique that can identify vibrational and rotational modes in a system. It can be used to detect changes in chemical bonding, which is what Roy will be looking for. Then people all over the world are going to concentrate on changing the Raman spectrum of the beaker of water. Roy will then collect a new Raman spectrum, and compare it to the old one. This is going to happen on April 26th.

The main problem I can see with this is that the experiment is totally uncontrolled. A beaker of water sitting around is going to dissolve gases and incorporate dust, particulates, skin cells and so on from the atmosphere surrounding it. It might also lose some of the dissolved gases in it to the atmosphere through time. Roy is apparently making no attempt to control for this at all.

Roy's previous work doesn't exactly give cause for confidence. The paper in Homeopathy claims to show differences in Raman spectra between homeopathic remedies. Essentially, all the paper does is present graphs, state that they look different, and leave it at that. There is no attempt to understand why the graphs might be different, or assign peaks to distinct vibrational/rotational modes, or show that the differences between the spectra are statistically significant, or anything.

I think I can confidently predict that the second spectrum will look subtly different from the first, and Roy will declare that we can change water structure with the power of our minds alone. The experiment will be totally worthless, but that won't stop people citing it for years to come as cast-iron evidence for telekinesis.

You've got to laugh

2008-04-07T13:21:00.007+01:00

I still tend to buy to the Observer on a Sunday, even though it's not very good (but the competition isn't up to much either). There's usually something to make me laugh, though. This week, I laughed like a drain at this story, about psychics, spiritualists and mediums being subject to the new European Consumer Protection Regulations. Apparently:

Promises to raise the dead, secure good fortune or heal through the laying on of hands are all at risk of legal action from disgruntled customers. Spiritualists say they will be forced to issue disclaimers, such as 'this is a scientific experiment, the results of which cannot be guaranteed'.

This really is pure comedy gold. I'm fairly sure that most of these 'scientific experiments' are going to fall short of the standards you would normally expect. For example, you would want to see adequate controls and blinding. I wonder if you could sue someone for describing something as a 'scientific experiment' when it is clearly no such thing?

Previously there was something called the Fraudulent Mediums Act (1951), which protected mediums against litigation, unless it could be proven that they acted with dishonest intent. As long as you genuinely believed the rubbish you were talking, you were in the clear. This is not an approach we would accept if, say, we were buying a used car. If Honest Pete's Motors told you "Sorry about that, guv, but I genuinely believed that there was nothing wrong with the gearbox on that Austin Allegro", you would not be happy. So why should spiritualists get away with it? According to the story:

Carole McEntee-Taylor, a spiritualist healer in Essex, said having to stand up and describe the invoking of spirits as an 'experiment' was forcing spiritualists to 'lie and deny our beliefs'. She added: 'No other religion has to do that'.

Perhaps if you have genuinely held religious beliefs, you shouldn't be attempting to make money out of them?

Homeopaths capitulate [by guest blogger Olaf Priol]

2008-03-31T11:44:00.002+01:00

A group of prominent homeopaths today admitted to 'several decades of flagrant self-deception and denial', concluding that there was no scientific evidence that homeopathy offered any benefits beyond the placebo effect.

Una Daleman, a well known US homeopath, issued the following statement:

"Friends, I've spent many years working as a homeopath, and I have always felt that the fact my patients felt better after treatment was proof that homeopathy did something. I also knew of several studies that seemed to show statistically significant evidence that homeopathy works. Scientists dismissed this as being placebo effect, or regression to the mean (whatever that is!), or the result of biased and useless studies with poor methodology, but I thought I knew better. However, as you all will know, I am nothing if not HUMBLE, willing to learn, and intellectually honest. So, after spending much time at skeptical blogs and websites, taking in the criticisms of the apparently positive studies I wanted to discuss, and evaluating the scientific evidence as a whole, I have to conclude that homeopathy is just about the most ridiculous pile of faux-magical and pseudoscientific garbage that humanity has constructed.

In particular, I want to totally repudiate the nonsensical study by Rustum Roy and colleagues on spectrophotometry of homeopathic remedies, which is so poorly conducted it shows nothing at all. I also want to acknowledge that a study in Chest that I have spammed all over the internet as evidence for homeopathy is invalid, because of differences between the placebo and treatment groups."

Dieter Wisher, president of the Guild of Homeopaths, added:

"After long reflection, it seems to me that Ben Goldacre was right about the evidence from meta-analyses after all, and they show no effect beyond placebo for homeopathy. The attempts by myself and others to explain these results through quantum entanglement (whatever that is!), the bad vibes of anti-homeopaths infecting experimental procedures, or the wrong kind of magic wand being used in the treatment group, were misguided attempts to avoid the conclusion that everything we believed in is nonsense.

As penitence, I will be travelling to Togo and working at a malaria clinic."

However, maverick US homeopath Dana Ullman said:

"It is EASY to assume that homeopathic medicines are akin to placebos if one has a superficial understanding of what homeopathy is and what good research has been conducted to evaluate it. I want to remind skeptics that good and serious scientists maintain a high level of HUMILITY about what they know and what they don't know. I am proud of my humility of what I know and what I don't know - particularly what I don't know. I can tell you that about a new study on homeopathy and water by a respected professor of material sciences, Rustum Roy, PhD (of Penn State University)....[continued p. 94]

Academic shenanigans

2008-03-14T13:18:00.001Z

This is the sort of thing I get up to when I really ought to be producing original research, but who can resist a vaguely pointless semantic argument?

Another Sinai field trip...

2008-03-14T12:35:00.007Z

Monday 3^rd March

Yes, my incredible jet-set lifestyle never lets up. Here I am, in the famous Moon Beach resort in hot and gravelly Sinai once again. This time Paul Woodman and I are leading a field trip, all by ourselves, for a crew of petroleum industry types who are working in the North Sea and the Gulf of Suez. Our task is to give them an overview of the structural style and stratigraphy of the Suez rift, for which we’re getting paid consultancy rates. This is probably the first trip to Sinai that I’m making more money than Sayed, our driver. For Paul, it’s a useful chance to look around the area before he starts writing up his PhD in earnest. For me, the best geologist is the one who has seen the most rocks, so it’s good to keep looking.

We fly from Manchester to Cairo via Amsterdam, and take in the bar in Manchester’s terminal 2. It’s bloody awful. They’ve run out of most of the beer they have (no cask, of course), it costs £6.25 for two pints, and the place is strewn with dirty plates and glasses. The Murphy’s pub at Schipol is far nicer, and I get a pint of Murphy’s and a chip butty. We arrive in Cairo at 2am, which is never any fun, but negotiate the visa queue, passport queue, baggage claim, customs and taxi drivers without too many problems. We manage to get a couple of hours sleep at the apartment of some friends of Paul’s in Maadi, Sayed picks us up at 8am, and we drive to Moon Beach, arriving at 12:30 and just in time for lunch. A fairly uneventful trip, as they go, the highlight being an articulated lorry passing us at about 100 km/h with a burst tire strewing rubber shreds all over the carriageway. As far as we can tell, the driver doesn’t stop, but these things have ceased to surprise me much.

We eat goatburgers (nominally beef, but we have our suspicions) and chips, and spend the afternoon scoping out the sites we’re going the take the trip to. At the first one, a pipeline is being laid by the side of the main road, and the area seems to have become a work site and general rubbish dump, but we think we can spend some time looking at early syn-rift stratigraphic relationships there. The second stop is part of Paul’s study area, and as such is plagued with Bedouin marijuana (‘bango’ in Egyptian Arabic) growers. Last time we were here, there was a very large bango field, but as usual the Bedouin were happy enough once they knew who we were and what we were doing. We need to check out the area to make sure that no-one is going to think we’re army or police and shoot us. This is the kind of thing I still can’t quite believe I’ve found myself doing. Today, though, the area is quiet, not a soul to be seen, and it seems that we can use this site as well.

By now, on a good two and a half hours sleep, I’m done thinking for the day, so we repair to Moon Beach, where I watch the sun go down over Wadi Araba. Even this has a purpose, though: I want to know what time sunset is, so we don’t have to expose ourselves to the hideous danger of driving at night in Egypt.

Wednesday March 5^th

So far so good. We spent yesterday checking out some of the sites we’re going to visit on the trip, to make sure the roads are OK and that there is no bango. It’s all good, except on Thursday we have to go through the police checkpoint outside Abu Zenima. We don’t particularly want to do that, because Sayed doesn’t have a desert pass and we only have tourist passes, but a road is blocked and we don’t have any options.

The field trip people arrive today. We go out in the morning to look at one of the sites we’ll visit, and arrive back at Moon Beach at 11am. The trip is supposed to have left Cairo at 8am, so we expect them here around 11:30 or so. They eventually show up at 2pm. Apparently, there was some misunderstanding with the drivers about who needed to be picked up at Cairo. Then there was a delay at the Ayun Musa checkpoint. This now seems to be a corrupt checkpoint, with the excuse to stop the jeeps relating to a fire extinguisher. I doubt that 1% of the vehicles that go through that checkpoint have working fire extinguishers. They also checked passports. I assume a bribe was being angled for.

We manage to get away from Moon Beach in short order and take in a couple of field stops. The last stop is the summit of Gebel Hammam Faraun, where you can get a panoramic view across the Hammam Faraun fault block. The road to the summit was originally built by the UN and there is now a Vodaphone mast there, so the road is quite well maintained, although very steep and winding. Even so, two of the jeeps get punctures on the way up, the drivers are very slow (“Tourist drivers,” says Sayed dismissively, “not for geologists”) and the ensuing delay leaves us driving through dusk to get back to Moon Beach. The companies are very keen that we get back before dark, for safety reasons (which will make sense if you’ve ever travelled on Egyptian roads in the dark). But, there’s no harm done, although it’s 6:15 when we get back and I’m wiped out. I think I’m a little sunstruck, or I’ve picked up some mild bug. I have a headache and don’t much feel like eating.

Even so, I manage a couple of beers with the field trip people, and eat some Moon Beach buffet, and I feel a little better. The only problem is that Moon Beach has no international phone line at present, so I have no way of contacting Jolan and letting her know I’m OK (and letting me know that she’s OK). Paul let me use his phone to send a text to our landline last night, but I don’t know if that is going to work and I’m worried. But as it stands, there’s nothing I can do about it. I have no mobile and there certainly isn’t any internet here. I think that I’m going to have to get a mobile, and that annoys me unreasonably. Clearly I’m tired, so I turn in.

Thursday March 6^th

Wadi Nukhul today, which is really my show, given that I’ve spent the last two and a half years working on it. We start by looking at some minor faults in the footwall of one of the major block-bounding structures. There’s a spectacular detachment horizon where the faults flatten out into a clay-rich layer. We go to the Wadi Nukhul east face, and as we drive we see a Bedouin hiking through the desert with a Kalashnikov. Who knows where he was going and why he needs a gun for the journey. He waves at us, in our fleet of brand new black jeeps, cheerily enough. The east face is one of the best bits of geology you’re ever likely to see. The participants are impressed.

Then we go for a walk through the type section of the Nukhul Formation, but no-one seems particularly interested in that. Perhaps none of the companies are targeting it, or perhaps it’s the heat, but it’s not setting pulses racing. In the afternoon, we look at the Nukhul fault zone itself, and look at the west face. It’s spectacular stuff, and colourful, and I would say that there are few better examples of rift initiation stratigraphy and structure anywhere.

Friday March 7th

Today is ‘Wadi Baba and the faulted Thebes’, as some wag has christened it. This is spectacular geology that can’t fail to impress, so we’re on solid ground. In the morning we look at the Nukhul Formation (rift initiation). In contrast to Wadi Nukhul, the Nukhul Formation here lies directly on Thebes limestone, and no Abu Zenima Formation is present. We look at a section near the footwall crest of the Nezzazat fault,where only the top offshore/shoreface facies of the Nukhul are deposited, and another section in the mouth of Wadi Baba where some of the estuarine tidal facies are developed, although they seem to be much more wave-dominated here. We also look at the syn-rift lower Rudeis Formation, which has fairly extreme lateral variations in thickness and facies. The best part of the day, though, is the afternoon, when we walk along the main Baba fault and then over what we call the cableway col. The cableway was built to carry manganese ore from the basement to the edge of the Baba Markha plain, where a narrow-gauge railway took it to Abu Zenima. We get spectacular views of the growth syncline in the hangingwall of the fault, and the reverse faults associated with it. We do this at the hottest part of the day, and it’s sweaty work, but well worth it. There’s enough mobile reception here for our resident reservoir engineer to receive a wrong number from the UK. Then we walk down the lower Rudeis section, which is much more proximal here, full of conglomerate and with spectacular slump units.

We finish the day at the cableway terminus, where both Israel and Egypt have billeted soldiers at one time or another. The buildings now are largely destroyed with walls pock-marked by bullet holes. The cableway itself is no longer functional, the Leeds steel rusting gently in the baking sun. There is graffiti in both Hebrew and Arabic, and the field trip participants take a few pictures.

At the checkpoint before Abu Zenima, one of the secret policemen wants to stop us, but his colleague tells him we’re from an onshore drilling rig that is on the Baba Markha plain not far from the checkpoint. Sayed says the magic word ‘petrole!’ and they let us through. Another good day, and no more checkpoints now until we head to Cairo.

At dinner, a couple of the guys tell me that they’ve got a lot out of the trip, more than most trips they’ve been on, which is good to hear. You want people to get something out of it.

Sunday 9^th March

The final day of the trip goes OK, although we haven’t really rehearsed what we’re going to do at the last stop. I try to summarise what we’ve done on the trip, but I’m totally knackered, and it doesn’t come over too well. The rocks we see are largely calc-arenites, not good reservoir rocks, and the reservoir engineer gets glummer as the day progresses.

We get back to Cairo, safe but not exactly well. I can’t claim that I don’t deserve it, having stayed up drinking vodka (which I never drink) pretty much all Saturday night after the end of the trip. Paul manages to get me up in the morning, and we’re on the road by 9am. I don’t sleep, amazingly enough, but I’m in a bit of a stupor most of the way. My voice has more or less gone, a combination of yelling at field trip participants and staying up late.

Just after we get to Suez, I see some stopped traffic up ahead, and something lying on the road. I imagine one of the hundreds of overloaded vans and pick-up trucks has lost part of its load, a common enough occurrence. As we get closer, I realise that it’s something much worse. The bundle lying in the road is a dead body, looking pretty shapeless and mercifully covered with a blanket by someone. A motorcycle is lying forlornly near the shoulder about a hundred yards up the road. All the colour drains from Sayed’s face; I’m looking pretty colourless anyway. An ambulance is just arriving, but it’s clear that there’s nothing they can do. We pull over a little way up the road, where an articulated lorry and a few cars are parked, but of course, there’s nothing we can do to help either. Sayed tells us the motorcycle rider was trying to pass between two trucks: one of the trucks clipped him, and he fell underneath the other one. The rest of the trip passes more or less in silence.

We get back to Cairo OK, and are staying with friends of Paul’s in Maadi. We visit GUPCO, where Paul worked for a while, get a bite to eat, and take a nap. I’m about ready to get home. We watch a movie and some DVDs of ‘Extras’, and snatch another 40 winks before the 4:15 am (really) KLM flight to Amsterdam. Except, when we get to the airport at 2 am, we discover that the flight is delayed for ten hours. KLM put us up in a hotel across from the airport. Paul is quoted 25 LE for the ride, all of a couple of hundred yards. In Maadi, 5LE will get you pretty much anywhere you need to go. There was really only one possible response. ‘Magnoon!’ [crazy!] says Paul, and we walk.

We get to Amsterdam Monday afternoon, and have a soothing pint of Murphy’s and a bacon and egg sandwich. I eventually make it home 11 hours late, but at least I make it home.

The first earthquake of spring

2008-02-27T14:14:00.004Z

Last night, the largest earthquake since 1984 struck Britain. There I was, sleeping, like any good scientist should be at that time of the day (unless they're writing a PhD) when I was woken by the bed shaking. Sometimes this is because of the enthusiastic couple who live in the flat upstairs, but this was different. The room shook for probably 20 to 30 seconds, accompanied by a low rumbling, and I realised it was probably an earthquake. I checked the time so I could look at the BGS website [pdf] this morning: it was 1am.

This morning I discovered that it was a magnitude 5.2 quake, with the epicentre near Market Rasen, Lincolnshire. 5.2 is a pretty hefty magnitude, and it's fortunate that the focus was 10 km deep; if it had been much shallower it could have caused a lot of damage. There was only one injury, a man in Wombwell, near Barnsley, who was hit by a falling chimney pot. As the man's father said 'of all the things to happen-an earthquake!'

If you felt the earthquake, the BGS have a questionaire to fill in. It only takes a few minutes, and they use the information to do science.

As far as the Earth's crust goes, the UK is pretty stable, but earthquakes are not that rare. They just tend to be small. The BGS has a list of earthquakes for the last 30 days here. You'll notice that there has been aftershock, magnitude 1.8, from the Market Rasen quake.

Update

The BGS now reckon that the depth of the main earthquake was more than 18 km. There have now been three aftershocks, with the largest at a magnitude of 2.2.

Antidepressant manufacturers sound a bit like homeopaths

2008-02-26T16:31:00.005Z

The papers today are full of the news of the meta-analysis of clinical trials of anti-depressant drugs (including Prozac) that shows such drugs only have a clinically significant effect for a very small subset of the most depressed patients. The authors also concluded that the larger effect compared to placebo for the most depressed patients was caused by those patients responding less well to placebo, not to the SSRI drugs being more effective for those patients. [I should add that if you are prescribed such drugs, you shouldn't stop taking them unless you are advised to by your GP].

Obviously, this is not good news for the companies who make these drugs. GlaxoSmithKline, who make Seroxat, said that the conclusions were at odds "with the very positive benefits seen in actual clinical practice" and that "This one study should not be used to cause unnecessary alarm". What the study is suggesting is that the benefits of the drugs seen in clinical practice may be partly related to the placebo effect, and that those benefits not related to the placebo effect are small and usually not clinically significant. That's the point that GSK need to address. But they don't, and not only that but they disingenuously describe a meta-analysis of 47 trials as 'one study'.

Eli Lilly (makers of Prozac) said that "Extensive scientific and medical experience has demonstrated that [Prozac] is an effective anti-depressant". This just ignores the evidence from the meta-analysis that says the opposite.

When I read these responses, I couldn't help but be reminded of the responses of homeopaths to the evidence from double-blinded, randomised, placebo-controlled trials that homeopathy doesn't work. They simply ignore that evidence and say that they know it works, through clinical experience or anecdotes.

It's disappointing, but perhaps not surprising, to see drug companies falling back on these unscientific statements. After all, homeopathy transparently doesn't work, but plenty of people still use it. Perhaps the companies are hoping for a similar outcome for their anti-depressants. The point here is not that these anti-depressant drugs are useless: clearly they do 'work', especially for the most depressed patients. The point is that for most people taking them, they don't work particularly well.

Netcetera fold like a cheap suit

2008-02-19T10:17:00.004Z

The mighty Quackometer blog has been pulled by its hosts, Netcetera, apparently because of ludicrous legal threats made by one Joseph Chikuele Obi. This follows a previous attempt by the Society of Homeopaths to get the website shut down.

Encouragingly, it seems that this will have no effect at all, as the Quackometer will shortly re-appear hosted by Positive Internet, who host Bad Science, amongst others.

Obviously, there ought to be some protection against genuinely defamatory material on the web, but there also ought to be some protection for bloggers against the sheer frivolity that has characterised the threats against the Quackometer. The Quackometer has provided a useful service by exposing quackery of all kinds, and it's a shame that its author, Dr Andy Lewis, has been subjected to such nonsense.

Update

Happily, the Quackometer is now back online and causing trouble for quacks and frauds, courtesy of Positive Internet. You can read all about the saga here.

The uses for terrorist atrocities

2008-02-18T12:23:00.005Z

I couldn't help but be struck by the Guardian front page last week, that included two stories related to the 7th July terrorist bombings on the London transport system in 2005.

The first story outlined how the Saudi Arabian government, being investigated under corruption allegations linked to the sale of military equipment by BAE systems, allegedly threatened that intelligence co-operation would be compromised unless the investigation was called off. This would threaten another 7/7 and lead directly to the "loss of British lives on British streets". Any government with a spine would have told the Saudis where to go, given threats made in an attempt to halt an ongoing criminal investigation would be a criminal offence. Disgracefully, our government caved in and ordered an end to the investigation. This just can't be right.

The second story featured George Bush, who claimed that the 7/7 bombings could justify torture.

"To the critics, I ask them this: when we, within the law, interrogate and get information that protects ourselves and possibly others in other nations to prevent attacks, which attack would they have hoped that we wouldn't have prevented? And so, the United States will act within the law. We'll make sure professionals have the tools necessary to do their job within the law ".

On being asked about whether Guantanamo Bay, Abu Ghraib and extraordinary rendition send the wrong signal to the world, Bush said "It should send the signal that America is going to respect law, but is going to take actions necessary to protect ourselves, and find information that may protect others -- unless, of course, people say, well, there's no threat, they're just making up the threat, these people aren't problematic. But I don't see how you can say that in Great Britain, after people came and blew up bombs in subways. I suspect the families of those victims understand the nature of killers".

So there you have it. America will respect the law, by using techniques such as waterboarding that are transparently illegal. Families of the 7/7 victims will support torture, because they understand the nature of killers. But what if they understand the nature of torturers too? Again, this just can't be right, can it?

The terrorist attacks of 7/7, in which 52 innocent people were killed, are now being used to justify torture and corrupt arms sales to dictatorial regimes. Nice.

Bad geology: another pseudojournal

2008-02-14T16:28:00.007Z

Generally, my own field of geology is not somewhere you find a lot of pseudo-science. So it's nice to be able to comment on some, via the folk at Answers Research Journal. Thanks to Michelle for the heads-up. The journal contains a paper on 'catastrophic granite formation', an attempt to show that granites can form very quickly, quick enough to be consistent with what creationists call 'flood geology', the theory that most of the geology we see was formed in the Genesis flood.

The author is one Andrew Snelling, proud holder of a doctorate in geology from the University of Sydney. It seems that Snelling is happy to use the conventional geological column when working as a consulting geologist, but happy to disregard it when propagandising in favour of creationism.

The 'journal' claims to be "a professional, peer-reviewed technical journal for the publication of interdisciplinary scientific and other relevant research from the perspective of the recent Creation and the global Flood within a biblical framework". This raises some questions. Who are the reviewers for the journal, and who reviewed this article? In geology it is standard practice for reviewers to be acknowledged in published papers; reviewers usually identify themselves to authors, unless there are strong reasons not to do so. No reviewers are identified in the article: in fact there are no acknowledgments at all. In the instructions for authors [PDF], prospective authors are asked to suggest at least three referees. The guidelines state that comments will be solicited by at least three reviewers. I wonder if only reviewers nominated by the authors are used? Certainly it's hard to imagine mainstream geoscientists agreeing to review for the journal, or agreeing to the publication of any articles on 'flood geology'.

Why is the journal restricted to "research from the perspective of the recent Creation and the global Flood within a biblical framework"? The instructions for authors suggest that work will be rejected if it is not "formulated within a young earth-young universe framework". This would be like, for example, a physics journal refusing to accept papers that were not formulated within a string-theory framework. No real scientific journal would restrict itself to publication of work that addresses a single hypothesis, because that wouldn't be science.

The journal is claimed to be a "technical journal", but there's a lot of language in the article that suggests otherwise. For example, in the introduction, Snelling writes that "Each recognizably distinctive granite mass, the boundary of which can be traced on the ground, is marked as a separate geologic unit called a pluton on a geologic map". In a section headed 'Magma Principles', Snelling writes that "The molten material which flows from volcanoes is known as lava and cools to form volcanic rocks. So lavas must be molten rocks; that is, they were originally rocks that melted deep inside the earth underneath volcanoes. When deep inside the earth, these molten rock materials are called magmas because they are slightly different in composition and physical properties due to the steam and gases they have dissolved in them that erupt separately from the lavas through volcanoes". There's nothing in particular wrong with these statements, but they read much more like excerpts from a GCSE level textbook than prose from a technical journal. This sort of basic information is never included in real geological journals: the assumption is that those who read them are already familiar with the basics. The article reads much more like a piece written for a general audience.

What of the actual science? It's actually not too bad in the main. The issues surrounding the production and emplacement of granite are quite well set out. The point of the article is that granites could be created and emplaced within a timescale of 6000-7000 years. This seems like a reasonable conclusion, at least in some cases, though I'm not convinced that it applies to every intrusive granite body on the planet. The problem is that Snelling then jumps to the conclusion that this is consistent with the Genesis account of creation. At this point, the article is no longer talking about science, but is trying to provide evidence for an answer that has already been decided on. The argument that the earth is a lot older than 7000 years does not depend on granites taking more than 7000 years to form and become emplaced. It relies on pretty much everything we know about geology, but particularly on radioisotopic dating. How does Snelling get around this? He simply states that isotopic dating is in gross error because it fails to account for the acceleration of decay. This is a load of rubbish, as you can see here.

The dead giveaway, though, is the section devoted to 'Evidence from radiohalos'. A radiohalo is supposedly a zone of crystal damage surrounding a crystal that contains radioactive substances, for example zircon crystals that are relatively common in granites. Snelling claims to be able to identify radiohalos associated with decay of three polonium isotopes. The argument is that the isotopes of polonium have very short half lives: 3.1 minutes (²¹⁸Po), 164 microseconds (²¹⁴Po), and 138 days (²¹⁰Po). Therefore cooling of the granite has to be rapid, because otherwise the polonium would decay too rapidly to form halos. Not to put too fine a point on it, this is drivel. There's a useful summary of why it's drivel here.

Snelling has come to an initially reasonable conclusion about the potential rapidity of formation of some granite bodies, and then extrapolated wildly to suggest that this is the case for all igneous rocks, and that most of what we know about geology is wrong. Still, he is happy to ignore that conclusion in his consultancy work and publications in real scientific journals. Like Homeopathy, Answers in Genesis is a pseudojournal: it's designed to look superficially like it is publishing real science "from a different perspective", but it is not. It is a creationist propaganda organ, and it is publishing creationist propaganda.