Infinite costs wouldn't be Nice

There's been a bit of a furore over the last couple of days about a decision not to approve a drug for treating advanced bowel cancer, Avastin, on the grounds that it would not be cost-effective. This has led to a huge amount of criticism, and headlines ranging from the relatively sober to the Mail's hysterical "Betrayal of the cancer patients: Rationing watchdog accused of talking down wonder drug's power to save lives".

The decision (or guidance: in fact, the final decision has not yet been made, with the guidance subject to consultation and appeal) was made by the National Institute of Health and Clinical Excellence (Nice), which decides on the cost-effectiveness of treatments and whether they should be available to patients on the NHS. According to trial data submitted by the drug manufacturer, Roche, the drug can extend the mean lifespan of people diagnosed with advanced bowel cancer from 19.9 months on chemotherapy alone, to 21.3 months with chemotherapy and avastin. This is by no means a miracle cure or wonder drug. This is a six-week gain, but it comes at a cost: the drug costs up to £21,000 per patient.

Barbara Moss, a survivor of bowel cancer, appears in both the BBC and Guardian articles. The Guardian says:

Mrs Moss said the National Institute for Health and Clinical Excellence (Nice) had put a "value on life" after it said the price was too high for the extra benefit it gives patients

While Mrs Moss is quoted in both pieces as saying:

It seems immoral to me that, as a result of negative NICE decisions like this one, people's choice of living or dying depends on whether they can afford a drug, because it isn't available to them on the NHS.

It's difficult not to have some sympathy for this point of view. But someone has to put a "value on life" (or in fact, a thing called a Quality Adjusted Life Year, QALY). Otherwise Roche could come up with a life-saving drug that cost £1 trillion per patient, and the NHS would have to fund it because you "can't put a value on life". Ultimately, the question is not just about avastin. There is only a limited amount of money for the NHS to spend on drugs, so if avastin gets funded something else (with probably greater benefit) doesn't get funded . These are clearly not easy decisions to make, which is why we have Nice to do it in the first place.

My point here is not that Nice have necessarily got this right (I'm not qualified to judge), or even that the QALY approach is the best possible one. The point is that Nice is doing a difficult, unpopular and necessary job. Accusing them of betraying cancer patients, when what they're actually trying to do is get the greatest benefit from a limited budget, is not really helpful to anyone.

Bad cycling: I'm the champion (but I'm not very fast)

Last night was the Stockport Clarion club hill-climb, on the infamous Cat and Fiddle road between Macclesfield and Buxton. I rode in this event last year, and had a good ride on a clear and dry day with a good tailwind. I had high hopes of bettering the 26:00 I did last year, but things didn't go exactly to plan.

There's a webcam attached to the side of the Cat and Fiddle pub, which looks over the finish of the course at the road summit. I had a look at it yesterday afternoon, and there wasn't much to see except cloud and massive wagons heading over to Buxton with their fog lights on. It didn't look like a lot of fun. Once I got home, I was having serious second thoughts about the whole enterprise as I got ready for the 18-mile ride to Macclesfield, with the rain rattling the windows in my flat. But I made myself head out, leaving in absolutely filthy weather and gnarly traffic for a fun rush-hour jaunt through Stockport. The weather gradually cleared up as I got towards Macclesfield, and I arrived with about ten minutes to spare before the 7pm start.

Unfortunately, the start had been put back to 7:30, so I ended up waiting around in the cold for 40 minutes. Will, the timekeeper, didn't make it until just before 7:30 because he had to pick someone up from the airport. So, dispensing with numbers, the four riders who had shown up got going as soon as possible. The weather was now dry, but cool and windy. I decided to ride with lights in case it was cloudy at the summit. I went off number 2, and it was one of those days when you know it's going to be rough as soon as you set off. I hadn't had enough of a warm-up, and my legs just didn't want to have any of it. I didn't really get settled in until I was past Walker Barn and had done the most difficult bit. By then, the two guys behind me had already gone past, but I had caught up with my minute man. Emerging into the open country beyond Walker Barn, the gale-force tailwind came into play, and the middle section of the course was very fast. I even had to touch the brakes on some of the corners. Then the last climb onto the top: I almost got blown off the road on a cross-wind section, and struggled past the timekeeper in 26:36, 36 seconds slower than last year. But, with the only other Stockport Clarion rider being my minute man, who I had passed about two and a half miles earlier, I was the club hill-climb champion. This is nice, especially as I was several times hill-climb champion at my former club, Birdwell Wheelers, but there's no pretending that I was particularly quick. Several riders in the club would have put minutes into me had they ridden, but like the lottery, you have to be in it to win it. Jolan managed to capture a webcam image of us at the top: I'm the guy in the red shirt at left...


My reward was a long, windy ride home, mainly in darkness, but I got to see the sun go down over Fiddlers Ferry power station as I headed back down into Macclesfield.

Bad cycling: I get round in 27:45

Following my New Year's Day adventures, I haven't done as much bike racing as I hoped I would this year. I was away In Norway all of June for work, and then the Stockport Clarion evening time-trials took a break during July. However, the last three Monday evenings I've been out on the Chelford 10-mile course, in an attempt to get some race fitness for next week's hill climb on the Cat and Fiddle. My aim was to get under 28 minutes for 10 miles, which is pretty modest. After a horrible day in the first event, where I was recovering from a migraine and should probably have stayed at home, I clocked 29:30 and wasn't optimistic. But the following week was better with 28:24, and last night I cracked it, just about, with 27:45. For the first time in these 10s, there were actually a couple of people slower than me. It was a good night, dry and sunny with a light headwind on the slightly longer outward leg, and times were generally fast-ish. There was a 21-minute ride, which is pretty quick for such a slow course.

Coming into Chelford on the homeward leg, there was an electronic sign that gives a read-out of your speed. I was grovelling up the slight incline at a bare 21 mph, so the days when I could get under 25 minutes seem long ago. One of the other riders went through at 27 mph. Still, I felt good, and I'm looking forward to the Cat and Fiddle next week.

"Dr" Nancy Malik is spamming my blog again...

In the comments to a couple of old posts, homeopathic apologist and internet numpty extraordinaire Nancy Malik says the following:

Studies in support of Homeopathy published in reputed journals

1. Scientific World Journal
http://www.ncbi.nlm.nih.gov/pubmed/17982565

2. Lancet
http://www.ncbi.nlm.nih.gov/pubmed/9310601

3. Neuro Psycho Pharmacology
http://www.nature.com/npp/journal/v27/n2/abs/1395862a.html // Bacopa Monnieri for memory

I thought I'd have a quick look and explain why they're nonsense. Unfortunately, this hasn't proven to afford much in the way of intellectual exercise.

The first paper is by Graunke et al., and concerns, I kid you not, the treatment of tadpoles with homeopathic thyroxin. This is a well-known bad homeopathy paper. The tadpoles in the treatment group were more developed than those in the control group at the start of the experiment, so it wasn't much of a surprise that they were more developed at the end too. There is more discussion of this dreadful rubbish here.

The second paper is the famed Linde et al. meta-analysis, published in 1997. While this paper does say "The results of our meta-analysis are not compatible with the hypothesis that the clinical effects of homeopathy are completely due to placebo", there are some other things to bear in mind:

1. The paper also says "However, we found insufficient evidence from these studies that homeopathy is clearly efficacious for any single clinical condition".

2. A 1999 paper by the same authors, using improved methodology and including new trials, states that "It seems...likely that our meta-analysis at least overestimated the effects of
homeopathic treatments".

3. A subsequent meta-analysis by Shang et al., published in the Lancet in 2005, using further improved methodology concluded that the results were compatible with homeopathy being a placebo.

Finally, the third study, by Roodenrys et al. in the journal Neuropsychopharmocology, is not about homeopathy at all, but rather about herbal medicine. In homeopathy, remedies are typically diluted such that it is very unlikely that they contain any of the original material: there is no active ingredient. In the Roodenrys study, what is being tested is brahmi, an Indian herb, of which the paper says:

Studies have shown that the herb contains many active constituents, including a number of alkaloids and saponins, however, the major constituents are the steroidal saponins, Bacosides A and B.

So it isn't entirely surprising that brahmi might have some effect.

From this fairly cursory glance at the studies provided by Nancy Malik, it's clear that she is from the Dana Ullman school of evaluating journal articles. This involves finding some papers that superficially appear to support your position, and then spamming them all over the internet. Luckily, for this approach there is no need to understand the articles, or even to read them. For people who think that magic water is medicine, that would be rather too much to expect.

Private Eye still believes in Wakefield...

As discussed in my previous post, and many other places all over the internet, the GMC ruling against Andrew Wakefield was damning. But not so much if you write for Private Eye, who have been defending Wakefield for years. So, in the light of the GMC findings that Wakefield behaved unethically, with "callous disregard" for the wellbeing of the children involved, and that parts of his research were fraudulent, is it time for the Eye to admit it was wrong?

Apparently not. In an "In the Back" piece, the Eye had the following to say:

Although the GMC said the hearing was not about vaccination and autism, it is fairly clear that the two and a half years of disciplinary proceedings were to bring the MMR debate to a conclusion. The three doctors were to some degree being accused (and found guilty) of causing a public health scare which led to a fall in vaccination rates, so damaging "herd immunity", particularly with regard to measles.

Nope. They were found guilty of conducting invasive diagnostic procedures, against the children's interests and without ethical approval; of not disclosing serious conflicts of interest; and of presenting false information in the publication of their research (which finally led to the Lancet retracting the paper).

Several parents of the children who featured in the team's original research paper, which was at the centre of the GMC case, stormed out of the hearing in angry protest at the findings - particularly the suggestion that their children's tests were not clinically necessary...They say they would have told the GMC that the treatment they received at the hospital helped their children's symptoms - but they were never called to testify.

Eh? We're not talking about treatment here, we're talking about invasive diagnostic tests. These were done without ethical approval, without the required expertise, and against the interests of the children. Perhaps we should just get rid of ethical approval, as it gets in the way of heroic doctors doing whatever they need to vunerable patients.

And finally:

None of this debate about conduct, however, changes the fact that no subsequent research has supported Wakefield's thesis of a possible link between the MMR vaccine and autism. Nor does it change the fact that despite previous attempts to justify the Eye's coverage at the time, some readers are still deeply critical of the magazine's reporting of the MMR debate between 2001 and 2007, when we wrote about the concerns of Wakefield, the families and their lawyers, and endorsed calls for more research (see Letters).

Well, indeed. So why is the Eye still apparently defending the indefensible? I still buy the Eye, for all the other good that they do, but they've made a mistake over MMR and they ought to be big enough to admit it. They would surely ask the same of any of the politicians and industry figures they regularly lampoon.

It's literature integrity week...

The integrity of the scientific literature is at issue in not one but two news stories this week. Firstly, the 1998 Lancet paper by Andrew Wakefield and several co-authors, which purported to show a link between autism and the MMR vaccination, was retracted [PDF] by the journal. Meanwhile, there is an ongoing brouhaha about a 1990 Nature paper that has some implications for climate change research.

I'm only going to deal with Wakefield here because, you know, I have other things to do. The retraction by the Lancet is really the final nail in the coffin of Wakefield's paper, which has been heavily criticised for numerous reasons. Of the thirteen authors, ten had previously retracted the interpretation that there was any link between autism associated with gastro-intestinal problems and the MMR vaccine. Only Wakefield himself and one P Harvey refused to sign up to this: the remaining author could not be contacted. It has also been known for some time that the methodology of the paper was flawed. So what suddenly changed this week?

For some time the General Medical Council (GMC) has been investigating Wakefield and two colleagues, Simon Murch and John Walker-Smith, over allegations relating to their research on autism. The GMC findings were released on January 28th, and can only be described as devastating. You can find the whole thing here [PDF], or there's a good summary at Respectful Insolence.

The GMC did not look at whether the research findings were right or wrong: rather they looked at the research methodology. Essentially, the major problems were these:

1. Wakefield had an undisclosed conflict of interest, because he was being paid by lawyers whose clients believed their children had been harmed by the MMR vaccine.

2. Wakefield ordered invasive diagnostic tests, including colonoscopies and lumbar punctures, that were unnecessary and not in the children's interests, and he had neither the required ethical approval nor the requisite expertise to order those tests.

3. Wakefield obtained blood samples at his son's birthday party by paying children £5 each. The GMC panel described Wakefield's actions as comprising "callous disregard for the distress and pain the children might suffer".

4. The 12 children were described as having been "consecutively referred", but this was not true.

Heres what the Lancet had to say:

Following the judgment of the UK General Medical Council’s Fitness to Practise Panel on Jan 28, 2010, it has become clear that several elements of the 1998 paper by Wakefield et al1 are incorrect, contrary to the findings of an earlier investigation.2 In particular, the claims in the original paper that children were “consecutively referred” and that investigations were “approved” by the local ethics committee have been proven to be false. Therefore we fully retract this paper from the published record.

So that's that. The problem with the paper is not that it was wrong; that has been known for some time. In fact, simply being wrong would not be a reason to retract the paper. Science often progresses by building on papers that were not quite right. No, the problem with the Wakefield Lancet paper was that it was fraudulent, unethical and incompetent, as well as being wrong. As a result, the takeup of MMR has fallen below the ~95% level at which herd immunity is maintained, and measels has once again been declared endemic in the UK. That's why you should try to avoid publishing fraudulent and unethical research. We'll let you off if your research is merely wrong, as long as it was honestly and competently wrong.

Wakefield eh, what a hero? Unfortunately, as we'll see in my next post, some people in the most unexpected places still believe in the Cult of Andy...

The Observer ignores the evidence on homeopathy

Homeopathy is in the news once again, following a campaign set up by Merseyside Skeptics in which skeptics took an overdose of homeopathic pills, demonstrating that there's nothing in them other than lactose, and then went to the pub for some non-homeopathic beer. The campaign was mainly aimed at high-street pharmacist Boots, whose professional standards director admitted to the science and technology select committee of the House of Commons that there was no evidence that homeopathy worked, but they were very happy to continue charging people money for it. The campaign, and a forthcoming report of the select committee on whether the NHS should fund homeopathy, gave the Observer an excuse to publish a feature article, in which Anushka Asthana and Robin McKie examine the supposed controversy about whether homeopathy works or not.

Of course, scientifically speaking there is no controversy over homeopathy. It's perfectly clear that there is no scientific reason why it ought to work, and that when it is tested in properly conducted trials it works no better than placebo. I grant you that this probably wouldn't make for a very satisfying Sunday newspaper feature, and it would certainly be rather too short to fill up all of page 30 of the Observer. But you might expect there to be some discussion of the actual evidence. Not really, though. Edzard Ernst gets to say that "I have now published more than 100 papers on homeopathy and I am quite clear about its efficacy: you may as well take a glass of water than a homeopathic medicine". The evidence of Jane Lawrence of the Royal Pharmaceutical Society to the House of Commons select committee on science and technology that "There is no basis for [homeopathic remedies] being effective" is also quoted. But then Cristal Sumner of the British Homeopathic Association gets away with saying "Homeopathy helps patients and is not a placebo effect", despite all the evidence showing exactly the opposite.

This is typical of how the press treats scientific issues; they are presented in a superficially even-handed way, but crucially there is no attempt to weigh the evidence. Again, there is no scientific controversy over homeopathy, but by reading the Observer article you could be forgiven for thinking there was considerable room for doubt.

The worst things about the article, though, are the two inset boxes (these only appear in the printed version of the article, not in the online version, as far as I can tell). The first box is headed "Common Renedies" and lists "Popular homeopathic remedies for sale in Britain", including Arnica for clearing up bruises, and mixed pollen for treating hayfever. As these are homeopathic, they contain no Arnica or pollen, and there is no evidence they do anything except lighten your wallet, but this is not mentioned in the box. The second box is headed "Case Study", and recounts the story of one Helen Llewelyn, who claims that homeopathy helped control her endometriosis. This is an anecdote; it tells us that Llewelyn feels better, but it doesn't tell us anything about why, especially as we know nothing of what real medicines she might have been taking. From this website, it looks as though Llewelyn's case history is rather complex, and she is presently using homeopathy in conjunction with several kinds of real medicine. For medical treatments, you need much better evidence than this to show that they work. In the case of homeopathy, the evidence exists and it shows that homeopathy doesn't work. But there isn't an inset box anywhere that sums up the useful evidence: just one uncontrolled case report.

Given that there is genuinely a debate about whether the NHS should fund homeopathy, it would be good for this debate to be informed by the best currently available evidence. That is exactly what the Observer fails to do: all we get is "he said, she said" appeals to various forms of authority, and an anecdote. No wonder the newspapers are struggling.

Medical Hypotheses row resurfaces

Last year, publishers Elsevier got into trouble with HIV-AIDS researchers, after Medical Hypotheses (an Elsevier journal) published two papers on the subject of AIDS: one by Peter Duesberg claiming that the AIDS epidemic in South Africa was overhyped, and another by Marco Ruggiero suggesting that the Italian health ministry did not believe that HIV was the sole cause of AIDS (blog posts at Bad Science and Respectful Insolence). The papers were deeply flawed, and were retracted by Elsevier pending an investigation into how they were published. The story has now resurfaced in the Times Higher Education Supplement (THES), because:

Prominent Aids researchers contacted Elsevier to object to the article and wrote to the US National Library of Medicine requesting that Medical Hypotheses be removed from the Medline citation database - an act that would exclude it from the mainstream scientific-communication network.

Elsevier have now convened an expert panel to decide on the future of Medical Hypotheses, with conclusions due by the end of 2010.

In fact, there is no great mystery as to how these flawed papers came to be published. Medical Hypotheses is not peer reviewed: instead, decisions on publication are taken solely by the journal's editor, Prof Bruce Charlton. Articles are often accepted within days, or even hours, of being submitted, suggesting there is little or no quality control on what gets published. Prof Charlton defends this process on two grounds: firstly, that there ought to be some outlet for speculative and bizarre ideas that will not be published by mainstream journals. Secondly, that Medical Hypotheses is a successful and influential journal. Here's what he has to say on the comments following the THES article:

The basic facts are that Medical Hypotheses - www.elsevier.com/locate/mehy - is explicitly and proudly editorially-reviewed (i.e. by me - not peer reviewed); aims to publish radical and revolutionary scientific ideas; and it is objectively a successful journal. It makes a profit, the Thomson ISI Impact Factor is 1.416 (much better than average, and rising), and I know from internal sources that there are half a million papers downloaded per year - which is equivalent download usage to the prestigious Journal of Theoretical Biology. Clearly, in spite or because of our policy to publish bold and sometimes bizarre ideas, Medical Hypotheses plays a significant role in medical science. Fact; not opinion. The editorial advisory board currently includes such respected figures as Nobelist Arvid Carlsson http://en.wikipedia.org/wiki/Arvid_Carlsson; Sir Roy Calne http://en.wikipedia.org/wiki/Roy_Calne; Antonio Damasio http://en.wikipedia.org/wiki/Antonio_Damasio and V.S. Ramachandran http://en.wikipedia.org/wiki/Vilayanur_S._Ramachandran . Past editorial advisors have included Sir Karl Popper and Nobelist Sir James Black. *** There are only two possible legitimate outcomes to the current process. Either: 1. Medical Hypotheses could continue as an influential, profitable and well-known editorially-reviewed journal with a radical mission. Or else: 2. The journal could be closed-down altogether, and the title abolished. But it would obviously not be ethically acceptable to launch a new ‘imposter’ journal - with utterly different editorial aims, procedures and personnel; yet retaining the 34 year established title of Medical Hypotheses.

As I keep saying, the impact factor of a journal tells you nothing about its quality. For example, here are three peer-reviewed pseudojournals that repeatedly publish abject nonsense and pdeudoscience, with their impact factors according to Journal Citation Reports:

• Homeopathy: 1.041
  

• Evidence-based Complementary and Alternative Medicine: 1.954
  

• Journal of Alternative and Complementary Medicine: 1.628

The articles in these journals are typically written by quacks, and are cited by other quacks writing in quack journals, giving a high-ish but meaningless impact factor. Perhaps Medical Hypotheses is also highly influential among pseudoscientists?

But the main point here is about radical and controversial hypotheses. I think most people would agree that these have their place in scientific discourse, and there ought to be somewhere to publish them. However, this isn't really what the argument is about. In this case, two fatally flawed papers were published with little or no scrutiny: these papers have potential global health implications. In the case of the Duesberg paper, reviews posted on the Denying AIDS blog show the major problems with the paper. There's a difference between publishing provocative ideas that might inspire new research, and ones that are just demonstrably wrong. While the likes of Peter Duesberg have the right to say what they like, they don't have the right to say it in a MEDLINE-indexed journal. This is not an argument about free speech, it's an argument about the integrity of the scientific literature. There may be a place for journals such as Medical Hypotheses, but there has to be some level of quality control. Otherwise, why should anyone take them seriously?

Bad cycling: New Years Day '10'

Not so much bad science, as bad cycling. Last year I returned to bike racing after more than 10 years away, riding a few club events organised by my club, Stockport Clarion. The triumph of my season was my not-too-bad performance in the club hill climb up the Cat and Fiddle. Since then, I've not really been on the bike too much, because it's dark and cold and I'm a bit of a wuss like that. I did have an ill-fated ride before Christmas, where I fell on a patch of ice just before the Great Stone Road roundabout in Stretford, and then almost got run over by the gritter that was following me. But Stockport Clarion traditionally organises a 10-mile time trial on New Years Day, and I thought it would be a good way to kick-start 2010.

It was an 11am start at Chelford, so I reckoned I had to leave about 9:30. This seemed less and less like a good idea as I headed off into the -4 weather, wearing pretty much every piece of bike kit I own. At least the roads were quiet. There was virtually no wind, but riding at about 15 mph was enough to freeze my hands within a few miles, despite my winter gloves. There was a bloke staggering about and yelling as I went through the edge of Didsbury. Hopefully he was on his way home. There was hardly anyone else about. I rattled through a deserted Alderley Edge and made it to the sign-on about 10:45, just as light snow began to fall. Dunc, the timekeeper, had a good story about the previous year's race, when it had also snowed. Apparently a rider behind him noticed that he was leaving two tyre tracks instead of one: he'd twisted his forks in a crash a few days before.

By the time 11:00 came around, there were still only two riders. It was now -3, and the news was that both Snake Pass and the Cat and Fiddle were closed because of drifting snow. One rider from Poynton had taken one look at the thermometer (-6) and decided to go back to bed. Having come this far, I thought we might as well have a ride anyway. Then the perenially late Will arrived, on a full time-trial machine, to make it three riders.

The Chelford course is a bit of a horror, as time trial courses go. But there are just too many traffic lights around Manchester for a selection of courses, so Chelford is what you get. It's hardly a dragstrip, but not really a sporting course either. It's flattish, narrow and typically busy with fast traffic, but being New Years Day it was relatively quiet. The road surface is pretty heavy and there's a couple of lumps to keep you honest, the main one being the railway bridge in Chelford. That probably doesn't sound too bad, but it's amazing how much effort it costs to maintain speed over the damned thing, with only about a mile to go to the finish.

I started number 2, and within a few hundred yards my feet were frozen. There's a roundabout after about a mile, where Ric the marshal was shivering and pacing about to keep warm, and then the turn is just outside Knutsford. By then I'd already been passed by Will at number 3, and was clearly losing time to Dan at number 1. By the time I got to the turn, I was starting to warm up, but my legs just wouldn't have it over the last 4 miles and I finished in 32:53. That's the slowest '10' I've ever done, a minute slower than the first one I ever did riding as a schoolboy under the banner of Penistone Grammar School. I rode back to the timekeeper's car, and took a drink from my bottle: there was ice in it.

Still, the winning time was 28:39, from a rider who would be doing 24 minute rides in summer, so I couldn't feel too bad. In any case, I didn't have time: to avoid hypothermia, I jumped back on the bike and set off home ASAP.

Hopefully there'll be more bad science in the year ahead (there's certainly no shortage of it about), and perhaps even some good science of my own. Whatever else happens, though, I'm certain that there'll be more bad cycling.

Fighting parliamentary committees...with your mind

You might remember me writing about the Intention Experiment in the past. The Intention Experiment hopes to harness the power of positive thinking to influence events. In one memorable example, they tried to influence the structure of water through the collected brainpower of a large collection of gullible numpties. Despite the experiment having a design likely to massively increase the chance of getting a false positive result, nothing astonishing happened.

Gimpy has now made my day by posting this story about homeopaths concerned that the British parliament's Science and Technology Committee will conclude that there is no evidence in favour of homeopathy. Instead of marshalling the evidence and trying to make a scientific case, the homeopaths are trying an intention experiment to influence the committee in favour of homeopathy.

There's just nothing to add to this: it's shear barking mad lunacy, and you have to think that maybe these people are their own worst enemies.

Tuesday is the wettest day of the week...?

It was a bit disappointing to get back from a week in the field in Sinai, to find my employers at the University of Manchester advertising the Manchester Science Festival with this story. Apparently, Tuesday is the wettest day of the week in Manchester.

Well, every day is rainy in Manchester. Perhaps predictably, if you look at the data this conclusion is based on, you can see that there isn't exactly an astounding difference in rainfall between different days of the week. Certainly, statistically significant differences have not been demonstrated. The research has some interesting things to say about how rainfall patterns seem to have changed over the last 30 years or so: Manchester is somehow managing to get wetter. This is consistent with warmer temperatures, as more water vapour can be moved around when temperatures are higher. This is interesting stuff, and excellent for illustrating local changes in climate for a science festival. So why emphasise that Tuesday is supposedly the wettest day of the week, when the data surely don't convincingly support that? I suppose the university press office thinks that wet Tuesdays are a more interesting story than local climate changes, but I don't think I would agree.

A quick google shows that the story has been picked up by the Express (Why Tuesday is the day you'll most likely need your umbrella) and the Telegraph (Tuesday is the rainiest day), which are clearly based around the press release. I think a better press release might have been headlined "Manchester getting rainier", which is interesting and also has the benefit of being supported by the data.

REF consultation document published

For anyone interested in how research funding is allocated (fascinating stuff, I know), a consultation document on the Research Excellence Framework (REF) is now available here. REF is the mooted replacement for the old Research Assessment Exercise (RAE), the last one of which was conducted in 2008. Enjoy...

Playing the game: the impact of research assessment

Yesterday I was sent this report, produced by the Research Information Network, in conjunction with the Joint Information Systems Committee, and entitled "Communicating knowledge: How and why UK researchers publish and disseminate their findings". The report used a literature review, bibliometric analysis, an online survey of UK researchers, and focus groups or interviews with researchers to look at how and why researchers put information into the public domain. Being an early-career researcher, I'm interested in this sort of thing: I know why I'm publishing and disseminating information, but it's interesting to see why everyone else is doing it. It's also interesting to see the extent to which research assessment in the UK - until recently the Research Assessment Exercise (RAE) and in future the mysterious Research Excellence Framework (REF) - influence the decisions that researchers make. What particularly struck me about the report was the number of times researchers talked about "playing games": the framework of research assessment is seen as a game to be played, with the needs of research being subordinated to the need to put in a good performance. This has important implications for the REF, in which bibliometric indicators are likely to play an important role.

The key point of the report is that there is some confusion among researchers about what exactly it is they're supposed to be doing. There are conflicting and unclear messages form different bodies about what sort of research contributions are valued. The perception is that the only thing that really counts in terms of research assessment is peer-reviewed journal articles. Other contributions, such as conference proceedings, books, book chapters, monographs, government reports and so on are not valued. As a result, the proportion journal articles compared to other outputs increased significantly between 2003 and 2008. A couple of comments by researchers quoted in the report (p.15):

[There is] much more emphasis on peer reviewed journals …Conferences, working papers and book chapters are pretty much a waste of time … Books and monographs are worth concentrating on if they help one demarcate a particular piece of intellectual territory.

There is a strong disincentive to publish edited works and chapters in edited works, even though these are actually widely used by researchers and educators in my field, and by our students.

This is certainly the impression I get from my own field. In fact, I have been advised by senior colleagues to target high-impact journals, rather than, for example, special publications. I have never received any formal guidance on what research outputs are expected of me, but the prevailing atmosphere gives the impression that it's all about journal articles. After publishing a couple of things from my PhD, it took another three years to publish anything from my first post-doc. I worried about that: it seemed that the numerous conferences and internal company reports and presentations I produced over that time counted for nothing career-wise.

The report makes it clear that, in the case of the RAE, it is more perceptios than the reality causing the problem: the RAE rules meant that most outputs were admissible, and all would be treated equally. But it's perceptions that drive the way researchers respond to research assessment. Clearer guidance is needed.

An interesting point brought up by the report is how, when there is more than one author for a journal article, the list of authors is arranged. In my field, authors are typically listed in order of contribution, so I was surprised to find that this is by no means always the case. In some fields, especially in the humanities and social sciences, authors are commonly listed alphabetically. In some cases, the leader of the research group is listed first, in other cases last. And there are various mixtures of listing by contribution, grant-holding and alphabetic order. There is even a significant minority where papers based on work done by students have the student's supervisor as first author! This means that there is no straightforward way of apportioning credit to multiple authors of a paper, something that David Colquhoun has already pointed out. This is a huge problem for any system of assessment based on bibliometrics.

The report also examines how researchers cite the work of other people. Other researcher's work should be cited because it forms part of the background of the new research, because it supports a statement made in the new paper, or as part of a discussion of how the new paper fits into the context of previous research. Crucially, this includes citing work with which the authors disagree, or that is refuted or cast into doubt in the light of the new work (p.30):

Citing somebody often indicates opposition / disagreement, rather than esteem and I am as likely to cite and critique work that I do not rate highly as work I value.

So any system that relies on bibliometric indicators is likely to reward controversial science as much as good science (not that those categories are mutually exclusive, but they don't completely overlap either).

Researchers are perfectly clear that a system based on bibliometrics will cause them to change their publication behaviour: 22% will try to produce more publications, 33% will submit more work to high-status journals, 38% will cite their collaborators work more often, while 6% will cite their competitors work less often. This will lead to more journal articles of poorer quality, a the decline of perfectly good journals that have low "impact", and corruption in citation behaviour. In general, researchers aren't daft, and they've clearly identified the incentives that would be created by such a system.

The report presents a worrying picture of research, and scientific literature, distorted by the perverse incentives created by poorly thought-out and opaque forms of research assessment. It can be argued that scientists who allow their behaviour to be distorted by these incentives are acting unprofessionally: I wouldn't disagree. But for individuals playing the game, the stakes are high. Perhaps we ought to be thinking about whether research is the place for playing games. It surely can't lead to good science.

I get e-mail

Got this today, sent out to academic and academic-related staff in my department:

Dear All,

Please find attached NSS results by Faculty, School and JACS Level 3 subjects. Also included is a mapping document to accompany the JACS report to assist you in understanding which programmes of study are included under each heading. The Word document, 'APPENDIX 06-Surveys - NSS Table EPS.doc' shows the data that will be included in the OPR documentation.

Please note that the data is FOR INTERNAL USE ONLY.

I have no idea what NSS, JACS or OPR mean, so this e-mail makes no sense to me whatsoever. I seem to be getting an increasing number of these things, all with acronyms I've never heard of.

What happens when you don't have peer review

Normally, when a scientific paper is submitted, it is subjected to scrutiny by two or more scientists working in a similar field. Only if the paper gets through this peer review process, and if corrections required by the reviewers have been made, does the paper actually get published. This process is by no means perfect: bad papers slip through, and good papers get blocked by over-zealous reviewers. But there are two examples this week of what can go wrong when papers are not peer reviewed.

Firstly, Ben Goldacre and Respectful Insolence discuss the case of two papers, recently published in Medical Hypotheses, that were so bad they were withdrawn by publishers Elsevier. Given that Elsevier happily publishes Homeopathy, the fanzine of the Faculty of Homeopathy, this should give pause for thought. Medical Hypotheses is a bit of an oddity: it does not send papers out for peer review. Rather, they are approved solely by the editor of the journal, one Bruce Charlton. It appears that many papers are approved within days, sometimes hours, of being submitted, suggesting that there is very little scrutiny of the papers.

The two papers are one by Duesberg et al., and one by Ruggiero et al., both of which seek to deny the magnitude of the AIDS crisis. Seth Kalichman of the Denying Aids blog did an experiment by sending the manuscript out for blind peer review. All three "reviewers" rejected the manuscript on the basis that it was filled with logical flaws and mis-representations of the published literature.


Elsevier says:

This Article-in-Press has been withdrawn pending the results of an investigation. The editorial policy of Medical Hypotheses makes it clear that the journal considers "radical, speculative, and non-mainstream scientific ideas", and articles will only be acceptable if they are "coherent and clearly expressed." However, we have received serious expressions of concern about the quality of this article, which contains highly controversial opinions about the causes of AIDS, opinions that could potentially be damaging to global public health. Concern has also been expressed that the article contains potentially libelous material. Given these important signals of concern, we judge it correct to investigate the circumstances in which this article came to be published online. When the investigation and review have been completed we will issue a further statement. Until that time, the article has been removed from all Elsevier databases. The Publisher apologizes for any inconvenience this may cause. The full Elsevier Policy on Article Withdrawal can be found at http://www.elsevier.com/locate/withdrawalpolicy.

The second example is a paper published in Proceedings of the National Academy of Sciences, amusingly known as PNAS. This is a venerable and respected journal, but it has a little-known wrinkle: members of the National Academy of Sciences are allowed to bypass formal peer review by "communicating" papers for other researchers. This is how the PNAS "Information for Authors" page describes the process:

An Academy member may “communicate” for others up to 2 manuscripts per year that are within the member's area of expertise. Before submission to PNAS, the member obtains reviews of the paper from at least 2 qualified referees, each from a different institution and not from the authors' or member's institutions. Referees should be asked to evaluate revised manuscripts to ensure that their concerns have been adequately addressed. The names and contact information, including e-mails, of referees who reviewed the paper, along with the reviews and the authors' response, must be included. Reviews must be submitted on the PNAS review form, and the identity of the referees must not be revealed to the authors. The member must include a brief statement endorsing publication in PNAS along with all of the referee reports received for each round of review. Members should follow National Science Foundation (NSF) guidelines to avoid conflict of interest between referees and authors (see Section iii). Members must verify that referees are free of conflicts of interest, or must disclose any conflicts and explain their choice of referees. These papers are published as “Communicated by" the responsible editor.

The paper in question is was submitted via this communication process. It was written by Donald Williamson, a retired academic from the University of Liverpool, and suggests that butterflies and caterpillars orginated as different species:

I reject the Darwinian assumption that larvae and their adults evolved from a single common ancestor. Rather I posit that, in animals that metamorphose, the basic types of larvae originated as adults of different lineages, i.e., larvae were transferred when, through hybridization, their genomes were acquired by distantly related animals.

The paper has been criticised on the basis that it contains no supporting data for what is, after all, a fairly extraordinary hypothesis. Not only that, but it turns out that it had previously been rejected by seven different journals.

In both Medical Hypotheses and PNAS, the defence seems to be that there needs to be some mechanism by which speculative ideas that go against current mainstream opinion can be presented and discussed. This seems fair enough, but is anything gained by publishing hypotheses that are not supported by any data, or papers that are logically flawed and contain mis-representations? In both these cases, it seems that the papers would not have been published had they been reviewed properly.

Soviet Manchester

There's an exhibition of maps of Manchester at the John Rylands Library just now, and I'm hoping to have a look at it next week when I take a much-needed break. One of the exhibits in particular is attracting a lot of attention: a map of the city prepared by the Soviets, for use in the event of an invasion of the UK.

It shows the roads - familiar to many Mancunians - which the Soviets felt were wide enough to carry tanks including Washway Road, the Mancunian Way, and Princess Road.

I biked down Washway Road last night on my way home from a training ride, and it's quite hard to imagine columns of Soviet tanks clanking along it towards Stretford. Though I suppose if there ever had been a Soviet invasion, Washway Road would be even more of a post-nuclear wasteland than it currently appears. Apparently, this map is only 35 years old, and it's very strange to think that so recently there were plans for a Soviet Manchester.

Washway Road, yesterday

Society of Homeopaths defends ineffective treatment for condition that kills 2 million children a year...

...surprise!

I'm not going to write a huge amount about this, because it is ably covered elsewhere. But the World Health Organisation finally came out today, and clearly stated that homeopathy should not be used for life-threatening conditions such as AIDS, malaria, TB and childhood diarrhoea.

The Society of Homeopaths (SoH) respond by saying that treating AIDS, malaria and TB would contravene their ethical guidelines. These guidelines are a joke, as you can see by perusing the Quackometer and Gimpy's blog. But apart from that, they say:

The Society of Homeopaths, the UK’s largest body of registered homeopaths, is concerned to learn, in an online article by the BBC (“WHO warns against homeopathy use’), that the World Health Organisation (WHO) has issued caution against the use of homeopathy for childhood diarrhoea following a letter by the charity Sense About Science.

They go on to cherry-pick and distort the research that has actually been conducted on homeopathy for childhood diarrhoea.

Well, who's surprised by that? Regular readers will be aware of just a few of the ways that homeopaths try to distort the evidence that homeopathy doesn't work. The point here is that the SoH, a supposedly professional organisation, is trying to defend a totally ineffective treatment for a disease that kills 2 million children every year. Homeopathy can't do anything to help, and using it instead of proper medical treatment could cost lives. I don't really mind if homeopaths sell ineffective sugar pills to the worried well in Alderley Edge, but this press release is delusional and irresponsible, and reasonable people should have no truck with this sort of dangerous nonsense.

A long trip to a closed pub

Several years ago, I used to do a lot of time trialling. Given that, at my peak fitness, I weighed 8 and a half stone and looked like a bundle of pipe cleaners tied together, my favourite races were always hill climbs. Hill climbs are essentially short time-trials up a hill, and they usually happen at the end of the season in September and October. In a typical season I would ride several around Sheffield, Monsal Head and Curbar Edge being particular favourites. Then in 1999 I went to Canada to study for my PhD, and I didn't race again until this year.

This year I joined Stockport Clarion CC, and I've been riding the odd weekday 10-mile time-trial with no great success. But the event I was really looking forward to was the club hill climb championships. Hill climb courses are typically between several hundred yards and a couple of miles long, and they're usually steep, with gradients of 10-20%. Just to be different, ours is on the Cat and Fiddle road, between Macclesfield and the eponymous pub that stands, hemmed in by peat bog, at the road summit. That's about 6.5 miles of climbing, but at an average gradient of less than 4%. There are steeper railway bridges around here, but there's still about 335 m of height gain involved. The course starts in the outskirts of Macclesfield, opposite a bus stop, and the first 3 miles or so is a fairly steady 4.8%. The next mile and a half is very bendy, contains some short downhill sections, and is nearly flat on average. The final mile and a half takes you up onto Axe Edge Moor proper, climbing at 3.5% or so.

I had never ridden up the climb before, but I had a gameplan of sorts. The first section was the bit where my light weight would give me an advantage, so I would ride hard, but not flat out: the key is not to overdo it, what with there still being 3 miles to go. I would then use the fast section to take a bit of a breather, riding at slightly less than 10-mile pace, since you don't gain much time there anyway. Then it would be eyeballs out over the last mile and a half to the summit. What gears would I use? Hell if I knew. I would figure it out as I went along.

The race was last night, and I actually really enjoyed it. The conditions could scarcely have been better. It was clear and dry, about 18 C, and there was a moderate tailwind. There were only 8 riders, and I was off Number 2, with a slower rider starting one minute before me. I started well, settling in very quickly, and once I had emerged from the trees at the bottom of the climb I could see my minute man almost all of the time. I was clearly gaining on him, and that gave me a psychological boost, as I finally reeled him in on a sharp right-hand bend about a mile and a half in, just before Walker Barn. I was first on the road now, a nice feeling, and my legs were holding up just fine. After Walker Barn, into the fast part of the course, I switched into the big ring, and tried to maintain a slightly-slower-than-10-mile-time-trial pace. My biggest problem here was gear selection. The gradient changes so often and there are so many sharp curves that I felt like I was using nearly every gear on the bike, and I was changing front rings too often: I should have picked one and stuck to it. But I still felt good, and I knew I had something left for the last mile and a half. No strategy involved here; just eyeballs out until the summit. There was a guy out for a training ride in front, and that gave me something to chase. Round the last bend, and there was the welcome sight of the Cat and Fiddle Inn. One last leg-breaking effort, and I was past the timekeeper in 26 minutes dead, gasping like a freshly landed fish. It was about a minute faster than I'd hoped for, so I couldn't have been happier.

It was glorious at the summit, with views across the Cheshire Plain and the Peak District. Unfortunately, the pub is closed on Monday nights, so there was no chance of a swift half before the 25 mile ride back home. At least the first seven or so was downhill.

Publication bias: medical science is looking for it. Are you?

Via the mighty Ben Goldacre's miniblog, here's a study that's just been published in the BMJ on the subject of publication bias. Publication bias typically occurs when studies with positive results are preferentially published over studies with negative results. There are many reasons why this might happen; drug companies don't want to publish negative trials of their drugs for obvious reasons, but there is also a tendency for journals to be uninterested in publishing negative trials, and for busy authors to be reluctant to write them up. There's nothing sexy about a negative trial.

In the BMJ study, Santiago Moreno and colleagues look at anti-depressants. This is a good area to look at, because of the availability of data that was submitted to the Federal Drug Administration in the United States. Legal requirements enforce submission of ALL data to the FDA, so the authors consider the FDA dataset to be unbiased (although not necessarily complete). This unbiased dataset can then be compared to the data that is available in published journal articles.

The comparison is done with our old friend the funnel plot. This plots the standard error for a trial against the size of the effect that the trial found. The authors of the BMJ study added a new twist by contouring the funnel plot for statistical significance: at a glance it can be seen where studies fall in terms of statistical significance.

The results are dramatic. Of the 74 trials registered with the FDA, 23 were not published. In the FDA data, there is a wide spread of results across the contours marking 1%, 5% and 10% levels of significance. When only the published data are considered, there is a clear cut-off at the contour for the 5% significance level, which is typically used in clinical trials to establish statistical significance. That strongly suggests that publication bias is a serious problem in the set of published trials: trials with statistically non-significant results have been systematically excluded.

What this means is that the published literature over-estimates the benefit of anti-depressants. It doesn't show that anti-depressants don't work: meta-analysis of the FDA data still shows a beneficial effect. The point is that the real benefit (as shown by the FDA data) is less than the benefit you would expect if you looked only at the published literature. Anti-depressants work less well than you might think, but they still work.

The problem is, of course, that most of the time we only have the published literature to work with. So the BMJ paper adds a useful visual technique for identifying publication bias as a likely problem, even if we don't have access a bias-free dataset for comparison. There is no doubt that biases exist in published data; the response from medical science, as seen here, is to try to identify and account for these biases. Regular readers will know that the response of CAM research is to manipulate the data in order to pretend that the problem doesn't exist.

I have a bad feeling about this...

I was idly browsing the front page of the Guardian's website, when I really ought to be doing increasingly tedious corrections to a paper, and I saw this. Eczema sufferer Peter Beech, after many years of decreasingly effective steroid treatments, is trying homeopathy. Beech also claims to be a sceptic of alternative medicine.

First things first. Is there any evidence that homeopathy is effective for treatment of eczema? A quick search on Pubmed showed this to be the most recent (2003) relevant review. Incidentally, Pubmed is a freely accessible service, unlike many of the journal databases used by academics, and anyone with an internet connection can do the same thing as me and come up with the presently available evidence in a couple of minutes. Here's what the summary says:

Alternative methods are commonly used in patients with dermatologic diseases, with homeopathy being one of the most common. Homeopathy was developed by Samuel Hahnemann (1755–1843) and is based on the law of similars and the law of infinitesimals. It is a regulatory therapy where high dilutions of particular compounds are thought to induce a counterreaction in the organism. In dermatology, homeopathy is often used in atopic dermatitis, other forms of eczema, psoriasis, and many other conditions. To date, however, there is no convincing evidence for a therapeutic effect. There are only a few controlled trials, most of them with negative results. The few studies with positive results have not been reproduced. Acceptance by the patient seems largely based on counseling and emotional care rather than on objective responses to the homeopathic drugs.

Not particularly convincing, is it?

Beech is at least unlikely to do himself any harm by trying homeopathy. However, real harm can result when people use homeopathy and avoid real medicine, as this tragic case in Australia shows. Nine-month old Gloria Thomas died after her eczema allowed an infection to get out of control. Her parents had been treating her with homeopathy instead of real medicine: they were later convicted of manslaughter by gross criminal negligence.

But the main reason that I have a bad feeling about this, is that even if Beech gets better, it won't prove anything about homeopathy (and the same goes if he doesn't see any improvement, to be fair). This is essentially an uncontrolled case report, and there would be no way of showing that any improvement resulted from the homeopathy, rather than, say, the natural cyclicity of eczema, or the placebo effect. Whatever happens to Beech, it isn't going to trump the evidence from the most recent systematic review of the evidence from clinical trials. So Beech's experiment cannot add anything to what we laughingly call the "debate" over the efficacy of homeopathy.

Still, good luck to him. I hope he does get some relief from his symptoms, but if he does the chances of it having anything to do with homeopathy are vanishingly small.