Medical Hypotheses row resurfaces

Last year, publishers Elsevier got into trouble with HIV-AIDS researchers, after Medical Hypotheses (an Elsevier journal) published two papers on the subject of AIDS: one by Peter Duesberg claiming that the AIDS epidemic in South Africa was overhyped, and another by Marco Ruggiero suggesting that the Italian health ministry did not believe that HIV was the sole cause of AIDS (blog posts at Bad Science and Respectful Insolence). The papers were deeply flawed, and were retracted by Elsevier pending an investigation into how they were published. The story has now resurfaced in the Times Higher Education Supplement (THES), because:

Prominent Aids researchers contacted Elsevier to object to the article and wrote to the US National Library of Medicine requesting that Medical Hypotheses be removed from the Medline citation database - an act that would exclude it from the mainstream scientific-communication network.

Elsevier have now convened an expert panel to decide on the future of Medical Hypotheses, with conclusions due by the end of 2010.

In fact, there is no great mystery as to how these flawed papers came to be published. Medical Hypotheses is not peer reviewed: instead, decisions on publication are taken solely by the journal's editor, Prof Bruce Charlton. Articles are often accepted within days, or even hours, of being submitted, suggesting there is little or no quality control on what gets published. Prof Charlton defends this process on two grounds: firstly, that there ought to be some outlet for speculative and bizarre ideas that will not be published by mainstream journals. Secondly, that Medical Hypotheses is a successful and influential journal. Here's what he has to say on the comments following the THES article:

The basic facts are that Medical Hypotheses - www.elsevier.com/locate/mehy - is explicitly and proudly editorially-reviewed (i.e. by me - not peer reviewed); aims to publish radical and revolutionary scientific ideas; and it is objectively a successful journal. It makes a profit, the Thomson ISI Impact Factor is 1.416 (much better than average, and rising), and I know from internal sources that there are half a million papers downloaded per year - which is equivalent download usage to the prestigious Journal of Theoretical Biology. Clearly, in spite or because of our policy to publish bold and sometimes bizarre ideas, Medical Hypotheses plays a significant role in medical science. Fact; not opinion. The editorial advisory board currently includes such respected figures as Nobelist Arvid Carlsson http://en.wikipedia.org/wiki/Arvid_Carlsson; Sir Roy Calne http://en.wikipedia.org/wiki/Roy_Calne; Antonio Damasio http://en.wikipedia.org/wiki/Antonio_Damasio and V.S. Ramachandran http://en.wikipedia.org/wiki/Vilayanur_S._Ramachandran . Past editorial advisors have included Sir Karl Popper and Nobelist Sir James Black. *** There are only two possible legitimate outcomes to the current process. Either: 1. Medical Hypotheses could continue as an influential, profitable and well-known editorially-reviewed journal with a radical mission. Or else: 2. The journal could be closed-down altogether, and the title abolished. But it would obviously not be ethically acceptable to launch a new ‘imposter’ journal - with utterly different editorial aims, procedures and personnel; yet retaining the 34 year established title of Medical Hypotheses.

As I keep saying, the impact factor of a journal tells you nothing about its quality. For example, here are three peer-reviewed pseudojournals that repeatedly publish abject nonsense and pdeudoscience, with their impact factors according to Journal Citation Reports:

• Homeopathy: 1.041
  

• Evidence-based Complementary and Alternative Medicine: 1.954
  

• Journal of Alternative and Complementary Medicine: 1.628

The articles in these journals are typically written by quacks, and are cited by other quacks writing in quack journals, giving a high-ish but meaningless impact factor. Perhaps Medical Hypotheses is also highly influential among pseudoscientists?

But the main point here is about radical and controversial hypotheses. I think most people would agree that these have their place in scientific discourse, and there ought to be somewhere to publish them. However, this isn't really what the argument is about. In this case, two fatally flawed papers were published with little or no scrutiny: these papers have potential global health implications. In the case of the Duesberg paper, reviews posted on the Denying AIDS blog show the major problems with the paper. There's a difference between publishing provocative ideas that might inspire new research, and ones that are just demonstrably wrong. While the likes of Peter Duesberg have the right to say what they like, they don't have the right to say it in a MEDLINE-indexed journal. This is not an argument about free speech, it's an argument about the integrity of the scientific literature. There may be a place for journals such as Medical Hypotheses, but there has to be some level of quality control. Otherwise, why should anyone take them seriously?

What happens when you don't have peer review

Normally, when a scientific paper is submitted, it is subjected to scrutiny by two or more scientists working in a similar field. Only if the paper gets through this peer review process, and if corrections required by the reviewers have been made, does the paper actually get published. This process is by no means perfect: bad papers slip through, and good papers get blocked by over-zealous reviewers. But there are two examples this week of what can go wrong when papers are not peer reviewed.

Firstly, Ben Goldacre and Respectful Insolence discuss the case of two papers, recently published in Medical Hypotheses, that were so bad they were withdrawn by publishers Elsevier. Given that Elsevier happily publishes Homeopathy, the fanzine of the Faculty of Homeopathy, this should give pause for thought. Medical Hypotheses is a bit of an oddity: it does not send papers out for peer review. Rather, they are approved solely by the editor of the journal, one Bruce Charlton. It appears that many papers are approved within days, sometimes hours, of being submitted, suggesting that there is very little scrutiny of the papers.

The two papers are one by Duesberg et al., and one by Ruggiero et al., both of which seek to deny the magnitude of the AIDS crisis. Seth Kalichman of the Denying Aids blog did an experiment by sending the manuscript out for blind peer review. All three "reviewers" rejected the manuscript on the basis that it was filled with logical flaws and mis-representations of the published literature.


Elsevier says:

This Article-in-Press has been withdrawn pending the results of an investigation. The editorial policy of Medical Hypotheses makes it clear that the journal considers "radical, speculative, and non-mainstream scientific ideas", and articles will only be acceptable if they are "coherent and clearly expressed." However, we have received serious expressions of concern about the quality of this article, which contains highly controversial opinions about the causes of AIDS, opinions that could potentially be damaging to global public health. Concern has also been expressed that the article contains potentially libelous material. Given these important signals of concern, we judge it correct to investigate the circumstances in which this article came to be published online. When the investigation and review have been completed we will issue a further statement. Until that time, the article has been removed from all Elsevier databases. The Publisher apologizes for any inconvenience this may cause. The full Elsevier Policy on Article Withdrawal can be found at http://www.elsevier.com/locate/withdrawalpolicy.

The second example is a paper published in Proceedings of the National Academy of Sciences, amusingly known as PNAS. This is a venerable and respected journal, but it has a little-known wrinkle: members of the National Academy of Sciences are allowed to bypass formal peer review by "communicating" papers for other researchers. This is how the PNAS "Information for Authors" page describes the process:

An Academy member may “communicate” for others up to 2 manuscripts per year that are within the member's area of expertise. Before submission to PNAS, the member obtains reviews of the paper from at least 2 qualified referees, each from a different institution and not from the authors' or member's institutions. Referees should be asked to evaluate revised manuscripts to ensure that their concerns have been adequately addressed. The names and contact information, including e-mails, of referees who reviewed the paper, along with the reviews and the authors' response, must be included. Reviews must be submitted on the PNAS review form, and the identity of the referees must not be revealed to the authors. The member must include a brief statement endorsing publication in PNAS along with all of the referee reports received for each round of review. Members should follow National Science Foundation (NSF) guidelines to avoid conflict of interest between referees and authors (see Section iii). Members must verify that referees are free of conflicts of interest, or must disclose any conflicts and explain their choice of referees. These papers are published as “Communicated by" the responsible editor.

The paper in question is was submitted via this communication process. It was written by Donald Williamson, a retired academic from the University of Liverpool, and suggests that butterflies and caterpillars orginated as different species:

I reject the Darwinian assumption that larvae and their adults evolved from a single common ancestor. Rather I posit that, in animals that metamorphose, the basic types of larvae originated as adults of different lineages, i.e., larvae were transferred when, through hybridization, their genomes were acquired by distantly related animals.

The paper has been criticised on the basis that it contains no supporting data for what is, after all, a fairly extraordinary hypothesis. Not only that, but it turns out that it had previously been rejected by seven different journals.

In both Medical Hypotheses and PNAS, the defence seems to be that there needs to be some mechanism by which speculative ideas that go against current mainstream opinion can be presented and discussed. This seems fair enough, but is anything gained by publishing hypotheses that are not supported by any data, or papers that are logically flawed and contain mis-representations? In both these cases, it seems that the papers would not have been published had they been reviewed properly.

On peer review

If you spend a lot of time hanging around at sceptical websites, you'll be aware that when evidence is demanded in support of a claim, it is usually demanded in the form of a peer-reviewed journal article. The suggestion is that peer review provides some sort of check on quality: any peer reviewed paper should meet a certain minimum standard. But is this really the case?

First of all, what does it mean to say that an article is peer reviewed? If an article is peer reviewed, it has been checked over by scientists who work in a similar field to the submitted article. A submitted article will usually be sent to two or three reviewers, who will each read the paper and submit a report to the journal editor. The editor will then decide whether the article merits publication.

On the face of it, this would seem to imply that certain standards are being met. But there is some evidence that this isn't necessarily the case. For example, this article (found via Ben Golacre's miniblog) suggests that in Chinese journals, only 5.9-7.7% of supposed randomised controlled trials reported in peer reviewed articles had adequate procedures for randomisation. A lack of adequate randomisation means that there is a good chance of introducing bias into your trial, and it ought to be one of the first things a reviewer would check. While the article specifically addresses trials published in Chinese journals, I don't think there's any compelling reason to think that things are dramatically different in what we laughingly call the west. Anecdotally, anyone who spends time wading through journals as part of their day job will be able to come up with several examples of utterly dreadful papers that should never have been published. This is without looking at pseudojournals, such as those that concentrate on complementary and alternative medicine, where articles on quackery are peer reviewed by other quacks.

So, if peer review can't tell you whether a trial described as randomised is in fact randomised, what can it tell you? Does it really act as any kind of guarantee of minimum quality? I would suggest not.

That is not to say that peer review is useless as it stands. In my fairly limited experience, papers that I have submitted have always been significantly improved by peer review. But surely there's a way of making peer review "fit for purpose", to use the current jargon?

This post was prompted by a discussion at the Bad Science forum, where the idea of applying industrial-style quality assurance to journal articles was raised. This would mean that there would be some sort of checklist that a reviewer would have to go through, and this would be checked to make sure it had been done. It would not be much use to do this informally; there would need to be some formal way of doing it.

In fact, this is not too far from what already happens, in many cases. I've just got a review back in which the reviewers have answered a series of yes/no questions (in addition to their more detailed written comments). For example, "Are the interpretations and conclusions sound and supported by the interpretation provided?", and "Is the paper logically organised?". For the example of trials published in Chinese journals above, there could be a question like "Is the methodology appropriate for testing the specified hypotheses?". Again, there would have to be some checks that this had been adequately done; this is really what the journal editor should be doing. At present, I think the role of the editor is often too weak. They do little more than send out and receive reviews. This is probably not surprising, given that editors are usually working more or less voluntarily and tend to have plenty of other things that they need to do. And it is not always the case: there are many excellent editors who make a lot of effort to engage with the papers they are handling, and the reviewer's comments on them. But if the role of editors were beefed up, such that they spend time formally checking that reviews have been carried out adequately, then peer review might actually provide the quality guarantee that we seem to think it should.

That might require actually paying editors and reviewers for their time. This would be a fairly radical step, but if it led to a professionalisation of the journal reviewing and editing process it would probably be a good thing. And if it led to a reduction in the number, and an increase in the quality, of papers published, that would not be a bad thing either.