Medical Hypotheses row resurfaces

Last year, publishers Elsevier got into trouble with HIV-AIDS researchers, after Medical Hypotheses (an Elsevier journal) published two papers on the subject of AIDS: one by Peter Duesberg claiming that the AIDS epidemic in South Africa was overhyped, and another by Marco Ruggiero suggesting that the Italian health ministry did not believe that HIV was the sole cause of AIDS (blog posts at Bad Science and Respectful Insolence). The papers were deeply flawed, and were retracted by Elsevier pending an investigation into how they were published. The story has now resurfaced in the Times Higher Education Supplement (THES), because:

Prominent Aids researchers contacted Elsevier to object to the article and wrote to the US National Library of Medicine requesting that Medical Hypotheses be removed from the Medline citation database - an act that would exclude it from the mainstream scientific-communication network.

Elsevier have now convened an expert panel to decide on the future of Medical Hypotheses, with conclusions due by the end of 2010.

In fact, there is no great mystery as to how these flawed papers came to be published. Medical Hypotheses is not peer reviewed: instead, decisions on publication are taken solely by the journal's editor, Prof Bruce Charlton. Articles are often accepted within days, or even hours, of being submitted, suggesting there is little or no quality control on what gets published. Prof Charlton defends this process on two grounds: firstly, that there ought to be some outlet for speculative and bizarre ideas that will not be published by mainstream journals. Secondly, that Medical Hypotheses is a successful and influential journal. Here's what he has to say on the comments following the THES article:

The basic facts are that Medical Hypotheses - www.elsevier.com/locate/mehy - is explicitly and proudly editorially-reviewed (i.e. by me - not peer reviewed); aims to publish radical and revolutionary scientific ideas; and it is objectively a successful journal. It makes a profit, the Thomson ISI Impact Factor is 1.416 (much better than average, and rising), and I know from internal sources that there are half a million papers downloaded per year - which is equivalent download usage to the prestigious Journal of Theoretical Biology. Clearly, in spite or because of our policy to publish bold and sometimes bizarre ideas, Medical Hypotheses plays a significant role in medical science. Fact; not opinion. The editorial advisory board currently includes such respected figures as Nobelist Arvid Carlsson http://en.wikipedia.org/wiki/Arvid_Carlsson; Sir Roy Calne http://en.wikipedia.org/wiki/Roy_Calne; Antonio Damasio http://en.wikipedia.org/wiki/Antonio_Damasio and V.S. Ramachandran http://en.wikipedia.org/wiki/Vilayanur_S._Ramachandran . Past editorial advisors have included Sir Karl Popper and Nobelist Sir James Black. *** There are only two possible legitimate outcomes to the current process. Either: 1. Medical Hypotheses could continue as an influential, profitable and well-known editorially-reviewed journal with a radical mission. Or else: 2. The journal could be closed-down altogether, and the title abolished. But it would obviously not be ethically acceptable to launch a new ‘imposter’ journal - with utterly different editorial aims, procedures and personnel; yet retaining the 34 year established title of Medical Hypotheses.

As I keep saying, the impact factor of a journal tells you nothing about its quality. For example, here are three peer-reviewed pseudojournals that repeatedly publish abject nonsense and pdeudoscience, with their impact factors according to Journal Citation Reports:

• Homeopathy: 1.041
  

• Evidence-based Complementary and Alternative Medicine: 1.954
  

• Journal of Alternative and Complementary Medicine: 1.628

The articles in these journals are typically written by quacks, and are cited by other quacks writing in quack journals, giving a high-ish but meaningless impact factor. Perhaps Medical Hypotheses is also highly influential among pseudoscientists?

But the main point here is about radical and controversial hypotheses. I think most people would agree that these have their place in scientific discourse, and there ought to be somewhere to publish them. However, this isn't really what the argument is about. In this case, two fatally flawed papers were published with little or no scrutiny: these papers have potential global health implications. In the case of the Duesberg paper, reviews posted on the Denying AIDS blog show the major problems with the paper. There's a difference between publishing provocative ideas that might inspire new research, and ones that are just demonstrably wrong. While the likes of Peter Duesberg have the right to say what they like, they don't have the right to say it in a MEDLINE-indexed journal. This is not an argument about free speech, it's an argument about the integrity of the scientific literature. There may be a place for journals such as Medical Hypotheses, but there has to be some level of quality control. Otherwise, why should anyone take them seriously?

Bad cycling: New Years Day '10'

Not so much bad science, as bad cycling. Last year I returned to bike racing after more than 10 years away, riding a few club events organised by my club, Stockport Clarion. The triumph of my season was my not-too-bad performance in the club hill climb up the Cat and Fiddle. Since then, I've not really been on the bike too much, because it's dark and cold and I'm a bit of a wuss like that. I did have an ill-fated ride before Christmas, where I fell on a patch of ice just before the Great Stone Road roundabout in Stretford, and then almost got run over by the gritter that was following me. But Stockport Clarion traditionally organises a 10-mile time trial on New Years Day, and I thought it would be a good way to kick-start 2010.

It was an 11am start at Chelford, so I reckoned I had to leave about 9:30. This seemed less and less like a good idea as I headed off into the -4 weather, wearing pretty much every piece of bike kit I own. At least the roads were quiet. There was virtually no wind, but riding at about 15 mph was enough to freeze my hands within a few miles, despite my winter gloves. There was a bloke staggering about and yelling as I went through the edge of Didsbury. Hopefully he was on his way home. There was hardly anyone else about. I rattled through a deserted Alderley Edge and made it to the sign-on about 10:45, just as light snow began to fall. Dunc, the timekeeper, had a good story about the previous year's race, when it had also snowed. Apparently a rider behind him noticed that he was leaving two tyre tracks instead of one: he'd twisted his forks in a crash a few days before.

By the time 11:00 came around, there were still only two riders. It was now -3, and the news was that both Snake Pass and the Cat and Fiddle were closed because of drifting snow. One rider from Poynton had taken one look at the thermometer (-6) and decided to go back to bed. Having come this far, I thought we might as well have a ride anyway. Then the perenially late Will arrived, on a full time-trial machine, to make it three riders.

The Chelford course is a bit of a horror, as time trial courses go. But there are just too many traffic lights around Manchester for a selection of courses, so Chelford is what you get. It's hardly a dragstrip, but not really a sporting course either. It's flattish, narrow and typically busy with fast traffic, but being New Years Day it was relatively quiet. The road surface is pretty heavy and there's a couple of lumps to keep you honest, the main one being the railway bridge in Chelford. That probably doesn't sound too bad, but it's amazing how much effort it costs to maintain speed over the damned thing, with only about a mile to go to the finish.

I started number 2, and within a few hundred yards my feet were frozen. There's a roundabout after about a mile, where Ric the marshal was shivering and pacing about to keep warm, and then the turn is just outside Knutsford. By then I'd already been passed by Will at number 3, and was clearly losing time to Dan at number 1. By the time I got to the turn, I was starting to warm up, but my legs just wouldn't have it over the last 4 miles and I finished in 32:53. That's the slowest '10' I've ever done, a minute slower than the first one I ever did riding as a schoolboy under the banner of Penistone Grammar School. I rode back to the timekeeper's car, and took a drink from my bottle: there was ice in it.

Still, the winning time was 28:39, from a rider who would be doing 24 minute rides in summer, so I couldn't feel too bad. In any case, I didn't have time: to avoid hypothermia, I jumped back on the bike and set off home ASAP.

Hopefully there'll be more bad science in the year ahead (there's certainly no shortage of it about), and perhaps even some good science of my own. Whatever else happens, though, I'm certain that there'll be more bad cycling.

Fighting parliamentary committees...with your mind

You might remember me writing about the Intention Experiment in the past. The Intention Experiment hopes to harness the power of positive thinking to influence events. In one memorable example, they tried to influence the structure of water through the collected brainpower of a large collection of gullible numpties. Despite the experiment having a design likely to massively increase the chance of getting a false positive result, nothing astonishing happened.

Gimpy has now made my day by posting this story about homeopaths concerned that the British parliament's Science and Technology Committee will conclude that there is no evidence in favour of homeopathy. Instead of marshalling the evidence and trying to make a scientific case, the homeopaths are trying an intention experiment to influence the committee in favour of homeopathy.

There's just nothing to add to this: it's shear barking mad lunacy, and you have to think that maybe these people are their own worst enemies.

Tuesday is the wettest day of the week...?

It was a bit disappointing to get back from a week in the field in Sinai, to find my employers at the University of Manchester advertising the Manchester Science Festival with this story. Apparently, Tuesday is the wettest day of the week in Manchester.

Well, every day is rainy in Manchester. Perhaps predictably, if you look at the data this conclusion is based on, you can see that there isn't exactly an astounding difference in rainfall between different days of the week. Certainly, statistically significant differences have not been demonstrated. The research has some interesting things to say about how rainfall patterns seem to have changed over the last 30 years or so: Manchester is somehow managing to get wetter. This is consistent with warmer temperatures, as more water vapour can be moved around when temperatures are higher. This is interesting stuff, and excellent for illustrating local changes in climate for a science festival. So why emphasise that Tuesday is supposedly the wettest day of the week, when the data surely don't convincingly support that? I suppose the university press office thinks that wet Tuesdays are a more interesting story than local climate changes, but I don't think I would agree.

A quick google shows that the story has been picked up by the Express (Why Tuesday is the day you'll most likely need your umbrella) and the Telegraph (Tuesday is the rainiest day), which are clearly based around the press release. I think a better press release might have been headlined "Manchester getting rainier", which is interesting and also has the benefit of being supported by the data.

REF consultation document published

For anyone interested in how research funding is allocated (fascinating stuff, I know), a consultation document on the Research Excellence Framework (REF) is now available here. REF is the mooted replacement for the old Research Assessment Exercise (RAE), the last one of which was conducted in 2008. Enjoy...

Playing the game: the impact of research assessment

Yesterday I was sent this report, produced by the Research Information Network, in conjunction with the Joint Information Systems Committee, and entitled "Communicating knowledge: How and why UK researchers publish and disseminate their findings". The report used a literature review, bibliometric analysis, an online survey of UK researchers, and focus groups or interviews with researchers to look at how and why researchers put information into the public domain. Being an early-career researcher, I'm interested in this sort of thing: I know why I'm publishing and disseminating information, but it's interesting to see why everyone else is doing it. It's also interesting to see the extent to which research assessment in the UK - until recently the Research Assessment Exercise (RAE) and in future the mysterious Research Excellence Framework (REF) - influence the decisions that researchers make. What particularly struck me about the report was the number of times researchers talked about "playing games": the framework of research assessment is seen as a game to be played, with the needs of research being subordinated to the need to put in a good performance. This has important implications for the REF, in which bibliometric indicators are likely to play an important role.

The key point of the report is that there is some confusion among researchers about what exactly it is they're supposed to be doing. There are conflicting and unclear messages form different bodies about what sort of research contributions are valued. The perception is that the only thing that really counts in terms of research assessment is peer-reviewed journal articles. Other contributions, such as conference proceedings, books, book chapters, monographs, government reports and so on are not valued. As a result, the proportion journal articles compared to other outputs increased significantly between 2003 and 2008. A couple of comments by researchers quoted in the report (p.15):

[There is] much more emphasis on peer reviewed journals …Conferences, working papers and book chapters are pretty much a waste of time … Books and monographs are worth concentrating on if they help one demarcate a particular piece of intellectual territory.

There is a strong disincentive to publish edited works and chapters in edited works, even though these are actually widely used by researchers and educators in my field, and by our students.

This is certainly the impression I get from my own field. In fact, I have been advised by senior colleagues to target high-impact journals, rather than, for example, special publications. I have never received any formal guidance on what research outputs are expected of me, but the prevailing atmosphere gives the impression that it's all about journal articles. After publishing a couple of things from my PhD, it took another three years to publish anything from my first post-doc. I worried about that: it seemed that the numerous conferences and internal company reports and presentations I produced over that time counted for nothing career-wise.

The report makes it clear that, in the case of the RAE, it is more perceptios than the reality causing the problem: the RAE rules meant that most outputs were admissible, and all would be treated equally. But it's perceptions that drive the way researchers respond to research assessment. Clearer guidance is needed.

An interesting point brought up by the report is how, when there is more than one author for a journal article, the list of authors is arranged. In my field, authors are typically listed in order of contribution, so I was surprised to find that this is by no means always the case. In some fields, especially in the humanities and social sciences, authors are commonly listed alphabetically. In some cases, the leader of the research group is listed first, in other cases last. And there are various mixtures of listing by contribution, grant-holding and alphabetic order. There is even a significant minority where papers based on work done by students have the student's supervisor as first author! This means that there is no straightforward way of apportioning credit to multiple authors of a paper, something that David Colquhoun has already pointed out. This is a huge problem for any system of assessment based on bibliometrics.

The report also examines how researchers cite the work of other people. Other researcher's work should be cited because it forms part of the background of the new research, because it supports a statement made in the new paper, or as part of a discussion of how the new paper fits into the context of previous research. Crucially, this includes citing work with which the authors disagree, or that is refuted or cast into doubt in the light of the new work (p.30):

Citing somebody often indicates opposition / disagreement, rather than esteem and I am as likely to cite and critique work that I do not rate highly as work I value.

So any system that relies on bibliometric indicators is likely to reward controversial science as much as good science (not that those categories are mutually exclusive, but they don't completely overlap either).

Researchers are perfectly clear that a system based on bibliometrics will cause them to change their publication behaviour: 22% will try to produce more publications, 33% will submit more work to high-status journals, 38% will cite their collaborators work more often, while 6% will cite their competitors work less often. This will lead to more journal articles of poorer quality, a the decline of perfectly good journals that have low "impact", and corruption in citation behaviour. In general, researchers aren't daft, and they've clearly identified the incentives that would be created by such a system.

The report presents a worrying picture of research, and scientific literature, distorted by the perverse incentives created by poorly thought-out and opaque forms of research assessment. It can be argued that scientists who allow their behaviour to be distorted by these incentives are acting unprofessionally: I wouldn't disagree. But for individuals playing the game, the stakes are high. Perhaps we ought to be thinking about whether research is the place for playing games. It surely can't lead to good science.

I get e-mail

Got this today, sent out to academic and academic-related staff in my department:

Dear All,

Please find attached NSS results by Faculty, School and JACS Level 3 subjects. Also included is a mapping document to accompany the JACS report to assist you in understanding which programmes of study are included under each heading. The Word document, 'APPENDIX 06-Surveys - NSS Table EPS.doc' shows the data that will be included in the OPR documentation.

Please note that the data is FOR INTERNAL USE ONLY.

I have no idea what NSS, JACS or OPR mean, so this e-mail makes no sense to me whatsoever. I seem to be getting an increasing number of these things, all with acronyms I've never heard of.