It's literature integrity week...

The integrity of the scientific literature is at issue in not one but two news stories this week. Firstly, the 1998 Lancet paper by Andrew Wakefield and several co-authors, which purported to show a link between autism and the MMR vaccination, was retracted [PDF] by the journal. Meanwhile, there is an ongoing brouhaha about a 1990 Nature paper that has some implications for climate change research.

I'm only going to deal with Wakefield here because, you know, I have other things to do. The retraction by the Lancet is really the final nail in the coffin of Wakefield's paper, which has been heavily criticised for numerous reasons. Of the thirteen authors, ten had previously retracted the interpretation that there was any link between autism associated with gastro-intestinal problems and the MMR vaccine. Only Wakefield himself and one P Harvey refused to sign up to this: the remaining author could not be contacted. It has also been known for some time that the methodology of the paper was flawed. So what suddenly changed this week?

For some time the General Medical Council (GMC) has been investigating Wakefield and two colleagues, Simon Murch and John Walker-Smith, over allegations relating to their research on autism. The GMC findings were released on January 28th, and can only be described as devastating. You can find the whole thing here [PDF], or there's a good summary at Respectful Insolence.

The GMC did not look at whether the research findings were right or wrong: rather they looked at the research methodology. Essentially, the major problems were these:

1. Wakefield had an undisclosed conflict of interest, because he was being paid by lawyers whose clients believed their children had been harmed by the MMR vaccine.

2. Wakefield ordered invasive diagnostic tests, including colonoscopies and lumbar punctures, that were unnecessary and not in the children's interests, and he had neither the required ethical approval nor the requisite expertise to order those tests.

3. Wakefield obtained blood samples at his son's birthday party by paying children £5 each. The GMC panel described Wakefield's actions as comprising "callous disregard for the distress and pain the children might suffer".

4. The 12 children were described as having been "consecutively referred", but this was not true.

Heres what the Lancet had to say:

Following the judgment of the UK General Medical Council’s Fitness to Practise Panel on Jan 28, 2010, it has become clear that several elements of the 1998 paper by Wakefield et al1 are incorrect, contrary to the findings of an earlier investigation.2 In particular, the claims in the original paper that children were “consecutively referred” and that investigations were “approved” by the local ethics committee have been proven to be false. Therefore we fully retract this paper from the published record.

So that's that. The problem with the paper is not that it was wrong; that has been known for some time. In fact, simply being wrong would not be a reason to retract the paper. Science often progresses by building on papers that were not quite right. No, the problem with the Wakefield Lancet paper was that it was fraudulent, unethical and incompetent, as well as being wrong. As a result, the takeup of MMR has fallen below the ~95% level at which herd immunity is maintained, and measels has once again been declared endemic in the UK. That's why you should try to avoid publishing fraudulent and unethical research. We'll let you off if your research is merely wrong, as long as it was honestly and competently wrong.

Wakefield eh, what a hero? Unfortunately, as we'll see in my next post, some people in the most unexpected places still believe in the Cult of Andy...

What do bibliometrics actually add to research evaluation?

Firstly, the reason that I haven't posted in an age is that I've been in Norway, interpreting seismic data for the new project I'm working on. Hopefully I can now post a bit more regularly, as I should actually be in Manchester for a few consecutive weeks, for the first time this year.

Regular readers will know that I like to whinge about the increasing use of statistical indicators (bibliometrics) to evaluate research performance. Previously in England, research performance has been evaluated by the Research Assessment Exercise, a cumbersome and involved system based around expert peer review of research. Currently, HEFCE (the body that decides how scarce research funding is allocated to English universities) is looking into replacing this with a cumbersome and involved system based around bibliometrics and "light-touch" peer review. To this end, a pilot exercise using bibliometrics and including 22 universities has been underway. An interim report on the pilot is now available.

Essentially, three approaches have been evaluated:

i) Based on institutional addresses: here papers are assigned to a university based on the addresses of the the authors, as stated in the paper. This would be cheap to do, as it would need no input from the universities.

ii) Based on all papers published by authors. In this approach, all papers written by staff selected for the 2008 RAE were identified. This requires a lot of data to be collected.

iii) Based on selected papers published by authors. Again, this approach used all staff selected for the 2008 RAE, but only used the most cited papers.

For each approach, the exercise was conducted twice: once using the Web Of Science (WoS) database, and once using Scopus. The results were then compared with those from the 2008 RAE.

Well, the results are interesting, if you like this sort of thing. It is clear that the results can be very different from those provided by the RAE, whichever method was used, although the "selected papers" method tends to give the closest results. It is also notable that the two different databases give different results, sometimes radically so; Scopus seems to consistently give higher values than WoS. Workers in some fields complained that they made more use of other databases, such as the arXiv or Google Scholar (it's worth noting that the favoured databases are proprietary, while the arXiv and Google Scholar are publically accessible).

In general, the institutions involved in the pilot preferred the "selected papers" method, but it seems that none of the methods produced particularly convincing results. According to the report (paras 66 and 67):

In many disciplines (particularly in medicine, biological and physical sciences and psychology), members reported that the ‘top 6’ model (which looked at the most highly cited papers only) generally produced reasonable results, but with a number of significant discrepancies. In other disciplines (particularly in the social sciences and mathematics) the results were less credible, and in some disciplines (such as health sciences, engineering and computer science) there was a more mixed picture. Members generally reported that the other two models (which looked at ‘all papers’) did not generally produce credible results or provide sufficient differentiation.

One of the questions here is what is meant by "reasonable" or "credible" results? The institutions involved in the pilot seem to assume that the best results are the ones that most closely match those of the RAE. I suspect this is because the large universities that currently receive the lion's share of research funding are not going to support any system that significantly changes the status quo.

The institutions involved in the pilot seem to think that bibliometrics would be most useful when used in conjunction with expert peer review. From the report:

Members discussed whether the benefits of using bibliometrics would outweigh the costs. Some found this difficult to answer given limited knowledge about the costs. Nevertheless there was broad agreement that overall the benefits would outweigh the costs – assuming a selective approach. For institutions this would involve a similar level of burden to the RAE and any additional cost of using bibliometrics would be largely absorbed by internal management within institutions. For panels, some members felt that bibliometrics might involve additional work (for example in resolving differences between panel judgements and citation scores); others felt that they could be used to increase sampling and reduce panels’ workloads.

According to the interim report, the "best" results (i.e. those most closely matching the results of the RAE) were obtained using a methodology that will have a similar administrative burden as the RAE. Even then the results had "significant discrepancies". So, if the aim of the pilot was to get similar results to the RAE with a lesser administrative burden, it seems that the pilot exercise has failed on both counts. So if bibliometrics don't seem to add much to the process, it's worth considering what they might take away. For which, see my previous post...

Bloody Elsevier

Some time ago, I had a paper on normal fault evolution in the Gulf of Suez accepted for publication in the Journal of Structural Geology. This is an Elsevier journal, and the paper duly went off to the Elsevier production people to be published. Now, one of the figures in the paper is a large and spectacularly detailed geological map of the study area, done in the late 1990s by my co-author and former University of Manchester post-doc Ian Sharp. This is an excellent piece of work in itself, and it had never been published; we decided that this paper would be a good place to finally publish it. The level of detail on this map is such that we wanted to reproduce it in colour, at A3 size. We knew that this would cost money, but the industrial sponsors of the work were happy to cover the costs.

After a long and generally fruitless attempt at corresponding with the Elsevier production department (which has been outsourced to India, incidentally), I finally received a PDF proof of the paper in which the geological map was reproduced at A3 size. All well and good. Until the final version of the paper was published [paywall: for God's sake, don't pay $31.50 for this...if you really want a copy, e-mail me and I'll send you a PDF], and the map was back to A4 size, with much of the fine detail lost as a result.

Gah!

Now, surely it isn't on for Elsevier to unilaterally make changes to an article without consulting the authors about it. I know some people who have been involved in editing this journal, and it seems they are unhappy with how it is being run by Elsevier. As Dr Aust points out, companies like Elsevier charge large amounts of money for papers, in just about the only example of publishing in which the authors don't want to be paid for producing all the content. Elsevier makes massive profits out of journal publishing, gets to hide all of the content behind ridiculous paywalls, and doesn't even make a particularly good job of the journal production. There must be a better way.

Does the REF add up to good science?

The RAE (Research Assessment Exercise) results from the 2008 were published back in December. You might have noticed this from the number of university websites that could be found frantically spinning the results. My very own University of Manchester, for example, is claiming that Manchester had broken into the “golden triangle” of UK research, that is, Oxford, Cambridge and institutions based in London. It seems that depending on the measure you pick, we’re anywhere between third and sixth place in the UK. Clearly these are excellent results, but whether we’re really up there with the Oxfords, Cambridges, Imperials and UCLs of the world I’m not sure.

In any case, that was the last ever RAE. It has been a fairly cumbersome process, involving expert peer review of the research contribution of research institutions, that has been a real burden on the academics who have had to administer it. I’m sure there are few who will mourn its passing. Now the world of English academia is waiting, like so many rats in an experimental maze, to find out what will replace the RAE. The replacement will be a thing called the Research Excellence Framework, or REF, and at this stage exactly what it will involve is fairly sketchy. However, it will be based on the use of bibliometrics (statistical indicators that are usually based on how much published work is cited in other publications) and “light-touch peer review”.

What kind of bibliometric indicators are we talking about? Last year HEFCE (the Higher Education Funding Council for England, the body that evaluates research and decides who gets scarce research funding) published a “Scoping study on the use of bibliometric analysis to measure the quality of research in UK higher education institutions” produced by the Centre for Science and Technology Studies at the University of Leiden, Netherlands. I’ve spent a fair amount of time reading through this, and in some ways I was encouraged. It’s clear that some thought has gone into creating bibliometric indicators that are as sensible as possible: I was dreading a crude approach based around impact factors, which have already done so much damage to the pursuit of good science. The authors of the “scoping study” came up with an “internationally standardised impact indicator”: I will abbreviate this as ISII for concision. The ISII takes the average number of citations for publications for the academic unit you are interested in (this might be a research group, an academic department or an entire university), and divides it by a weighted, field-specific international reference level. The reference level is calculated by taking the average number of citations for all publications in a specific field: if the publication falls under more than one field (as many will in practice), the reference level can be calculated as a weighted average of the number of citations generated by publications in all the fields in question. So, if the ISII for your research group comes out as 1, you’re average, if above 1, better than the average, and if below 1, worse than the average. The authors of the scoping study say that they regard the ISII as being “the most appropriate research performance indicator”, and suggest that a value of >1.5 indicates a scientifically strong institution. They also suggest a threshold of 3.0 to identify research excellence. It seems that the HEFCE is expecting to adopt the ISII as the main research performance indicator, according to their FAQs, where they say “We propose to measure the number of citations received by each paper in a defined period, relative to worldwide norms. The number of citations received by a paper will be 'normalised' for the particular field in which it was published, for the year in which it was published, and for the type of output”. However, they are still deciding what thresholds they will use to decide which institutions are producing high-quality research.

All well and good. If you insist that bibliometric indicators are necessary, this is probably as good a way as any of generating those data. However, there are some problems here, as well as philosophical difficulties with the entire approach.

Firstly, what is it we are trying to measure? In theory, what HEFCE wants to do is evaluate research quality. But the ISII does not directly measure research quality. Like any indicator based on citation rates, it is measuring the “impact” of the research: how many other researchers published papers that cited the research. It ought to be clear that while this should reflect quality to some degree, there are significant confounding factors. For example, research that is done in a highly active topic is likely to be cited more than research in which fewer groups are working. This does not mean that work in less active topics is of intrinsically lower quality, or even that it is less useful.

Secondly, there is an assumption that the be-all and end-all of scientific research is publication in peer-reviewed journals that are indexed in the Web of Science citation database published by Thomson Scientific. This a proprietary database that lists articles in the journals that it indexes, and also tracks citations. Criteria for journals to be included are not in the public domain (although the scoping report suggests these are picked based on their citation impact, p. 43). A number of journals that I would not consider to be scientifically reputable are included. For example, under the heading of Integrative and Complementary Medicine, the 2007 Journal Citation Reports (a database that compiles bibliometric statistics for journals in the citation database) includes 12 journals, including Evidence Based Complementary and Alternative Medicine (impact factor 2.535!) and the Journal of Alternative and Complementary Medicine (impact factor 1.526). This reinforces the point made above: it would be possible to publish outright quackery in either of these journals, have it cited by other quacks in the quackery that they publish, and get a respectable rating on the ISII. The ISII can’t tell you that this is a vortex of nonsense: it only sees that other authors have cited the work. It is also true that not all journals are included in the citation index: for example, in my own field the Bulletin of Canadian Petroleum Geology fails to make the cut, although it has always published good quality research. Although the authors of the scoping report make clear that it is possible to expand bibliometrics beyond the citation database, this will take much more effort and it seems that HEFCE will not take this route. So we will be relying on a proprietary and opaque database to make decisions on future research funding. A further point is that it is not clear how open access publications will be incorporated in the citation index: in principle there is no reason that this can’t happen, but can we be sure it will?

Thirdly, there is the assumption that research output can only be evaluated in terms of published articles in peer-reviewed journals. I’m not sure that this accurately reflects the actual research output of many scientists. For example, most of us put a lot of effort into presentations at scientific conferences, chapters in books, or government reports that will never make it into a citation database. This has become a problem for things like, in my own field, the special publications of the Geological Society of London. These are volumes that collect recent research on specific topics, and they generally contain excellent research. But they aren’t included in citation databases and they have no impact factor. This has led to a lack of interest in publishing results in these special publications, because they don’t tick the right boxes in terms of publication metrics. This is surely a bad thing. A similar problem occurs with things like government open-file reports. These are not, in general, pieces of world-class, cutting edge research. But that does not mean that they are useless or that they have no value. For example, good regional geological work can allow mineral exploration to be better targeted, benefiting the local economy. Yet that kind of work is ignored in a framework that only considers journal articles: HEFCE says only that “We accept that citation impact provides only a limited reflection of the quality of applied research, or its value to users. We invite proposals for additional indicators that could capture this”. To me, research quality and value cannot be measured by bibliometric indicators. It can only be evaluated by reading the research, understanding its context within the totality of pre-existing research, and understanding how it contributes to new understanding. That is, it can only be evaluated through peer review.

Which brings me to my fourth point; there are some questions about the role of peer review within the REF. HEFCE says that “the scoping study recommends that experts with subject knowledge should be involved in interpreting the data. It does not recommend that primary peer review (reading papers) is needed in order to produce robust indicators that are suitable for the purposes of the REF”. However, I’m not convinced that this accurately summarises what is written in the scoping report, which says “In the application of indicators, no matter how advanced, it remains of the utmost importance to know the limitations of the method and to guard against misuse, exaggerated expectations of non-expert users, and undesired manipulations by scientists themselves…Therefore, as a general principle we state that optimal research evaluation is realised through a combination of metrics and peer review. Metrics, particularly advanced analysis, provides the tools to keep the peer review process objective and transparent. Metrics and peer review both have their strengths and limits. The challenge is to combine the two methodologies in such a way that the strengths of one compensates for the limitations of the other”.

Finally, there is a hint of conflict of interest in the preparation of the scoping report by the Centre for Science and Technological Studies: according to their website, the centre is involved in selling "products" based on its research and development in the area of bibliometric indicators. Their report in favour of bibliometric indicators might allow them to drum up significant business from HEFCE.

At present, the proposals for the REF are at a fairly early stage, but the use of bibliometric indicators seems to be entrenched, and there will be a pilot exercise on bibliometric indicators this year. However, this is based on “expert advice” that consists of a single report from an organisation that makes money by creating bibliometric indicators. While academia in general might welcome the proposals on the grounds that they will be less burdensome than the RAE and give everyone more time to do research, I don’t think many academics will be kidding themselves that the bibliometric indicators involved actually tell us much about research quality and usefullness.

Pediatrics publishes a critique of an "anti-vaccine" book

I've tended not to write anything about vaccines, partly because it's a long way from my field of expertise, and partly because it brings out the internet loons in greater numbers than almost anything else. Still, I ran across this via Ben Goldacre's miniblog, and it overlaps with my interest in scientific publishing: Pediatrics, the official journal of the American Academy of Pediatrics, has published an article by Paul Offit and Charlotte Moser. The article is highly critical of "The Vaccine Book: Making the Right Decision for Your Child", by Dr. Robert Sears, also known in media-friendly mode as Dr. Bob in his role as one part of "America's family of pediatricians". This is something that I haven't seen before: the journal of a learned society publishing a rebuttal of a popular health book. The article goes through Dr. Sears's book, and picks out misleading statements, flawed logic and questionable assumptions, particularly concentrating on the alternative vaccine schedules suggested in the book.

It seems unlikely that worried parents will pick up this article; they are much more likely to read the book. But perhaps by putting these arguments in one place in Pediatrics, it makes it easier for pediatricians themselves to counter the arguments against vaccines that they hear from parents. It might even be possible to give parents the article to look through. And many of the arguments in Dr. Bob's book ARE arguments against vaccines, in spite of the protestations of Dr. Bob himself. The paper is not written in overly technical language, although it does contain a lot of references that are unlikely to be easily available to the average concerned parent. It gives pediatricians a chance to say that actually mainstream scientific opinion does not agree with Dr. Bob. This is unlikely to do much to sway those who've drunk the anti-vaccine kool-aid and think that Dr. Offit is an unscrupulous shill of big pharma, willing to cause lasting harm to untold numbers of children in order to make money (just do a google search to see what I mean). But it might sway sensible people who have inevitably picked up doubts from the unbelievably incompetent reporting of the science surrounding vaccines in the media.

Perhaps it could be worth putting a similar article debunking the myths on MMR into something like the BMJ or the Lancet? Or perhaps something similar already exists and I've missed it?

Academic shenanigans

This is the sort of thing I get up to when I really ought to be producing original research, but who can resist a vaguely pointless semantic argument?

Bad geology: another pseudojournal

Generally, my own field of geology is not somewhere you find a lot of pseudo-science. So it's nice to be able to comment on some, via the folk at Answers Research Journal. Thanks to Michelle for the heads-up. The journal contains a paper on 'catastrophic granite formation', an attempt to show that granites can form very quickly, quick enough to be consistent with what creationists call 'flood geology', the theory that most of the geology we see was formed in the Genesis flood.

The author is one Andrew Snelling, proud holder of a doctorate in geology from the University of Sydney. It seems that Snelling is happy to use the conventional geological column when working as a consulting geologist, but happy to disregard it when propagandising in favour of creationism.

The 'journal' claims to be "a professional, peer-reviewed technical journal for the publication of interdisciplinary scientific and other relevant research from the perspective of the recent Creation and the global Flood within a biblical framework". This raises some questions. Who are the reviewers for the journal, and who reviewed this article? In geology it is standard practice for reviewers to be acknowledged in published papers; reviewers usually identify themselves to authors, unless there are strong reasons not to do so. No reviewers are identified in the article: in fact there are no acknowledgments at all. In the instructions for authors [PDF], prospective authors are asked to suggest at least three referees. The guidelines state that comments will be solicited by at least three reviewers. I wonder if only reviewers nominated by the authors are used? Certainly it's hard to imagine mainstream geoscientists agreeing to review for the journal, or agreeing to the publication of any articles on 'flood geology'.

Why is the journal restricted to "research from the perspective of the recent Creation and the global Flood within a biblical framework"? The instructions for authors suggest that work will be rejected if it is not "formulated within a young earth-young universe framework". This would be like, for example, a physics journal refusing to accept papers that were not formulated within a string-theory framework. No real scientific journal would restrict itself to publication of work that addresses a single hypothesis, because that wouldn't be science.

The journal is claimed to be a "technical journal", but there's a lot of language in the article that suggests otherwise. For example, in the introduction, Snelling writes that "Each recognizably distinctive granite mass, the boundary of which can be traced on the ground, is marked as a separate geologic unit called a pluton on a geologic map". In a section headed 'Magma Principles', Snelling writes that "The molten material which flows from volcanoes is known as lava and cools to form volcanic rocks. So lavas must be molten rocks; that is, they were originally rocks that melted deep inside the earth underneath volcanoes. When deep inside the earth, these molten rock materials are called magmas because they are slightly different in composition and physical properties due to the steam and gases they have dissolved in them that erupt separately from the lavas through volcanoes". There's nothing in particular wrong with these statements, but they read much more like excerpts from a GCSE level textbook than prose from a technical journal. This sort of basic information is never included in real geological journals: the assumption is that those who read them are already familiar with the basics. The article reads much more like a piece written for a general audience.

What of the actual science? It's actually not too bad in the main. The issues surrounding the production and emplacement of granite are quite well set out. The point of the article is that granites could be created and emplaced within a timescale of 6000-7000 years. This seems like a reasonable conclusion, at least in some cases, though I'm not convinced that it applies to every intrusive granite body on the planet. The problem is that Snelling then jumps to the conclusion that this is consistent with the Genesis account of creation. At this point, the article is no longer talking about science, but is trying to provide evidence for an answer that has already been decided on. The argument that the earth is a lot older than 7000 years does not depend on granites taking more than 7000 years to form and become emplaced. It relies on pretty much everything we know about geology, but particularly on radioisotopic dating. How does Snelling get around this? He simply states that isotopic dating is in gross error because it fails to account for the acceleration of decay. This is a load of rubbish, as you can see here.

The dead giveaway, though, is the section devoted to 'Evidence from radiohalos'. A radiohalo is supposedly a zone of crystal damage surrounding a crystal that contains radioactive substances, for example zircon crystals that are relatively common in granites. Snelling claims to be able to identify radiohalos associated with decay of three polonium isotopes. The argument is that the isotopes of polonium have very short half lives: 3.1 minutes (²¹⁸Po), 164 microseconds (²¹⁴Po), and 138 days (²¹⁰Po). Therefore cooling of the granite has to be rapid, because otherwise the polonium would decay too rapidly to form halos. Not to put too fine a point on it, this is drivel. There's a useful summary of why it's drivel here.

Snelling has come to an initially reasonable conclusion about the potential rapidity of formation of some granite bodies, and then extrapolated wildly to suggest that this is the case for all igneous rocks, and that most of what we know about geology is wrong. Still, he is happy to ignore that conclusion in his consultancy work and publications in real scientific journals. Like Homeopathy, Answers in Genesis is a pseudojournal: it's designed to look superficially like it is publishing real science "from a different perspective", but it is not. It is a creationist propaganda organ, and it is publishing creationist propaganda.

My own bad science

Well, that's science for you. A paper that I worked long and hard on was sent out into the world, and I've got it back torn to shreds and covered in red ink. Yes, my paper was rejected. These things happen.

One of the reviewers thought the paper was OK, while the other one thought it had some serious flaws. The editor agreed with the second reviewer. I can't say that I'm happy about it, but I have to admit that the criticisms are not unfair. Still, I think the observations in the paper are useful and interesting, so I'm going to mount a salvage operation, and attempt to re-submit the paper.

It's not nice to have your work criticised, but of course, that's how science works. You submit your work for rigorous examination by other experts in your field, and only the best stuff makes it through. If the paper does eventually get published, it will be much stronger as a result.

At least, that's what I'm telling myself, but I can't pretend that it doesn't feel like a kick in the teeth...

How wrong is it to speculate?

Just some thoughts that have been festering since the Hurlbert and Ling paper in Current Biology, the one that showed that women preferred pink hues when compared with men. I wrote about it, Ben Goldacre wrote about it, and I think more or less everyone with an internet connection wrote something about it somewhere.

The general gist of opinion on the article seemed to be that it was an example of bad science, because of the author's speculation that their results had some sort of evolutionary psychology explanation, perhaps that women needed to pick out red berries against a green background in the days when the men were out killing stuff and hauling it back to the cave. I would agree that the data presented in the paper don't support that interpretation. On the other hand, the authors clearly identified that hypothesis as speculative. I think it's worth asking whether speculation can have a place in scientific literature.

In my view, the answer is yes. In the case of the Hurlbert and Ling paper, data was presented showing that women prefer pink hues. The authors indulged in some speculation as to why this might be the case. It should now be the task of scientists to try and devise studies that could refute that speculation. It follows the pattern make observations, formulate hypothesis to explain observations, make more observations to test hypothesis. So I think the speculation in the paper is scientifically defensible. In this context, Kaj Sand-Jensen's famous paper on "How to write consistently boring scientific literature" is always worth a look, especially the section headed "remove most implications and every speculation". Scientific caution is generally sensible, but if taken too far it can mean that possible leads are not followed up.

The problem with that approach is the way such studies get reported in the media. As we've seen, most of the stories on the paper suggested that the authors had confirmed an evolutionary psychological explanation for gender-based differences in colour preference. In fact, they had simply showed that gender-based differences in colour preference exist. With scientists getting brownie points for media engagement, it is inevitable that the differences between results and speculation get lost.

Denouncing your own publications as 'crap'

What happens when you have an academic system that prioritises quantity of publications over everything else? What happens when drug companies need eminent scientists to conclude that their stuff works? Predictably, you get perverse results. Like this one, detailed here and here (and all over the interweb).

In this case, it seems as if the putative author of the ghost-written work is happy to disown it. I can imagine similar cases where the author would be happy to put the resulting bogus publication on their CV, because it's nice to have a career. It's a vortex of shit, folks.

More on Rao et al. homeopathy tomfoolery

If anyone out there has read the stuff on this blog (here and here) about the recent special issue of the pseudojournal Homeopathy dealing with the memory of water, and the Rao et al. paper in it, they'll know there are some serious problems with the paper. If you're really interested in finding out just how rubbish this supposedly peer-reviewed paper is, I recommend the discussions of it on the JREF forum. Now Rolfe, one of the contributors to the forum, has drafted a letter to Peter Fisher, the editor of Homeopathy, pointing out the errors and problems with the presented UV-vis data. This includes my observations of graph tomfoolery, as well as many other points picked up by JREF contributors Rolfe, JJM and Pipirr. We haven't made any comment on the Raman spectroscopy data, which seems to be equally problematic (see here, look for comment No. 11). The letter is available here.

Assuming there's any response at all, watch this space...

Pseudoscience and impact factors

Just out of interest, I looked up the journal 'Homeopathy' (see below) in ISI's Journal Citation Reports. It isn't listed, which cheered me up a bit. However, the Journal of Alternative and Complementary Medicine is listed, and has a impact factor of 1.104. In comparison, the Canadian Journal of Earth Sciences, a solid and respected regional earth science journal, has an impact factor of 0.943. So perhaps it would be a better career move to try to publish some pseudoscience that purports to prove the efficacy of homeopathic treatments, than to try to produce a solid and useful piece of earth science research.

I'm sure this is not the intention, but including such pseudo-journals in these rankings has the effect of making them seem respectable. After all, the numbers don't lie, right?

A cracking example of a pseudojournal

It turns out there is a journal called Homeopathy. It looks very like a proper journal. It is peer reviewed, it is indexed in the Web of Science, and it is published by Elsevier. If you look at the page layout, it looks not unlike the Journal of Structural Geology; it has that journal format to it. But this is definitely not a scientific journal. It is a pseudo-journal that publishes pseudo-science, and its reviewers are pseudo-scientists.

The journal has recently published a special issue on the memory of water (as so often, Ben Goldacre's site is responsible for the heads up). This is an important topic for homeopaths, because it provides a way in which homeopathy might work. The water that contains the homeopathic 'remedy' is diluted such that none of the 'remedy' can possibly remain, so homeopaths fall back on the notion that the water somehow structurally 'remembers' the remedy. Some proper scientific work that got published in Nature suggested that the actual length of time water can structurally remember anything is on the order of 50 femtoseconds, so this is probably a non-starter. Those brave homeopaths have taken on the blinkered scientific establishment and had a go, nonetheless.

Oh, it's a shambles.

In it, Martin F Chaplin has a paper entitled 'The Memory of Water: an overview'. But this paper contains no evidence of water having a memory at all. Chaplin's examples of 'evidence for water memory' have nothing to do with water 'remembering' a solute that has been diluted out of existence. For example, 'human taste is quite capable of telling the difference between two glasses of water, processed in different ways (eg one fresh and one undrunk for several days.)' Yes, that’s because they will contain different amounts of dissolved gases from the atmosphere. What would be really interesting is if you could taste the difference, say, between two 30C dilutions of different solutes prepared under identical conditions. Chaplin also mentions a 'memory effect' related to clathrate formation. Again, this isn't relevant to whether a solute that is diluted out of existence has any effect on the water structure. And again, there is a mention of 'slow equilibration' of solutions. But this isn’t relevant to cases where no solute is present.

Then there are the 'experimental papers'. The paper by Elia et al. is so poorly organised and written that it isn't clear what they did. However, they do say 'It is important to emphasise that, from the studies so far conducted, we cannot derive reproducible information concerning the influence of the different degrees of homeopathic dilution or the nature of the active principle (solute) on the measured physicochemical parameters'. What this seems to say is that none of the results they present are reproducible, so they can't really be considered to represent results in any scientifically meaningful sense.

The paper by Rao et al. seems to have some serious problems with the reported spectroscopy, which is dealt with here. My personal favourite section is where the paper suggests that 'it can be argued' that the succusion process (shaking of the homeopathic remedy between dilution steps) could produce pressures of up to 10 kbar. Perhaps this point can be argued, but the authors don't make any attempt to argue it. They also state that grinding in a mortar and pestle can produce pressures up to 20 kbar, for which they give a reference to a paper by Dachille and Roy, which is number 22 in their reference list. In fact, their reference no. 22 is to a Bates et al. paper in Science, which looks at high-pressure forms of Germanium. The paper contains no mention of mortar and pestle at all. This sort of bad referencing would be picked up by any thorough review. But anyway, 10 to 20 kbar? Now, as a geologist, I know that pressure in the crust increases by approximately a third of a kbar per kilometre, depending on rock density. Thus 10 kbar is equivalent to a depth of 30 km, and 20 kbar is equivalent to a depth of 60 km. So if I shake up a homeopathic solution, I can get a pressure equivalent to 30 km of rock, and if I grind up some shale in a mortar and pestle, I end up with a blueschist facies metamorphic rock. 10 kbar is on the order of 10,000 atmospheres. This claim seems extremely unlikely, and again is the sort of thing that would be picked up by any half-way competent reviewer.

The paper by Louis Rey seems interesting, but there are issues with reproducibility. None of the graphs contain error bars. Although the author says that his results have been replicated by another lab, he doesn't seem to have tried to replicate them himself. It is just stated that some graphs look different, without any discussion of how statistically significant the differences might be. Again, it's difficult to know whether the paper shows anything at all.

Then there is a paper by Vybiral and Voracek, which looks at some interesting physics of water. However, the authors conclude that the 'autothixotropy' effect that they observe is absent when they use de-ionised water. Thus their effect is related to ions in the water, and not any 'water memory' effect which would be due to ions that were previously in the water, but are no longer there.

After that there are some 'theoretical' papers, which contain no new evidence, and some bizarre excursions into the world of quantum mechanics, which luckily are dealt with here.

In short, the entire issue seems to contain no good evidence in support of its title 'The Memory of Water is a Reality'. And yet this will now be cited by homeopaths for evermore as peer-reviewed scientific evidence for how homeopathy might work. Of course, all the evidence suggests that homeopathy doesn't work at all, so there's no need to explain how it works in the first place. What a waste of time.

To give Homeopathy its due though, it did publish a paper by Jose Teixeira, entitled 'Can water possibly have a memory? A sceptical view'. The paper concludes that 'any interpretation calling for ‘memory’ effects in pure water must be totally excluded', on the grounds that 'the longest life of any structure observed in liquid water is of the order of 1 ps'. So much time and effort could have been saved.

Edit: There's a far better overview than the above somewhat amateur attempt at Philip Ball's blog, which is why he writes for Nature and I don't. I can't link directly to the post itself, but look for the August 03 2007 entry entitled 'A bad memory'. There are also comments from Martin F Chaplin there.

More metrics nonsense

Since I posted my last ramblings on metrics, I went to the European Association of Geoscientists and Engineers (EAGE) conference in London. I was presenting a talk in a session titled 'Best of Geolsoc'. The session comprised a bunch of talks that had been presented at a Geological Society of London conference earlier in the year: the Geolsoc had asked us to reprise the meeting at the EAGE.

There was some discussion about possible publications that might arise from the meeting. The Geolsoc was interested in putting out a special publication, but most people were more interested in putting out a thematic set of papers in the Journal of the Geological Society. Why? Because a Geolsoc special publication is a book, it doesn't have an impact factor, and, at least in the eyes of academics working at British institutions, it doesn't count.

This, surely, is just plain nuts. The Geolsoc special publications are among the most respected publications in the earth sciences. They are widely read, highly cited, and in general contain benchmark papers presenting solid and interesting science. That otherwise sensible people should be wary of publishing in them on the basis of metrics is a clear indication that we're following the wrong approach. As I said previously, there is enough material on this subject for people to understand the problems if they want to. Indeed, it was clear that many at the meeting recognised the problems of bad metrics being used to evaluate individuals. If we all know it's bollocks, why do we put up with it?

Metrication

On those rare occasions when I actually have a paper to submit for publication, I tend to consider the most appropriate journal for the article I've written. I weigh up factors such as the likely readership of the paper, the readership of the journal, the journal's reputation for rapid peer review and editorial processes, and so on. I have colleagues who say that I ought to take things like the impact factor into account, because publishing in journals with high impact factors is important for my career. The question is, why?

The impact factor is calculated using a publication database by Thomson Scientific, and published on the ISI Web of Knowledge. The database is proprietary, but my employer subscribes to it. The impact factor basically works by counting the number of citations in the year to articles published in a particular journal over the last two years, and dividing by the total number of articles published over the last two years. So the 2006 numbers, which will be published at the end of this month, are derived by counting the number of citations from articles published in 2006 to articles published in the journal in question in 2004 and 2005, and dividing by the total number of articles published in the journal in question in 2004 and 2005. The impact factor applies to the journal as a whole, and not the individual papers published in it.

Why do I hate the impact factor so? There are several reasons. Firstly, it is statistically dubious. Journals with high impact factors get most of their citations from a relatively small number of highly cited papers. For example, David Colquhoun writes that in Nature in 1999 the most cited 16% of papers accounted for 50% of citations (Nature 423, p. 479). In other words, the citation rate of an individual paper is uncorrelated to the impact factor of a journal. This is the most obvious statistical drawback. Several other sources of bias are described in Seglen (1997; British Medical Journal 314, p.497). Eugene Garfield, who invented the impact factor, also agrees that the impact factor should not be used to evaluate individuals (Garfield 1998; Der Unfallchirurg 48, p. 413).

Another 'metric' that has been suggested to evaluate the contribution of scientists is the h-index. In this scheme, an author will have an h-index of h such that they have published h papers that have each been cited at least h times. It has been pointed out that Einstein, had he died in early 1906, would have an h of only 4 or 5, despite the revolutionary nature of the work he had published before that date. This makes me slightly happier about my own h-index of 0, being early in my career and having published two papers that have not yet been around long enough to be cited.

Really though, statistical arguments about biases in various metrics miss the central point, which is that it is impossible to evaluate the scientific worth of an article without reading and understanding it. This can only be done well by people who have expertise in the field of study. In other words, it can only be done well by peer review.

In the UK, academic researchers are evaluated through the Research Assessment Exercise (RAE), which has traditionally been based on peer review. In the upcoming RAE in 2008, there will be a 'shadow' metrics-based exercise running alongside the traditional peer-review based process. In RAEs after 2008, metrics will be used as the main measure of the scientific worth of individual researchers. There have been many criticisms of the traditional RAE. David Colquhoun has written "All of us who do research (rather than talk about it) know the disastrous effects that the Research Assessment Exercise has had on research in the United Kingdom: short-termism, intellectual shallowness, guest authorships and even dishonesty" (Nature 446, p. 373). The situation is hardly going to be improved by relying on a metrics-based approach, as authors inevitably play the system in order to inflate their rankings and progress their careers.

What is perhaps most disappointing is that in general scientists themselves seem to have failed to critically examine metrics such as the impact factor. There is enough material out there in the public domain (the sources cited here are only a sample) for anyone to understand the problems, if they're interested in finding out.

Journal madness

A recent survey shows a remarkable disconnect between what scientists think the system of journal publication should do, and how the scientists act in practice. The survey, entitled "New journal publishing models: an international survey of senior researchers", asked 5,513 senior journal authors their opinions of the current system of scholarly communication. In one part of the survey, participants were asked a series of paired questions relating to journal publications. For me two findings particularly stood out. Firstly, on journal subscription costs. Participants were asked how much they agreed with the statements 'high prices make it difficult to access the journals literature' and 'I publish in affordable journals'. Only 20.7% agreed with both statements.

Of course, how much of a problem high journal prices are to you depends on who you are. As a researcher at the University of Manchester, a large institution with a library that can afford high journal subscription fees, this is not a problem for me at all. If I were working at a less wealthy institution, perhaps in Asia or Africa, access to journal articles would be a significant problem. And as a member of the general public, I would be unlikely to be able to afford to subscribe to a large number of journals. If you wanted to look at my own article on salt deposits in New Brunswick in the Canadian Journal of Earth Sciences, it would cost you £16.85 to download it. This seems crazy, as the journals pay nothing for content and much of the reviewing and editing is done on a voluntary basis. Surely research work that is done at universities should be freely available to the public. After all, they have (in most cases) paid for the work to be done. If the point of doing research is to serve society, then what purpose is served by not making research findings freely available? While journal authors recognise there is a problem here, they generally do not act to mitigate the problem by publishing in affordable (or open access) journals.

Secondly, participants were asked how much they agreed with the statements 'too much research is being published' and 'I publish more than I ought to'. Although the growth in published literature was a major concern, only 8.5% agreed with both statements! So again, while a problem is recognised, journal authors themselves refuse to act in ways that might help solve the problem (i.e. by publishing less).

So why the disconnect? To put it simply, the system is broken. Careers in academia depend on publishing as many papers as possible, and in journals that have high 'impact factors' or high levels of citations by other journal papers. This inevitably leads to an elevation of quantity over quality, and researchers tweaking research findings to publish in journals with high impact factors, rather than considering the most appropriate journal for the paper.

There is surely an urgent need to create a more sensible system of scholarly communication. In the meantime, perhaps those of us who write journal articles can put more thought into whether we are contributing to the madness.