Little evidence for effectiveness of scientific peer review

Despite its widespread use and costs, little hard evidence exists that peer review improves the quality of published biomedical research, concludes a systematic review from the international Cochrane Collaboration.

Yet the system, which has been used for at least 200 years, has only recently come under scrutiny, with its assumptions about fairness and objectivity rarely tested, say the review authors. With few exceptions, journal editors—and clinicians—around the world continue to see it as the hallmark of serious scientific endeavour.

Published last week, the review is the third in a series from the Cochrane Collaboration Methods Group. The other reviews look at the grant application process and technical editing.

Only the latter escapes a drubbing, with the reviewers concluding that technical editing does improve the readability, accuracy, and overall quality of published research.

The Cochrane reviewers based their findings on 21 studies of the peer review process from an original trawl of only 135. These were drawn from a comprehensive search of biomedical print and online databases, and information received from bodies such as the World Association of Medical Editors.

Almost half of the available research focused on the effects of concealing the identity of reviewers and/or authors, which, the Cochrane authors conclude, has little impact on quality. Few studies assessed the impact of peer review on the importance, usefulness, relevance, or quality of research. Only one small study tested the validity of the peer review procedure itself.

On the basis of the current evidence, “the practice of peer review is based on faith in its effects, rather than on facts,” state the authors, who call for large, government funded research programmes to test the effectiveness of the system and investigate possible alternatives.

“As the information revolution gathers pace, an empirically proven method of quality assurance is of paramount importance,” they contend.

Professor Tom Jefferson, who led the Cochrane review, suggested that further research might prove that peer review, or an evolved form of it, worked. At the very least, it needed to be more open and accountable.

But he said that there had never even been any consensus on its aims and that it would be more appropriate to refer to it as “competitive review.”

Not only did peer review pander to egos and give researchers licence to knife each other in the back with impunity, he said, but it was also “completely useless at detecting research fraud” and let editors off the hook for publishing poor quality studies.

In the latest report from the Committee on Publication Ethics, Professor Peter Lachmann, until recently president of the UK Academy of Medical Sciences, comments: “Peer review is to science what democracy is to politics. It's not the most efficient mechanism, but it's the least corruptible.”