The role of letters in reviewing research

The publication of medical research is prone to error. Misuse of statistics, selective citation of published work, misquotation of references, and overenthusiasm in the search for positive findings are crimes, often committed unwittingly, that can escape peer review before a paper is published.1 Only after publication can a piece of research be exposed to the sort of critical review, by journal readers, that can either establish its place or consign it to the dustbin. The potential of correspondence as a form of peer review is supported by editors2 - the BMJ reserves its letters pages almost exclusively for comment on published material - but it remains underdeveloped and undervalued by clinicians, academics, teachers, and many journals.

In four specialist journals examined by Spodick and Goldberg only 2% of the space was devoted to letters from readers.3 Four general journals gave 15% of the space to letters, but fewer than half referred to original papers. Dr Steven Spiro, editor of Thorax, says he receives only about four letters an issue (the journal received only 37 in 1993). He encourages letters referring to original research and publishes most that come in, usually with a response from the authors. Last year the Annals of Rheumatic Diseases, a monthly journal, received only 84 letters and the British Journal of Ophthalmology, also a monthly, received only 37.

The BMJ received 4075 letters last year - an average of only 78 each week from a circulation of 110 000 - and published 1800, but only one of the 10 articles attracting the most comment was original research.4 An earlier audit of BMJ correspondence showed that in 20 weeks in 1989 the journal received fewer than two letters for each original paper published.5 Naturally some research attracts more comment than others. The BMJ received 18 letters responding to a series of three papers on cholesterol, but most attract little or none.*RF 6-8*

Two things contribute to the underdevelopment of letters pages as a platform for correcting scientific error and contributing to peer review. Firstly, in the assessment of academic performance scientific papers carry great weight while critical analysis in the form of correspondence carries little or none. The intellectual challenge and educational benefits of preparing a concise critical evaluation of published research need to be more widely recognised and rewarded. Letters, like papers, should have data, analysis, insights, and ideas. Secondly, patchy indexing and inadequate linkage of letters to papers make it difficult for authors reviewing a subject to search for, cite, and summaries letters and other comment. This is particularly important when consideration of ensuing comment is central to the interpretation of a paper. For example, a study of the Bristol Cancer Help Centre9 was heavily criticised in several of the 15 letters in response published in the Lancet. One letter, on behalf of the funding agency, withdrew support for the main conclusions of the research.10

Inadequate linkage of papers to comment and corrections ensures that the impact of a paper endures while the effect of relevant comment does not. A search of the Science Citation Index shows nine citations for the Bristol study,9 one for the 15 letters, and none for the letter from the funding agency.10 In 1992 the BMJ published a paper suggesting that elective delivery by caesarean section of the breech fetus at term was far safer than spontaneous vaginal delivery.11 Nineteen letters were subsequently published in response which robustly challenged the paper's conclusions. Three subsequent articles cited the paper, none cited the critical comment. Finally, Bhopal criticised the design, data, interpretation, and conclusions of a paper on snoring and disease.^12,13 The Science Citation Index lists 70 citations for the paper and one for the letter.

Thomasson and Stanley recognised this phenomenon in 1955,14 commenting in Science that, “Buried in scholarly journals, critical notes are increasingly likely to be overlooked with the passage of time, while the studies to which they pertain, having been reported more widely, are apt to be rediscovered.” Their communication has itself been ignored while the same issues still plague science. This work was cited only twice in 1981-92.

If all searches of published reports are to include relevant letters, corrections, and other comment, a system will need to be developed systematically and reliably to link papers with other relevant material. The first step must be comprehensively to index all letters responding to original research. Currently, BMJ letters are indexed and linked to the related paper under both author and subject heading. The US National Library of Medicine has since 1989 linked comment, corrections, and retractions to papers on its Medline database, but only “substantive comment” is indexed and the library limits linkages to comments that were published in the same journal as the related paper. Further, letters are indexed independently, possibly even under different subject headings so a reviewer must undertake a separate search to find them. On line journals now offer electronic linkage, and traditional journals need to develop systems too.15

The simple idea of a unique number for each paper, to be attached by authors or editors to related correspondence, should be evaluated. Reviewers citing a paper could then search easily for related material using the unique identifier. Ideally interlibrary loan requests for papers should be for the paper and its related commentary; a unique identifier would ease the task of storing and retrieving material published at different times, by different authors, and in different journals. Already a unique number is entered as part of the identification details of a paper on Medline, but it is not used for this purpose. Currently there is no effective means of neutralising the effect of misleading or fraudulent published material. Linking articles to critical comment, firstly in researchers' minds and, secondly, in the retrieval of papers will help to achieve this.

If published critical comment is considered integral to research should it not be peer reviewed? The BMJ does not peer review letters but allows authors of papers to respond to critical letters in the same issue of the journal. Peer review can be slow, and the aim is to publish the debate generated by research while it is still fresh in readers' minds. Letters from the United Kingdom are not considered unless they arrive within six weeks, but that deadline is waived for any letters coming from outside the United Kingdom or for reports of serious flaws or frank errors.

Raising the status of letters pages as a forum for scientific criticism requires a new perspective from the readers and writers of science: that an original scientific paper is simply incomplete without the accompanying published comment. More and better quality critical comment that is easily accessible to reviewers will follow.