Biomedical funding decisions should be audited

EDITOR—Grant et al note that organisations that fund biomedical research assume that the research they support will lead to an eventual improvement in health.1 Because clinical guidelines represent one of the final links between basic research and actions to improve health, they looked at which studies were cited in guidelines.

Although their analysis is a valuable move away from the naive use and abuse of citation counts and impact factors, they may have attempted to bridge too great a distance in assessing which publications in the serial peer reviewed literature were cited in guidelines. Guidelines should be based on systematic reviews of all the studies relevant to particular clinical questions. They should not be based on the biased subsets of reports of primary research included in bibliographic databases or those that are sufficiently concise to be published in serial journals.2

It would be helpful if Grant et al would indicate the extent to which references to systematic reviews were cited in the guidelines they studied. The “payback” from primary studies might then be studied by assessing their contribution to these systematic reviews. For example, was a primary study judged to be of sufficiently high quality to have been included in a systematic review at all? If so, what contribution did it make to the totality of the relevant evidence?

Grant et al suggest that an alternative to the retrospective approach that they used for assessing payback would be “to identify a body of basic research published some time ago and follow its subsequent knowledge flow.” A more informative approach would be to identify a body of basic research funded some time ago. Payback could then be assessed not only in terms of whether it led to an eventual improvement in health but also whether it was completed and published.

Failed research and failure to publish successful research are costs to the public. Yet I am not aware of any public or charitable organisation that funds biomedical research that routinely publishes audits of its investment decisions using criteria such as these.

Authors' reply

EDITOR—Chalmers asks what proportion of references cited in the guidelines we studied were systematic reviews. The answer is 68/2501 (2.7%). We made this calculation by using the keywords “systematic” (for systematic review) and “meta” (for meta-analysis) in the title of the publication. Interestingly, there was no difference between publications in peer reviewed journals (56/2043; 2.7%) and the so called grey literature (12/458; 2.6%). Although Chalmers argues that clinical guidelines should be based on systematic reviews of the literature, these data show that authors of guidelines are citing the primary research.

The Cochrane Database of Systematic Reviews could provide a useful resource in undertaking applied bibliometric studies. Chalmers notes that the prospective tracing of funded research, as opposed to that only published, may be more informative in auditing the outcomes of public or charitable funding of biomedical research. We agree.