BMJ 1998;317:1185-1190 ( 31 October )

Papers

The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials

Regina Kunz, registrar, ^a Andrew D Oxman, director. ^b

^a Department of Nephrology, Charité, Berlin, Germany, ^b Health Services Research Unit, National Institute of Public Health, Oslo, Norway

Correspondence to: Dr Oxman andrew.oxman{at}labmed.uio.no

	Abstract

Top Abstract Introduction Methods Results Discussion References

Objective To summarise comparisons of randomised clinical trials and non-randomised clinical trials, trials with adequately concealed random allocation versus inadequately concealed random allocation, and high quality trials versus low quality trials where the effect of randomisation could not be separated from the effects of other methodological manoeuvres.
Design Systematic review.
Selection criteria Cohorts or meta-analyses of clinical trials that included an empirical assessment of the relation between randomisation and estimates of effect.
Data sources Cochrane Review Methodology Database, Medline, SciSearch, bibliographies, hand searching of journals, personal communication with methodologists, and the reference lists of relevant articles.
Main outcome measures Relation between randomisation and estimates of effect.
Results Eleven studies that compared randomised controlled trials with non-randomised controlled trials (eight for evaluations of the same intervention and three across different interventions), two studies that compared trials with adequately concealed random allocation and inadequately concealed random allocation, and five studies that assessed the relation between quality scores and estimates of treatment effects, were identified. Failure to use random allocation and concealment of allocation were associated with relative increases in estimates of effects of 150% or more, relative decreases of up to 90%, inversion of the estimated effect and, in some cases, no difference. On average, failure to use randomisation or adequate concealment of allocation resulted in larger estimates of effect due to a poorer prognosis in non-randomly selected control groups compared with randomly selected control groups.
Conclusions Failure to use adequately concealed random allocation can distort the apparent effects of care in either direction, causing the effects to seem either larger or smaller than they really are. The size of these distortions can be as large as or larger than the size of the effects that are to be detected.

Key messages Empirical studies support using random allocation in clinical trials and ensuring that the allocation process is concealedthat is, that assignment is impervious to any influence by the people making the allocation The effect of not using concealed random allocation can be as large or larger than the effects of worthwhile interventions On average, failure to use concealed random allocation results in overestimates of effect due to a poorer prognosis in non-randomly selected control groups compared with randomly selected control groups, but it can result in underestimates of effect, reverse the direction of effect, mask an effect, or give similar estimates of effect The adequacy of allocation concealment may be a more sensitive measure of bias in clinical trials than scales used to assess the quality of clinical trials It is a paradox that the unpredictability of randomisation is the best protection against the unpredictability of the extent and direction of bias in clinical trials that are not properly randomised

	Introduction

Top Abstract Introduction Methods Results Discussion References

Observational evidence is clearly better than opinion, but it is thoroughly unsatisfactory. All research on the effectiveness of therapy was in this unfortunate state until the early 1950s. The only exceptions were the drugs whose effect on immediate mortality were so obvious that no trials were necessary, such as insulin, sulphonamide, and penicillin.¹

"The basic idea, like most good things, is very simple."¹ Randomisation is the only means of controlling for unknown and unmeasured differences between comparison groups as well as those that are known and measured. Random assignment removes the potential of bias in the assignment of patients to one intervention or another by introducing unpredictability. When alternation or any other preset plan (such as time of admission) is used, it is possible to arrange to enter a patient into a study at an opportune moment. With randomisation, however, each patient's treatment is assigned according to the play of chance. It is a paradox that unpredictability is introduced into the design of clinical trials by using random allocation to protect against the unpredictability of the extent of bias in the results of non-randomised clinical trials.

Despite this simple logic, and many examples of harm being done because of delays in conducting randomised trials, there are limitations to the use of randomised trials, both real and imagined, and scepticism about the value of randomisation.^2-5 We believe this scepticism is healthy. It is important to question assumptions about research methods, and to test these assumptions empirically, just as it is important to test assumptions about the effects of health care. In this paper we have attempted systematically to summarise empirical studies of the relation between randomisation and estimates of effect.

	Methods

Top Abstract Introduction Methods Results Discussion References

We included four types of comparisons in our review: randomised clinical trials versus non-randomised clinical trials of the same intervention, randomised clinical trials versus non-randomised clinical trials across different interventions, adequately concealed random allocation versus inadequately concealed random allocation in trials, and high quality trials versus low quality trials in which the specific effect of randomisation or allocation concealment could not be separated from the effect of other methodological manoeuvres such as double blinding. Both descriptive and analytical assessments of the relation between the use of random allocation and estimates of effect are included, based on cohorts or meta-analyses of clinical trials.

We identified studies from the Cochrane Review Methodology Database,⁶ other methodological bibliographies, Medline, and SciSearch, and by hand searching journals, personal communication with methodologists, and checking the reference lists of relevant articles. These searches were conducted up to July 1998. Potentially relevant citations were retrieved and assessed for inclusion independently by both authors. Disagreements were resolved by discussion.

We used the following criteria to appraise the methodological quality of included studies: Were explicit criteria used to select the trials? Did two or more investigators agree regarding the selection of trials? Was there a consecutive or complete sample of clinical trials? Did the study control for other methodological differences such as double blinding and complete follow up? Did the study control for clinical differences in the participants and interventions in the included trials? Were similar outcome measures used in the included trials? The overall quality of each study was summarised as: no important flaws, possibly important flaws, or major flaws.

View this table: [in this window] [in a new window]   Table 1. Randomised controlled trials (RCTs) compared with non-randomised controlled trials (non-RCTs) of the same intervention

For each study one of us (RK) extracted information about the sample of clinical trials, the comparison that was made, the type of analysis undertaken, and the results, and the other checked the extracted data against the published article. The reported relation between randomisation and estimates of effect was recorded and, if possible, converted to the relative overestimation or underestimation of the relative risk reduction. We prepared tables for each type of comparison to facilitate a qualitative analysis of the extent to which the included studies yielded similar results, and heterogeneity in the included studies was explored both within and across comparisons.

In summarising the results we have assumed that evidence from randomised trials is the reference standard to which estimates from non-randomised trials are compared. However, as with other gold standards, randomised trials are not without flaws, and this assumption is not intended to imply that the true effect is known, or that estimates derived from randomised trials are always closer to the truth than estimates from non-randomised trials.

	Results

Top Abstract Introduction Methods Results Discussion References

We have identified 18 cohorts or meta-analyses that met our inclusion criteria, totalling 1211 clinical trials.^7-24 Efforts to develop an efficient electronic search strategy using Medline have thus far not been successful due to poor indexing. Searches for studies that cited Colditz and colleagues,¹⁵ Miller and colleagues,¹⁶ Chalmers and colleagues,¹⁸ or Schulz and colleagues¹⁹ using SciSearch yielded seven additional studies. Searches using SciSearch for studies that cited the other studies meeting our inclusion criteria did not yield any other additional studies. Exploratory hand searching of three methodological journals (Controlled Clinical Trials, Statistics in Medicine, and the Journal of Clinical Epidemiology) for four years (1970, 1980, 1990, and 1995) yielded a single relevant study published in 1990. The 18 included studies were published in 14 different journals. The majority of studies were identified through personal communication with methodologists and through bibliographies and reference lists.

Randomised trials versus non-randomised trials of the same intervention
Table 1 summarises the eight studies comparing randomised clinical trials and non-randomised clinical trials of the same intervention. In five of the eight studies, estimates of effect were larger in non-randomised trials. Outcomes in the randomised treatment groups and non-randomised treatment groups were frequently similar, but worse outcomes among historical controls spuriously increased the estimated treatment effects. One study found comparable results for both allocation procedures, and two studies reported smaller treatment effects in non-randomised studies. In one study the smaller estimate of effect was due to a poorer prognosis for patients in the non-randomised treatment groups. The deviation of the estimates of effect for non-randomised trials compared with randomised trials ranged from an underestimation of effect of 76% to an overestimation of effect of 160%.

View this table: [in this window] [in a new window]   Table 2. Randomised controlled trials (RCTs) compared with non-randomised controlled trials (non-RCTs) across different interventions

View this table: [in this window] [in a new window]   Table 3. Trials with adequately concealed allocation compared with inadequately concealed allocation

Randomised trials versus non-randomised trials across different interventions
The evidence from comparisons across different interventions and various study designs (randomised controlled trials and non-randomised controlled trials, crossover designs, and observational studies) is less clear (table 2). In all three studies several study designs and clinical conditions were combined and their diverse outcomes converted to a standardised effect size. There was substantial clinical heterogeneity, and there were many other factors that could distort or mask a possible association between randomisation and estimates of effect. No consistent relation between study design or quality and the magnitude of the estimates of effect was detected.

Adequately concealed allocation versus inadequately concealed allocation
Concealed random allocation to treatmentthat is, blinding of the randomisation schedule to prevent subversion by the investigators or trial participantsshould ensure protection against biased allocation. Chalmers and colleagues found that within randomised controlled trials failure adequately to conceal allocation was associated with larger imbalances in prognostic factors and larger treatment effects (table 3).¹⁸ They reported a more than sevenfold overestimation of the treatment effect in trials with inadequately concealed allocation. They did not, however, control for other methodological factors in their descriptive analysis.¹⁸ Schulz and colleagues conducted a multivariate analysis that controlled for blinding and completeness of follow up, which yielded similar results.¹⁹ They found that inadequately concealed random allocation (for example, alternation) compared with adequately concealed random allocation (for example, assignment by a central office) resulted in estimates of effect (odds ratios) that were on average 40% larger.

High quality trials versus low quality trials
Considerable differences in the observed treatment effect were detected when the results of high quality studies were compared with those of low quality studies in the context of systematic reviews of specific health care (table 4). In these studies the estimates of effect were distorted in both directions and even caused the alarming situation of a harmful intervention associated with a reduction in pregnancies (odds ratio 0.5, on the basis of high quality studies) seeming beneficial in low quality studies (odds ratio 2.6, on the basis of low quality studies). In two meta-analyses, low quality studies consistently underestimated the beneficial effect of the intervention being evaluated by 27% to 100%, and an effective treatment could have been discarded based on the results of low quality studies.

View this table: [in this window] [in a new window]   Table 4. Studies of high quality trials compared with low quality trials

Methodological quality
The methodological quality of the studies included in this review varied. Four studies met all of our criteria. ^{19 21-23} Three of these assessed the impact of bias on the effect of a specific healthcare intervention as part of a systematic review, and the analysis was performed as part of a subgroup analysis to test the robustness of the overall finding.^21-23 The other 14 studies had one or more methodological flaws including not controlling for other methodological manoeuvres ^{16 18 22 27} or clinical differences. ^{7 13-17 20 24}

	Discussion

Top Abstract Introduction Methods Results Discussion References

It has proved difficult to develop efficient search strategies for locating empirical methodological studies such as the ones included in this review. Although we believe it is unlikely that there are many published methodological studies such as the ones by Sacks and colleagues,⁸ Schulz and colleagues,¹⁹ Chalmers and colleagues,¹⁸ and Emerson and colleagues²⁰ that we have not identified, there may be unpublished or ongoing studies like these that we have not identified, and it is likely that there are many meta-analyses that meet the inclusion criteria for this review that we have not identified. The Cochrane Library contains 428 completed reviews and 397 protocols, and there are over 1700 entries in the database of abstracts of reviews of effectiveness.²⁶ We have not systematically gone through all of these meta-analyses. An expanded version of this review will be published in the Cochrane Library and kept up to date through the Cochrane Empirical Methodological Studies Methods Group.²⁷ Additional studies will be added to the review, and any errors that are identified will be corrected.

We have not included comparisons between randomised controlled trials and cohort studies,²⁸ case-control studies, ^{29 30} or evaluations of effectiveness using large healthcare administrative databases,³ although some of the studies in this review included observational studies. Observational studies often provide valuable information that is complementary to the results of clinical trials. For example, case-control studies may be the best available study design for evaluating rare adverse effects, and large database studies may provide important information about the extent to which effects that are expected based on randomised clinical trials are achieved in routine practice. However, it is important to remember that it is only possible to control for confounders that are known and measured in observational studies, and we should be wary of hubris and its consequences in assuming that we know all there is to know about any disease.

As with any review the quality of the data is limited by the quality of the studies that we have reviewed. Most of the studies included in the review had one or more methodological flaws. In many of the included comparisons, particularly those between randomised controlled trials and historically controlled trials, methodological differences other than randomisation may account for some of the observed differences in estimates of effect. ^{7-9 13 18}

Four of the studies met all of our criteria for assessing methodological quality, ^{19 21-23} and one study in particular provided strong support for the conclusion that clinical trials that lack adequately concealed random allocation produce estimates of effect that are on average 40% larger than clinical trials with adequately concealed random allocation, but that the degree and the direction of this bias varies widely.¹⁹ This study also shows the potential contribution that systematic reviews, and notably the Cochrane Database of Systematic Reviews, can make towards developing an empirical basis for methodological decisions in evaluations of health care. Currently this empirical basis is lacking, and many methodological debates rely more on logic or rhetoric than evidence. Analyses such as the one undertaken by Schulz and colleagues, in which methodological comparisons are made among trials of the same intervention, are likely to yield more reliable results than comparisons that are made across different interventions which, not surprisingly, tend to be inconclusive.^15-17

We have assumed that, in general, differences between randomised trials and non-randomised trials or between trials with adequately concealed random allocation and inadequately concealed random allocation are best explained by bias in the non-randomised controlled trials and inadequately concealed trials. This assumption is supported by findings of large imbalances in prognostic factors as well. However, it is possible that randomised controlled trials can sometimes underestimate the effectiveness of an intervention in routine practice by forcing healthcare professionals and patients to acknowledge their uncertainty and thereby reduce the strength of placebo effects. ^{4 25 31} It is also possible that publication bias can partly explain some of the differences in results observed in studies such as the one by Sacks and colleagues.⁸ This would be the case if randomised trials are more likely to be published regardless of the effect size, than historically controlled trials. However, we are not aware of any evidence that supports this hypothesis, and the available evidence shows consistently that randomised trials, like other research, are also more likely to be published if they have results that are considered significant.^32-35

Several explanations for discrepancies between estimates of effect derived from randomised trials and non-randomised trials are possible. For example, it can be argued that estimates of effect might be larger in randomised trials if the care provided in the context of trials is better than that in routine practice, assuming this is the case for the treatment group and not the control group. Similarly, strict eligibility criteria might select people with a higher capacity to benefit from a treatment, resulting in larger estimates of effect in randomised trials than non-randomised trials with less strict eligibility criteria. If, for some reason, patients with a poor prognosis were more likely to be allocated to the treatment group in non-randomised trials then this would also result in larger estimates of effect in randomised trials. Conversely, if patients with a poor prognosis were more likely to be allocated to the control group in non-randomised trials, as often seems to be the case based on the results of this review, this would result in larger estimates of effect in the non-randomised trials.

Conclusion
Overall, this review supports using random allocation in clinical trials and ensuring that the randomisation schedule is adequately concealed. The effect of not using random allocation with adequate concealment can be as large or larger than the effects of worthwhile interventions. On average, non-randomised trials and randomised trials with inadequately concealed allocation result in overestimates of effect. This bias, however, can go in either direction, can reverse the direction of effect, or can mask an effect.

For those undertaking clinical trials this review provides support for using randomisation to assemble comparison groups.²⁵ For those undertaking systematic reviews of clinical trials, this review provides support for considering sensitivity analyses based on the adequacy of allocation concealment in addition to or instead of on the basis of overall quality scores, which may be less sensitive measures of bias.

As Cochrane stated: "The [randomised controlled trial] is a very beautiful technique, of wide applicability, but as with everything else there are snags."¹ Those making decisions on the basis of clinical trials need to be cautious of small trials (even when they are properly randomised) and systematic reviews of small trials both because of chance effects and the risk of biased reporting. ^{36 37} It is also possible to introduce bias into a trial despite allocation concealment. ^{19 38} Finally, even when the risk of error due to either bias or chance is small, judgments must be made about the applicability of the results to individual patients ^{39 40} and about the relative value of the probable benefits, harms, and costs. ^{41 42}

	Acknowledgments

We thank Alex Jadad, Steve Halpern, and David Cowan for help in locating studies, Dave Sackett and Iain Chalmers for encouragement and advice, Mike Clarke for reviewing the manuscript, Annie Britton and other colleagues for provision of their bibliographies on research methodology, and the investigators who conducted the studies we reviewed.

Contributors: RK and ADO contributed to the preparation of the protocol and the final manuscript and assessed the relevance and methodological quality of retrieved reports. RK prepared the first drafts of the protocol and the paper, undertook the majority of the searches with help from David Cowan, Steve Halpern, Alex Jadad, and collected data from the included studies. ADO checked the collected data against the original reports. Both authors will act as guarantors for the paper.

Funding: Norwegian Ministry of Health and Social Affairs.

Competing interests: None declared.

	References

Top Abstract Introduction Methods Results Discussion References

1. Cochrane AL. Effectiveness and efficiency: random reflections on health services. London: Nuffield Provincial Hospitals Trust, 1972:20-25.
2. Committee for Evaluating Medical Technologies in Clinical Use. Assessing medical technologies. Washington DC: National Academy Press, 1985:76-78.
3. US Congress, Office of Technology Assessment. Identifying health technologies that work: searching for evidence, OTA-H-608. Washington DC: US Government Printing Office, 1994:41-51.
4. Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ 1996; 312: 1215-1218[Free Full Text].
5. Weiss CH. Evaluation. Methods for studying programs and policies. 2nd ed. Upper Saddle River: Prentice Hall, 1998:229-233.
6. Cochrane Review Methodology Database. In: Clarke M, Carling C, Oxman AD, eds. The Cochrane Library. Oxford: Update Software, 1998Issue 3.
7. Chalmers TC, Matta RJ, Smith Jr H, Kunzler AM. Evidence favoring the use of anticoagulants in the hospital phase of acute myocardial infarction. N Engl J Med 1977; 297: 1091-1096[Abstract].
8. Sacks H, Chalmers TC, Smith Jr H. Randomized versus historical controls for clinical trials. Am J Med 1982; 72: 233-240[Medline].
9. Diehl LF, Perry DJ. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid? J Clin Oncol 1986; 4: 1114-1120[Abstract/Free Full Text].
10. Reimold SC, Chalmers TC, Berlin JA, Antman EM. Assessment of the efficacy and safety of antiarrhythmic therapy for chronic atrial fibrillation: observations on the role of trial design and implications of drug related mortality. Am Heart J 1992; 124: 924-932[Medline].
11. Recurrent Miscarriage Immunotherapy Trialists Group. Worldwide collaborative observational study and meta analysis on allogenic leukocyte immunotherapy for recurrent spontaneous abortion. Am J Reprod Immunol 1994; 32: 55-72.
12. Watson A, Vandekerckhove P, Lilford R, Vail A, Brosens I, Hughes E. A meta-analysis of the therapeutic role of oil soluble contrast media at hysterosalpingography: a surprising result? Fertil Steril 1994; 61: 470-477[Medline].
13. Pyorala S, Huttunen NP, Uhari M. A review and meta-analysis of hormonal treatment of cryptorchidism. J Clin Endocrinol Metab 1995; 80: 2795-2799[Abstract].
14. Carroll D, Tramer M, McQuay H, Nye B, Moore A. Randomization is important in studies with pain outcomes: systematic review of transcutaneous electrical nerve stimulation in acute postoperative pain. Br J Anaesth 1996; 77: 798-803[Abstract/Free Full Text].
15. Colditz GA, Miller JN, Mosteller F. How study design affects outcomes in comparisons of therapy. I: medical. Stat Med 1989; 8: 441-454[Medline].
16. Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II: surgical. Stat Med 1989; 8: 455-466[Medline].
17. Ottenbacher K. Impact of random assignment on study outcome: an empirical examination. Control Clin Trials 1992; 13: 50-61[Medline].
18. Chalmers TC, Celano P, Sacks HS, Smith Jr H. Bias in treatment assignment in controlled clinical trials. N Engl J Med 1983; 309: 1358-1361[Abstract].
19. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408-412[Abstract/Free Full Text].
20. Emerson JD, Burdick E, Hoaglin DC, Mosteller F, Chalmers TC. An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. Control Clin Trials 1990; 11: 339-352[Medline].
21. Imperiale TF, McCullough AJ. Do corticosteroids reduce mortality from alcoholic hepatitis? A meta analysis of the randomized trials. Ann Intern Med 1990; 113: 299-307.
22. Nurmohamed MT, Rosendaal FR, Buller HR, Dekker E, Hommes DW, Vandenbroucke JP, et al. Low molecular weight heparin versus standard heparin in general and orthopaedic surgery: a meta-analysis. Lancet 1992; 340: 152-156[Medline].
23. Khan KS, Daya S, Jadad A. The importance of quality of primary studies in producing unbiased systematic reviews. Arch Intern Med 1996; 156: 661-666[Abstract/Free Full Text].
24. Ortiz Z, Shea B, Suarez Almazor ME, Moher D, Wells GA, Tugwell P. The efficacy of folic acid and folinic acid in reducing methotrexate gastrointestinal toxicity in rheumatoid arthritis. A meta-analysis of randomized controlled trials. J Rheumatol 1998; 25: 36-43[Medline].
25. Chalmers I. Assembling comparison groups to assess the effects of health care. J R Soc Med 1997; 90: 379-386[Medline].
26. NHS Centre for Reviews and Dissemination. Database of abstracts of reviews of effectiveness. The Cochrane Library. Oxford: Update Software , 1998Issue 3.
27. Cochrane Empirical Methodological Studies Methods Group. The Cochrane Library. Oxford: Update Software , 1998Issue 3.
28. Forgie MA, Wells PS, Laupacis A, Fergusson D. Preoperative autologous donation decreases allogeneic transfusion but increases exposure to all red blood cell transfusion: results of a meta-analysis. Arch Intern Med 1998; 158: 610-616[Abstract/Free Full Text].
29. Colditz GA, Brewer TF, Berkey CS, Wilson ME, Burdick E, Fineberg HV, et al. Efficacy of BCG vaccine in the prevention of tuberculosis. Meta analysis of the published literature. JAMA 1994; 271: 698-702[Abstract/Free Full Text].
30. Stieb D, Frayha HH, Oxman AD, Shannon HS, Hutchison BG, Crombie F. The effectiveness and usefulness of Haemophilus influenzae type b vaccines: a systematic overview (meta-analysis). Can Med Assoc J 1990; 142: 719-732[Abstract].
31. Kleijnen J, Gøtzsche P, Kunz RH, Oxman AD, Chalmers I. So what's so special about randomisation? In: Maynard A, Chalmers I, eds. Non-random reflections on health services research: on the 25th anniversary of Archie Cochrane's effectiveness and efficiency. London: BMJ Publishing Group, 1997:93-106.
32. Dicksersin K, Min YI. NIH clinical trials and publication bias. Online J Curr Clin Trials [serial online] 1993; document No 50.
33. Dickersin K. How important is publication bias? A synthesis of available data. AIDS Education and Prevention 1997; 9(suppl A): 15-21[Medline].
34. Stern JM, Simes RJ. Publication bias: evidence of delayed publication of clinical research projects. BMJ 1997; 315: 640-645[Abstract/Free Full Text].
35. Ioannidis JPA. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998; 279: 281-286[Abstract/Free Full Text].
36. Counsell CE, Clarke MJ, Slattery J, Sandercock PAG. The miracle of DICE therapy for acute stroke: fact or fictional product of subgroup analysis? BMJ 1994; 309: 1677-1681[Abstract/Free Full Text].
37. Egger M, Davey SG, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: 629-634[Abstract/Free Full Text].
38. Guyatt GH, Sackett DL, Cook DJ, for the Evidence-Based Working Group. Users' guides to the medical literature, II: how to use an article about therapy or prevention, A: are the results of the study valid? JAMA 1993; 270: 2598-2601[Abstract/Free Full Text].
39. Dans AL, Dans LF, Guyatt GH, Richardson S. Users' guides to the medical literature:ow to decide on the applicability of clinical trial results to your patient. JAMA 1998; 279: 545-549[Free Full Text].
40. Cochrane Methods Working Group on Applicability and Recommendations. The Cochrane Library. Oxford: Update Software , 1998Issue 3.
41. Guyatt GH, Sackett DL, Cook DJ, for the Evidence-Based Working Group. Users' guides to the medical literature, II: how to use an article about therapy or prevention, B: what were the results and will they help me in caring for my patients? JAMA 1994; 270: 59-63.
42. Oxman AD, Flottorp S. An overview of strategies to promote implementation of evidence based health care. In: Silagy C, Haines A, eds. Evidence based practice. London: BMJ Books, 1998:91-109.

(Accepted 29 September 1998)

---

© BMJ 1998
CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Relevant Articles

Use of primary care electronic medical record database in drug efficacy research on cardiovascular outcomes: comparison of database and randomised controlled trial findings

Richard L Tannen, Mark G Weiner, and Dawei Xie
BMJ 2009 338: b81. [Abstract] [Full Text] [PDF]

Perinatal mortality and other severe adverse pregnancy outcomes associated with treatment of cervical intraepithelial neoplasia: meta-analysis

M Arbyn, M Kyrgiou, C Simoens, A O Raifu, G Koliopoulos, P Martin-Hirsch, W Prendiville, and E Paraskevaidis
BMJ 2008 337: a1284. [Abstract] [Full Text] [PDF]

Oral misoprostol for induction of labour at term: randomised controlled trial

Jodie M Dodd, Caroline A Crowther, and Jeffrey S Robinson
BMJ 2006 332: 509-513. [Abstract] [Full Text] [PDF]

Fifty years of randomised controlled trials
BMJ 1998 317: 0. [Full Text] [PDF]

Concealed random allocation protects against bias in clinical trials
BMJ 1998 317: 0. [Full Text]

This article has been cited by other articles:

• Ene-Iordache, B., Carminati, S., Antiga, L., Rubis, N., Ruggenenti, P., Remuzzi, G., Remuzzi, A. (2009). Developing Regulatory-compliant Electronic Case Report Forms for Clinical Trials: Experience with The Demand Trial. J. Am. Med. Inform. Assoc. 16: 404-408 [Abstract] [Full text]  
• Bhandari, M., Morshed, S., Tornetta, P. III, Schemitsch, E. H. (2009). Design, Conduct, and Interpretation of Nonrandomized Orthopaedic Studies: A Practical Approach. (All) Evidence Matters. JBJS 91: 1-1 [Full text]  
• Hoppe, D. J., Schemitsch, E. H., Morshed, S., Tornetta, P. III, Bhandari, M. (2009). Hierarchy of Evidence: Where Observational Studies Fit in and Why We Need Them. JBJS 91: 2-9 [Abstract] [Full text]  
• Simunovic, N., Sprague, S., Bhandari, M. (2009). Methodological Issues in Systematic Reviews and Meta-Analyses of Observational Studies in Orthopaedic Research. JBJS 91: 87-94 [Abstract] [Full text]  
• Riley, M., Bearn, D. R. (2009). A systematic review of clinical trials of aligning archwires. J. Orthod. 36: 42-51 [Abstract] [Full text]  
• Sjauw, K. D., Engstrom, A. E., Vis, M. M., van der Schaaf, R. J., Baan, J. Jr, Koch, K. T., de Winter, R. J., Piek, J. J., Tijssen, J. G.P., Henriques, J. P.S. (2009). A systematic review and meta-analysis of intra-aortic balloon pump therapy in ST-elevation myocardial infarction: should we change the guidelines?. Eur Heart J 30: 459-468 [Abstract] [Full text]  
• Tannen, R. L, Weiner, M. G, Xie, D. (2009). Use of primary care electronic medical record database in drug efficacy research on cardiovascular outcomes: comparison of database and randomised controlled trial findings. BMJ 338: b81-b81 [Abstract] [Full text]  
• French, J., Gronseth, G. (2008). Invited Article: Lost in a jungle of evidence: We need a compass. Neurology 71: 1634-1638 [Full text]  
• Gronseth, G., French, J. (2008). Invited Article: Practice parameters and technology assessments: What they are, what they are not, and why you should care. Neurology 71: 1639-1643 [Full text]  
• Onslow, M., Jones, M., O'Brian, S., Menzies, R., Packman, A. (2008). Defining, Identifying, and Evaluating Clinical Trials of Stuttering Treatments: A Tutorial for Clinicians. AJSLP 17: 401-415 [Abstract] [Full text]  
• Martens, E. P, Pestman, W. R, de Boer, A., Belitser, S. V, Klungel, O. H (2008). Systematic differences in treatment effect estimates between propensity score methods and logistic regression. Int J Epidemiol 37: 1142-1147 [Abstract] [Full text]  
• Arbyn, M, Kyrgiou, M, Simoens, C, Raifu, A O, Koliopoulos, G, Martin-Hirsch, P, Prendiville, W, Paraskevaidis, E (2008). Perinatal mortality and other severe adverse pregnancy outcomes associated with treatment of cervical intraepithelial neoplasia: meta-analysis. BMJ 337: a1284-a1284 [Abstract] [Full text]  
• Thomson, A. (2008). The Enteral vs Parenteral Nutrition Debate Revisited. JPEN J Parenter Enteral Nutr 32: 474-481 [Abstract] [Full text]  
• Karanicolas, P. J., Kunz, R., Guyatt, G. H. (2008). Point: Evidence-Based Medicine Has a Sound Scientific Base. Chest 133: 1067-1071 [Full text]  
• Pannu, N., Klarenbach, S., Wiebe, N., Manns, B., Tonelli, M., for the Alberta Kidney Disease Network, (2008). Renal Replacement Therapy in Patients With Acute Renal Failure: A Systematic Review. JAMA 299: 793-805 [Abstract] [Full text]  
• Morshed, S., Bhandari, M. (2008). Clinical Trial Design in Fracture-Healing Research: Meeting the Challenge. JBJS 90: 55-61 [Abstract] [Full text]  
• Yang, K. (2007). A Review of Yoga Programs for Four Leading Risk Factors of Chronic Diseases. Evid Based Complement Alternat Med 4: 487-491 [Abstract] [Full text]  
• Shrier, I., Boivin, J.-F., Steele, R. J., Platt, R. W., Furlan, A., Kakuma, R., Brophy, J., Rossignol, M. (2007). Should Meta-Analyses of Interventions Include Observational Studies in Addition to Randomized Controlled Trials? A Critical Examination of Underlying Principles. Am J Epidemiol 166: 1203-1209 [Abstract] [Full text]  
• Nicholl, J. (2007). Case-mix adjustment in non-randomised observational evaluations: the constant risk fallacy. J. Epidemiol. Community Health 61: 1010-1013 [Abstract] [Full text]  
• Adams, S. G., Smith, P. K., Allan, P. F., Anzueto, A., Pugh, J. A., Cornell, J. E. (2007). Systematic Review of the Chronic Care Model in Chronic Obstructive Pulmonary Disease Prevention and Management. Arch Intern Med 167: 551-561 [Abstract] [Full text]  
• Bonovas, S., Filioussi, K., Tsavaris, N., Sitaras, N. M. (2006). Statins and Cancer Risk: A Literature-Based Meta-Analysis and Meta-Regression Analysis of 35 Randomized Controlled Trials. JCO 24: 4808-4817 [Abstract] [Full text]  
• Foley, N. C, Bhogal, S. K, Teasell, R. W, Bureau, Y., Speechley, M. R (2006). Estimates of Quality and Reliability With the Physiotherapy Evidence-Based Database Scale to Assess the Methodology of Randomized Controlled Trials of Pharmacological and Nonpharmacological Interventions. ptjournal 86: 817-824 [Abstract] [Full text]  
• Gartlehner, G., Hansen, R. A., Carson, S. S., Lohr, K. N. (2006). Efficacy and Safety of Inhaled Corticosteroids in Patients With COPD: A Systematic Review and Meta-Analysis of Health Outcomes. Ann Fam Med 4: 253-262 [Abstract] [Full text]  
• Dodd, J. M, Crowther, C. A, Robinson, J. S (2006). Oral misoprostol for induction of labour at term: randomised controlled trial. BMJ 332: 509-513 [Abstract] [Full text]  
• Bonovas, S., Filioussi, K., Tsavaris, N., Sitaras, N. M. (2005). Use of Statins and Breast Cancer: A Meta-Analysis of Seven Randomized Clinical Trials and Nine Observational Studies. JCO 23: 8606-8612 [Abstract] [Full text]  
• Whelan, T., Levine, M. (2005). In Reply:. JCO 23: 6267-6267 [Full text]  
• Stengel, D., Bauwens, K., Rademacher, G., Mutze, S., Ekkernkamp, A. (2005). Association between Compliance with Methodological Standards of Diagnostic Research and Reported Test Accuracy: Meta-Analysis of Focused Assessment of US for Trauma. Radiology 236: 102-111 [Abstract] [Full text]  
• Hartling, L., McAlister, F. A., Rowe, B. H., Ezekowitz, J., Friesen, C., Klassen, T. P. (2005). Challenges in Systematic Reviews of Therapeutic Devices and Procedures. ANN INTERN MED 142: 1100-1111 [Abstract] [Full text]  
• van Exel, N.J.A., Koopmanschap, M.A., Scholte op Reimer, W., Niessen, L.W., Huijsman, R. (2005). Cost-effectiveness of integrated stroke services. QJM 98: 415-425 [Abstract] [Full text]  
• Lee, A., Cheng, F. F. K., St Leger, L. (2005). Evaluating health-promoting schools in Hong Kong: development of a framework. HEALTH PROMOT INT 20: 177-186 [Abstract] [Full text]  
• Hargrove, P., Lund, B., Griffer, M. (2005). A Guideline for Applying Systematic Reviews to Child Language Intervention. Communication Disorders Quarterly 26: 226-235 [Abstract]  
• Weisburd, D., Eck, J. E. (2004). What Can Police Do to Reduce Crime, Disorder, and Fear?. The ANNALS of the American Academy of Political and Social Science 593: 42-65 [Abstract]  
• Pitkin, R. M., Scott, J. R. (2004). Evaluating Gynecological Surgical Procedures: Trials and Tribulations. JAMA 291: 1503-1504 [Full text]  
• Lin, S Y-C, Davey, R C, Cochrane, T (2004). Community rehabilitation for older adults with osteoarthritis of the lower limb: a controlled clinical trial. Clin Rehabil 18: 92-101 [Abstract]  
• Marks, H. M (2003). Rigorous uncertainty: why RA Fisher is important. Int J Epidemiol 32: 932-937 [Full text]  
• Eddleston, M., Wilks, M.F., Buckley, N.A. (2003). Prospects for treatment of paraquat-induced lung fibrosis with immunosuppressive drugs and the need for better prediction of outcome: a systematic review. QJM 96: 809-824 [Abstract] [Full text]  
• Chalmers, I. (2003). Trying to do more Good than Harm in Policy and Practice: The Role of Rigorous, Transparent, Up-to-Date Evaluations. The ANNALS of the American Academy of Political and Social Science 589: 22-40 [Abstract]  
• Turlik, M. A., Kushner, D., Stock, D. (2003). Assessing the Validity of Published Randomized Controlled Trials in Podiatric Medical Journals. J. Am. Podiatr. Med. Assoc. 93: 392-398 [Abstract] [Full text]  
• Maher, C. G, Sherrington, C., Herbert, R. D, Moseley, A. M, Elkins, M. (2003). Reliability of the PEDro Scale for Rating Quality of Randomized Controlled Trials. ptjournal 83: 713-721 [Abstract] [Full text]  
• Sudlow, C. L M, Counsell, C. E (2003). Problems with UK government's risk sharing scheme for assessing drugs for multiple sclerosis. BMJ 326: 388-392 [Full text]  
• Jubb, A M (2002). Palliative care research: trading ethics for an evidence base. J. Med. Ethics 28: 342-346 [Abstract] [Full text]  
• van der Schaaf, I. C., Ruigrok, Y. M., Rinkel, G. J.E., Algra, A., van Gijn, J. (2002). Study Design and Outcome Measures in Studies on Aneurysmal Subarachnoid Hemorrhage. Stroke 33: 2043-2046 [Abstract] [Full text]  
• ZIGURAS, S.J., STUART, G.W., JACKSON, A.C. (2002). Assessing the evidence on case management. Br. J. Psychiatry 181: 17-21 [Abstract] [Full text]  
• Ciliska, D., Cullum, N., Marks, S. (2001). Evaluation of systematic reviews of treatment or prevention interventions. Evid. Based Nurs. 4: 100-104 [Full text]  
• Chalmers, I. (2001). Comparing like with like: some historical milestones in the evolution of methods to create unbiased comparison groups in therapeutic experiments. Int J Epidemiol 30: 1156-1164 [Abstract] [Full text]  
• McMahon, A. D. (2001). Observation and Experiment with the Efficacy of Drugs: A Warning Example from a Cohort of Nonsteroidal Anti-inflammatory and Ulcer-healing Drug Users. Am J Epidemiol 154: 557-562 [Abstract] [Full text]  
• Ioannidis, J. P. A., Haidich, A.-B., Pappa, M., Pantazis, N., Kokori, S. I., Tektonidou, M. G., Contopoulos-Ioannidis, D. G., Lau, J. (2001). Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies. JAMA 286: 821-830 [Abstract] [Full text]  
• Wiebe, S., Blume, W. T., Girvin, J. P., Eliasziw, M., the Effectiveness and Efficiency of Surgery for Te, (2001). A Randomized, Controlled Trial of Surgery for Temporal-Lobe Epilepsy. NEJM 345: 311-318 [Abstract] [Full text]  
• Renders, C. M., Valk, G. D., Franse, L. V., Schellevis, F. G., van Eijk, J. Th.M., van der Wal, G. (2001). Long-Term Effectiveness of a Quality Improvement Program for Patients With Type 2 Diabetes in General Practice. Diabetes Care 24: 1365-1370 [Abstract] [Full text]  
• Evans, A., Kalra, L. (2001). Are the Results of Randomized Controlled Trials on Anticoagulation in Patients With Atrial Fibrillation Generalizable to Clinical Practice?. Arch Intern Med 161: 1443-1447 [Abstract] [Full text]  
• Bhandari, M., Guyatt, G. H., Swiontkowski, M. F. (2001). User's Guide to the Orthopaedic Literature: How to Use an Article About a Surgical Therapy. JBJS 83: 916-926 [Abstract] [Full text]  
• Devereaux, P.J., Manns, B. J., Ghali, W. A., Quan, H., Guyatt, G. H. (2001). Reviewing the reviewers: the quality of reporting in three secondary journals. CMAJ 164: 1573-1576 [Abstract] [Full text]  
• Donnelly, C A, Cox, D R (2001). Mathematical biology and medical statistics: contributions to the understanding of AIDS epidemiology. Stat Methods Med Res 10: 141-154 [Abstract]  
• Bigby, M. (2001). Challenges to the Hierarchy of Evidence: Does the Emperor Have No Clothes?. Arch Dermatol 137: 345-346 [Full text]  
• Weisburd, D., Lum, C. M., Petrosino, A. (2001). Does Research Design Affect Study Outcomes in Criminal Justice?. The ANNALS of the American Academy of Political and Social Science 578: 50-70 [Abstract]  
• Kunz, R., Khan, K. S., Neumayer, H.-H., Sacks, H. S., Liu, P.-Y., Anderson, G., Crowley, J. J., Friedman, H. S., Smith, R. P., Meier, P., Benson, K., Hartz, A. J., Concato, J., Shah, N., Horwitz, R. I. (2000). Observational Studies and Randomized Trials. NEJM 343: 1194-1197 [Full text]  
• Guyatt, G. H., Haynes, R. B., Jaeschke, R. Z., Cook, D. J., Green, L., Naylor, C. D., Wilson, M. C., Richardson, W. S., for the Evidence-Based Medicine Working Group, (2000). Users' Guides to the Medical Literature: XXV. Evidence-Based Medicine: Principles for Applying the Users' Guides to Patient Care. JAMA 284: 1290-1296 [Abstract] [Full text]  
• Barton, S. (2000). Which clinical studies provide the best evidence?. BMJ 321: 255-256 [Full text]  
• Benson, K., Hartz, A. J. (2000). A Comparison of Observational Studies and Randomized, Controlled Trials. NEJM 342: 1878-1886 [Abstract] [Full text]  
• Concato, J., Shah, N., Horwitz, R. I. (2000). Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs. NEJM 342: 1887-1892 [Abstract] [Full text]  
• Dickinson, K., Bunn, F., Wentz, R., Edwards, P., Roberts, I. (2000). Size and quality of randomised controlled trials in head injury: review of published studies. BMJ 320: 1308-1311 [Abstract] [Full text]  
• Adetugbo, K., Williams, H. (2000). How Well Are Randomized Controlled Trials Reported in the Dermatology Literature?. Arch Dermatol 136: 381-385 [Abstract] [Full text]  
• Mariani, L., Marubini, E. (2000). Content and Quality of Currently Published Phase II Cancer Trials. JCO 18: 429-429 [Abstract] [Full text]  
• Goodacre, S. W (2000). Should we establish chest pain observation units in the UK? A systematic review and critical appraisal of the literature. Emerg. Med. J. 17: 1-6 [Abstract] [Full text]  
• Roberts, I., Diguiseppi, C. (1999). Smoke alarms, fire deaths, and randomised controlled trials. Inj. Prev. 5: 244-245 [Full text]  
• Juni, P., Witschi, A., Bloch, R., Egger, M. (1999). The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis. JAMA 282: 1054-1060 [Abstract] [Full text]  
• McKee, M., Britton, A., Black, N., McPherson, K., Sanderson, C., Bain, C. (1999). Methods in health services research: Interpreting the evidence: choosing between randomised and non-randomised studies. BMJ 319: 312-315 [Full text]  
• Chalmers, I. (1998). Unbiased, relevant, and reliable assessments in health care. BMJ 317: 1167-1168 [Full text]  

Online poll

Find out more >>