Learning in practice

Cost effectiveness of continuing professional development in health care: a critical review of the evidence

C A Brown, research fellow ^a, C R Belfield, assistant director for research ^b, S J Field, regional postgraduate dean ^c. 

^a Centre for Research in Medical and Dental Education, School of Education, University of Birmingham, Birmingham B15 2TT, ^b National Center for the Study of Privatization in Education, Teachers College, Columbia University, New York, USA, ^c West Midlands Deanery, Birmingham

Correspondence to: C A Brown C.A.Brown{at}bham.ac.uk

Continuing professional development (CPD) for healthcare professionals is an important strategic instrument for improving health. The Department of Health identifies CPD as a way of maintaining standards of care; improving the health of the nation; and recruiting, motivating, and retaining high quality staff.¹ To this end, direct NHS spending on continuing professional development in 1999-2000 was about £1bn ($1.6bn). ^{2 3} If we regard CPD as any method to improve health professionals' skills the total resources devoted to it are probably much greater, particularly with the recent increased participation in response to the need for recertification and revalidation.⁴ To ensure the maximum gain from participation in CPD, these resources must be used efficiently.

To assess the efficiency of participating in CPD, economic criteria are needed. Resources for health care are scarce, and money spent on CPD could otherwise be used for direct patient care. These opportunity costs are explicitly considered in the economic methods of cost benefit analysis and cost effectiveness analysis. The literature contains various reviews of cost effectiveness analysis in both health care ^{5 6} and education.⁷ Such articles explain why cost effectiveness analysis (or another method of economic evaluation) is essential and how such evaluation should be undertaken, and they clearly define the set of economic terms (such as cost benefit analysis, cost effectiveness analysis, rate of return, and opportunity cost) that need to be incorporated into this type of research. Casebeer et al highlighted the need for economic evaluation of CPD activities,⁸ but they emphasised the use of cost benefit analysis, which requires monetary values to be assigned to measures of effectiveness. Cost benefit analysis is generally used to ascertain whether an intervention should be undertaken. Cost effectiveness analysis is used to decide which interventions (out of a number of alternatives) should be undertaken.

However, cost effectiveness analysis in education research is rare. ^{7 9 10} This is partly because of limited training for researchers, antipathy toward (economic) analysis that might constrain policy, and the dearth of significant results in studies of educational effectiveness.⁹ The quality of such research is also often poor: Clune found that only 1% of 541 "cost-effectiveness" studies of elementary and secondary education between 1991 and 1996 could be considered reliable, with strong design and analysis.¹⁰ In contrast, economic evaluation of healthcare technologies is increasing, and the methods for making such analysis are rapidly evolving.¹¹ (Even here, however, critical reviews identify a substantial number of weak cost benefit and cost effectiveness analyses.^12-14)

There is a sizeable literature on the effectiveness of CPD interventions (over 100 randomised controlled trials are thoroughly reviewed by Davis et al ^{15 16} ), but the evidence on the cost effectiveness of CPD has not been systematically investigated. In this article we therefore investigate the quantity and quality of the evidence on the cost effectiveness of CPD for healthcare professionals.

	Methods

Top Methods Results Conclusions References

Search strategy
The aim of our search strategy was to identify evaluations of CPD interventions that included some form of economic analysis. We searched the main bibliographic databases (Medline, CINAHL, Web of Science, BIDS, ERIC, University of York Centre for Reviews and Dissemination, and the Research and Development Resource Base in CME at the University of Toronto) from their inception and without any language restrictions. We then searched relevant papers for additional references and attempted to identify grey literature (via internet searches, for example). The key words we used in the search included one term relating to CPD and education ("[continuing] professional development," "[continuing professional] medical education," and "postgraduate medical education") and one relating to costs ("costs," "efficiency," "cost-benefit," and "cost-effectiveness").

Inclusion criteria
We included studies in the review if they considered an educational intervention of CPD undertaken by health professionals. There are several different definitions of CPD, but we adopted an inclusive definition"post-registration acquisition of skills or knowledge by healthcare professionals." This definition includes formal programmes as well as use of guidelines, mentoring, and independent study.

We included any outcomes that measured the costs and effectiveness of the intervention, although we excluded papers described as costs studies and cost saving exercises that failed to report the cost of the intervention. Neither type of study allows an economic appraisal of the effectiveness of the intervention. We applied no methodological restrictions, such as restricting our review to studies that used randomised or controlled experimental techniques.

Review method of selected studies
We reviewed the studies in terms of their educational aims, the mode of education, and the results, and we evaluated each for the appropriate use of economic methods. Specifically, we used the six criteria of Udvarhelyi et al,¹² which were designed to assess health technologies. The economic methods for evaluating health technologies and educational interventions are sufficiently similar that this approach is legitimate.

	Results

Top Methods Results Conclusions References

Results of search
Our search revealed nine papers, of which seven were cost effectiveness studies and two were cost benefit analyses (table 1).^17-26 This is obviously a meagre crop of literature with limited scope for generalisation.

Results of review
Table 2 shows the results of our appraisal of the studies. On average, the studies met about half of the six criteria (a similar result to that reported by Udvarhelyi et al,¹² where the average healthcare technology study met three criteria). Specifically, most studies failed to apply a full appraisal of costs (probably the most important criterion), discounting, or sensitivity analysis, and only five studies provided a summary measure of cost effectiveness.

Two of the papers applied a cost benefit analysis, allowing rates of return to the education programme to be calculated, although neither paper offered such calculations. ^{25 26} Rates of return allow the (discounted) money costs and benefits of an intervention to be compared. To calculate the rate of return, the excess of benefits over costs is divided by the costs of the intervention. For these two studies, the rates of return were 39%²⁶ and greater than 10 000%.²⁵ We could also calculate rates of return for two of the cost effectiveness studies: these were 460% and 63%. ^{22 23} Although these rates may be accurate for these particular interventions, they seem extremely high when compared with the base interest rate used by the Bank of England (currently 4%, which includes a premium that adjusts for inflation). Moreover, one of these studies²⁶ did not investigate what most health professionals would consider CPD; instead, it used the human capital model of education²⁷ to compare the economic gain arising from two specialties.

Ideally, the evidence base should allow for comparison of the results of papers that identify the same healthcare outcome. This is possible for four of the nine studies across two different healthcare outcomes. Given the appraisal in table 2, however, any inference drawn from these comparisons can be only tentative. Two studies compared the cost effectiveness of different training and support conditions for increasing screening for and treatment of alcohol addiction. ^{17 18} The results are conflicting, as the first study found training with no physician support to be the most cost effective mode whereas the second found the training with support to be most cost effective.

The studies by Modell et al ^{19 20} and Stevens et al²⁴ addressed increasing the use of screening services. Modell et al found that effectiveness increased after formal general practitioner education, but so did costs. Knowledge of the availability of funding (or a cost benefit analysis) is therefore required to ascertain whether the education should be implemented. Stevens et al found no significant difference in effectiveness between the control and intervention groups (with increased costs for the intervention arm because of outreach visits). In this instance, the additional expenditure was clearly not cost effective. However, Modell et al included the resource implications (costs of additional screenings) in their analyses, whereas Stevens et al did not.

Finally, two papers seemed to use an inappropriate method of economic analysis. Morriss et al calculated the rate of return to seminar based education (460%), as the main aim of the study was to reduce direct healthcare costs.²² While the study itself offers a calculation of cost effectiveness, the aim of the research and lack of a comparison educational mode suggest that it should be reclassified as a cost benefit analysis.

Steele et al tested two different educational interventions to determine which was the most cost effective at reducing outpatient prescribing costs.²³ Given that the outcomes were therefore measured in monetary units (dollars saved), a cost benefit approach is appropriate, as cost effectiveness cannot be calculated. Although a rate of return to the outreach intervention can be calculated (peer comparison feedback showed no difference in prescribing costs and thus had a negative rate of return), the cost of the intervention was not fully included. Any rate of return estimate would therefore overstate the benefit of the intervention.

	Conclusions

Top Methods Results Conclusions References

We reviewed the literature that assesses the cost effectiveness of CPD interventions in health care. Our literature search shows that economic evaluations of CPD are rare. Furthermore, the evidence that does exist is not consistent in its approach to costing or analysis. This leaves no scope for a full systematic review of a particular educational intervention, and a single trial of an intervention does not allow decisions to be made on the strength of the results. Overall, it is impossible to draw any feasible conclusions regarding the cost effectiveness of different modes of CPD for healthcare professionals.

The external validity of the existing studies may be impaired for many reasons. This raises concerns over the extent to which the results can be generalised and used to inform policy. Both costs and technologies change over time, along with differences in input prices and technologies across settings (and especially across countries). Also, the studies mainly focused on intermediate rather than final outcomes: any effects of CPD on patient health therefore have to be inferred from changes in the behaviour of physicians.

Interpreting the results is difficult since uniform methods of costing or analysis were not applied. The "ingredients" included in the costs analysis were not identified in a standard form: development costs and the opportunity costs of the participants needed to be included. In addition, the resource implications of an educational intervention (such as for additional screenings) would need to be included in a cost benefit analysis but not in a cost effectiveness analysis. Often, the evidence seemed to be directed at both forms of analysis, despite these analyses being methodologically and purposively distinct.

Many of the studies were inadequate to meet the objective of deciding which is the most cost effective mode of CPD. The aim of cost effectiveness analysis is to aid decisions between interventions, and thus studies need to test the relative cost effectiveness of two or more alternatives. Several of the studies compared just one educational mode with a control group of standard care. Were monetary values attached to the outcomes and a cost benefit analysis applied, a decision could be made on whether the intervention should be implemented. Otherwise, a value judgment is required to determine if, say, the cost per additional screening represents value for money. When two or more educational designs are compared cost effectiveness can be more readily understood if the educational goal is held constant. The cost effectiveness analysis will then discriminate between the types of CPD.

Implications of our study
The literature on the effectiveness of CPD continues to expand, but effectiveness is not a sufficient criterion for implementation. For scarce resources to be devoted to CPD, the relative cost effectiveness of different educational interventions must be established, and those offering the most value for money must be implemented. An investment in high quality evaluations would therefore reap health benefits for the public and ease policy makers' decisions about resource allocation. Cost effectiveness analysis must be applied to studies of educational effectiveness, and this should be possible with the methods detailed by Levin and McEwan⁷ or Drummond and Jefferson.⁵ At present, notwithstanding the substantial resource commitment to CPD, evidence on the cost effectiveness of CPD is completely inadequate.

	Acknowledgments

We thank the BMJ's referee and the editorial board for their helpful comments.

Contributors: CAB, who is guarantor for the study, collected the data and helped analyse the data and write the article. CRB helped analyse the data and write the article. SJF helped write the article.

	Footnotes

Funding: Our research is supported by the Department of Health.

Competing interests: None declared.

	References

Top Methods Results Conclusions References

1.	Department of Health. Our healthier nation: a contract for health. London: Stationery Office, 1998.
2.	Belfield CR, Brown CA, Thomas HR, Field S, Cooper RF. Cost-effective continuing professional development in the NHS. London: Department of Health, 2001.
3.	Budgetary arrangements for funding professional education and vocational training in the general dental and community services of the NHS. London: Department of Health, NHS., 1999. (Letter 28 April 1999)
4.	Peck C, McCall M, McLaren B, Rotem T. Continuing medical education and continuing professional development: international comparisons. BMJ 2000; 320: 432-435[Free Full Text].
5.	Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ 1996; 313: 275-283[Free Full Text].
6.	Drummond MF. Methods for the economic evaluation of health care programmes. Oxford: Oxford University Press, 1997.
7.	Levin H, McEwan. Cost-effectiveness analysis. London: Sage, 2000.
8.	Casebeer L, Raichle L, Kristofco R, Carillo A. Cost-benefit analysis: review of an evaluation methodology for measuring return on investment in continuing education. J Contin Educ Health Prof 1997; 17: 224-227.
9.	Levin HM. Waiting for Godot. Cost-effectiveness analysis in education. In: Light RJ, ed. Evaluation findings that surprise: New directions for evaluation. San Francisco: Jossey-Bass, 2001:55-65.
10.	Clune W. Methodological strength and policy usefulness of published research on cost-effectiveness in education [draft report]. In: Madison, WI: University of Wisconsin, 1999.
11.	Gray A. Economic evaluation. In: Dawes M, Davies P, Gray A, Mant J, Seers K, Snowball R, eds. Evidence-based practice: a primer for health care professionals. London: Churchill Livingstone, 1999:101-110.
12.	Udvarhelyi IS, Colditz GA, Rai A, Epstein AM. Cost-effectiveness and cost-benefit analyses in the medical literature. Are the methods being used correctly? Ann Intern Med 1992; 116: 238-244.
13.	Zarnke KB, Levine MAH, O'Brien BJ. Cost-benefit analyses in the health care literature: don't judge a study by its label. J Clin Epidemiol 1997; 50: 813-822[CrossRef][Web of Science][Medline].
14.	Barber JA, Thompson SG. Analysis and interpretation of cost data in randomised controlled trials: review of published studies. BMJ 1998; 317: 1195-1200[Abstract/Free Full Text].
15.	Davis DA, Thompson MA, Oxman AD, Haynes B. Changing physician performance: a systematic review of the effect of continuing medical education strategies. JAMA 1995; 274: 700-705[Abstract/Free Full Text].
16.	Davis D. Does CME work? An analysis of the effect of educational activities on physician performance or health care outcomes. Int J Psychiatry Med 1998; 28: 21-39[Web of Science][Medline].
17.	Gomel MK, Wutzke SE, Hardcatle DM, Lapsley H, Reznik RB. Cost-effectiveness of strategies to market and train primary health care physicians in brief intervention techniques for hazardous alcohol use. Soc Sci Med 1998; 47: 203-211.
18.	Kaner EFS, Lock CA, McAvoy BR, Heather N, Gilvarry E. A RCT of three training and support strategies to encourage implementation of screening and brief alcohol intervention by general practitioners. Br J Gen Pract 1999; 49: 353-356[Web of Science][Medline].
19.	Modell M, Wonke B, Anionwu E, Khan M, See Tai S, Lloyd M, et al. A multidisciplinary approach for improving services in primary care: randomised controlled trial of screening for haemoglobin disorders. BMJ 1998; 317: 788-791[Abstract/Free Full Text].
20.	Beech R, Khan M, Modell M. The costs and consequences of a practice based genetic screening facilitator [commissioned report]. In: London: United Medical and Dental Schools of Guy's and St Thomas' Hospitals, 1998.
21.	Morrison J. Improving quality of referral: a cost-effectiveness evaluation of clinical guidelines for infertility management across the interface [commissioned report]. In: Glasgow: University of Glasgow, 1999.
22.	Morriss R, Gask L, Ronalds C, Downes-Grainger E, Thompson H, Leese B, et al. Cost-effectiveness of a new treatment for somatized mental disorder taught to GPs. Fam Pract 1998; 15: 119-125[Abstract/Free Full Text].
23.	Steele MA, Bess DT, Franse VL, Graber SE. Cost-effectiveness of two interventions for reducing outpatient prescribing costs. DICP 1989; 23: 497-500[Abstract].
24.	Stevens SA, Cockburn J, Hirst S, Jolley D. An evaluation of education outreach to general practitioners as part of a statewide cervical screening program. Am J Public Health 1997; 87: 1177-1181[Abstract/Free Full Text].
25.	Rutz W, Carlsson P, von Knorring L, Walinder J. Cost-benefit analysis of an educational program for general practitioners by the Swedish Committee for the prevention and treatment of depression. Acta Psychiatr Scand 1992; 85: 457-464[Web of Science][Medline].
26.	Prashker MJ, Meenan RF. Sub-specialty training: is it financially worthwhile? Ann Intern Med 1991; 115: 715-719.
27.	Becker GS. Human capital. Chicago: University of Chicago Press, 1993.

(Accepted 13 August 2001)