Introduction

Top Abstract Introduction Participants and methods Results Discussion References

The UK government, as part of a bid to modernise health services and to make them more convenient to use, has introduced NHS walk-in centres and commissioned their evaluation by an independent team. Our study forms one component of that evaluation. We aimed to determine whether walk-in centres, staffed mainly by nurses, provide adequate and safe clinical care to a range of patients, and how quality of care in these centres compares with that provided in general practice and by NHS Direct.

The only previous evaluation of UK walk-in centres found quality of care disappointing overall, some care positively dangerous, and considerable variation between sites.¹ The study considered just eight centres, with no comparative group and several other flaws in its methods.²

Several methods exist for evaluating quality of clinical care. Systematic biases are associated with indirect methods, such as review of medical records or self-reporting by clinicians.³ Direct methods of observation, such as video recording, require large numbers of consultations to be reliable, because of the variable case mix seen in clinical practice. In addition, clinicians may change their behaviour when observed. The use of covert standardised patients for quality assessment is gaining favour because it avoids these methodological problems.

Standardised patients are people trained to portray a clinical scenario for teaching or research purposes. They have been used in the United States for over 30 years and are the subject of a major body of educational research. ^{4 5} Several studies have employed them to explore what occurs in clinical practice, including studies concerned with quality assurance.^6-11 Whereas this use of standardised patients is novel in the United Kingdom, experience from elsewhere suggests it is a valid and reliable approach.¹² We used standardised patients to compare care in walk-in centres with that in the primary care settings of general practice and NHS Direct: we focus on what is necessary, rather than ideal, with respect to patient care.

	Participants and methods

Top Abstract Introduction Participants and methods Results Discussion References

Recruitment of clinical sites
For logistical reasons we approached sites in three geographical areas, in and around Bristol, Birmingham, and London. We invited general practices involved in research or teaching to participate, and we approached walk-in centres and NHS Direct sites on the basis of their geography. Twenty out of 25 (80%) walk-in centres, 11 out of 12 (92%) NHS Direct sites, and 24 out of 62 (39%) practices agreed to participate: we included the first 20 practices to respond. Twelve practices (19%) declined to take part, and the remainder did not respond.

Ethical approval for our study was obtained from a multicentre research ethics committee. Participating sites all gave informed consent for contact with standardised patients but were unaware when they would consult. Participating sites were ensured anonymity: outcome data were presented such that individuals could not be identified.

Selection of clinical scenarios
We chose five clinical scenarios, largely to represent problems likely to be presented by patients to walk-in centres (box 1). The scenario on postcoital contraception was intended to assess management of a common, straightforward problem, the scenario on chest pain to assess ability to exclude a potentially serious diagnosis and reassure accordingly, and the scenario on sinusitis to assess issues around antibiotic prescribing. The scenario on headache was devised to assess ability to explore psychosocial issues, and the scenario on asthma to assess the history taking on drugs and awareness of the side effects of the drugs.

Derivation of assessment criteria
We assessed clinical care against prospectively determined standards. We constructed lists of essential criteria for the adequate management of each scenario by a stepwise procedure, based on the Delphi process.¹³ The research team identified potential criteria for essential actions relevant to the five scenarios, by review of relevant literature and consensus based guidelines. We posted lists of potential criteria to a consensus panel of nine general practitioners and five nurse practitioners. Criteria were included as items in the final checklists if, after two rounds, 93% or more members of the panel agreed they were essential. Checklists consisted of eight to 17 items, depending on scenario and setting, with items grouped into "history taking," "examination," and "diagnosis, advice, and treatment." For each scenario, checklists for the walk-in centres and general practices were identical, with the NHS Direct checklist containing all but the examination items.

Standardised patient consultations
Fifteen professional role players involved in medical education played standardised patients. They were chosen to match the roles assigned to them in terms of age and sex. Five role players, each portraying one scenario, worked in each locality. Each role player visited a particular walk-in centre or general practice once, but owing to the smaller number of sites for NHS Direct, contacted one NHS Direct site up to three times. Overall, 305 contacts were planned, 100 in walk-in centres, 100 in practices, and 105 with NHS Direct. Consultations took place over a 13 week period from July to September 2001. Standardised patients visited the general practices as temporary residents to avoid the logistical problems of registration as a new patient. They completed checklists immediately after consultations.

Training of role players
The role players were provided with detailed information about the sites and personal data relating to their roles. They received a day's training in role portrayal and clinician assessment, during which data on accuracy of portrayal and reliability of assessment were collected.

Accuracy of portrayal of standardised patients
Accuracy of portrayal, defined as the proportion of predefined clinical features presented correctly, was measured by two observers independently completing a performance item checklist during the final round of role playing.¹⁴ Across all scenarios, 1038 of 1164 (89%) clinical features were portrayed correctly. Consistency of portrayal was maintained by role players watching a videotape of their performance before each consultation during the study.

Reliability of assessment by standardised patients
Reliability of assessment was measured by comparing role players' responses to each checklist item with gold standard responses. These were the responses of two general practitioners who independently watched videotaped consultations from the training day and completed the checklists blind to role players' responses. Across all scenarios and role players =0.8. The reliability between role players was measured by the role players independently assessing each other's videotaped consultations. The responses of two role players observing the same videotaped scenario were compared. Across all scenarios and role players =0.9. The reliability of the role player over time was measured by role players assessing their training day videotapes half way through the study and comparing responses to those on the training day: =0.7.

Validity of standardised patients
Participating clinicians were encouraged to contact the study team, anonymously via an answer machine, if they suspected a consultation with a standardised patient. Twenty three calls were madefive about true standardised patients and 18 about true patients. Thus the detection rate was 1.7% (5 of 297), and the positive predictive value was 21.7% (5 of 23).

Analysis
For each consultation we calculated a score representing the proportion of essential criteria fulfilled for all items and separately for the three subgroups of items (history taking, examination, and diagnosis, advice, and treatment). We calculated the means of the all item and subgroup scores for each of the three primary care settings, with 95% confidence intervals calculated with design weighted survey estimators. We then undertook estimation of differences between mean scores for the three settings by using multivariable regression models, with setting and scenario included as fixed effects. We accounted for non-independence of repeat observations within individual centres in two alternative ways: models were estimated with design weighted survey estimators and with individual centre included as a random effect, as implemented in Stata.¹⁵ For each model we then added the interaction between primary care setting type and scenario and tested for significance. Where this interaction was significant we estimated separate models excluding first NHS Direct consultations,(allowing comparison of walk-in centres and practices) and second general practice consultations (allowing comparison of walk-in centres and NHS Direct). Within each scenario we estimated mean scores, mean score differences between settings, and 95% confidence intervals for these statistics by using design weighted survey estimators. Scores for the three subgroups of items were not normally distributed, so we repeated all analyses with the Wilcoxon rank sum test. In no case was the result of the non-parametric test significantly different to the analyses presented in tables 1 and 2. Finally, we calculated the variation in scores (standard deviation) within each setting.

The results of the design weighted models and random effects models were similar in all cases, and thus only the design weighted analyses are presented as these tended to be slightly more conservative than the random effects models. For the models based on all items and those restricted to history taking and diagnosis, advice, and treatment, the interactions between primary care setting and scenario were significant. Thus, we present the results of these analyses and analysis of examination items separately for comparisons between walk-in centres and general practice and then between walk-in centres and NHS Direct.

	Results

Top Abstract Introduction Participants and methods Results Discussion References

Data were collected on 297 of the planned 305 consultations, 99 in each setting. One consultation in general practice and six with NHS Direct were not completed owing to problems with access, and data on one consultation were lost in the post. Most (91%) consultations in walk-in centres were exclusively with a nurse, whereas most (96%) in general practice were with a doctor. A minority (3%) of consultations with NHS Direct involved a call handler rather than a nurse.

Quality of care

Walk-in centres versus general practice
Considering all scenarios together, walk-in centres achieved a significantly greater mean score for all essential items than did general practices, but there were between scenario differences. Quality of care for postcoital contraception and asthma was significantly better in walk-in centres than in general practices, that for sinusitis and headache was similar in the two settings, and that for chest pain was better in general practice, although not significantly so (table 1).

Walk-in centres versus NHS Direct
Considering all scenarios together, walk-in centres achieved a significantly greater mean score for all items than did NHS Direct, again with between scenario differences. Quality of care for postcoital contraception and asthma was significantly better in walk-in centres, with no significant difference for other scenarios (table 1). Overall, and for postcoital contraception and asthma, walk-in centres scored significantly better on history taking, with the same pattern for diagnosis, advice, and treatment (table 2).

Referral
About a quarter (26%) of consultations in walk-in centres and four fifths (82%) with NHS Direct resulted in referral (table 3). Patients were advised to go to an emergency department in 5% of consultations in walk-in centres and 13% with NHS Direct, but not from consultations in general practice.

Between site variability
Considering all scenarios, the variability of quality scores, expressed as standard deviation, was highest for NHS Direct and lowest for general practice (table 4). However, variability of history taking for all scenarios was higher in general practice. Variability of quality in the three settings varied across scenarios. NHS Direct's high variability was largely the result of the postcoital contraception scenario, where referral may have substituted for consistent performance on essential items.

Access
Contacting NHS Direct was sometimes time consuming and unsatisfactory. About one quarter (25 of 99) of completed calls involved "call back" (mean wait 33 minutes). Three consultations with NHS Direct were not completed owing to length of wait for "call back." Fewer problems were reported accessing general practice and walk-in centres.

	Discussion

Top Abstract Introduction Participants and methods Results Discussion References

Walk-in centres provided equivalent if not better quality clinical care than both general practice and NHS Direct, for the range of clinical conditions assessed. Substantial variation existed in quality between scenarios, with walk-in centres performing particularly well for postcoital contraception and asthma. They performed significantly less well than general practice in examination of chest pain and diagnosis, advice, and treatment of sinusitis. These inadequacies do not necessarily provide evidence of poor quality care. Firstly, examination may be of secondary importance in the overall management of musculoskeletal chest pain. Secondly, significantly lower antibiotic use by walk-in centres for sinusitis may reflect acknowledged difficulties identifying patients who will benefit from antibiotics. Walk-in centres' better performance was particularly noticeable for history taking, perhaps owing to the longer consultations undertaken in this setting.¹⁶

Considerable variability exists in the quality of care provided by individual sites. Variability among walk-in centres is greater than among practices, possibly due to variation in training, experience, and competencies of nurses working in walk-in centres.¹⁷ However, variability of history taking is greater in general practice, perhaps owing to the use of clinical assessment software in walk-in centres.

Although walk-in centres provide better quality of care, both the centres and NHS Direct referred a proportion of standardised patients elsewhere. Some of the referrals were to emergency departments, which may not be appropriate for the conditions studied and raises questions about the impact on workload of other providers, as addressed elsewhere in the evaluation.

Comparison with other studies
The accuracy of portrayal (90%) by standardised patients compared well with previous studies, as did their reliability (=0.7 to 0.9). ^{14 18} Detection (1.7%) and positive predictive value (21.7%) also compared favourably.^18-21

Although checklists were meant to consist of essential items, the proportions of items achieved in all three settings were low, a finding consistent with another study that assessed general practitioners against peer determined standards.²² Performance may not reflect competence, as clinicians exhibit efficiency by only carrying out what is necessary at a particular moment.²²

Methodological issues
The study's main limitations are non-random sampling of participating sites, use of a limited number of scenariossome more discriminating than othersand use of novel assessment checklists. Participating sites, particularly general practices, were likely to be more interested in the research question and may have provided a higher quality of care, possibly attenuating the study findings. Scenarios were chosen as typical of those seen in walk-in centres and because they were appropriate for portrayal by standardised patients. Scenarios necessitating the presence of abnormal findings or potentially involving certain types of physical examination or referral to third parties could not be included. The finding that walk-in centres offer safe care cannot necessarily be extrapolated to all clinical conditions, although we aimed to assess important, generalisable aspects of clinical care.

The ability of the scenarios to discriminate between settings was variable. Scores on the scenario for postcoital contraception were highly variable both overall and for NHS Direct in particular, perhaps because NHS Direct's management of this scenario always involved referral. Consequently, performance on history taking and advice and treatment may have been less thorough.

Scenarios did not involve children and elderly patients, who attend general practice most often. Nor did they assess some of the supposed strengths of general practicefor example, continuity and availability of past medical records. Also, the methodology, requiring all standardised patients to be temporary residents, did not lend itself to the assessment of some of these strengths. Thus, the findings do not suggest care in general practice is inferior to walk-in centres, rather that walk-in centres perform adequately and safely compared with controls.

The development of assessment checklists used by standardised patients in previous studies has received little attention, although their construction is crucial to the reliability and validity of assessment.²³ Checklists used in this study seem to have high face validity and content validity, although few data were collected on other properties, such as reproducibility.

Conclusion
Standardised patients have a useful role in the assessment of quality in the UK primary care setting. Also, walk-in centres provide adequate, safe clinical care to a range of patients, compared with general practice and NHS Direct. However, the cost effectiveness of walk-in centres and their impact on workload of other healthcare providers requires further assessment.