Advancing the objective structured clinical examination: sequential testing in theory and practice

Introduction

Models of short term remediation for failing students are typically associated with improvement in candidate performance at retest, yet are costly to deliver (particularly for performance retest with Objective Structured Clinical Examinations (OSCEs)). There is increasing evidence that these traditional models are associated with longitudinal underperformance of candidates.

Methods

Rather than a traditional OSCE model, sequential testing involves a shorter ‘screening’ test format, with an additional ‘sequential’ test for candidates who fail to meet the standards of the screening test. For those tested twice, overall pass/fail decisions are then made on the full sequence of tests. The impact of sequential assessment on student performance cost of assessment delivery and overall reliability was modelled using prior data from the final, graduating OSCE of an undergraduate medical degree programme.

Results

The initial modelling predicted significant improvements to reliability in the critical area, reflected in pilot results (with 14% of students, n=228, required to sit the sequential OSCE). One student (0.4%) was identified as a false positive, i.e. under the old system would have passed the OSCE, but failed on extended testing. Nine students (4%) who would have required OSCE retests under the prior system passed the full sequence and were therefore able to graduate at the normal time without loss of earnings. Overall reliability was estimated as 0.79 for the full test sequence, with significant cost saving realised.

Discussion

Introducing sequential testing for OSCEs increases reliability for borderline students since the increased number of observations implies that ‘observed’ student marks are closer to the ‘true’ marks. However, the station level quality of the assessment needs to be sufficiently high for the full benefits in terms of reliability to be achieved. Introduction of such a system has financial benefits, good validity inferences and has proved acceptable to students and other stakeholders.