Going beyond hawks and doves – measuring degrees of examiner misalignment in OSCEs

Background

Minimising examiner differences in scoring in OSCEs is key in supporting the validity of the assessment outcomes. However, the common classification of extreme examiners as ‘hawks’ or ‘doves’ can be overly simplistic. Rather, it is the difference in combined patterns of scoring/grading across OSCE circuits that better indicate poor levels of agreement between examiners - this misalignment can unfairly advantage particular groups of candidates in comparison with others in other circuits.

Methods and materials

In this study, a new measure of differences in examiner scoring is presented that quantifies the different combined patterns of scoring in global grades and station total scores for pairs of examiners assessing in the same station but in different circuits. Over 10,000 separate station administrations from 2016 to 2024 are analysed from the UK exam for international medical graduates who want to work in the NHS (PLAB2). The new misalignment measure is based on calculating the area between separate examiners’ individual borderline regression lines.

Results and conclusions

Particular station examples are presented where alignment between examiners is excellent and where it is poor. Longitudinal evidence suggests that average misalignment has declined over time suggesting that a range of interventions/developments of PLAB2 to improve calibration between examiners, and scoring practices more generally, have had some success. Variation in misalignment does not vary much between different types of station (e.g. standard/prescription/practical/simulation).

In challenging the ‘hawks’/‘doves’ paradigm, this paper contributes to the theoretical debate around the nature of examiner stringency in OSCEs. It also presents a new empirical misalignment measure which can be used to provide additional validity evidence for an OSCE-type assessment. Further work is needed to develop the metric to larger scale OSCEs.