Investigating disparity between global grades and checklist scores in OSCEs

Background: When measuring assessment quality, increasing focus is placed on the value of station level metrics in the detection and remediation of problems in the assessment. Aims: This paper investigates how disparity between checklist scores and global grades in an OSCE can provide powerful new insights at the station level whenever such disparities occur, and develops metrics to indicate when this is a problem. Method: This retrospective study uses OSCE data from multiple examinations to investigate the extent to which these new measurement of disparity complements existing station level metrics. Results: In stations where existing metrics are poor, the new metrics provide greater understanding of the underlying sources of error. Equally importantly, stations of apparently satisfactory ‘quality’ based on traditional metrics are shown to sometimes have problems of their own – with a tendency for checklist score ‘performance’ to be judged stronger than would be expected from the global grades awarded. Conclusions: There is an ongoing tension in OSCE assessment between global holistic judgements and the necessarily more reductionist, but arguably more objective, checklist scores. This paper develops methods to quantify the disparity between these judgements, and illustrates how such analyses can inform ongoing improvement in station quality.