Estimating and comparing the reliability of a suite of workplace-based assessments: an obstetrics and gynaecology setting

This paper reports on a study that compares estimates of the reliability of a suite of workplace based assessment forms as employed to formatively assess the progress of trainee obstetricians and gynaecologists. The use of such forms of assessment is growing nationally and internationally in many specialisms, but there is little research evidence on comparisons by procedure/competency and form-type across an entire specialty. Generalisability theory combined with a multilevel modelling approach is used to estimate variance components, G-coefficients and standard errors of measurement across 13 procedures and three form-types (mini-CEX, OSATS and CbD). The main findings are that there are wide variations in the estimates of reliability across forms, although there is little evidence that reliability varies systematically by form-type. However, the results suggest that the guidance on assessment within the specialty does not always allow for enough forms per trainee to ensure that the reliability of the assessment process is adequate. Methodologically, the problems of accurately estimating reliability in these contexts through the calculation of variance components and, crucially, their associated standard errors are considered. The importance of the use of appropriate methods in such calculations is emphasised, and the unavoidable limitations of research naturalistic settings are discussed.