Homer, MS orcid.org/0000-0002-1161-5938, Pell, G, Fuller, R et al. (1 more author) (2016) Quantifying error in OSCE standard setting for varying cohort sizes: A resampling approach to measuring assessment quality. Medical Teacher, 38 (2). pp. 181-188. ISSN 0142-159X
Abstract
Background: The use of the borderline regression method (BRM) is a widely accepted standard setting method for OSCEs. However, it is unclear whether this method is appropriate for use with small cohorts (e.g. specialist post-graduate examinations).
Aims and methods: This work uses an innovative application of resampling methods applied to four pre-existing OSCE data sets (number of stations between 17 and 21) from two institutions to investigate how the robustness of the BRM changes as the cohort size varies. Using a variety of metrics, the ‘quality’ of an OSCE is evaluated for cohorts of approximately n=300 down to n=15. Estimates of the standard error in station-level and overall pass marks, R2 coefficient, and Cronbach’s alpha are all calculated as cohort size varies.
Results and conclusion: For larger cohorts (n>200), the standard error in the overall pass mark is small (less than 0.5%), and for individual stations is of the order of 1-2%. These errors grow as the sample size reduces, with cohorts of less than 50 candidates showing unacceptably large standard error. Alpha and R2 also become unstable for small cohorts. The resampling methodology is shown to be robust and has the potential to be more widely applied in standard setting and medical assessment quality assurance and research.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | (c) 2015, Informa UK Ltd. This is an Accepted Manuscript of an article published by Taylor & Francis in Medical Teacher on 24 April 2015 , available online: http://dx.doi.org/10.3109/0142159X.2015.1029898 |
Keywords: | Borderline regression; bootstrapping; cohort size; error; OSCE; resampling; standard setting |
Dates: |
|
Institution: | The University of Leeds |
Academic Units: | The University of Leeds > Faculty of Education, Social Sciences and Law (Leeds) > School of Education (Leeds) The University of Leeds > Faculty of Medicine and Health (Leeds) > School of Medicine (Leeds) > Leeds Institute of Medical Education > Medical Education Unit (Leeds) |
Depositing User: | Symplectic Publications |
Date Deposited: | 15 Apr 2015 10:51 |
Last Modified: | 28 Sep 2020 15:46 |
Published Version: | http://dx.doi.org/10.3109/0142159X.2015.1029898 |
Status: | Published |
Publisher: | Taylor & Francis |
Identification Number: | 10.3109/0142159X.2015.1029898 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:84990 |