Homer, M orcid.org/0000-0002-1161-5938, Fuller, R, Hallam, J orcid.org/0000-0002-1044-0515 et al. (1 more author) (2020) Shining a spotlight on scoring in the OSCE: Checklists and item weighting. Medical Teacher, 42 (9). pp. 1037-1042. ISSN 0142-159X
Abstract
Introduction: There has been a long-running debate about the validity of item-based checklist scoring of performance assessments like OSCEs. In recent years, the conception of a checklist has developed from its dichotomous inception into a more ‘key-features’ and/or chunked approach, where ‘items’ have the potential to become weighted differently, but the literature does not always reflect these broader conceptions.
Methods: We consider theoretical, design and (clinically trained) assessor issues related to differential item weighting in checklist scoring of OSCEs stations. Using empirical evidence, this work also compares candidate decisions and psychometric quality of different item-weighting approaches (i.e. a simple ‘unweighted’ scheme versus a differentially weighted one).
Results: The impact of different weighting schemes affect approximately 30% of the key borderline group of candidates, and 3% of candidates overall. We also find that measures of overall assessment quality are a little better under the differentially weighted scoring system.
Discussion and conclusion: Differentially weighted modern checklists can contribute to valid assessment outcomes, and bring a range of additional benefits to the assessment. Judgment about weighting of particular items should be considered a key design consideration during station development and must align to clinical assessor expectations of the relative importance of sub-tasks.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2020 Informa UK Limited, trading as Taylor & Francis Group. This is an author produced version of an article published in Medical Teacher. Uploaded in accordance with the publisher's self-archiving policy. |
Keywords: | assessment quality; checklist design; item weighting; OSCE scoring |
Dates: |
|
Institution: | The University of Leeds |
Academic Units: | The University of Leeds > Faculty of Medicine and Health (Leeds) > School of Medicine (Leeds) > Leeds Institute of Medical Education > Medical Education Unit (Leeds) The University of Leeds > Faculty of Education, Social Sciences and Law (Leeds) > School of Education (Leeds) |
Depositing User: | Symplectic Publications |
Date Deposited: | 03 Jun 2020 19:44 |
Last Modified: | 05 Jul 2022 08:05 |
Status: | Published |
Publisher: | Taylor & Francis |
Identification Number: | 10.1080/0142159X.2020.1781072 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:161400 |