Homer, MS orcid.org/0000-0002-1161-5938 and Darling, JC orcid.org/0000-0001-7176-5957 (2016) Setting standards in knowledge assessments: comparing Ebel and Cohen via Rasch. Medical Teacher, 38 (12). pp. 1267-1277. ISSN 0142-159X
Abstract
Introduction: It is known that test-centered methods for setting standards in knowledge tests (e.g. Angoff or Ebel) are problematic, with expert judges not able to consistently predict the difficulty of individual items. A different approach is the Cohen method, which benchmarks the difficulty of the test based on the performance of the top candidates.
Methods: This paper investigates the extent to which Ebel (and also Cohen) produces a consistent standard in a knowledge test when comparing between adjacent cohorts. The two tests are linked using common anchor items and Rasch analysis to put all items and all candidates on the same scale.
Results: The two tests are of a similar standard, but the two cohorts are different in their average abilities. The Ebel method is entirely consistent across the two years, but the Cohen method looks less so, whilst the Rasch equating itself has complications – for example, with evidence of overall misfit to the Rasch model and change in difficulty for some anchor items.
Conclusion: Based on our findings, we advocate a pluralistic and pragmatic approach to standard setting in such contexts, and recommend the use of multiple sources of information to inform the decision about the correct standard.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2016, University of Leeds. Informa UK Limited, trading as Taylor & Francis Group. This is an Accepted Manuscript of an article published by Taylor & Francis in Medical Teacher on 20 Sep 2016, available online: http://dx.doi.org/10.1080/0142159X.2016.1230184. Uploaded in accordance with the publisher's self-archiving policy. |
Keywords: | Rasch; Ebel; Cohen; standard setting; medical; knowledge test |
Dates: |
|
Institution: | The University of Leeds |
Academic Units: | The University of Leeds > Faculty of Education, Social Sciences and Law (Leeds) > School of Education (Leeds) |
Depositing User: | Symplectic Publications |
Date Deposited: | 01 Sep 2016 09:55 |
Last Modified: | 20 Sep 2018 16:28 |
Published Version: | http://dx.doi.org/10.1080/0142159X.2016.1230184 |
Status: | Published |
Publisher: | Taylor & Francis |
Identification Number: | 10.1080/0142159X.2016.1230184 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:104117 |