Assessor experience, not rubric type, determines grading reliability in biosciences coursework

Introduction: Previous research on whether holistic or analytical scoring rubrics yield higher reliability and validity has been mixed and inconclusive, with little of it based on empirical data. This study addressed this gap by comparing scores from assessors with varying experience levels who used both a holistic and an analytical scale descriptor to grade undergraduate bioscience essays, complemented by a qualitative survey of assessor perceptions. The goal was to determine which method resulted in more consistent inter-rater agreement.

Methods: For the study, independent assessors (two per essay) scored essays using either holistic (n = 212) or analytical (n = 62) scale descriptors, over four consecutive years. Each assessor provided both a holistic and an analytical score for every essay. The agreement between the scores was then calculated.

Results: The results showed no significant difference in scores awarded holistically versus analytically for the same essay, regardless of the scale descriptor used or the assessor's experience. However, a key finding was that experienced assessors had a higher agreement with the final awarded grade than their less-experienced counterparts.

Discussion: This suggests that the experience of the assessor, rather than the specific scoring rubric or guidance provided, is the primary factor in determining the reliability of the grade awarded.