Checco, A. orcid.org/0000-0002-0981-3409, Roitero, K., Maddalena, E. et al. (2 more authors) (2017) Let’s Agree to Disagree: Fixing Agreement Measures for Crowdsourcing. In: Dow, S. and Kalai, T., (eds.) Proceedings of the Fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP-17). Fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP-17), 23–26 October 2017, Québec City, Québec, Canada. AAAI Press , Palo Alto, California , pp. 11-20.
Abstract
In the context of micro-task crowdsourcing, each task is usually performed by several workers. This allows researchers to leverage measures of the agreement among workers on the same task, to estimate the reliability of collected data and to better understand answering behaviors of the participants. While many measures of agreement between annotators have been proposed, they are known for suffering from many problems and abnormalities. In this paper, we identify the main limits of the existing agreement measures in the crowdsourcing context, both by means of toy examples as well as with real-world crowdsourcing data, and propose a novel agreement measure based on probabilistic parameter estimation which overcomes such limits. We validate our new agreement measure and show its flexibility as compared to the existing agreement measures.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Editors: |
|
Copyright, Publisher and Additional Information: | © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). |
Keywords: | crowdsourcing; inter-rater agreement; reliability |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 25 Oct 2017 14:13 |
Last Modified: | 16 Apr 2018 11:23 |
Published Version: | https://aaai.org/ocs/index.php/HCOMP/HCOMP17/paper... |
Status: | Published |
Publisher: | AAAI Press |
Refereed: | Yes |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:122865 |