Checco, A., Bates, J. orcid.org/0000-0001-7266-8470 and Demartini, G. (2018) All That Glitters is Gold -- An Attack Scheme on Gold Questions in Crowdsourcing. In: Proceedings of The 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2018). The 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2018), 05-08 Jul 2018, Zurich, Switzerland. AAAI Press
Abstract
One of the most popular quality assurance mechanisms in paid micro-task crowdsourcing is based on gold questions: the use of a small set of tasks of which the requester knows the correct answer and, thus, is able to directly assess crowd work quality. In this paper, we show that such mechanism is prone to an attack carried out by a group of colluding crowd workers that is easy to implement and deploy: the inherent size limit of the gold set can be exploited by building an inferential system to detect which parts of the job are more likely to be gold questions. The described attack is robust to various forms of randomisation and programmatic generation of gold questions. We present the architecture of the proposed system, composed of a browser plug-in and an external server used to share information, and briefly introduce its potential evolution to a decentralised implementation. We implement and experimentally validate the gold detection system, using real-world data from a popular crowdsourcing platform. Finally, we discuss the economic and sociological implications of this kind of attack.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2018 Association for the Advancement of Artificial Intelligence. This is an author-produced version of a paper subsequently published in HCOMP 2018 Proceedings. Uploaded in accordance with the publisher's self-archiving policy. |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 17 May 2018 08:43 |
Last Modified: | 19 Dec 2022 13:49 |
Published Version: | https://www.aaai.org/ocs/index.php/HCOMP/HCOMP18/p... |
Status: | Published |
Publisher: | AAAI Press |
Refereed: | Yes |
Related URLs: | |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:130654 |