Checco, A. orcid.org/0000-0002-0981-3409, Bates, J. orcid.org/0000-0001-7266-8470 and Demartini, G. (2019) Quality control attack schemes in crowdsourcing. In: Kraus, S., (ed.) Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. 28th International Joint Conference on Artificial Intelligence, 10-16 Aug 2019, Macao. International Joint Conferences on Artifical Intelligence (IJCAI) , pp. 6136-6140. ISBN 9780999241141
Abstract
An important precondition to build effective AI models is the collection of training data at scale. Crowdsourcing is a popular methodology to achieve this goal. Its adoption introduces novel challenges in data quality control, to deal with under-performing and malicious annotators. One of the most popular quality assurance mechanisms, especially in paid micro-task crowdsourcing, is the use of a small set of pre-annotated tasks as gold standard, to assess in real time the annotators quality. In this paper, we highlight a set of vulnerabilities this scheme suffers: a group of colluding crowd workers can easily implement and deploy a decentralised machine learning inferential system to detect and signal which parts of the task are more likely to be gold questions, making them ineffective as a quality control tool. Moreover, we demonstrate how the most common countermeasures against this attack are ineffective in practical scenarios. The basic architecture of the inferential system is composed of a browser plug-in and an external server where the colluding workers can share information. We implement and validate the attack scheme, by means of experiments on real-world data from a popular crowdsourcing platform.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Editors: |
|
Copyright, Publisher and Additional Information: | © 2019 IJCAI and the Authors. |
Keywords: | Humans and AI: Human Computation and Crowdsourcing; Machine Learning Applications: Applications of Unsupervised Learning; Machine Learning: Online Learning; Multidisciplinary Topics and Applications: Information Retrieval |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 28 Jun 2021 10:07 |
Last Modified: | 28 Jun 2021 10:07 |
Status: | Published |
Publisher: | International Joint Conferences on Artifical Intelligence (IJCAI) |
Refereed: | Yes |
Identification Number: | 10.24963/ijcai.2019/850 |
Related URLs: | |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:175203 |