Difallah, D.E., Demartini, G. and Cudré-Mauroux, P. (2016) Scheduling Human Intelligence Tasks in Multi-Tenant Crowd-Powered Systems. In: WWW '16 Proceedings of the 25th International Conference on World Wide Web. WWW2016, 11-15 Apr 2016, Montreal, Canada. ACM ISBN 978-1-4503-4143-1
Abstract
Micro-task crowdsourcing has become a popular approach to effectively tackle complex data management problems such as data linkage, missing values, or schema matching. However, the backend crowdsourced operators of crowd-powered systems typically yield higher latencies than the machine-processable operators, this is mainly due to inherent efficiency differences between humans and machines. This problem can be further exacerbated by the lack of workers on the target crowdsourcing platform, or when the workers are shared unequally among a number of competing requesters; including the concurrent users from the same organization who execute crowdsourced queries with different types, priorities and prices. Under such conditions, a crowd-powered system acts mostly as a proxy to the crowdsourcing platform, and hence it is very difficult to provide effiency guarantees to its end-users. Scheduling is the traditional way of tackling such problems in computer science, by prioritizing access to shared resources. In this paper, we propose a new crowdsourcing system architecture that leverages scheduling algorithms to optimize task execution in a shared resources environment, in this case a crowdsourcing platform. Our study aims at assessing the efficiency of the crowd in settings where multiple types of tasks are run concurrently. We present extensive experimental results comparing i) different multi-tenant crowdsourcing jobs, including a workload derived from real traces, and ii) different scheduling techniques tested with real crowd workers. Our experimental results show that task scheduling can be leveraged to achieve fairness and reduce query latency in multi-tenant crowd-powered systems, although with very different tradeoffs compared to traditional settings not including human factors.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2016 International World Wide Web Conferences Steering Committee. This is an author produced version of a paper subsequently published in WWW '16 Proceedings of the 25th International Conference on World Wide Web. Uploaded in accordance with the publisher's self-archiving policy. |
Keywords: | Crowdsourcing; Scheduling; Crowd-Powered System |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 19 Feb 2016 16:08 |
Last Modified: | 19 Dec 2022 13:32 |
Published Version: | http://dx.doi.org/10.1145/2872427.2883030 |
Status: | Published |
Publisher: | ACM |
Refereed: | Yes |
Identification Number: | 10.1145/2872427.2883030 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:94722 |