Difallah, D., Checco, A. orcid.org/0000-0002-0981-3409, Demartini, G. et al. (1 more author) (2019) Deadline-aware fair scheduling for multi-tenant crowd-powered systems. ACM Transactions on Social Computing, 2 (1). pp. 1-29. ISSN 2469-7818
Abstract
Crowdsourcing has become an integral part of many systems and services that deliver high-quality results for complex tasks such as data linkage, schema matching, and content annotation. A standard function of such crowd-powered systems is to publish a batch of tasks on a crowdsourcing platform automatically and to collect the results once the workers complete them. Currently, these systems provide limited guarantees over the execution time, which is problematic for many applications. Timely completion may even be impossible to guarantee due to factors specific to the crowdsourcing platform, such as the availability of workers and concurrent tasks. In our previous work, we presented the architecture of a crowd-powered system that reshapes the interaction mechanism with the crowd. Specifically, we studied a push-crowdsourcing model whereby the workers receive tasks instead of selecting them from a portal. Based on this interaction model, we employed scheduling techniques similar to those found in distributed computing infrastructures to automate the task assignment process. In this work, we first devise a generic scheduling strategy that supports both fairness and deadline-awareness. Second, to complement the proof-of-concept experiments previously performed with the crowd, we present an extensive set of simulations meant to analyze the properties of the proposed scheduling algorithms in an environment with thousands of workers and tasks. Our experimental results show that, by accounting for human factors, micro-task scheduling can achieve fairness for best-effort batches and boosts production batches.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2019 Association for Computing Machinery. This is an author-produced version of a paper subsequently published in ACM Transactions on Social Computing. Uploaded in accordance with the publisher's self-archiving policy. |
Keywords: | Crowdsourcing Systems; Priority; Human Factors; Task Scheduling; Deadline |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield) |
Funding Information: | Funder Grant number European Commission - Horizon 2020 732328 |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 18 Jun 2020 09:49 |
Last Modified: | 30 Jun 2020 08:04 |
Status: | Published |
Publisher: | Association for Computing Machinery (ACM) |
Refereed: | Yes |
Identification Number: | 10.1145/3301003 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:143193 |