Micro-task Crowdsourcing has been used for different purposes: creating training data for machine learning algorithms, relevance judgments for evaluation of information systems, sentiment analysis, language translation, etc. In this paper we focus on the use of crowdsourcing as core component of data-driven systems. The creation of hybrid human-machine systems is a highly promising direction as it allows leveraging both the scalability of machines over large amounts of data as well as keeping the quality of human intelligence in the loop to finally obtain both efficiency and effectiveness in data processing applications. Such a hybrid approach is a great opportunity to develop systems that are more powerful than purely machine-based ones. For example, it is possible to build systems that can understand sarcasm in text at scale. However, when designing such systems it is critical to take into account a number of dimensions related to human behavior as humans become a component of the overall process. In this paper, we overview existing hybrid human-machine systems presenting commonalities in the approaches taken by different research communities. We summarize the key challenges that one has to face in developing such systems as well the opportunities and the open research directions to make such approaches the best way to process data in the future.
CORE (COnnecting REpositories)
CORE (COnnecting REpositories)