Maynard, D.G. and Bontcheva, K. (2016) Challenges of Evaluating Sentiment Analysis Tools on Social Media. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). 10th edition of the Language Resources and Evaluation Conference, 23-28 May 2016, Portorož. LREC , pp. 1142-1148. ISBN 978-2-9517408-9-1
Abstract
This paper discusses the challenges in carrying out fair comparative evaluations of sentiment analysis systems. Firstly, these are due to differences in corpus annotation guidelines and sentiment class distribution. Secondly, different systems often make different assumptions about how to interpret certain statements, e.g. tweets with URLs. In order to study the impact of these on evaluation results, this paper focuses on tweet sentiment analysis in particular. One existing and two newly created corpora are used, and the performance of four different sentiment analysis systems is reported; we make our annotated datasets and sentiment analysis applications publicly available. We see considerable variations in results across the different corpora, which calls into question the validity of many existing annotated datasets and evaluations, and we make some observations about both the systems and the datasets as a result.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2016 The Author(s). This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial Licence (http://creativecommons.org/licenses/by-nc/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. You may not use the material for commercial purposes. |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Funding Information: | Funder Grant number EUROPEAN COMMISSION - FP6/FP7 DECARBONET - 610829 EUROPEAN COMMISSION - HORIZON 2020 687847 |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 05 May 2016 10:55 |
Last Modified: | 13 Mar 2018 09:22 |
Published Version: | http://www.lrec-conf.org/proceedings/lrec2016/summ... |
Status: | Published |
Publisher: | LREC |
Refereed: | Yes |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:98769 |