Villa, R. and Halvey, M. (2013) Is relevance hard work? Evaluating the effort of making relevant assessments. In: Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM SIGIR 2013, 28 Jul - 01 Aug 2013, Dublin, Ireland. ACM , 765 - 768.
Abstract
The judging of relevance has been a subject of study in information retrieval for a long time, especially in the creation of relevance judgments for test collections. While the criteria by which assessors' judge relevance has been intensively studied, little work has investigated the process individual assessors go through to judge the relevance of a document. In this paper, we focus on the process by which relevance is judged, and in particular, the degree of effort a user must expend to judge relevance. By better understanding this effort in isolation, we may provide data which can be used to create better models of search. We present the results of an empirical evaluation of the effort users must exert to judge the relevance of document, investigating the effect of relevance level and document size. Results suggest that "relevant" documents require more effort to judge when compared to highly relevant and not relevant documents, and that effort increases as document size increases. Copyright © 2013 ACM.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2013 ACM. This is an author produced version of a paper subsequently published in SIGIR '13 Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval . Uploaded in accordance with the publisher's self-archiving policy. |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 27 Feb 2014 15:17 |
Last Modified: | 19 Dec 2022 13:26 |
Published Version: | http://dx.doi.org/10.1145/2484028.2484150 |
Status: | Published |
Publisher: | ACM |
Refereed: | Yes |
Identification Number: | 10.1145/2484028.2484150 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:77920 |