Al-Maskari, A., Sanderson, M. and Clough, P. (2008) Relevance Judgments between TREC and Non-TREC Assessors. In: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. SIGIR '08, July 20-24, 2008, Singapore, Singapore. ACM , New York, USA , pp. 683-684. ISBN 978-1-60558-164-4
Abstract
This paper investigates the agreement of relevance assessments between official TREC judgments and those generated from an interactive IR experiment. Results show that 63% of documents judged relevant by our users matched official TREC judgments. Several factors contributed to differences in the agreements: the number of retrieved relevant documents; the number of relevant documents judged; system effectiveness per topic and the ranking of relevant documents.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Keywords: | TREC relevance assessment, user study |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield) |
Depositing User: | Repository Officer |
Date Deposited: | 19 Nov 2008 10:34 |
Last Modified: | 08 Feb 2013 16:58 |
Published Version: | http://dx.doi.org/10.1145/1390334.1390450 |
Status: | Published |
Publisher: | ACM |
Identification Number: | 10.1145/1390334.1390450 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:4510 |