White Rose University Consortium logo
University of Leeds logo University of Sheffield logo York University logo

Relevance Judgments between TREC and Non-TREC Assessors

Al-Maskari, A., Sanderson, M. and Clough, P. (2008) Relevance Judgments between TREC and Non-TREC Assessors. In: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. SIGIR '08, July 20-24, 2008, Singapore, Singapore. ACM , New York, USA , pp. 683-684. ISBN 978-1-60558-164-4

[img] Text

Download (70Kb)


This paper investigates the agreement of relevance assessments between official TREC judgments and those generated from an interactive IR experiment. Results show that 63% of documents judged relevant by our users matched official TREC judgments. Several factors contributed to differences in the agreements: the number of retrieved relevant documents; the number of relevant documents judged; system effectiveness per topic and the ranking of relevant documents.

Item Type: Proceedings Paper
Keywords: TREC relevance assessment, user study
Institution: The University of Sheffield
Academic Units: The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield)
Depositing User: Repository Officer
Date Deposited: 19 Nov 2008 10:34
Last Modified: 08 Feb 2013 16:58
Published Version: http://dx.doi.org/10.1145/1390334.1390450
Status: Published
Publisher: ACM
Identification Number: 10.1145/1390334.1390450
URI: http://eprints.whiterose.ac.uk/id/eprint/4510

Actions (repository staff only: login required)