Al-Maskari, A., Sanderson, M. and Clough, P. (2008) Relevance Judgments between TREC and Non-TREC Assessors. In: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. SIGIR '08, July 20-24, 2008, Singapore, Singapore. ACM , New York, USA , pp. 683-684. ISBN 978-1-60558-164-4
This paper investigates the agreement of relevance assessments between official TREC judgments and those generated from an interactive IR experiment. Results show that 63% of documents judged relevant by our users matched official TREC judgments. Several factors contributed to differences in the agreements: the number of retrieved relevant documents; the number of relevant documents judged; system effectiveness per topic and the ranking of relevant documents.
|Keywords:||TREC relevance assessment, user study|
|Institution:||The University of Sheffield|
|Academic Units:||The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield)|
|Depositing User:||Repository Officer|
|Date Deposited:||19 Nov 2008 10:34|
|Last Modified:||08 Feb 2013 16:58|