White Rose University Consortium logo
University of Leeds logo University of Sheffield logo York University logo

Forming test collections with no system pooling

Sanderson, M. and Joho, H. (2004) Forming test collections with no system pooling. In: 27th ACM Conference of the Special Interest Group in Information Retrieval. Conference of the Special Interest Group in Information Retrieval 2004, 25-29 July 2004, University of Sheffield. ACM New York, NY, USA , Sheffield, United Kingdom , pp. 33-40. ISBN 1-58113-881-4


Forming test collection relevance judgments from the pooled output of multiple retrieval systems has become the standard process for creating resources such as the TREC, CLEF, and NTCIR test collections. This paper presents a series of experiments examining three different ways of building test collections where no system pooling is used. First, a collection formation technique combining manual feedback and multiple systems is adapted to work with a single retrieval system. Second, an existing method based on pooling the output of multiple manual searches is re-examined: testing a wider range of searchers and retrieval systems than has been examined before. Third, a new approach is explored where the ranked output of a single automatic search on a single retrieval system is assessed for relevance: no pooling whatsoever. Using established techniques for evaluating the quality of relevance judgments, in all three cases, test collections are formed that are as good as TREC.

Item Type: Proceedings Paper
Keywords: Test collection formation, evaluation of qrel sets.
Institution: The University of Sheffield
Academic Units: The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield)
Depositing User: Repository Officer
Date Deposited: 08 Sep 2008 11:46
Last Modified: 15 Sep 2014 03:39
Status: Published
Publisher: ACM New York, NY, USA
URI: http://eprints.whiterose.ac.uk/id/eprint/4585

Actions (repository staff only: login required)