White Rose University Consortium logo
University of Leeds logo University of Sheffield logo York University logo

The relationship of word error rate to document ranking

Mang Shou, X., Sanderson, M. and Tuffs, N. (2004) The relationship of word error rate to document ranking. In: Proceedings of the AAAI Spring Symposium Intelligent Multimedia Knowledge Management Workshop. AAAI Technical Report SS-03-08. AAAI-IMKM, March 24–26, 2003, Stanford University, California. , pp. 28-33. ISBN 1-57735-190-8


Download (181Kb)


This paper describes two experiments that examine the relationship of Word Error Rate (WER) of retrieved spoken documents returned by a spoken document retrieval system. Previous work has demonstrated that recognition errors do not significantly affect retrieval effectiveness but whether they will adversely affect relevance judgement remains unclear. A user-based experiment measuring ability to judge relevance from the recognised text presented in a retrieved result list was conducted. The results indicated that users were capable of judging relevance accurately despite transcription errors. This lead an examination of the relationship of WER in retrieved audio documents to their rank position when retrieved for a particular query. Here it was shown that WER was somewhat lower for top ranked documents than it was for documents retrieved further down the ranking, thereby indicating a possible explanation for the success of the user experiment.

Item Type: Proceedings Paper
Institution: The University of Sheffield
Academic Units: The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield)
Depositing User: Repository Officer
Date Deposited: 23 Sep 2008 15:20
Last Modified: 08 Feb 2013 16:56
Status: Published
URI: http://eprints.whiterose.ac.uk/id/eprint/4710

Actions (repository staff only: login required)