Hopfgartner, F. orcid.org/0000-0003-0380-6088, Balog, K., Lommatzsch, A. et al. (4 more authors) (2019) Continuous evaluation of large-scale information access systems : a case for living labs. In: Ferro, N. and Peters, C., (eds.) Information Retrieval Evaluation in a Changing World: Lessons Learned from 20 Years of CLEF. Springer , pp. 511-543. ISBN 9783030229474
Abstract
A/B testing is currently being increasingly adopted for the evaluation of commercial information access systems with a large user base since it provides the advantage of observing the efficiency and effectiveness of information access systems under real conditions. Unfortunately, unless university-based researchers closely collaborate with industry or develop their own infrastructure or user base, they cannot validate their ideas in live settings with real users. Without online testing opportunities open to the research communities, academic researchers are unable to employ online evaluation on a larger scale. This means that they do not get feedback for their ideas and cannot advance their research further. Businesses, on the other hand, miss the opportunity to have higher customer satisfaction due to improved systems. In addition, users miss the chance to benefit from an improved information access system. In this chapter, we introduce two evaluation initiatives at CLEF, NewsREEL and Living Labs for IR (LL4IR), that aim to address this growing “evaluation gap” between academia and industry. We explain the challenges and discuss the experiences organizing these living labs.
Metadata
Item Type: | Book Section |
---|---|
Authors/Creators: |
|
Editors: |
|
Copyright, Publisher and Additional Information: | © 2019 Springer. This is an author-produced version of a chapter subsequently published in Information Retrieval Evaluation in a Changing World. Uploaded in accordance with the publisher's self-archiving policy. |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 28 Jun 2019 13:26 |
Last Modified: | 14 Aug 2021 00:38 |
Published Version: | https://www.springer.com/gb/book/9783030229474 |
Status: | Published |
Publisher: | Springer |
Refereed: | Yes |
Identification Number: | 10.1007/978-3-030-22948-1_21 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:146335 |