Kille, B., Lommatzsch, A., Hopfgartner, F. orcid.org/0000-0003-0380-6088 et al. (2 more authors) (2017) A stream-based resource for multi-dimensional evaluation of recommender algorithms. In: SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR'17 : 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, 07-11 Aug 2017, Tokyo, Japan. ACM Digital Library , pp. 1257-1260. ISBN 9781450350228
Abstract
Recommender System research has evolved to focus on developing algorithms capable of high performance in online systems. This development calls for a new evaluation infrastructure that supports multi-dimensional evaluation of recommender systems. Today's researchers should analyze algorithms with respect to a variety of aspects including predictive performance and scalability. Researchers need to subject algorithms to realistic conditions in online A/B tests. We introduce two resources supporting such evaluation methodologies: the new data set of stream recommendation interactions released for CLEF NewsREEL 2017, and the new Open Recommendation Platform (ORP). The data set allows researchers to study a stream recommendation problem closely by "replaying" it locally, and ORP makes it possible to take this evaluation "live" in a living lab scenario. Specifically, ORP allows researchers to deploy their algorithms in a live stream to carry out A/B tests. To our knowledge, NewsREEL is the first online news recommender system resource to be put at the disposal of the research community. In order to encourage others to develop comparable resources for a wide range of domains, we present a list of practical lessons learned in the development of the dataset and ORP.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2017 The Authors. This is an author-produced version of a paper subsequently published in SIGIR '17: Proceedings. Uploaded in accordance with the publisher's self-archiving policy. |
Keywords: | streams; recommender system; multi-dimensional benchmarking |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 28 Jun 2021 10:36 |
Last Modified: | 28 Jun 2021 10:36 |
Status: | Published |
Publisher: | ACM Digital Library |
Refereed: | Yes |
Identification Number: | 10.1145/3077136.3080726 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:175303 |