Stevenson, M. orcid.org/0000-0002-9483-6006 and Bin-Hezam, R. (2024) Stopping methods for technology assisted reviews based on point processes. ACM Transactions on Information Systems, 42 (3). pp. 1-37. ISSN 1046-8188
Abstract
Technology Assisted Review (TAR), which aims to reduce the effort required to screen collections of documents for relevance, is used to develop systematic reviews of medical evidence and identify documents that must be disclosed in response to legal proceedings. Stopping methods are algorithms which determine when to stop screening documents during the TAR process, helping to ensure that workload is minimised while still achieving a high level of recall. This paper proposes a novel stopping method based on point processes, which are statistical models that can be used to represent the occurrence of random events. The approach uses rate functions to model the occurrence of relevant documents in the ranking and compares four candidates, including one that has not previously been used for this purpose (hyperbolic). Evaluation is carried out using standard datasets (CLEF e-Health, TREC Total Recall, TREC Legal), and this work is the first to explore stopping method robustness by reporting performance on a range of rankings of varying effectiveness. Results show that the proposed method achieves the desired level of recall without requiring an excessive number of documents to be examined in the majority of cases and also compares well against multiple alternative approaches.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2023 Association for Computing Machinery. This is an author-produced version of a paper subsequently published in ACM Transactions on Information Systems. Uploaded in accordance with the publisher's self-archiving policy. |
Keywords: | Technology assisted review; TAR; total recall; stopping criteria; point processes; Cox Process; Poisson Process |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 10 Nov 2023 15:41 |
Last Modified: | 01 Nov 2024 11:24 |
Status: | Published |
Publisher: | Association for Computing Machinery (ACM) |
Refereed: | Yes |
Identification Number: | 10.1145/3631990 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:205161 |