Computer-assisted screening in systematic evidence synthesis requires robust and well-evaluated stopping criteria

Nearly two decades of research have outlined the potential for machine learning to reduce the effort required to screen documents as part of a systematic review. Despite this, using this method to reduce the human labour required for screening is not yet sanctioned in systematic review guidelines, due to uncertainty over the risk of missing relevant studies.

Stopping criteria exist to manage the uncertainty of missing studies when using machine learning to assist systematic review screening. They define the point at which abstracts are no longer screened by humans but excluded as a result of machine learning predictions. However, they vary greatly in utility and reliability. Those that offer statistically informed estimates of the risk of missing relevant studies are seldom used. The evidence synthesis platforms which implement machine learning-assisted screening typically do not implement stopping criteria. They leave the choice of when to stop up to the user, or recommend heuristic criteria that cannot give a theoretically sound estimate of the risk of missing relevant studies.

The state of affairs described above means that many systematic reviews either do not make use of machine learning and thus forego potential work savings, or use machine learning in ways that imply large and unquantified risks of missing relevant studies.

In this commentary, we make five recommendations for fulfilling the promise of work savings through the application of machine learning in systematic review screening. We discuss concrete steps that systematic review technology researchers, guideline-makers, tool developers, and the wider systematic review community should implement towards reaching this goal.