Weakly supervised active learning for abstract screening leveraging LLM-based pseudo-labeling

Abstract screening is a notoriously labour-intensive step in systematic reviews. AI-aided abstract screening faces several grand challenges, such as the strict requirement of near-total recall of relevant studies, lack of initial annotation, and extreme data imbalance. Active learning is the predominant solution for this challenging task, which however is remarkably time-consuming and tedious. To address these challenges, this paper introduces a weakly supervised learning framework leveraging large language models (LLM). The proposed approach employs LLMs to score and rank candidate studies based on their adherence to the inclusion criteria for relevant studies that are specified in the review protocol. Pseudo-labels are generated by assuming the top T % and bottom B% as positive and negative samples, respectively, for training an initial classifier without manual annotation. Experimental results on 28 systematic reviews from a well-established benchmark demonstrate a breakthrough in automated abstract screening: Manual annotation can be eliminated to safely reducing 42-43% of screening workload on average and maintaining near-perfect recall — the first approach that has succeeded in achieving this strict requirement for abstract screening. Additionally, LLM-based pseudo-labelling significantly improves the efficiency and utility of the active learning regime for abstract screening.