Yet another problem with systematic reviews: a living review update

Background: In February 2023, the Journal of Clinical Epidemiology published ‘The Problems with Systematic Reviews: A Living Systematic Review.’ In updating this living review for the first time a new problem and several themes relating to research culture have emerged.

Methods: Literature searches were rerun to identify articles published or indexed between May 2022 and May 2023. Thematic analysis coded articles and problems across four domains of systematic review conduct (1. comprehensive, 2. rigour, 3. transparent, 4. objective).

Results: One hundred fifty-two newly included articles bring the total number of relevant articles to 637. A new problem (the lack of gender diversity of systematic review author teams) brings the total number of problems with systematic reviews up to 68. This update also reveals emerging themes such as: fast science from systematic reviews on COVID-19; the failure of citation of methodological or reporting guidelines to predict high-quality methodological or reporting quality; and the influence of vested interests on systematic review conclusions. These findings coupled with a proliferation of research waste from “me-too” meta-research articles highlighting well-established problems in systematic reviews underscores the need for reforms in research culture to address the incentives for producing and publishing research papers. This update also reports where the identified flaws in systematic reviews affect their conclusions drawing on 77 meta-epidemiological studies from the total 637 included articles. These meta-meta-analytic studies begin the important work of examining which problems threaten the reliability and validity of treatment effects or conclusions derived from systematic reviews.

Conclusion: This living review has captured an emerging theme in the published literature relating to the composition of the review author team and highlights a potential effect on the equity reporting of the systematic reviews. We recommend that meta-research endeavors evolve from merely documenting well-established issues to understanding lesser-known problems or consequences to systematic reviews.