The Pandora’s box of evidence synthesis and the case for a living evidence synthesis taxonomy

Have we, as an evidence-based health community, opened the Pandora’s box of evidence synthesis? There now exists a plethora of overlapping evidence synthesis approaches and duplicate, redundant and poor-quality reviews.1–4 After years of advocating for the need for systematic reviews of the evidence, there is a risk that this message been disseminated too widely and has been misinterpreted in this process. We have reached a point where in some fields more reviews exist than clinical trials, where same topic reviews are being conducted in parallel, and evidence syntheses possess limited utility for decision-making because of their poor quality or poor reporting.To paraphrase the late Douglas Altman,5 it is possible we are now at a stage where we need less reviews, better reviews and reviews done for the right reason—as opposed to the current state of mass production (approximately 80 reviews per day).6

How have we arrived at this point - and is it a point of no return? One obvious reason is that over time, systematic review methods (and evidence synthesis more broadly) have become increasingly demanding, complex and multifaceted. This evolution in review methods has caused, and continues to cause, great confusion for both novice and experienced synthesisers of evidence.7–10 For example, confusion persists between scoping reviews and systematic reviews and the correct application of these approaches to evidence synthesis. This, in turn, results in scoping reviews assessing the effects of interventions when they are neither intended nor equipped to do so.11 This confusion is evident with other evidence synthesis methodologies, including scoping reviews and mapping reviews, where further guidance may be needed to help with the appropriate choice of methodologies.12 Indeed, it has already been argued that the current proliferation of review types is creating challenges for the terminology for describing such reviews, creating fundamental issues for evidence synthesisers.7 Although the utility of an encompassing toolkit for evidence synthesis to assist in answering an array of complex and multifaceted questions is to be welcomed, the resultant confusion (and complexity) associated with this expansion cannot be overlooked. With supervisors encouraging increasing numbers of novice researchers to undertake systematic reviews (fuelled by the misplaced idea of a quick or easy publication that requires little [if any] infrastructure support [or ethics approval] or as a requirement of doctorate studies or grant applications), and funders and frameworks rightly promoting interventional research be developed based on the evidence, we are likely to encounter further proliferation of misplaced, misconducted and redundant evidence synthesis projects.13

We are concerned about this current state of affairs within the field of evidence synthesis, but believe we have not yet reached the point of no return. As such, this article discusses some of the pitfalls associated with an ever-expanding toolkit for evidence synthesis (likened to the opening of Pandora’s Box) and discuss potential solutions for improving the cohesiveness of evidence synthesis.