White Rose University Consortium logo
University of Leeds logo University of Sheffield logo York University logo

Do health technology assessments comply with QUOROM diagram guidance? An empirical study

Hind, D. and Booth, A. (2007) Do health technology assessments comply with QUOROM diagram guidance? An empirical study. BMC Medical Research Methodology, 7. p. 49. ISSN 1471-2288

Full text not available from this repository.



The Quality of Reporting of Meta-analyses (QUOROM) statement provides guidance for improving the quality of reporting of systematic reviews and meta-analyses. To make the process of study selection transparent it recommends "a flow diagram providing information about the number of RCTs identified, included, and excluded and the reasons for excluding them". We undertook an empirical study to identify the extent of compliance in the UK Health Technology Assessment (HTA) programme.


We searched Medline to retrieve all systematic reviews of therapeutic interventions in the HTA monograph series published from 2001 to 2005. Two researchers recorded whether each study contained a meta-analysis of controlled trials, whether a QUOROM flow diagram was presented and, if so, whether it expressed the relationship between the number of citations and the number of studies. We used Cohen's kappa to test inter-rater reliability.


87 systematic reviews were retrieved. There was good and excellent inter-rater reliability for, respectively, whether a review contained a meta-analysis and whether each diagram contained a citation-to-study relationship. 49% of systematic reviews used a study selection flow diagram. When only systematic reviews containing a meta-analysis were analysed, compliance was only 32%. Only 20 studies (23% of all systematic reviews; 43% of those having a study selection diagram) had a diagram which expressed the relationship between citations and studies.


Compliance with the recommendations of the QUOROM statement is not universal in systematic reviews or meta-analyses. Flow diagrams make the conduct of study selection transparent only if the relationship between citations and studies is clearly expressed. Reviewers should understand what they are counting: citations, papers, studies and trials are fundamentally different concepts which should not be confused in a diagram.

Item Type: Article
Copyright, Publisher and Additional Information: © 2007 Hind and Booth; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Institution: The University of Sheffield
Academic Units: The University of Sheffield > Faculty of Medicine, Dentistry and Health (Sheffield) > School of Health and Related Research (Sheffield)
Depositing User: Sheffield Import
Date Deposited: 05 Oct 2009 10:58
Last Modified: 08 Oct 2009 10:08
Published Version: http://www.biomedcentral.com/1471-2288/7/49
Status: Published
Publisher: Biomed Central
Identification Number: 10.1186/1471-2288-7-49
URI: http://eprints.whiterose.ac.uk/id/eprint/9666

Actions (repository staff only: login required)