More than two thirds of meta-analyses available to critical care physicians have major methodological flaws

More than two thirds of meta-analyses available to critical care physicians have major methodological flaws and cannot be reliably used to guide clinical practice. Research published in Critical Care concludes that the overall quality of meta-analyses in this field is poor, but has improved over time, possibly due to the publication of the Quality of Reporting of Meta-analyses statement (QUOROM) in 1999.

Anthony Delaney from the Royal North Shore Hospital, Australia, Chip Doig, from the University of Calgary, Canada, and colleagues, studied 139 reports of meta-analyses addressing critical care topics published between January 1994 and December 2003. They assessed whether data from the reports met any of nine of the criteria listed in the Overview Quality Assessment Questionnaire(OQAQ), the only instrument available to researchers to grade the quality of articles. The reports were given an overall score on a scale of 1-to 7 according to the results of this assessment, with a score of 1 to 4 indicating major flaws in the meta-analysis and a score of 5 or more indicating only minimal or minor flaws.

Delaney et al.'s results show that 69% of the reports score below 4 and the average score was 3.3. The most common flaws were failure to report whether a comprehensive literature search was conducted and failure to report how bias in the inclusion of studies was avoided, with only 35% of studies fulfilling these criteria.

Before the QUOROM statement, the proportion of reports adequately reporting the validity of the studies included in the meta-analysis was 39%, but it increased to 52% after publication of the statement.

Delaney et al. found similar results in other areas such as emergency medicine, anaesthesia and general surgery.

Physicians view meta-analyses as a highly reliable resource. This study shows that critical care clinicians should carefully evaluate meta-analyses before considering applying the results of these studies in their clinical practice.