Grey literature in meta-analyses of randomized trials of health care interventions

Flooded with new health information every week, patients and their doctors look for ways to sort the best from the rest.

The gold standard for medical studies is the randomized controlled trial or RCT, where patients are randomly sorted into groups to compare different types of treatment for the same condition.

But just like paper money, these valuable health studies are only as “golden” as the data that back them up. Three new reviews in the latest issue of The Cochrane Library offer a behind-the-scenes look at how RCTs and major research reviews come together.

The Cochrane Library is a publication of The Cochrane Collaboration, an international organization that evaluates medical research. Systematic reviews draw evidence-based conclusions about medical practice after considering both the content and quality of existing medical trials on a topic.

The first new review asks, “Are randomized controlled trials risky for patients?”

Some patients are uneasy about participating in an RCT because they feel like “guinea pigs,” undergoing what they see as an unproven treatment and possibly getting worse care than they might outside of a trial. But the evidence suggests RCT patients are not putting their health at risk.

Dr. Gunn Elisabeth Vist and colleagues compared the health outcomes of more than 30,000 patients treated in a RCT with more than 20,000 similar patients outside of a trial who received the same treatment.

The two groups of patients fared similarly, which means that participating in a RCT “is unlikely to be harmful,” said Vist, of the Norwegian Knowledge Centre for Health Services.

Vist and colleagues also found no strong evidence that patients in a RCT received better- than-normal care.

Dr. James Lubowitz, an orthopedic surgeon and director of the Taos Orthopaedic Institute in New Mexico, said some patients might feel drawn to trials by the idea that the treatments tested in them are “experimental.”

Patients might think certain procedures “are state-of-the-art because they're newer and perhaps hyped, when in fact the newer procedure is not validated or proven, which is why you're doing the study in the first place,” he said.

Lubowitz is participating in four RCTs now and he called the work “personally gratifying and intellectually stimulating.” Still, other doctors can be reluctant to recruit patients for trials.

In a second Cochrane review, a group led by Dr. Jennifer Rendell of the University of Oxford discovered that many doctors are leery of participating in RCTs because they feel participation might have a negative effect on the doctor-patient relationship.

In their review of 11 studies, the authors also found that clinicians who agreed to join a trial because they knew the researchers behind the study were actually less likely to recruit patients than those who joined the trial for other reasons.

Rendell and colleagues called the finding “counterintuitive,” saying, “Researchers need to be alert to the possibility that some clinicians may feel obligated to sign up to a trial without necessarily being motivated to recruit.”

Most systematic reviews of the medical literature — including the Cochrane reviews —depend mainly on studies published in journals for their data. But reviewers ignore a slew of “gray literature” from conference abstracts, unpublished dissertations and government reports at their peril, say Dr. Sally Hopewell of the UK Cochrane Centre and colleagues in a third new review.

Reviews based only on published studies can show bias, because “those trials with significant or interesting results are more likely to be published than those trials with non-significant, non-interesting results,” Hopewell said.

Published trials “tend to be larger and show an overall greater treatment effect than gray trials,” the reviewers found. In three of the studies they examined, the average treatment effects were 9 percent larger in trials published in journals than in gray trials.

The Cochrane Collaboration is an international nonprofit, independent organization that produces and disseminates systematic reviews of health care interventions and promotes the search for evidence in the form of clinical trials and other studies of interventions. Visit http://www.cochrane.org for more information.

Rendell JM, et al. Incentives and disincentives to participation by clinicians in randomised controlled trials. The Cochrane Database of Systematic Reviews 2007, Issue 2.

Vist GE, et al. Outcomes of patients who participate in randomized controlled trials compared to similar patients receiving similar interventions who do not participate. The Cochrane Database of Systematic Reviews 2007, Issue 2.

Hopewell S, et al. Grey literature in meta-analyses of randomized trials of health care interventions. The Cochrane Database of Systematic Reviews 2007, Issue 2.