The limits of observational data in determining outcomes from cancer therapy

Certain biases may exist in observational studies that compare outcomes of different cancer therapies, making the results questionable.

That is the conclusion of a new study published in the June 1, 2008 issue of CANCER, a peer-reviewed journal of the American Cancer Society. The research suggests that observational studies should include more thorough information and should be better designed to minimize inaccuracies.

Clinical trials are considered the gold standard for demonstrating the effectiveness of new treatments for cancer, but observational studies, which do not involve randomization but where available data are nonetheless analyzed to make treatment comparisons, have also been used to provide information on how well patients respond to particular drugs. Many investigators perform these types of studies by analyzing data from the Surveillance, Epidemiology and End Results (SEER) Tumor Registry, a national population-based cancer registry that collects cancer-related information.

To determine the accuracy of observational studies on cancer treatments, Dr. Sharon H. Giordano of the University of Texas MD Anderson Cancer Center in Houston and her colleagues compared the effectiveness of different cancer therapies in terms of prolonging survival in patients, using data from the SEER registry. They presented several examples, including re-analyses of previously published data. In all cases, they came up with improbable results, indicating how easy it is to generate questionable results when conducting an observational study.

In their first analysis, the researchers looked at data on a hormone therapy called androgen deprivation in men with stage III prostate cancer. Randomized clinical trials have shown that androgen deprivation can improve survival in these patients. When the investigators analyzed data from the SEER registry of more than 5,000 men, they found that men treated with androgen deprivation actually had a higher risk of death from prostate cancer than men who did not receive the therapy.

Dr. Giordano and her team next re-analyzed data from a previously published study of more than 43,000 men with localized prostate cancer who were treated compared with men who were not treated. Like the original study, the researchers’ analysis revealed that men who were treated for prostate cancer experienced lower mortality rates. However, they also found that in many cases, the cause of death was due to something other than prostate cancer, such as diabetes or pneumonia.

Finally, the investigators re-analyzed data from a previously published study on the effects of fluorouracil-based chemotherapy for colon cancer. They came to the same conclusion as the original research study—that chemotherapy for node positive colon cancer is associated with improved survival. However, they found that the link between the treatment and survival was strongest for non-cancer deaths, which presumably are not related.

The authors attributed the improbable results found in their three analyses to selection biases when patients are treated. For example, selection bias occurs when patients with poorer prognoses are more likely to receive a more efficacious drug, or when patients with better underlying health are more likely to receive a more toxic treatment because they are more likely to tolerate it.

The authors concluded that their findings “suggest that the results of observational studies of treatment outcomes should be viewed with caution.” They recommended that analyses of observational data should at a minimum attempt to segregate patient outcomes into those that could possibly be due to the treatments vs. those that could not. Many observational studies on cancer treatments only report death rates from all causes and do not specify cancer-related deaths.