Only by analyzing results using a statistical method that considers several factors at once -- for instance, overall heart attack risk, rather than just individual risk factors such as age -- will researchers truly be able to see how the risks and benefits of a treatment vary from patient to patient, the study suggests.
This kind of analysis, called multivariable or risk-stratified analysis, would require more statistical know-how. But it could save lives and money, and allow patients to weigh a treatment's true risks and benefits to their health before starting treatment.
"Most studies currently emphasize the average risk and average benefit found in the study, but the average trial participant might get much less benefit than average, or even be harmed," says lead author Rodney Hayward, M.D. "If 9 people are in a room with Bill Gates, the average net worth of people in the room will be several billion dollars even if everyone else in the room is in serious debt.
"Similarly, in a clinical trial, if a small group of patients gets a very large amount of benefit, the treatment can have average benefit even when most people in the trial received no benefit and some are harmed," he says. "We believe our study clearly demonstrates that the current way medical treatments are evaluated is inadequate to detect this problem, that this problem is not rare, and that mandating risk-stratified analysis has the potential to decrease the risk of promoting wasteful and unsafe treatments." Hayward directs the VA Center for Practice Management and Outcomes Research and is a professor of medicine and public health at U-M.
In the paper, he and his colleagues present real data from clinical studies, and hypothetical simulation models. They also report what they found in a review of papers in major medical journals, which showed that only 4 percent of clinical trials used risk-stratified analysis.
Among the treatments that should have been studied with risk-stratified analysis, they show, is the clot-busting drug tPA. A major 1993 study showed that heart attack patients who received it were much less likely to die than those who got another drug.
But when Hayward's colleague David M. Kent, M.D., M.Sc., now at Tufts University, analyzed the data from this study in a risk-stratified way, he found major differences in effectiveness of tPA. In fact, his analysis shows that 25 percent of the patients in the original study accounted for more than 60 percent of all the benefit in the entire study. Meanwhile, half the patients received little or no benefit -- and some had such a high risk of brain bleeding from tPA that there was net harm.
"A risk-stratified analysis can also find that we are underestimating the benefits of a treatment," says Hayward. For example, a major study of tPA for stroke treatment showed no benefit among any group of patients treated between three and five hours after the start of their stroke. But Kent and others showed by risk-stratified analysis that many patients who had a low risk of brain bleeding did indeed receive significant benefit from tPA even three to five hours after a stroke.
These real-world examples are among those described in the new paper alongside statistical simulations that further illustrate how "average" trial results can be misleading.
"A treatment that decreases risk of stroke by 25 percent will have 100 times more absolute benefit for someone with a one in 10 risk of having a stroke than for someone with a one in 1,000 risk," notes Timothy Hofer, M.D., M.Sc., a U-M associate professor and VA research scientist who worked on the statistical simulation models for the study. "In fact, if that treatment also harms even one in 1,000 people due to serious medication-related adverse events, giving that treatment to the low-risk person you will harm four people for every one person it helps."
"Risk stratification of study results will also provide better information to help doctors talk with their patients about whether or not a certain treatment is right for them, considering their individual circumstances," says Hayward. It could also save health care dollars by decreasing overuse of medical treatments, which have become increasingly expensive. "It is quite possible that a substantial proportion of runaway health care costs is related to advocates using average benefits of treatments to promote use of these treatments to large numbers of people who receive little benefit from them."
Of course, to make risk-stratified analysis possible, it's important to have good models, or statistical tools, that take into account the underlying factors that can make someone more or less prone to certain diseases or health incidents. For instance, in order to tell whether a new drug increases the risk of heart attack, researchers would need to use a model that calculates baseline risk of heart attack considering multiple risk factors, such as age, smoking history, cholesterol, family history, blood pressure, diabetes, past heart-related problems, and other factors.