Historically, what death rates have been published?
The death rates of individual heart surgeons have been published in England and several US states for some years. A high profile investigation into high death rates at a certain cardiac centre motivated this in England. For other surgery types, death rates have been reported only at the hospital or trust level. NHS England’s “Everyone Counts” policy, introduced this Summer, means that outcomes of ten types of surgery will be reported at the individual surgeon level.
What are the potential benefits of publishing individual surgeons’ mortality rates?
The information could lead to better outcomes by improving quality of care, and could help patients make choices about their own care.
What sparked your study into the effects of publishing death rates for individual surgeons? Had this ever been studied before?
The policy to roll out the reporting of outcomes of individual surgeons beyond heart surgeons concerned us. Heart surgeons operate on a large number of patients each year but other types of surgeons operate on lower numbers of patients. If the total number of patients treated by a surgeon is very low then their death rate is very inaccurate. A surgeon with poor performance can then slip through the net because there is not enough evidence to say with any sort of certainty that their death rate is too high. There is a danger that this will lead to false complacency amongst surgeons carrying out few procedures. Our study investigated, for four specialties, the proportion of surgeons in England carrying out enough procedures to have a reasonable chance of identifying poor performance. This policy has not been investigated in this way before.
Which types of surgery did your research study the mortality rates of?
- Adult heart surgery
- Hip fracture surgery
- Removal of a section of the bowel (bowel resection) in patients with bowel cancer
- Removal of the oesophagus or stomach (oesophagectomy/gastrectomy) in patients with cancer of the oesophagus or stomach
How frequently are the types of surgery you studied performed?
In England each year there are approximately 35,000 adult heart operations, 60,000 operations for hip fracture, 20,000 bowel resections for bowel cancer and 2,000 oesophagectomies/gastrectomies for cancer of the oesophagus or stomach.
What were your findings?
For the majority of heart surgeons the number of operations carried out is large enough that there is a reasonable chance of identifying those who are not performing well. For many other types of surgery, the majority of surgeons do not carry out enough procedures to have a reasonable chance of identifying those with poor performance. For bowel cancer resection and oesophagectomy/gastrectomy, less than one-fifth of surgeons carry out enough procedures to have a reasonable chance of identifying poor performance, when this is defined as a death rate double the national rate, even when data are pooled over three years.
What are the dangers of publishing individual surgeons’ mortality rates?
The danger we identify in our study is false complacency. Rather than leading to improved quality of care, outcomes based on small numbers may be misinterpreted as signalling no problem, and this could lead to inaction. Another more widely discussed problem is falsely identifying surgeons as having poor performance when they have just had a run of bad luck. For surgeons this may lead to unfair damage to their reputations and could lead to legal action. It is also recognised that reporting their results in this way may lead to some surgeons ‘gaming’ the system e.g. selecting lower risk patients for surgery. Inaccurate estimates of surgeon performance may also cause unnecessary alarm to patients. A death rate for a surgeon of 10% may worry patients, even if the estimate is based on such small numbers that there is no reason to believe their performance is poor.
What impact do you think your findings will have?
With the wide audience that our message has now reached we hope that people will be appropriately cautious when interpreting the published figures. Our article also includes guidelines to improve the way that results are reported, so that the major pitfalls of reporting the outcomes of individual doctors are avoided. This has relevance to the future development of policy in England and globally as other countries consider the public reporting of the performance of doctors.
What are your further research plans?
We plan to continue our work in tackling the issues around monitoring the performance of healthcare providers. There is currently a drive to put more health statistics into the public domain. It is crucial that these statistics are analysed and reported in a meaningful way, so that they benefit the public rather than cause damage.
Where can readers find more information?
NHS England’s “Everyone Counts”policy:
https://www.england.nhs.uk/
About Dr Kate Walker
Kate Walker is a lecturer in medical statistics at the London School of Hygiene & Tropical Medicine.
She received her doctoral degree in 2006 from University College London.
She is based in the Clinical Effectiveness Unit at the Royal College of Surgeons where her current projects include statistical issues in performance monitoring, and various topics on the management and outcomes of patients with bowel cancer. These have included adjustment for patient risk in order to make fair comparison between providers of bowel cancer surgery, a comparison of open and key hole surgery for bowel cancer, and the reversal of “temporary” colostomies in patients with rectal cancer. She carries out a national audit of bowel cancer care.
Further details of her research can be found at:
Renaissance of "food as medicine" in modern clinical trials