Does Leeds General Infirmary have a death rate "twice the national average"?

UPDATE, 9 April 2013

Today Sir Bruce Keogh, Medical Director of the NHS, admitted that he had acted on "inaccurate data" in suspending children's heart surgery at Leeds General Infirmary. Yesterday NHS England said that the unit would resume operations from Wednesday.

NHS England has now completed the first stage of its review into patient safety at the unit. According to Sir Bruce, he based his original decision to halt child surgery at Leeds General Infirmary on poor quality mortality data that the hospital itself had submitted.

--

NHS England ordered the department to shut its doors a fortnight ago after it developed concerns about the death rate among its patients. It made its announcement in the wake of Robert Francis' damning report into the scandal at Mid Staffs NHS trust.

At the time of its closure, Sir Bruce had said that the mortality rate at the unit was double the national average. However, a cardiologist at the hospital had then claimed that the data behind this statistic was flawed: "We are", she said, "confident that our mortality figures are well within what would be expected."

"Safety first"?

Sir Bruce, who is leading the government's investigation into NHS trusts with 'outlying' mortality rates, said that he was forced to act because the death rate at the department was "twice the national average". He added that this was one of a number of reasons for his decision - for example, there had also been reports of inexperienced doctors being placed in charge of the unit.

Sir Bruce appears to have used data provided by the National Institute of Cardiovascular Outcomes Research (NICOR), which compiles detailed mortality statistics for cardiology departments. The most recent data, which has been passed on to Sir Bruce, isn't yet available to the public.

However Dr John Gibbs, who is Chair of the committee that's responsible for auditing the data, has emphasised that the mortality data, which hasn't yet been finalised and analysed, wasn't ready to be published. He suggested that the media furore has discredited their statistics: "I think nobody will trust us again. It's dreadful."

There's a degree of care required when it comes to interpreting and comparing a hospital's mortality rates. Hospitals with specialist units (like the paediatric heart surgery department at Leeds General Infirmary) are likely to treat patients who are more severely ill or who require more complicated medical interventions.

This means that the mortality rate at one of these hospitals might be higher even though it employs doctors with the highest level of expertise. The mortality rates therefore need to be statistically adjusted to account for a hospital's intake of patients.

For monitoring hospital care more generally, the NHS in England publishes Hospital Episode Statistics (HES). This is separate data to that produced by NICOR. However, the HES mortality rate data has not been adjusted for specific conditions.

Professor Sir Brian Jarman, who has privileged access to the HES data because of his involvement with health intelligence company Dr Foster, has used this HES data to produce an adjusted mortality rate for Leeds General Infirmary. He says that the mortality rate at Leeds General Infirmary was "not significantly higher than at other units".

From the information that he has supplied, we can see that the Standardised Mortality Ratio (SMR) at the hospital in question was 98 and therefore could be described as "average". The SMR is calculated against a baseline of 100 so any figure above this number potentially indicates a high or "above average" mortality rate, while any figure below this number suggests the opposite. 

Sir Bruce has used a different set of statistics to inform his decision and Professor Jarman's data is not part of an official statistical release. What we have essentially amounts to an ad-hoc briefing with no accompanying description of how the data has been adjusted, what it can tell us and what it can't. (There is, however, some more technical information on Professor Jarman's website.)

Incorrect and incomplete data?

Without the specific figures, we aren't able to put the "double death rate" claim into its proper context. We don't yet know how many deaths occurred at the cardiac unit at Leeds General Infirmary and how this compares to the total number of operations performed. We also don't have equivalent figures for other hospitals.

If we're dealing with small numbers (as is common, with this type of specialist surgery), then the mortality rate is more easily influenced by even a minor increase or decrease. This means that, compared to when we're handling a large data set, any deviation from the average is more likely to be down to chance.

As it stands, without any official release to accompany the data, there is too much that we don't know. 

Trial by media?

Sir Bruce said that a number of factors informed his decision to suspend operations at the children's cardiac unit. However, it's hardly surprising that the media has led with the line that it was a higher than average death rate that led to the closure of the department. 

It's clear that there are conflicting sets of data and that not all of the data currently meets accepted standards. As it stands, operations at the cardiac unit of Leeds General Infirmary have been suspended on the basis of data that is in 'first draft' form. To generate, to standardise and then to compare mortality rates between hospitals is a highly complex task.

What we as the public don't see is how data is 'cleaned up', double-checked and confidence-tested before it is made public. The data that informed Sir Bruce's decision to close the unit is effectively 'provisional'.

The UK Statistics Authority publishes a code of practice in which it states that no National Statistics report should be made public or shared with the media before it's officially published (Protocol 2.8). However, it's unclear whether the data from NICOR, which has been cited in support of the double death rate claim, is officially subject to these guidelines. NICOR's data is not kite-marked as 'National Statistics'. But as it receives government funding, there's an argument that it should be following the code of practice that applies to other public bodies.

It is easy to see why NHS managers are opting for a cautious approach - when lives might be on the line, the stakes couldn't be higher. However, from what we know of the numbers so far, it looks like the closure of the children's cardiac unit was a precautionary measure. It's clear that further analysis of the data is needed before the experts can agree on what it tells us.

It's not yet apparent how damaging the whole episode has been. A headline in a newspaper is not usually caveated, and even when journalists do acknowledge that there are nuances to a set of data, it's often too little, too late. In this case a community's confidence in its local hospital - and, more broadly, the public's confidence in the NHS - may have been irrevocably undermined.

***

We'd like to thank Alison Macfarlane from City University London, who provided valuable help in the writing of this article. However, any errors are the responsibility of Full Fact.

If you would like to join Full Fact's network of experts, please email our Communications Officer Phoebe Arnold.