What to know about St. Mary’s children’s heart surgery mortality rates

Update July 1st, 2015: St. Mary’s has posted a new risk adjusted mortality rate of 4.7% for its Pediatric Congenital Cardiac Surgery program. You can read St. Mary’s statement here.

CNN’s investigation of pediatric open heart mortality rates at St. Mary’s Medical Center in West Palm Beach, Florida, showed that the hospital had a much higher death rate than the national average. Both the hospital and state of Florida have responded with documents that criticize CNN’s methodology.

We wanted to further explain how mortality rates are calculated. We also want to help readers better understand the responses and our reporting and how to find more information about mortality data.

Following are 12 key questions and answers:

1. St. Mary’s Medical Center says it has a 5.3% risk-adjusted mortality rate. How did the hospital arrive at that number?

The hospital doesn’t say.

Two numbers are necessary to calculate a mortality rate: How many surgeries were performed and how many patients died. St. Mary’s doesn’t give either in its response.

Many other hospitals are clear about these two figures.

For example, from July 2010 through June 2014, Johns Hopkins All Children’s Heart Institute performed 883 congenital heart surgeries and had 30 deaths. Divide 30 by 883 and multiply by 100 and you have a mortality rate of 3.4%.

Duke University Hospital did 828 surgeries during that same period and had 25 deaths. Do the math, and that’s a 3% mortality rate.

While many other hospitals are transparent about the precise number of surgeries and the number of patients who died, St. Mary’s has not made that data public.

2. Does the hospital say anything about the number of surgeries performed?

Yes. It has a bar graph titled “Volume of STS Congenital Heart Surgery Procedures/Patients at St. Mary’s Medical Center.” It shows that from July 2011 until June 2014, it did 132 procedures on 90 patients.

3. Is this the number of surgeries the hospital used to calculate its mortality rate?

We can’t tell. Clearly some patients had more than one surgery, but it’s not clear exactly how many.

It’s important because the Society of Thoracic Surgeons says if a patient has more than one surgery during a hospitalization, you only count the first surgery, which is known as the “index operation.”

To make matters even more complicated, St. Mary’s told the Palm Beach Post newspaper that St. Mary’s determined its mortality rate “through 70 eligible [Society of Thoracic Surgeons] cases.” This doesn’t appear to match the 132 procedures done on 90 patients listed in St. Mary’s news release on June 7. No time frame is mentioned for these 70 cases in the Palm Beach Post account. As we’ll explain below, the time frame is important.

Bottom line: To calculate a mortality rate, you must know how many index operations were done, and we have no idea. St. Mary’s declined to respond to CNN’s questions about their June 7 news release.

4. And what about deaths? Has St. Mary’s disclosed those?

No. In its news release, St. Mary’s was silent about the specific number of deaths in its program.

The hospital told the Palm Beach Post it had seven deaths but doesn’t mention in what time period the deaths occurred.

5. Does it matter in what period these deaths occurred?

Yes. Death rates naturally go up and down at any hospital. It’s important to look at as long a time frame as possible to make sure you’re not capturing a time with a particularly high rate or a particularly low rate.

St. Mary’s is unclear about the time frame for its self-reported 5.3% risk-adjusted mortality rate. In the June 7 news release, the hospital says the time frame is “for the four-year period ending on June 30, 2014.” That would appear to mean going back to July 1, 2010, but the pediatric heart surgery program didn’t start until the fall of 2011.

You might think, then, that the data begins in the fall of 2011 when the program opened its doors, but not necessarily. St. Mary’s first appears as a participant with the Society in a report published June 2014. When joining, the Society gives hospitals the choice of including past data or not. The society says the hospital did not have to count all the deaths in 2011, 2012 or 2013 if it didn’t want to. That’s the time period for which CNN was able to obtain data on the deaths of six babies.

6. St. Mary’s states that its mortality rate is risk-adjusted. What does “risk-adjusted” mean?

The Society of Thoracic Surgeons reports data two ways: as a raw mortality rate (10 deaths out of 100 surgeries = a 10% mortality rate) and as a risk-adjusted mortality rate. The risk-adjusted rate takes into consideration several factors, such as how sick the patients were before treatment and the complexity of the procedures to treat them, according to the Society.

7. St. Mary’s has sharply criticized CNN for not using a risk-adjusted mortality rate in its story. Why didn’t CNN use such a figure?

Only St. Mary’s knows its true risk-adjusted mortality. We asked for the hospital’s outcomes repeatedly, and St. Mary’s wouldn’t divulge it. It was not included in the data CNN obtained from the state of Florida.

8. So CNN reported a raw mortality for the hospital?

Correct.

Here’s how we came up with it: Take a look at this document and scroll down to page 6. It’s a report that St. Mary’s is required to file with the state of Florida, and it shows that the hospital did 27 pediatric open heart surgery cases in 2012. Scroll down further and you’ll see a report that they did one open heart surgery in 2011 and 20 such procedures in 2013.

Together, that totals 48 open heart surgery cases from 2011 to 2013.

CNN ascertained that at least six babies died from 2011 to 2013 after having open heart surgery at St. Mary’s. (CNN spoke with parents of all six babies). Six deaths out of 48 cases is a 12.5% mortality rate.

9. Is a raw mortality rate useful?

The Society of Thoracic Surgeons thinks so. Nineteen hospitals have made their data public on the Society’s website, and it shows their raw rate (they call it “observed”) and their risk-adjusted rate (they call it “Adj. Rate”)

Most hospitals’ websites report the raw rate, not the risk-adjusted rate.

Here’s an example: From 2010 through 2013, Children’s Hospital of Philadelphia performed 2,103 open heart surgery procedures. It had 26 deaths in 2010, 21 deaths in 2011, 27 deaths in 2012 and 28 deaths in 2013. It reports a mortality rate of 4.9% for these procedures. This is a raw rate.

10. St. Mary’s has accused CNN of reporting an “inaccurate” mortality rate because we used “only a subset of the procedures performed by the program and reported to the Society of Thoracic Surgeons.” What’s that about?

There are two types of heart surgeries: open and closed. Open means the patient goes on a bypass machine, which does the work of the heart and lungs during the operation.

CNN asked St. Mary’s for its data for both open and closed heart surgeries, including the number of deaths and the number of surgeries performed. The hospital would not give it.

The hospital is required to report to the state about the number of pediatric open heart surgeries it performs. The state is required to share that data with reporters on request.

Throughout our reporting, CNN has been very clear that we calculated a mortality rate for pediatric open heart cases at St. Mary’s based on this data.

We compared the hospital’s mortality for such cases – 12.5% from 2011 to 2013 – to the pediatric open heart surgery mortality rate nationwide, which is 3.3%. The rate we calculated for St. Mary’s is not risk-adjusted. We compared our calculation to the national mortality rate for pediatric open-heart surgeries, which comes from the Society of Thoracic Surgeons and is also not risk-adjusted.

11. The state of Florida says its data shows St. Mary’s risk-adjusted mortality rate is 4.58%, a lower number than the hospital cites. Can you explain that?

That number comes from Florida’s Agency for Healthcare Administration. Here’s the AHCA rates.

“I would like to think the AHCA data is correct,” said Tiffany Cowie, a spokeswoman for the Florida Department of Health. “Florida does a great job with their data.”

But there’s a big problem with the Florida data in this particular situation, according to a doctor who helped design the database Florida uses.

CNN knows of six babies who died after heart surgery at St. Mary’s from 2011 to 2013. Three of those babies died at St. Mary’s, and three died after being transferred to other hospitals in life-threatening conditions. Those transferred babies – half the deaths – would not be factored into the St. Mary’s mortality rate according to the rules set up by the federal Agency for Healthcare Research and Quality, which manages the database.

“If St. Mary’s transferred out patients who subsequently died, their mortality on the Florida data would be artificially low,” said Dr. Patrick Romano, a professor of medicine and pediatrics at University of California, Davis, and a member of the Quality Indicators Enhancement Team that works with AHRQ.

There’s another problem, too.

The Florida data – called “administrative” data – is based on hospitals’ billing systems, and as many patients know, billing mistakes can happen.

Four studies have shown that the validity of coding used in these administrative databases for congenital heart surgery is “poor,” according to Dr. Jeffrey Jacobs, chair of the Society of Thoracic Surgeons National Database Work Force.

“These [administrative] databases can be wildly misleading,” added Dr. Robert Wachter, a professor and associate chairman of medicine at the University of California, San Francisco, and editor of one of the leading textbooks on patient safety. “The general statement ‘garbage in, garbage out’ is right.”

12. The Florida Department of Health says it is “disappointed” by the way CNN “inaccurately” calculated mortality rates for our story. What’s the department’s complaint?

The department spokesperson, Tiffany Cowie, said CNN should have used risk-adjusted data instead of raw data, even though CNN explained that the Society of Thoracic Surgeons and the nation’s top pediatric heart surgery programs in the country use raw data.

Risk-adjusted data from the Society of Thoracic Surgeons is also useful, but it is kept private by St. Mary’s.