Dollars, denominators, and risk adjustment

Because not everyone who reads this blog reads the comments, I wanted to highlight these particularly insightful observations about Mike’s post on denominators for CLABSI (emphasis mine):

"The thought experiment works with the assumption these two ICUs are indistinguishable except for the frequency of CVC use. Historically, I think the justification for comparative rates using CVC denominators was a no-brainer. These devices were critical to saving lives, and the variations in device utilization probably reflected differences in patient populations, even within similar types of locations. Accounting for the overwhelming primary risk (the CVC) made sense, since these devices were critical to care. The problem you’re outlining now is very real -- as the clinical environment has proven that a lot of the variations in CVC use may in fact be personal preference. Just like the argument with CAUTIs (where foley use is deemed less critical to care) to use a patient-day denominator is strong, we may be at a time where the CLABSI argument is as strong. Improving the classification of ICU types, by more objective criteria than currently used in NHSN (i.e. the 80% rule), would really advance the comparative metric substantially, and likely provide more valid risk adjustment with patient-day denominators than we currently have with these archaic classification schemes (e.g., "med-surg icu"). Advancing the use of composite administrative data to classify patient locations to a more objective, reliable, and granular level, based on fractions of patient-days that have key underlying diagnosis, procedures, etc. is greatly needed."

Given the millions of dollars that are now at stake based upon a hospital’s performance on healthcare-associated infection (HAI) metrics, it’s hard to overemphasize the pressure that is now being placed on the NHSN definitions, and the importance of ensuring that the definitions keep pace with evolving approaches to patient care. When I was a medical resident (yes, way back then), the presence of a CVC was a good indicator of severity of illness and likely served well as built-in risk adjustment for the broad categories of ICU. The same cannot be said now; the device utilization ratios (and percentile ranks compared across NHSN units) vary markedly between different ICU types in our hospital, and do not correlate well with illness severity. And as we’ve learned with CAUTI, the device days that are most amenable to reduction (the “low hanging fruit”) are always the lowest risk device days.

Denominators matter

Let's perform a thought experiment. At St. Eligius Hospital there are two ICUs. These two ICUs have the same number of beds, the same number of patient days (12,000/year), and the same case mix index. In fact, they're essentially identical, except that ICU A has an annual CLABSI rate of 2.7/1,000 central line days and ICU B has a CLABSI rate of 5.0/1,000 central line days. Which ICU is better performing with regards to CLABSI? Well, without any other data to consider, we'd be greatly tempted to conclude that ICU A is the better performer since it's CLABSI rate is nearly one-half that of ICU B. Now, let's add another piece of information: ICU B focused on reducing central line placement as a safety intervention--so at year's end, ICU A had 7,500 central line days and ICU B had 3,000 central line days. This means that ICU A finished the year with 20 CLABSIs, and ICU B had 15. Now it's clear that ICU B is the better performer despite having the higher rate.

This is not just a theoretical problem. During my first rotation on the Infectious Diseases Consultation Service at the University of Iowa last year, I was struck by the low prevalence of central lines in the medical ICU. Turns out my perception was spot on--when I looked at our NHSN data, I saw that 3 of our 5 adult ICUs have central line utilization ratios less than the 15th percentile nationally. This is not an accidental occurrence; clinicians in those ICUs have worked hard to avoid placement of devices that are associated with infection. The problem is that the central lines that do get placed in these units are concentrated in a group of patients that are sicker and more likely to develop CLABSI, since the less sick patients will be managed without a central line. Moreover, the denominator is reduced. And the result is higher CLABSI rates. Here, no good deed goes unpunished.

But there's an easy fix. Instead of using device days as the denominator, use patient days. In our thought experiment, we would see that ICU A would have a CLABSI rate of 1.7/1,000 patient days and ICU B would have a rate of 1.2/1,000 patient days. The better performer (ICU B) will now have the lower rate, as expected. Makes sense, no? CDC should move to address this given the financial penalties hospitals now face based on CLABSI rates. Changing the denominator would provide an incentive for hospitals to aggressively reduce device insertion. And since NHSN has collected patient days for decades, there would be no loss of long-term trending. Lastly, use of patient-days as a denominator produces a patient-centered metric. Think about it: do we really care at what rate catheters become infected? No! Our focus should be on what rate of and how many patients become infected, which is also more intuitive for providers at the sharp edge of patient care.

Using NHSN C. difficile Infection Rates? Mind your denominator!

Over here in the US hinterland we're completing a systematic review of MDRO outcomes for CDC in cooperation with investigators in Salt Lake City. At the moment we're tackling C. difficile and are busily pouring through the literature. We've come across many good studies, such as an ICHE paper from early 2013 by Gase and colleagues from the New York State Dept. of Health that compared NY State CDI surveillance to NHSN in 30 hospitals. The authors noted an 80% agreement between the methods and thus recommended that NY State adopt the NHSN LabID method because of ease of implementation.

Building on that study, Haley and colleagues also from the NY State Dept of Health completed an analysis of the sources of bias in NHSN "Hospital Onset" CDI rate calculations using data from 124 NY hospitals. Their findings were published in the January 2014 issue of ICHE and were accompanied by a nice editorial by two of my former Maryland colleagues Jessina McGregor and Anthony Harris. The NY authors looked at how auditing, including outside labs, age adjustment and exclusion of "patient days not at risk in the denominator" would improve the calculation of hospital-onset CDI rates. As you can see by the portion of Table 2 that I pasted below, most of the corrections had minimal impact on the average hospital-onset CDI rates.  However, "exclusion of patient-days not at risk" had a huge impact on the calculated HO-CDI rate. The correct rate after controlling for all factors was 11.6/10,000 patient days; however, excluding auditing or outside labs, or age adjustment had minimal impact, whereas not excluding patient days not at risk from the denominator led to a rate that was 45% lower (6.4/10,000 pt-days).

The reason that eliminating "patient-days not at risk" from the denominator had such a huge impact is that the CDC NHS definition excludes CDI cases that occur in the first three days from the numerator but does not exclude patient-stays less than three days from the denominator. For example, a patient that stays only two days would not be at risk from contributing a HO-CDI case to the numerator but contributes their patient-days to the denominator.

This has several important implications.  One, not removing the patient days not at risk results in reported CDI rates that were much lower than they actually are. This occurs since many if not most patients have stays that are shorter than 4 days.  Second, as the authors state, "HO-CDI rates at hospitals with shorter LOS are biased downward more than the rates at hospitals with longer LOS." It seems to me that this artificially hurts the rates at tertiary-care and academic medical centers more than it would smaller community hospitals. We always hear how academic hospitals are falling behind, but it may have something to do with how rates are calculated, especially if we are including the wrong patient-days in the denominator.  It seems like this would be an easy fix - hospitals could just exclude the first three days from their patient-day calculations.  I hope this happens.