De-implementation and Noninferiority in Infection Control Studies

De-implementation or "stopping practices that lack supporting evidence" is a popular topic in infection control circles. In fact, just yesterday I read a discussion where the authors suggested we no longer need to practice hand hygiene after removing gloves when caring for patients with CDI. I guess there aren't randomized trials - you can't be serious!

Which brings me to a recent review in the NEJM by Laura Mauri and Ralph D'Agostino titled "Challenges in the Design and Interpretation of Noninferiority Trials."  This review is very well written - perhaps required reading for epidemiology students well written. In infection control, it is important to recognize that most de-implementation studies are really non-inferiority trials. For example, when we discontinue contact precautions, we are really suggesting that "stopping contact precautions" is non-inferior to continuing contact precautions in preventing MDRO transmission - of course ignoring that compliance with contact precautions is probably so poor that they are basically the same intervention!

In the contact precautions example, we would be testing whether stopping contact precautions "is not worse than the control (continuing contact precautions) by an acceptably small amount, with a given degree of confidence." The null hypothesis would be that discontinuing contact precautions leads to higher transmission of MDRO (i.e. is worse) and rejection of the null hypothesis is used to support the claim that discontinuing CP is noninferior. Here I suggest you stare at Figure 1 for a bit (probably easier to read in the paper with the description of each condition, but I have included it below anyway)

Further discussion about the design and analysis of these trials is way beyond the scope of a humble blog post; however, the authors include nice descriptions of methods for deriving noninferiority margins, the "constancy assumption" and statistical analysis approaches. But their 6th and 7th components of noninferiority trials are worth mentioning from an infection control standpoint:

6) Adequate ascertainment of outcomes: The authors write that "incomplete or inaccurate ascertainment of outcomes, as a result of loss to follow-up, treatment crossover or nonadherence, or outcomes that are difficult to measure or subjective, may cause the treatments being compared to falsely appear similar."  I would suggest that studies that seek to de-implement contact precautions that do not include admission/discharge surveillance cultures seeking to detect transmission events fail this criteria.

7) Issues with "Intention-to-Treat" in noninferiority designs: In a superiority studies (typical RCTs), intention-to-treat analysis, where anyone who receives the treatment is included even if they get one dose, is the gold standard. The authors write: "In a noninferiority study, however, if some patients did not receive the full course of the assigned treatment, an intention-to-treat analysis may produce a bias toward a false positive conclusion of noninferiority by narrowing the difference between the treatments. In some instances, a per-protocol analysis, which excludes patients who did not meet the inclusion criteria or did not receive the randomized, per-protocol assignment, may be preferable in a noninferiority trial. However, a per-protocol analysis may include fewer participants and introduce postrandomization bias. In general, both the intention-to-treat and per-protocol data sets are important. We suggest analyzing both sets and examining the results for consistency."

Just some things to think about as we read the coming wave of de-implementation studies in infection control including diagnostic stewardship.

Wrong answer

This morning I stumbled upon this piece, Wrong Answer (free full text here), by Rachel Aviv in The New Yorker. It's an old article from 2014, but a wonderfully written, compelling, sad tale. It's the story of how high stakes standardized testing of middle school students in economically disadvantaged neighborhoods in Atlanta led to cheating by teachers. It focuses on Damany Lewis, a superb teacher totally committed to his students, who tirelessly worked to improve his students' math knowledge and was successful in doing so, but not successful enough to hit an unreachable goal. Responding to increasing pressure to raise testing scores, he and other teachers began to change the answers on students' tests. We all know that cheating is unethical, and at first glance I bet most of us would argue to punish those involved, but read this entire piece (warning: it's long), and you're likely to soften your stance. The consequences of not meeting unreasonable targets were so severe that the teachers felt compelled to cheat in the best interest of their students.

Now take this article from the education setting into the world of healthcare epidemiology, and if you're like me, there will be chills going down your spine as you read it. It should be required reading for anyone who works in healthcare quality or the key stakeholders in this space, from those at the front lines to those who work in professional societies, and to those who create policy at the state or national level. There are also lessons here for patients and patient advocates.

Donald Campbell

What happened in Atlanta shouldn't surprise us. In 1979, Donald Campbell, a psychologist, published a paper, the crux of which has become known as Campbell's law. I wasn't aware of this until I read Aviv's article. It states: "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." Moving this to our world, you can delete the word "social" in Campbell's law and take a look at Dan Sexton's commentary, Casablanca Redux, from 2012. Here's an excerpt:

Our informal discussions with other hospital epidemiologists, our experience in evaluating the source of infection in hundreds of bacteremic intensive care unit (ICU) patients, and common sense have led us to suspect that many hospitals do not accurately report their true rates of CLABSI, using current NHSN definitions. In some cases this may reflect an unwillingness of local staff to accept these definitions as accurate or fair; in other situations it may reflect an unconscious desire to hedge or reduce their rate of CLABSI to avoid criticism and negative consequences from their local supervisors in the press, clinicians, or the general public who review their publicly reported data... If clinicians inappropriately or illogically fear or anticipate negative feedback about the rate of CLABSI in their institutions, they may consciously or subconsciously fail to obtain blood culture results for every patient with a possible or likely BSI. Simply put: no culture equals no infection, using standard definitions of CLABSI. 

At some level, we are all complicit in this. And depending on the action, it may not be the wrong thing to do. In fact, it may benefit the patient. For example, better diagnostic stewardship in the form of appropriately ordering fewer urine cultures not only lowers CAUTI rates but reduces antibiotic utilization with several resultant benefits. Still it's important to note that the primary impetus for this was to lower HAI rates. We take into consideration how a new diagnostic test may impact HAI rates and may even allow that to impact the decision to implement (see an excellent paper by Dan on this here). We may allow clinicians to censor infections that infection preventionists have detected even though the cases meet NHSN definitions. And at the extreme, hospitals may engage in practices that may harm patients in order to reduce publicly reported HAI rates. In a recent publication on how physicians in training view quality initiatives, a dirty secret was elicited from a resident during a focus group at an academic medial center: “There’s like the central line infection protocols…. If you suspect that anybody has any type of bacteremia, you don’t do a blood culture, you just do a urine culture and pull the lines … we just don’t even test for it because the quality improvement then like marks you off.” 

While reading Rachel Aviv's paper, I wondered: Do we ever ignore results (i.e., infection rates) that seem too good to be true like the educational administrators in Atlanta did? Do we critically analyze surprisingly good results to the same degree as we do surprisingly bad results? In Damany Lewis' case did the end justify the means? Is there ever a situation where I could be pushed to a similar point as Lewis?

The Atlanta school system harmed students and teachers in a thoughtless quest to improve quality. There were no winners. Sadly, the response to the cheating scandal was to raise the stakes for test scores even higher. With pay for performance the same is happening in health care.

Chlorhexidine bathing outside the ICU: Await the ABATE!

Daily chlorhexidine (CHG) bathing has become routine in many ICUs. Given that more healthcare-associated infections (HAIs) (including more central-line associated bloodstream infections (CLABSIs)) occur outside of the ICU, many hospitals have also implemented this practice on general medicine and surgical wards. However, to this point there are few data to support the effectiveness of CHG bathing outside of ICUs. 

So I was very excited to hear Susan Huang present the results of the Active Bathing to Eliminate Infection (ABATE) study at IDWeek last week. Similar to the REDUCE MRSA study, this 53 hospital cluster-randomized trial took place in the Hospital Corporation of America (HCA) system. Units were randomized to routine care or decolonization (this consisted of daily CHG [4% rinse-off shower or 2% leave-on bed bath] with addition of mupirocin nasal ointment for 5 days if + for MRSA by history or culture/screen) for a 21-month intervention period (after collecting baseline data for 12 months). The primary outcome was MRSA or VRE clinical isolates, and the main secondary outcome was any bloodstream isolate attributed to the unit (for common commensals, 2 or more + cultures).

With the requisite reminder that it is always best to wait for the peer-reviewed publication to make firm conclusions, the results presented at IDWeek suggest the likely take-home from this study: No measurable benefit in the entire population, but significant reductions in MRSA/VRE clinical cultures and bloodstream infection in the subgroup with devices (central lines, midlines, and lumbar drains). This subgroup represented 12% of the study population but accounted for 34% of all MRSA/VRE events and 59% of bloodstream infections. The additional reductions in the decolonization arm for this subgroup (compared with routine care) were 32% for MRSA/VRE cultures and 28% for BSI (both highly statistically significant). 

Once published, these findings will leave infection prevention programs with some interesting decisions. A quick take might be, “OK, let’s just use CHG in those non-ICU patients with devices (or just central lines).” However, this isn’t what the ABATE trial evaluated—it showed a substantial reduction in MRSA/VRE/BSI outcomes in patients with devices when everyone else was also receiving CHG (+/- mupirocin). To assume that the decolonization of the non-device population had no beneficial effect on those with devices is to discount any potential role for reduction in pathogen transmission between non-device and device patients. Another tricky question has to do with the role of mupirocin—adding this agent to CHG for known MRSA carriers without knowing how important this component of the intervention was adds some logistical complexity, cost, and antimicrobial resistance concerns (the investigators are also doing the microbiology work to assess for CHG and mupirocin resistance emergence). 

We’ll revisit this study once it is published and all the details are available. For now we should congratulate Susan and the entire ABATE trial team for another tremendous contribution!

IDWeek 2017 is in the books

I'm back from IDWeek 2017 in San Diego -- always good to reconnect with old friends, meet new colleagues, and hear the latest in infection prevention and antibiotic stewardship science and practice.  This year I noted an increase in pro-con and clinical controversies sessions, which I think are so valuable for those of us struggling day-to-day with issues where the evidence base is absent or opinion is conflicting.   If you didn't get a chance to attend, fear not, as I am sure there will be more conference-related news emerging over the coming weeks (all 6 of your favorite bloggers made an appearance). 

Of course, it's never too early to think about next year -- IDWeek 2018 heads back to the left coast, this time it's San Francisco, October 3-7, 2018 (mark your calendars!).  I'm among the SHEA contingent for the planning committee (along with Hilary), and we're very interested in program suggestions -- please send them our way.

I'll leave you with one meeting tidbit -- Shelley Magill presented the results from the CDC's 2015 HAI Prevalence study, a follow up to their 2011 survey that was published in the NEJM.  In the 4 years since the first survey (using the same hospitals included previously):

• The HAI prevalence rate among hospitalized acute care patients fell from 4.1% to 3.2% (a 22% decrease)
• Central line and urinary catheter use were both significantly lower
• Healthcare-associated UTIs and SSIs significantly decreased

Nice to see some positive news to reflect the extensive HAI prevention efforts nationwide.  Of note, antimicrobial use stayed stable, reinforcing the need for increased antibiotic stewardship efforts.  Given the enhanced focus in this area, I am hopeful that the next prevalence survey will show improvement on that end as well.

Unwarping the playing field

Easing comfortably into my role as a "blogger," I've realized how easy it is to adopt a few key platform issues that tend to drive one a bit mad.  This blog isn't so fond of CAUTI, contact precautions, or devices that blow moist, warm particles over sterile fields (but then again, who is?).  We love diagnostic stewardship, influenza vaccination (um, mostly), and fecal transplantation.

Add advocating for better risk-adjustment of publically-reported HAI performance to my list.  A few months ago, I blogged about this issue and poor reporting validation by CMS -- now some excellent papers on improving risk adjustment have emerged, both from many FOB (friends of the blog) with senior authorship by Anthony Harris and his group at Maryland.  One focuses on SSI and one on CLABSI.  Their methodology is very similar and has some key features:

• They used comorbid conditions that are components of the Charlson and Elixhauser comorbidity indices
• These conditions were captured by diagnostic discharge coding that are currently routinely collected and submitted to CMS, limiting the data collection burden 
• They used conditions identified using Delphi consensus methodology from a survey of ID and infection prevention experts, providing some clinical credibility to the process

The authors examined the model performance and assessed changes in hospital rankings when compared to the traditional NHSN models. For SSI, a model containing procedure type, patient age, race, smoking history, diabetes, liver disease, obesity, renal failure and malnutrition showed good discrimination, and 86% of hospitals changed ranks within the cohort when the risk-adjusted model was used -- with 4 hospitals changing by >10 ranking spots.  For CLABSIs within the ICU, a model using coagulopathy, paralysis, renal failure, malnutrition and age showed improved predictability when compared to the NHSN ICU model, and 45% of hospitals changed ranking.  The authors note the clear limitations, including some of the challenges with using coded data, but these papers are important advances in the area of publically-reported HAI data.

You can read some of my more detailed thoughts in the accompanying editorial for the CLABSI paper (shameless plug).  At a time when there are many consequences for a hospital's performance on these surveillance metrics (e.g. last year my hospital received a draft quality incentive contract from a private insurer that required our large, tertiary care center to have ZERO of the Big 6 reported HAIs, to the tune of several million dollars in incentives), leveling the playing field to adjust for those factors that lead to HAIs that are beyond the control of the hospital is essential.  Thankfully, the CDC and HICPAC have recently chartered a new NHSN work group (disclaimer: Hilary and I serve on this group) and the issue of improved risk adjustment seems to be a major emphasis - fingers crossed that the field will start to level soon.

When prevention success stagnates? Treat the patient!

Prevention is paramount – I do believe this, and I know that is so much of what healthcare epidemiologist strive for; however we often become very myopic and focus exclusively on “modifiable risk factors.” It is refreshing to read a nicely done epidemiologic study to illustrate what an impact the infectious disease community can have by improving the way we approach patient treatment. When reading the recent article in JAMA IM by Michihiko Goto and colleagues I was expecting a nice ecologic study showing an impressionistic picture of how improved treatment processes correlate with improved MRSA bacteremia mortality - but our VA colleagues working with big data have painted more of a realistic 

Le Déjeuner sur l’herbe Painting 

by Édouard Manet, 

1863 Musée d’Orsay, Paris

Google Arts & Culture

than an impressionistic picture. Using the VA database, capturing deaths occurring during both the inpatient stay and the post-discharge period, they quantify improved survival at the patient level is driven by improved processes of care for S. aureus bloodstream infection (Association of Evidence-Based Care Processes With Mortality in Staphylococcus aureus Bacteremia at Veterans Health Administration Hospitals, 2003-2014 JAMA IM).

I have been pushing for transitioning efforts to prevent S. aureus (more specifically MRSA) bacteremia to the post-discharge setting, worried that the recent reductions observed among hospital-onset MRSA BSI are not being realized in the post-acute care setting. We’re getting stuck; in fact we have been stuck for a while at preventing community-associated MRSA BSI (See figure). 

Goto and colleagues provide some clarity to preventing deaths related to MRSA BSI (and S. aureus BSI overall) even during periods of prevention stagnation such as we may be in currently. Of note, the incidences of healthcare-associated and hospital-onset MRSA BSI decreased in VHA hospitals between 2003 and 2014, whereas the incidence of CA bacteremia was stable. This is identical to trends illustrated nationally using CDC's EIP data, suggesting the VA analysis may be reflective of what is going on nationally.

Goto utilized the national Veterans Health Administration (VHA) health care system to first determine how to best risk adjust mortality; and then determine the independent effect of each of three pillars of guideline directed processes of care for managing S. aureus bacteremia (SAB): (1) appropriate antibiotic therapy, (2) echocardiography, and (3) consultation with ID specialists. They report lower risk-adjusted mortality among patients with SAB when they received (1), (2), or (3), and there was a nice dose-response relationship between the number of care processes and mortality. They estimate “57.3% of the decrease in risk-adjusted mortality among patients with SAB between 2003 and 2014 could be attributed to increased use of these evidence-based care processes.”

Their paper also sheds some light on the importance of capturing post-discharge data when quantifying mortality related to processes or infections related to the hospital setting!. Risk-adjusted mortality decreased from 23.5% in 2003 to 18.2% in 2014, regardless of MRSA, MSSA, and place of acquisition (Figure).

From Supplemental Figures, Goto et. al.

I was caught by the discrepancy between these mortality rates and the mortality reported by the CDCs Emerging Infections Program invasive MRSA Surveillance, reported around 12%. The latter is limited to in-hospital or 30 day mortality, whichever comes first. Recent data at IDWeek by one Emerging Infection Program site identified another 30% of deaths among patients with MRSA BSI occurred post-discharge. The Goto paper captures these deaths, making their conclusions more realistic. In addition, their risk adjustment for mortality included over 13 comorbidities, timing of infection, and susceptibility. They did an outstanding job of trying to evaluate the relative importance of each care process while accounting for changes/absence/presence of these underlying predictors of mortality (both inpatient and post-discharge). They even did a sensitivity analysis to account for early deaths, before these care processes could occur. Furthermore, the impact of receiving the care processes was similar regardless if the SAB was hospital-onset or community-onset!

The bottom line – following evidence base care processes does save lives. Their analysis support their conclusion that “there is a need for continued implementation of quality improvement initiatives to increase the adoption of these evidence-based care processes for patients with SAB.” Let’s be bold and call these what they are: performance measures! Here we have a potential metric (proportion of SAB receiving said care process), that are closely linked to improved survival, and can be captured electronically in an objective manner (at least in the VA!) improving the reliability of these metrics across facilities. I was glad to find through a google search that IDSA recently (December 2016) urged CMS to move in this direction -- although the details of the status of this proposed measure (#407) are not clear to me.

While we anticipate novel therapeutics for MRSA BSI including immunotherapy adjuvants or vaccines to become available in the not to distant future; quality improvement efforts in the evidence-based processes of care for SAB now will likely improve our patient’s outcome. I hope we can turn our attention here soon -- and have hospitals rewarded for doing so.

ICD Coding and MDRO - If you don't bill for it, it doesn't exist

Fact 1: Infectious Disease specialists are among lowest paid physicians in the US

Fact 2: Many infectious diseases, including HAI, are absent from ICD billing codes or are poorly coded, and thus the true population health impact of infections, particularly MDROs, is invisible

Hypothesis: If we could improve ICD codes for infectious diseases, ID salaries would increase and the field could be saved from extinction

Many of us on the blog have lamented about the current state of ID with a particular focus on low relative salaries compared to other medical subspecialties and concerns about the annual fellowship match. If we focus on academic ID, the folks who supposedly will train the next generation, we also need to worry about low levels of NIH funding for anything other than HIV research; a trend that is slowly improving. (Fact 1, above)

Our group has looked at the accuracy of ICD billing codes for both HAI and MRSA and the results are not pretty. Michi Goto completed a systematic review of ICD code accuracy for CDI, SSI, VAP/VAE, CLABSI, CAUTI, post-procedure pneumonia and MRSA. He found that apart from CDI and orthopedic SSI, the codes have poor sensitivity and specificity. Marin Schweizer looked at the validity of the V09 code pre-2008 for MRSA and found it to be a very poor at detecting proven incident MRSA infection. (Fact 2, above)

Since many of those studies were completed, there have been new codes added for CLABSI (October 2011), VAE/VAP (October 2008), MRSA infections (October 2008), and post-procedure pneumonia (October 2012). So there is some hope that HAI and MRSA will become better recognized. But what about MDROs?  Investigators from Wash U (including co-blogger Hilary) just published a research letter in ICHE that calculated the sensitivity of ICD-9 codes for various MDRO at their hospital between 2006 and 2015. The gold-standard was a culture at a sterile site or BAL/brochial wash culture with an MDR-Enterobacteriaceae, Enterococcus spp., Staphylococcus aureus, Pseudomonas aeruginosa, or Acinetobacter spp.

As you can see in their Table 1, apart from MRSA (after 2008 code added) and P. aeruginosa, ICD organism coding had poor sensitivity and MDRO/V09 codes were - terrible. The authors concluded: "ICD-9-CM diagnosis codes cannot be used to estimate the burden of MDRO infections in hospitals."  I think we'd all agree. I would have liked to see more information on the specificity and positive/negative predictive values of individual codes (I understand this was a Research Letter).  I'm not sure what the ultimate solution is, but perhaps SHEA or IDSA (or ASM) could work to update ICD-10 codes and come up with ways to encourage accurate coding for infectious diseases. If we don't make sure infections "exist" in administrative data, the field of infectious diseases might not exist for long.

image source