Upon Further Review: Reexamining the Illinois MRSA Active Surveillance Mandate

“Thinking without the positing of categories and concepts in general would be as impossible as breathing in a vacuum” – Albert Einstein (1949)

A couple weeks ago, Scott highlighted a study in CID by Lin et al. that sought to estimate the benefits of the 2007 Illinois state-wide mandate of MRSA active surveillance cultures in ICU settings. The post was titled “Good Intentions Does not Always Mean Good Policy” and concluded “There may be many reasons the hospitals in Illinois overall are seeing an estimated 30% decrease in their hospital-onset MRSA BSI (as most states are) since the 2010 NHSN baseline, but admission screening isn’t one of them.” 

I would like to list several reasons why I think we should reconsider the study authors’ conclusions. And if you skip to the end, you will read why I think this study might make more valid conclusions about the lack of benefits of CHG bathing.

Let’s start with validity from Shadish (2001) et al: “We use the term validity to refer to the proximate truth of an inference. When we say something is valid, we make a judgement about the extent to which relevant evidence supports that inference as being true of correct.” Cook and Campbell (1979) outlined four components of validity: statistical conclusion validity, internal validity, construct validity and external validity. I’ve written about their validity typology here, if you’re interested. 

Now let’s review the Illinois study methods. They included data from 25 ICUs and completed eight, one-day point prevalence surveys AFTER the mandate was initiated (twice annually 2008-2011 and annually in 2012 and 2013). There was no concurrent control group.

Thus, this quasi-experimental study design has very low internal validity. It has no measurement before the intervention (sometimes called historical controls) and no concurrent controls. Shadish labeled this design as a “one-group post-test only design” and summarized its limitations with “this design is rarely useful.”

The Illinois study also lacks statistical validity, since it is underpowered to detect a benefit of active surveillance. Shadish lists low statistical power as the first threat to statistical validity since “the experiment may incorrectly conclude that the relationship between treatment and outcome is not significant.” Since the Illinois study is a negative one – claiming active surveillance for MRSA didn’t work, power is particularly important. If you jump ahead to my ICAAC abstract, you will begin to see why the study is likely very underpowered. Point prevalence is very insensitive to changes in acquisition or transmission, so you would need very large studies to see a benefit.

I’ll admit that a statewide study has strong external validity – that is generalizability. 

But let’s focus on construct validity. Construct validity - what Einstein was hinting at in the quote above - describes whether a study measures what it claims to be measuring. For example, if a study claims that active surveillance for MRSA and isolation doesn’t prevent MRSA transmission, that study better measure MRSA transmission. 

Let’s pause here. Now some might say, we don’t care about transmission, we just care about MRSA prevalence or MRSA infections or MRSA CLABSI (or even deaths in Avengers movies). Yet, active surveillance for MRSA doesn’t work like that – it prevents transmission between patients. 

But what if measuring MRSA point prevalence was good enough at detecting MRSA transmission? Thought experiment: what if we found PCN allergy alerts in the EMR annoying and so we eliminated them? To see if this was safe, we then checked to see if anyone was having a PCN allergic reaction every St Patrick’s Day for the next 5 years. Good enough, right? Probably not. Why would we be comfortable saying yearly point prevalence is an adequate way of measuring the benefits of MRSA transmission? I don’t think we should and here’s why:

Back in 2002 I presented an abstract at ICAAC titled: “Point Prevalence and Clinical Culture Positivity of Vancomycin Resistant Enterococci are Poor Estimates of Infection Control Intervention Impact.” This study was based on the VRE model that we eventually published in CID (2004). Anyway, we modeled VRE transmission in the ICU under a condition where active surveillance compliance on admission increased from 60% to 100% and I assumed that isolation prevented 71% of transmissions. This is a math model, so we know that the intervention worked– but could we detect it? The answer was yes but only if we used admission/discharge screening cultures.  If we used point prevalence, like the Illinois study, we would falsely claim that the intervention didn’t work 54% of the time. However, if we used admission/discharge cultures, we would correctly determine that the intervention worked 96% of the time. Here is our conclusion:

“Point prevalence or clinical culture positivity often failed to detect a benefit due to stochastic fluctuations in prevalence and high prevalence of VRE in patients entering the ICU. Studies to assess the benefits of active surveillance for VRE should measure new incident cases. Relying on point prevalence or clinical culture positivity to assess the benefits of infection control interventions may underestimate the magnitude of their benefit and may be responsible for a persistent bias against the broader institution of active surveillance. The benefits of active surveillance and other infection control interventions are probably underestimated.”  

Not bad for 16 years ago – replace VRE with MRSA and you can see why the study by Lin et al cannot be used to evaluate the benefits of active surveillance. Our conclusion was partially driven, as we said, by high rates of VRE colonization on admission. What did Lin et al say about their study: “we assessed MRSA prevalence in a region where MRSA is widely endemic both in the community and within healthcare facilities.” Thus, the MRSA situation in Illinois fits closely with what we modeled.

In conclusion – the study by Lin et al should not be used to claim active surveillance was ineffective in Illinois or elsewhere. When drawing inferences, it is important to remember all four components of validity described by Cook and Campbell 40 years ago. We still need to figure out why MRSA has declined by 30% in Illinois and elsewhere. I do wonder why we are so quick to claim CLABSI bundles, CHG bathing (see below) or other interventions have been driving these MRSA reductions and not active surveillance. If I had to guess, it’s something yellow and not the data.

-----

Side note: In this same study, the number of hospitals using CHG bathing in their ICUs went from 5 (20%) to 17 (68%). It appears the authors could use these same data and methods to show that CHG doesn’t prevent MRSA in ICUs settings. Interestingly since CHG bathing works at the level of transmission and also on individual patients’ decolonization, point prevalence data would have higher construct validity for evaluating CHG. The study might even be better powered to detect a benefit of CHG.

Lack of Financial Incentives for Preventive Interventions

One of the barriers to infection control and antimicrobial stewardship is the lack of financial support from third-party payers, such as insurance companies and medicare. For example, wouldn't it be great if we could bill for hand hygiene every time it was practiced correctly? The end result of this lack of support is that society is willing to pay thousands for a CABG surgery, for example, but not willing to spend a penny on preventing a surgical site infection.

There is a new perspective in the NEJM by two physician behavioral economists, Katherine Pryor and Kevin Volpp, that examines barriers to putting preventive services on equal financial footing with treatments. While their comments largely focus on patient-targeted (vs population targeted) preventive services like lifestyle-modification for diabetes prevention or smoking cessation, they highlight many barriers that could apply equally to population-level interventions like infection control. I've highlighted a few:

"...treatments determined by the Food and Drug Administration (FDA) to be safe and effective are usually covered by insurers regardless of their cost, but preventive services have been held to a higher standard: they are often assessed on the basis of whether they generate a positive return on investment and save money in the short term. This disparity leads to overprovision of treatments and underprovision of preventive services"

"Even Medicare — which typically covers beneficiaries for life — holds preventive services to a higher standard, applying cost-effectiveness analyses when making coverage decisions about preventive services but not treatments. This double standard has resulted in coverage of cost-ineffective therapies with prices of up to hundreds of thousands of dollars per quality-adjusted life-year, including treatments of questionable benefit"

The perspetive is worth a careful read. Would also be nice to see a similar commentary describing these barriers from an infection control perspective too.

Good Intentions Does not Always Mean Good Policy

How often do negative studies influence our behavior, or better yet our policies? For those of you that are familiar with the work I have published, you know that I published a lot of material focused on MRSA; emerging resistance, community-emergence, burden of disease, attributable cost, risk factors, and on.  I was in a position at CDC to access and synthesize a lot of data, with a goal of putting the problem in perspective and ideally affect policy. Well intended as it was, I remember very clearly in mid-2007 when policy got way ahead of the science. Two independent (but related) events occurred on October 16-17, 2007 that led to several years of a watershed of policy developments. Although I give a huge amount of credit to the very passionate and important patient advocates and consumers that built momentum for the policies – but with hindsight the policy inertia was really overcome when a senior student at Staunton River High School died on October 16 from MRSA sepsis—MRSA he acquired in the community. The press linked that death to Dr. Elizabeth Bancroft’s editorial that same week stating “…more people die of MRSA in the U.S. than of AIDS” published on October 17. Many of us see much of the public reporting and mandatory reporting policies have opened up real pathways for additional hospital resources to invest in HAI prevention. However all of us should recognize some policies of that era are likely in place that really should be re-examined. 

One of these is the Illinois 210 ILCS 83/ legislation requiring all patients admitted to intensive care units be screened for MRSA by nasal active surveillance testing (AST). Lin and colleges just published a negative study with a lot of important findings. To many, the findings will not be a surprise (CID May 15 2018, pp 1535-1539)

• Lin worked with 51 intensive care units at 25 hospitals over 5 years starting within months of enactment of this mandate to evaluate any changes in ICU MRSA prevalence through periodic point prevalence surveys performed by trained study staff during the time of this mandate. The study was a quasi-experimental time series evaluation but without a real before observation group and no control group. However, I believe that any impact would have been additive over time – the first year would have been a sort of wash in period for an intervention as broad in participation as this.  They sampled 3909 patients having the power to even detect an absolute difference in carriage as small as a 1.9% change in prevalence (eg, 10% vs 8.1%) – but they detected none. No change in prevalence of MRSA on these patients. 
  

• Compliance was high overall (93%), admission prevalence was comparable to other studies (9.7%), and overall, at any given survey of known positive patients and unknown, 11.1% were positive in any given month, in any given year of this study.  Sure, time to placement of contact precautions lagged from test turnaround time or from time to test result to actual placement of precautions, but most notably the mandated testing was only 84% sensitive compared to best testing methods  employed by the study investigators. This is the real world after all.

• While these ICUs have invested time, effort, and money into these admission swabbing and targeted placement of contact precautions, the prevalence of MRSA carriage has not budged in these intensive care unit patients.

There may be many reasons the hospitals in Illinois overall are seeing an estimated 30% decrease in their hospital-onset MRSA BSI (as most states are) since the 2010 NHSN baseline, but admission screening isn’t one of them. Maybe its CLABSI prevention, or that uptake of the percentage of study patients receiving CHG baths. However, this study suggests it was not the mandated AST for all ICU patients admitted to the ICU. These patients are bringing their MRSA in with them, let’s free up staff time to prevent the infections.

I know there are many major federal policies we all can be passionate about changing or starting, these are crazy days. But when the scientific evidence is so strong illustrating that a very well-intended policy regarding use of nursing and infection control resources does not have the intended impact – change it. Nursing care can better be spent caring for patients, practicing best infection control for all patients in these intensive care units. 

Prevention in Portland

Last week, several of your friendly neighborhood bloggers (Hilary, Scott, Mike, and yours truly) were among the attendees at this year's SHEA Spring Meeting, held in Portland, OR, and it was great to see how this iteration of the meeting has matured in just a few years.  The Planning Committee, headed by Matt Linam and Judy Guzman-Cottrill (who also hooked us up with some awesome Gram-positive and Gram-negative tailor-made SHEA Voodoo Donuts), did an outstanding job weaving the three traditional training courses (Healthcare Epi, Antibiotic Stewardship, and IP in Post-Acute and Long-Term Care -- check them out next year if you've never attended!) with many general meeting sessions that included important updates, pro-con debates, and impressive science.  The theme of the general track focused on communication, and the plenary sessions included pragmatic discussions of messaging to diverse stakeholders, to the media (with NPR's Joe Palca), and to government stakeholders through advocacy. 

I'm still unpacking everything I enjoyed about the meeting, but to highlight a few things:

• The meeting size was just right to network with old colleagues well as the growing number of new faces in the field
• The posters were held in a converted parking garage, which felt very old school SHEA ("We don't need no fancy ballroom to learn science!")
• Antibiotic stewardship remains very hot, and it was great to see the innovation and engagement on this very important issue (and the continued presence of our pharmacy colleagues at this meeting).  Also nice to see the emergence of diagnostic stewardship at the meeting.  There were several posters on reducing unnecessary urine cultures and C. difficile testing, for example.
• Not to be outdone by stewardship, infection prevention topics hit on some key challenges, including how to influence and change behavior (including the first? ever pro-con where a vegetable was introduced as a dueling weapon), barriers and innovations to improving use of PPE and hand hygiene, and discussions about risk-adjustment of reported HAI data, the limitations of surveillance definitions, and the possibility or folly of "chasing zero."
• Seeing the finalists for the SHEA Epi Competition expertly field many tough questions about their proposed projects -- and congrats to Valerie Vaughn from Univ. of Michigan on winning the $20,000 award for her project entitled "Antibiotic Overuse at Hospital Discharge."
• Getting to watch Scott's son, Daniel, present  like a pro as probably the youngest SHEA presenter in history (on an impressive look at CRE and transfer networks in Atlanta)

There's so much more to hit on, I invite my co-bloggers to jump in! Suffice to say it was a very successful meeting and a very nice partner to our fall extravaganza, IDWeek (Oct 3-7, 2018 in San Fran - register now!).  Next year, it's in Boston (April 24-26, 2019), so block your schedules!  Perhaps Eli will host a 10th blogiversary celebration?

A Research Agenda for MDRO Prevention

Of course, I don't need to explain the clinical importance of multi-drug resistant bacterial pathogens to readers of this blog. I probably don't need to remind you that "more research is needed" either - that's why we have controversies! But, I should probably point you to five papers recently published in ICHE that outline the future research agenda for MDRO prevention in the US Veterans Health Administration (the VA).

For our non-US readers, the VHA is the largest integrated healthcare system in the United States with over 130 acute care facilities, 1000 outpatient clinics, numerous long-term care facilities and 9 million enrolled patients. The VA has been a leader in medical and health services research for decades and has been well-ahead of the curve in application of interventions to prevent MDRO including its MRSA prevention bundle and antibiotic stewardship initiative.

To continue the VA's success in MDRO prevention and link future research questions to the greatest clinical need, we invited a multidisciplinary group with 37 participants to Iowa City in September 2016. The aim of the panel was to outline the VHA's research agenda for MDRO prevention. Dan Livorsi describes the process we used to identify the domains and research questions in an introductory editorial. The outlined research agenda was broad in scope and included efficacy, effectiveness and implementation questions. In addition, many of these questions are broadly applicable to study in non-VA and non-US hospitals. We are all more alike than different.

Research questions fell into four domains:

1. Transmission dynamics: Resistant pathogens are spread via human hands and environmental surfaces. Disrupting this transmission is essential to controlling MDROs.

2. Antimicrobial stewardship: Strategies to reduce and improve the use of antimicrobials will slow the emergence of resistant pathogens.

3. Microbiome: There may be ways to manipulate or augment the human microbiome to eradicate or prevent colonization with resistant pathogens.

4. Special populations: Strategies need to be tailored to patient populations with distinct underlying conditions and in nontraditional care settings.

All 5 papers are open access. Thanks ICHE!  And thank you to the brilliant group of VA investigators, clinicians and operational partners who traveled to Iowa City and contributed to this effort. We all hope it's helpful.

Remember me if I forget (to blog)

#SHEA2018 has just started in Portland and since it's SHEA Spring, it's our blogiversary! Back in 2009, it all started with this post by Dan. We've been on quite a run - 1788 posts, numerous guest commentaries and a good share of controversy.

But you've probably noticed that we've been a little light on posts since around the end of 2016. This is for a variety of reasons, many of which you might guess - busy schedules, enrolling in degree programs, submitting grants etc. The biggest reason that I've rarely posted is that I deleted my twitter app. Since twitter is where I found many interesting ideas for posts, I've lacked some motivation.

But, 2018 is a new year - our 10th year  - and with new bloggers and light at the end of many tunnels, expect big things. I'll be off to Singapore for a good part of the year to learn about MDRO control there - I can imagine some "virtual postcards" from the Uncle Traveling Matt of the blog.

Finally, I saw the Tedeschi Trucks Band in Davenport last night. Amazing show. And Charlie Parr opened for them. Charlie is an amazing blues guitarist from Minnesota - he's a character in the coolest sense and told great stories. I can't get this song out of my head, so I thought I'd share it with you. Have a great time at SHEA - I can't be there. Please, remember me and remember our blog.

Great news from CDC! Help me to understand it...

Today our colleagues at the CDC’s Division of Healthcare Quality Promotion (DHQP) and the Tennessee and Iowa Departments of Public Health released an interesting MMWR Vital Signs report highlighting CDC’s efforts to control emerging antibiotic resistance (AR) phenotypes (main focus being carbapenem-resistance in Enterobacteriaceae (CRE) and Pseudomonas aeruginosa (CRPA)). The release is accompanied by a helpful summary of CDC’s efforts to better coordinate the detection and control of new and emerging resistance threats (including, of course, not only ESBL and CRE but Candida auris, colistin resistance, glycopeptide-resistance in S. aureus, etc.). Check it out!

My major takeaway: investments in laboratory detection of AR are beginning to pay off. By improving their diagnostic capabilities, more regional and state labs are now able to help health care facilities confirm, characterize and respond to pathogens that display unusual resistance phenotypes. This is unquestionably a good thing, though it still requires local facilities to be able to detect phenotypes of concern—for example, many hospitals don’t have ready access to MALDI-TOF or sequencing, and many do not identify Candida to species level routinely from non-sterile sources. How long might C. auris spread in such a facility prior to its detection? So although the AR Lab Network (ARLN) is a major step forward, we still need to emphasize that individual hospitals must invest in improved and timely AR diagnostics--and we ought to ensure they receive updated guidance about how best to detect new or emerging pathogens of concern.

Now for the data itself: the report presents NHSN CAUTI and CLABSI data from 2006-2015 on the percentage of all isolates of E. coli and K. pneumoniae with ESBL and CRE phenotypes over time. The ESBL phenotype was consistently detected in 16-19% of isolates over time, with no major change noted. However, the % CRE declined steadily, from a peak of 10.6% in 2007 down to 3.1% in 2015 (see Figure below). This is despite the fact that CLSI breakpoints for carbapenem susceptibility were lowered during the surveillance period, which should have caused more isolates to meet phenotypic criteria for resistance. And although the CDC published CRE-specific control guidance in 2009, the decline in %-resistant preceded this guidance (which would obviously not have been immediately implemented across the country, likely delaying any impact for 1-2 years). This also coincides with the spread of carbapenemase-producing Enterobacteriaceae (CPE) across the US. So I’m left wondering: why is the % CRE declining as a cause of CAUTI and CLABSI in acute care hospitals?

 

Is it related to the general improvements in infection prevention practices that have accompanied public reporting and pay-for-performance? Would such improvements differentially impact CRE over other pathogens (keep in mind this isn’t a rate, it’s a proportional decrease)? It’s clear that the epicenter of CRE/CPE seems not be in acute care but rather “post-acute care” as is also noted in the other part of the report (the results of CPE screening by ARLN labs for 9 months of 2017)….but I’d still expect increases in post-acute care or other health care settings to eventually be reflected in the NHSN CAUTI/CLABSI data. Granted, the NHSN data represents a relatively small number of organisms (for example, Clare Rock and colleagues demonstrated that CLABSI represent only 6% of all hospital-onset bacteremias).

I remain perplexed. And happy to hear any thoughts on this!