Wrong answer

This morning I stumbled upon this piece, Wrong Answer (free full text here), by Rachel Aviv in The New Yorker. It's an old article from 2014, but a wonderfully written, compelling, sad tale. It's the story of how high stakes standardized testing of middle school students in economically disadvantaged neighborhoods in Atlanta led to cheating by teachers. It focuses on Damany Lewis, a superb teacher totally committed to his students, who tirelessly worked to improve his students' math knowledge and was successful in doing so, but not successful enough to hit an unreachable goal. Responding to increasing pressure to raise testing scores, he and other teachers began to change the answers on students' tests. We all know that cheating is unethical, and at first glance I bet most of us would argue to punish those involved, but read this entire piece (warning: it's long), and you're likely to soften your stance. The consequences of not meeting unreasonable targets were so severe that the teachers felt compelled to cheat in the best interest of their students.

Now take this article from the education setting into the world of healthcare epidemiology, and if you're like me, there will be chills going down your spine as you read it. It should be required reading for anyone who works in healthcare quality or the key stakeholders in this space, from those at the front lines to those who work in professional societies, and to those who create policy at the state or national level. There are also lessons here for patients and patient advocates.

Donald Campbell

What happened in Atlanta shouldn't surprise us. In 1979, Donald Campbell, a psychologist, published a paper, the crux of which has become known as Campbell's law. I wasn't aware of this until I read Aviv's article. It states: "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." Moving this to our world, you can delete the word "social" in Campbell's law and take a look at Dan Sexton's commentary, Casablanca Redux, from 2012. Here's an excerpt:

Our informal discussions with other hospital epidemiologists, our experience in evaluating the source of infection in hundreds of bacteremic intensive care unit (ICU) patients, and common sense have led us to suspect that many hospitals do not accurately report their true rates of CLABSI, using current NHSN definitions. In some cases this may reflect an unwillingness of local staff to accept these definitions as accurate or fair; in other situations it may reflect an unconscious desire to hedge or reduce their rate of CLABSI to avoid criticism and negative consequences from their local supervisors in the press, clinicians, or the general public who review their publicly reported data... If clinicians inappropriately or illogically fear or anticipate negative feedback about the rate of CLABSI in their institutions, they may consciously or subconsciously fail to obtain blood culture results for every patient with a possible or likely BSI. Simply put: no culture equals no infection, using standard definitions of CLABSI. 

At some level, we are all complicit in this. And depending on the action, it may not be the wrong thing to do. In fact, it may benefit the patient. For example, better diagnostic stewardship in the form of appropriately ordering fewer urine cultures not only lowers CAUTI rates but reduces antibiotic utilization with several resultant benefits. Still it's important to note that the primary impetus for this was to lower HAI rates. We take into consideration how a new diagnostic test may impact HAI rates and may even allow that to impact the decision to implement (see an excellent paper by Dan on this here). We may allow clinicians to censor infections that infection preventionists have detected even though the cases meet NHSN definitions. And at the extreme, hospitals may engage in practices that may harm patients in order to reduce publicly reported HAI rates. In a recent publication on how physicians in training view quality initiatives, a dirty secret was elicited from a resident during a focus group at an academic medial center: “There’s like the central line infection protocols…. If you suspect that anybody has any type of bacteremia, you don’t do a blood culture, you just do a urine culture and pull the lines … we just don’t even test for it because the quality improvement then like marks you off.” 

While reading Rachel Aviv's paper, I wondered: Do we ever ignore results (i.e., infection rates) that seem too good to be true like the educational administrators in Atlanta did? Do we critically analyze surprisingly good results to the same degree as we do surprisingly bad results? In Damany Lewis' case did the end justify the means? Is there ever a situation where I could be pushed to a similar point as Lewis?

The Atlanta school system harmed students and teachers in a thoughtless quest to improve quality. There were no winners. Sadly, the response to the cheating scandal was to raise the stakes for test scores even higher. With pay for performance the same is happening in health care.

The Playing Field is Still Warped . . .

Over the past decade, public reporting of facility-specific HAI performance has had a dramatic impact on infection prevention programs.  The increased awareness of HAI prevention by an increasing diversity of stakeholders (administrators, payors, patients, etc.) has arguably led to increased emphasis on HAI prevention and, in many cases, more resources to infection prevention programs.  These changes were very apparent at my institution where we increased our IP staff from 3 infection preventionists, 2 hospital epidemiologists and one administrative assistant to 9 IPs, 2 data analysts, 1 chart abstractor, 1 program coordinator, and partial support for 4 HEs (not including the antibiotic stewardship program support).  HAI performance is front and center among our annual quality goals, tied to department chair incentives, and the days since the last HAI are posted publically on our inpatient units for all to see.  Most importantly, our frontline healthcare workers understand and routinely discuss what were once surveillance acronyms like "CLABSI" and "CAUTI."  We've seen remarkable reductions in HAIs (and, more importantly, the associated patient harm) during this time (e.g. CLABSI in our ICUs have reduced 80% in the past 8 years). 

Our story is not unique.  Many hospitals have noted marked reductions in HAI rates.  One could argue that many, if not all, of the "low" and even "middle" hanging fruits have been tackled, and we are starting to reach the area where uncontrollable differences in patient risk factors/case mix may lead to different HAI performance across facilities.  With HAI performance tied to increasing financial consequences, however, the need to better insure a level and fair playing field across facilities is growing.  In this context, the recent report from the HHS Office of Inspector General on the CMS HAI reporting is very interesting.  The report focused on the validation of reported HAI data.  They found that while sufficient data were validated (as per regulatory requirements) and 99% of reviewed hospitals passed validation (only 6 failed), concerns were raised regarding how hospitals were selected for validation:

"However, CMS’s approach to selecting hospitals for validation for payment year 2016 made it less likely to identify gaming of quality reporting (i.e., hospitals’ manipulating data to improve their scores). CMS did not include any hospitals in its targeted sample on the basis of their having aberrant data patterns. Targeting hospitals with aberrant patterns for further review could help identify inaccurate reporting and protect the integrity of programs that make quality-based payment adjustments."

Gaming strategies that may be employed include overculturing (to designate an infection as POA), underculturing (if no blood cultures are collected . . . voila! No CLABSIs!), and adjudication/clinician veto ("I know that met the definition for SSI, but it was just a seroma . . . that I treated with antibiotics . . . uh, prophylactically . . . yeah, that's it!).  We have no clue how widespread these practices may be, but the OIG report notes that the current validation strategy should be enhanced to better capture gaming.  With the growing financial consequences placed on HAI prevention, it is paramount that everyone plays fair to better level the playing field.  Now, if we could also get more patient risk factors into the SIR models . . .

Public Reporting - Do we need big brother?

I was fortunate to be an invited speaker at the 2016 ECCMID meeting in Amsterdam last week. My topic was "Monitoring Process of Care: Do We Need Big Brother?" I used the opportunity to take a big picture view of public reporting of HAI and MDRO data in the US and Europe and a closer look at the selection of process versus outcomes measures for reporting. I've posted my slides below and you can also listen to my talk on ECCMID's website. As I believe Dan stated earlier, I hope that all meetings evolve to allow free/open access to presentations like ECCMID has.

What the HAC?!

The University of Wisconsin (UW) Hospital is an excellent institution with a stellar infection prevention program. In fact, last year they won the prestigious U.S. Department of Health and Human Services (HHS) Partnership in Prevention Award, which recognizes “prevention leaders in the U.S…who have achieved wide-scale reduction and progress toward elimination of targeted health care associated infections.” Sadly, this achievement may not be enough to keep HHS from levying financial penalties against UW for high infection rates. 

Yes, the Hospital Acquired Condition (HAC) scores have come out, and have been generating a fair bit of media coverage, focused on those hospitals most likely to face financial penalties. Funny thing, though—the hospitals most likely to lose money under this program share a lot of characteristics:

“who is getting penalized? Large, urban, public, teaching hospitals in the Northeast with lots of poor patients. Who is not getting penalized? Small, rural, for-profit hospitals in the South. Here are the data from the multivariable model: The chances that a large, urban, public, major teaching hospital that has lots of poor patients (i.e. top quartile of DSH Index) will get the HAC penalty? 62%. The chances that a small, rural, for-profit, non-teaching hospital in the south with very few poor patients will get the penalty? 9%.”

Interesting. Explanations for these findings include: (1) small size, rural location, southern region and for-profit status magically translate to higher-quality, safer care, or (2) this HAC metric is bullshit, as it obviously doesn’t adequately control for myriad variables that are associated with the score but that are not indicators of quality and safety. What variables? Intensity and accuracy of surveillance, and variation in infection risk of the different patient populations, for starters.

I can overlook bullshit when it brings more attention (and resources) to the critical task of infection prevention. Unfortunately, this particular form of bullshit does the opposite (unfairly punishing already cash-strapped hospitals with financial penalties). As others have pointed out, the current HAC metric is well-intentioned but obviously flawed, and in desperate need of fixing.  

More on CLABSI validation

A new study in the American Journal of Infection Control looks at validation of publicly reported CLABSI (central line associated bloodstream infection) data in New York. To my knowledge, New York has done the most work on validation, so it's worth taking a look at this paper. The authors analyzed CLABSI data from 2007 through 2010. By year, the sensitivity of reported data varied from 68% to 74%, and for the entire time period was 71%. Of concern, there appeared to be no improvement over time, and for 2010 sensitivity for medical ICUs was only 46%. Specificity ranged from 90 to 99%. So, about 30% of CLABSIs are not reported in a state that has focused on validity. It makes you wonder how bad validity may be in states that have done little work on this. It's interesting that consumer advocacy groups don't seem to be too concerned about the validity issue, and to some degree neither do payers. A consumer advocate once told me, "Just give us all the data. We don't care if it's correct." Caveat emptor!

Graphic: WhyWeSuffer.com

HICPAC lays down the letter of the law

This week's Annals of Internal Medicine has a paper entitled "Public Reporting of Health Care–Associated Surveillance Data: Recommendations From the Healthcare Infection Control Practices Advisory Committee." It's a somewhat misleading title, as the paper isn't about public reporting, it's about what happens in hospitals before HAI data are publicly reported. The primary message is this: if a case meets criteria for HAI sensu stricto, it's an HAI, dammit! Don't go asking a doctor for her clinical opinion on whether it's an infection and then erasing that case from your line list if she says, "doesn't look like an infection to me."

I have mixed feelings about this. On the one hand, in order to allow for valid inter-facility comparisons of infection rates, everyone has to play by the same rules. I think we all get that. But it's incredibly frustrating to review a case that is clearly not an HAI, yet be forced to label it as such. And it's not an uncommon occurrence. Last week, my IPs brought me a case of a patient admitted with pneumonia and because the patient's condition worsened after admission, we were forced to label the case a possible VAP. This morning, an IP from another state emailed me a case of a patient who was admitted with an infected wound, went to the OR, and subsequently developed a surgical site infection, which had to be categorized as an HAI. The IP notes, "the same patient, without the surgery, would have a community acquired wound infection and would not be counted as an HAI."

Even more commonly, hospitals with large oncology populations see many cases of bloodstream infections with enteric flora in patients who just so happen to have a central line. While I give CDC credit for now allowing us to classify cases as mucosal barrier injury related bloodstream infections, it's of little value, as these infections are still publicly reported as CLABSIs. And to add insult to injury, those poor IPs in Pennsylvania have to send patients a letter telling them they suffered an HAI that wasn't an HAI.

All of these problems with post-ascertainment veto and adjudication are occurring because the stakes are high. Most of the time, this is done in good faith, I believe. There is a big push in hospitals to hold staff accountable for adverse events, and it really stings to have the finger pointed at you for an event that was not preventable, or maybe not even an event. It undermines the credibility of IPs and hospital epidemiologists with clinicians when you call a single positive VRE blood culture in a neutropenic leukemic patient a CLABSI. To mitigate that, I find myself appearing at committee meetings to explain that while this case technically meets the criteria for CLABSI, all evidence tells us this is an infection not related to the central line, and is in fact, not preventable. Some hospitals keep two sets of books--the official publicly reported set, and the internally "correct" set.

So while I agree with HICPAC in spirit, this paper only addresses a part of the problem. It ignores the fact that we need definitions with more specificity, and those definitions are needed now. The mucosal barrier injury infection definition has been validated, so let's use it for public reporting. Task IPs to send in descriptions of cases where the definitions are not working, catalog them, categorize them, and start fixing the definitions in a timely manner. Allow for relatively rapid tweaking of definitions instead of acting as if definitions are carved in stone and represent some absolute truth. While we will never be able to have perfect case definitions in the murky world of medicine, fixing the underlying problem to the degree that it can be fixed would decrease the drive for post-ascertainment veto and adjudication. Or how about embracing adjudication but have it occur at a central level? In a world of electronic communication it wouldn't be that difficult.

And one last thing: I think that any official statement from HICPAC should be in the public domain, not behind a journal paywall. Perhaps this paper is also posted somewhere on CDC's website, but I was not able to locate it, if it is indeed there.

Why validation is important

A new study in the American Journal of Infection Control describes the validation of publicly reported central line associated bloodstream infection (CLABSI) data in Colorado. The study encompassed a review of data from ICUs at 35 acute care hospitals and 8 LTACHs for the first quarter of 2010. Charts of 519 patients with positive blood cultures were reviewed. Results of this study, as well as 3 other statewide CLABSI validation projects are summarized below.

	Sensitivity	Specificity
Connecticut	48%	99%
New York	74%	95%
Oregon	72%	99%
Colorado	83%	99%

These 4 studies show that 17-53% of CLABSI cases were misclassified as non-CLABSIs, primarily due to labeling CLABSIs as secondary bacteremias. Interestingly, the Colorado study attempted to delineate the underlying causes of misclassification. Two reasons were found to be significant: lack of an electronic medical record or data mining software, and review of potential cases by an infectious diseases physician (i.e., post-surveillance certification).

Given all the attention paid to publicly reported HAI data and the important implications with regards to reimbursement, it's imperative that hospitals produce valid data. However, validation projects remain infrequent due to their cost. We have a long way to go...

When is a CLABSI not a CLABSI?

We’ve blogged often about problems with how current NHSN definitions are applied in the era of public reporting and "zero" cheerleading. The results of a recent EIN survey are now available online, and confirm that we all ought to take publicly-reported CLABSI data with a few large grains of salt. When presented with a vignette that clearly meets the NHSN CLABSI definition, fewer than half of respondents reported that they would definitely call a CLABSI when the organism is VRE or Klebsiella, with more than a quarter stating they would classify the case as a secondary bacteremia (secondary, presumably, to gastroenteritis in a patient with diarrhea and fecal leukocytes). Most reported that their hospitals used clinical judgment (e.g. consensus panels, clinician vetoes) in their CLABSI surveillance, and 75% of respondents wanted at least some subjectivity to remain in any CLABSI definition. Needless to say, these findings spell trouble for public reporting and a level playing field for inter-hospital comparisons.

Working the refs

As the Miami Heat and Oklahoma City Thunder prepare to do battle in the NBA finals, we should recognize the importance of home court advantage and the pressure placed on referees by coaches and players. There are data that home teams benefit from fewer fouls, and everyone knows coaches and players who are good at “working the refs” to influence their calls. There are also data that as the pressure mounts (game 7 in a 7 game series), referee influence is even greater.

Now, as a mental exercise, picture your infection preventionists (IPs) in referee outfits. It shouldn’t be surprising that as the pressure mounts for hospitals to eliminate reportable HAIs, the refs (those who “make the calls” as to whether an event meets the NHSN criteria for an HAI) are under increasing pressure. We’ll soon publish survey data revealing how commonly hospitals use “consensus methods” (e.g. adjudication panels that include clinical leaders and/or hospital administrators) or even allow clinicians to “veto” HAI calls. These approaches all drive HAI rates lower, as I know of no hospitals where clinicians or hospital administrators bring cases to the IPs to ask why they failed to report them as HAIs. They are also corrosive of the prevention culture, and contribute to IP burnout.

However, even if all hospitals stopped these practices immediately, the increasing pressure to demonstrate HAI elimination would remain a problem. The celebrations units have when they reach “zero” for a period of time, and the massive disappointment when a single VAP or CLABSI ruins the celebration, are akin to the crowds and coaches during game 7, cheering a call for the home team and booing a call for the opposition. And in those centers where financial rewards and penalties accrue to unit directors based upon HAI rates, the pressure is even greater. 

The answer? Eliminate subjectivity in HAI definitions, and move to objective definitions that are amenable to electronic reporting. These HAI events may no longer correlate well with the infectious disease syndromes we diagnose and treat at the bedside, but as long as they are associated with important adverse outcomes (length of stay, mortality) and are preventable, they should suffice (see VAC vs. VAP).

Myopic consequences

The May issue of American Journal of Infection Control has a survey of infection prevention programs in US hospitals assessing the impact of the CMS policy to restrict payment for hospital-acquired central line associated bloodstream infections (CLABSI), catheter associated UTI (CAUTI), and selected surgical site infections. Five hundred hospitals were randomly selected to participate and the response was 64%.

While the survey contained some good news, primarily that hospitals were making efforts to get urinary and central venous catheters removed more quickly, 3 disturbing findings were uncovered:

• 32% of respondents reported a shift in resources to address the policy, as only 15% of hospitals increased resources to infection prevention programs to meet the mandate. While not directly addressed in this paper, the implication is that hospitals stopped performing surveillance for certain infections in order to perform surveillance for CAUTI (more on this below).
• 27% reported that in response to the policy it has become routine practice to obtain urine cultures on patients admitted with urinary catheters in order to determine whether an infection was present on admission
• 13% reported that it has become routine practice to obtain blood cultures on patients admitted with central venous catheters

Obtaining cultures on patients without signs or symptoms of infection is a terrible practice. Any positive results are extremely likely to be contaminants or colonizers, yet it is likely that a sizable fraction of these patients may receive unwarranted antibiotic therapy. This increases the cost of care, adds to the problem of antibiotic resistance, and increases the risk for C. difficile infection.

A companion paper in Medical Care Research and Review by some of the same authors is a qualitative study using semi-structured interviews with 36 infection preventionists to assess the impact of the Medicare policy. In addition to the issues noted in the survey, a few other unintended consequences emerged:

• Providers intentionally not documenting infections in the medical record
• Surgeons refusing to operate on patients at high risk of infection
• IPs were concerned that in order to meet the mandate to perform surveillance for CAUTIs, they stopped surveillance for infections that were deemed to be clinically more important in their hospitals

To those of us who work in the field of infection prevention, none of this is surprising. In fact, we predicted all of these things would happen. It doesn't take a lot of imagination to think through a proposed mandate to envision adverse unintended consequences.

Graphic:  Pie and Coffee

Don't Believe the Pennsylvania Hospital-Acquired Infection Report

When reports include claims that are clearly wrong, I wouldn't trust their entirety.  That's the case with the recently released "Impact of Healthcare-associated Infections in Pennsylvania 2010."  Dan and Mike have every reason to be surprised by the very very low infection rates. I too hope that more details concerning the validation of the HAIs will be provided soon.

However, I don't think we have to wait for more details to see that the report is seriously flawed. The error is so obvious (and so frequently repeated in other reports), that it should almost make us laugh.  The serious error is that the Pennsylvania report attempts to estimate the costs and excess length of stay associated with hospital infections by including the outcomes that manifest BEFORE the infection. 

For a better explanation, here is what we said in the methods of a recent Archives of Internal Medicine paper: "Longer hospital stays and higher costs associated with HAI cases may, in part, be due to extended preinfection hospital exposure. Because extended LOS is an independent risk factor for infection, the preinfection LOS of patients with HAIs may be expected to exceed that of similar patients who did not acquire an HAI. Attributing preinfection LOS to HAIs would overstate the true costs of HAIs...A study of 490 nosocomial sepsis cases from 8 tertiary care centers found that the mean preonset LOS was approximately 40% of the total LOS for these hospitalizations." 

To quote from the PA reports methods: "The average payment reported is for the entire length of stay, and not just for the treatment related to the infection."  It's like they are FLAUNTING THEIR ERROR. When the reports states "the estimated average Medicare fee-for-service payment for hospital stays for patients who acquired an HAI was $21,378...(and) the estimated average Medicare fee-for-service payment for those without an HAI was $6,709", we know we can't trust the estimates.  In fact, if we can't trust those, why should we trust any of it?

My copy just went into the trash.

Attention, Pennsylvania IPs and hospital epidemiologists!

I share Mike’s amazement at the low overall prevalence of healthcare-associated infections (HAIs) in recent reports from Pennsylvania (1% of all admitted patients for the years 2009 and 2010). Especially since the 5-10% estimates aren’t just from older studies. Recent studies-- in fact, virtually every study performed outside of high-stakes public reporting mandates--report similar estimates (see this WHO summary, pages 12-15, from which the I pasted part of a Figure to the right, with PA added in red). As Mike points out, an amazing achievement.

But I have to stay true to the name of this blog, so for the sake of argument I will now assume a highly impolite and cynical stance regarding this report. Here goes: I don’t believe it.

I think we have reason to assume that a healthy proportion of CLABSIs are “adjudicated away” by many centers, and lord only knows how many other HAIs (VAP, anyone?) vanish into the ether of consensus review, clinician veto, administrative adjustment, or just plain old bad surveillance. Recall also the SHEA 2011 presentation reporting evidence for gaming in the earlier PA data (using methods that can also be criticized, but the “difference-in-differences” analysis was persuasive enough for me to accept that there was both a real reduction in HAI rates, and evidence for gaming, since the PA public reporting mandate).

Now that I’ve thrown down the gauntlet, Pennsylvania, please convince me that these data are accurate. The report references validation—how was it done? Was your own hospital subject to audit of HAI rates? How many records were reviewed, if chart review was performed? What were the credentials of the auditors? As a “consumer” of health care, would you base your decisions about where to receive care on facility level data from your state reporting program?

Pretty damn good!

The 2010 report on healthcare associated infections in Pennsylvania was recently released. Since Pennsylvania mandates the reporting of all HAIs, this probably represents the most robust data that currently exist. The most important statistic in the report, I think, is that 1.1% of patients admitted to acute care hospitals developed an infection. That shows tremendous progress since previous studies estimated that 5-10% of patients developed nosocomial infections. Of the infections reported in Pennsylvania, over half were surgical site infections, 20% were urinary tract infections, and bloodstream infections accounted for approximately 10%.

Graphic:  Leica News

Sunshinism: Protecting Patients or Further Destroying Physician Trust?

Yesterday, Dan posted about the new rules mandating the reporting of all physician payments from drug and device manufacturers. Generally, I've been in favor of increased transparency if, as Dan said, the "information is detailed and accurate." However, I think any system like this can and will be gamed and I suspect the results will be more destructive than constructive. For example, funding for "research" is thought to be less conflicting than direct payments to physicians for giving canned talks.  However, what about "research" support that pays for each patient enrolled and what if that payment goes directly to the enrolling physicians pocket, as would be the case in private practice? Is this such a bad thing that it needs to be constrained?

I suspect there are many other examples of how we won't be able to interpret the reports that are generated from these new rules.  I even suspect that the eventual approach to determining financial conflicts will be through opening up every physician's tax return.  That way, we can look at the true financial impact to the individual.  Pharmaceutical research that goes to a university and doesn't directly increase a physician's deans-approved salary would thus not appear on a tax return.  What about physicians that own stock in Pharma?  Wouldn't that be a more important conflict? You can see where this is going.  So someday soon, all physicians will have to share their tax returns with their patients perhaps by posting them in their waiting rooms or websites.

However, do you really think this will help root out conflicts?  What are the negative externalities of such an approach?  I suspect it will root out the caring physicians who don't want to appear to be in the pocket of pharma even if they are involved in highly important clinical studies. Which gets me to why I've been moved to write this post...

In today's NYT, David Brooks and Gail Collins debate the call to release Mitt Romney's tax return.  In the column, Brooks makes some important points, which I think are worth at least pondering in regards to the new payment disclosure rules and other examples of "sunshinism."

Brooks: "...there is a misbegotten ideology haunting the land, the ideology of sunshinism. This is the belief that everything should be made public. Sunshinism is a destructive ideology. Forcing people to financially undress in public is just one of those incursions that repels decent people..."

Could these new rules further mistrust of the medical community?  Is society better off when a patient doesn't want see an ID doc because she made $5000 enrolling patients in a trial of a new antibiotic? If she won't enroll patients, who will?

Which is a greater conflict for a physician? (a) $50,000 investigator initiated grant to a University (b) $5000 direct payment for giving a canned talk (c) $5000 for enrolling patients in a trial or (d) $50,000 stock in a pharmaceutical company that won't be disclosed under the new rules?

OK, so I think I've built a solid enough straw man.

Central line + positive blood culture = CLABSI (not!)

There's a thoughtful commentary in a recent issue of Clinical Infectious Diseases by Tom Fraser and Steve Gordon at Cleveland Clinic on problems related to CDC's central line associated bloodstream infection (CLABSI) case definition. We've blogged about this before. The definition is old and was designed to maximize sensitivity long before anyone thought about public reporting. But the issues of poor specificity of this definition are haunting many of us, particularly those who work at hospitals with cancer centers. Unfortunately, neutropenic cancer patients not uncommonly have translocation of enteric flora across their intestinal mucosa and the resulting bloodstream infection in the presence of a central line forces us to label these as CLABSIs, even though these infections are not at all related to the central line. Ten years ago no one cared about this surveillance technicality. Now, in the era of public reporting this is a big problem. In fact, nearly every "CLABSI" in the medical ICU of my hospital falls into this category. Fraser and Gordon show us how this is handled at their hospital with a modification to the CDC definition that is used for internal purposes. Hopefully relief is on the way. CDC is very interested in this issue and has assembled a committee that is actively evaluating the issue.

SHEA's Consumer Retort

Kudos to SHEA for publishing a statement on Consumer Reports' article on teaching hospitals and hospital quailty ratings, which include infection prevention metrics. Here's the money quote:

Buried in the methodology, the publisher of Consumer Reports agrees that comparisons must be done carefully, but the article does not reflect this caution. Instead, the article draws broad conclusions about the quality and safety of care throughout entire health systems based on one measurement gathered from a single unit in each hospital.

While Consumer Reports may know how to evaluate refrigerators, they have a long way to go in order to produce a high quality assessment of health care.

Surveillance bias and public reporting

This week’s JAMA has a commentary from Haut and Pronovost that’s worth a read, on the topic of surveillance bias. This is Epidemiology 101 for those of us who live and breathe surveillance and prevention, but unfortunately it is not very well understood by patients and public policy makers. So when data like this garbage dump from Consumers Reports are released and generate media attention, more harm than good result. Good hospitals are unfairly maligned, and undoubtedly some hospitals that game their rates or just perform poor surveillance are rewarded. Call me a stupid consumer, but if I require ICU care and have to choose between Vanderbilt, Virginia Commonwealth, University of Maryland and one of these hospitals, I’m choosing one of the first three. Now I'm sure this list includes excellent hospitals, but I have no (zero) confidence in the data presented (i.e. I'm 'getting to zero' confidence).

We’ve blogged plenty about the challenges of public reporting and establishing a level playing field, so I’ll refer readers to these prior posts. As Haut and Pronovost point out, to ignore standardized, accurate and fair measurement (to include external validation) is both “reckless and unjust”. It is also true that “to be done appropriately, quality measurement is expensive”. Cheap shortcuts, like using ICD-9 coding data, simply prove the maxim that ‘you get what you pay for’.

Maryland report on hospital associated complications: Don't waste your time reading it

This week, NPR and the Washington Post ran stories on a new report on healthcare associated complications in Maryland. The report can be viewed here. Based on the results of the report, nine hospitals are required to pay penalties due to higher than average rates of complications. Eleven of the report's 49 indicators are infectious complications, such as infection related to central venous catheters. However, what is most important to know is that the source of the data for the report is administrative claims (ICD-9 codes which were developed for billing purposes). We've blogged before about how notoriously inaccurate these codes are for determining whether patients experienced healthcare associated infections. This is because case ascertainment is performed by abstractors with little medical training using case definitions that were not designed for surveillance purposes. Last year, Pennsylvania, the state with the most comprehensive mandatory reporting program for healthcare associated infections in the country, abandoned use of administrative claims data and required that all hospitals use CDC surveillance methodology. Particularly when hospitals are going to be punished by fines and bad publicity, valid methods must be used. I noted in the report that there were 431 cases of "moderate infectious," at a cost of over $6 million. What in the world is "moderate infectious"? I don't think you could find an infectious disease doctor anywhere who could tell you what that is because there is no such thing. Those of us who work in hospital epidemiology understand the need for public reporting because our society values transparency and accountability. We get it. But public reporting is a two-way street that requires a commitment on the part of public agencies to insure that the data generated are obtained via state-of-the-art methods and risk adjusted in order to produce the most valid reports for the public. In other words, it's about playing fairly.

Learning to count...

There's an old joke about asking an accountant "what's 2+2?" and the accountant responds "what do you want it to be?"

Unfortunately, CDC has some creative math rules of its own. So for hospital epidemiologists, 1 + x = 1 when it comes to counting central line days. That is, for patients who have more than one central line, only one line can be counted per day for the denominator in calculation of central line associated bloodstream infection (CLABSI) rates. It's as if only one of the three central lines in the acutely ill ICU patient poses a risk to the patient. Magically, the other two are immune.

In the February issue of Infection Control and Hospital Epidemiology, there is a study by the Hopkins group that examined the effect of the one-catheter-per-day rule and found that counting only one catheter falsely overestimated their CLABSI rate by 36% in 3 surgical ICUs.

Two years ago, our group presented a very similar study at SHEA done in our medical and surgical trauma ICUs. We found that the CDC rule falsely overestimated our CLABSI rate by 20%.

In the era of mandatory public reporting of HAIs, it's imperative that everything be done to produce the most valid data for consumers. I'm baffled that CDC has been so slow to respond to these issues. The focus seems to be on validating surveillance using the methodology as is, rather than modifying the methodology to make it more valid.

California's Healthcare Associated Infections Report: For what it's worth...

California has just released its first statewide report on healthcare associated infections (you can view it here). The metrics reported are healthcare associated VRE bloodstream infections per 1,000 inpatient days, healthcare associated MRSA bloodstream infections per 1,000 inpatient days, and CLABSI in ICUs per 1,000 central line days. The report has major problems as evidenced by the disclaimer on every table of rates that says that the data should not be compared between hospitals, which is generally the whole purpose of public reporting. However, since the reporting period for this report ended, the state mandated that all hospitals join NHSN, which they anticipate will improve the quality of the data reported.