Public Reporting - Do we need big brother?

I was fortunate to be an invited speaker at the 2016 ECCMID meeting in Amsterdam last week. My topic was "Monitoring Process of Care: Do We Need Big Brother?" I used the opportunity to take a big picture view of public reporting of HAI and MDRO data in the US and Europe and a closer look at the selection of process versus outcomes measures for reporting. I've posted my slides below and you can also listen to my talk on ECCMID's website. As I believe Dan stated earlier, I hope that all meetings evolve to allow free/open access to presentations like ECCMID has.

Clamoring for CLAMBI

Last month we blogged on updated NHSN surveillance definitions and we bemoaned the fact that CLAMBIs (central line associated mucosal barrier injury bloodstream infections) are not being separated from CLABSIs for public reporting, and more importantly for the CMS pay-for-performance programs. These infections are particularly common in patients with hematologic malignancies, are due to the translocation of enteric flora into the bloodstream, and unlike true CLABSIs are not preventable. A new paper in Infection Control and Hospital Epidemiology from Northwestern University demonstrates why this is important.

All cases of CLABSI were identified over a 14-month period on 2 inpatient hem/onc/BMT units (72 beds). The cases were further subdivided into "true" CLABSIs (i.e., not associated with mucosal barrier injury) and CLAMBIs. A total of 66 infections were identified, of which 47 (71%) were CLAMBIs. E. coli, enterococci and viridans streptococci accounted for 62% of the pathogens isolated.

The authors note that at the present time CLABSIs identified outside of ICUs are not publicly reported nationally; however, the CLAMBI patients spillover into ICUs. At Northwestern, 12% of ICU CLABSIs were determined to actually be CLAMBIs.

Is it any wonder that tertiary care hospitals are disproportionately affected by CMS penalties? This is just one of many reasons.

Montgomery County is now in the loop!

The next time a contagion sweeps through the NIH Clinical Center, Montgomery County officials will be on it, thanks to a new agreement between NIH, Maryland and Montgomery County. The back-story is that Montgomery County officials were unhappy that they weren’t informed promptly about the deadly KPC outbreak at NIH.

This raises the question of when a hospital should communicate with public health officials (and the public generally) about fairly common SNAFUs. At any given time, 5-10% of hospitals are dealing with clusters or outbreaks of multiple-drug resistant gram negative bacteria (KPCs, ESBLs, MDR-Acinetobacter, etc.), and even more are in the midst of MRSA, VRE, fungal or other outbreaks. The population at risk during these outbreaks is pretty clearly defined, and doesn’t include the general public. General notification can generate media frenzy, free-floating panic and anxiety, and waste precious time and resources for the personnel trying to contain the outbreak (responding to media, doing damage control of various types, etc.). Furthermore, most states don’t include common healthcare associated bacterial pathogens among their legally reportable diseases.

However, as state and local public health officials become increasingly involved in HAI issues, it would be wise to establish explicit criteria for when healthcare facilities should report clusters and outbreaks. Provided they have sufficient funding (which they currently do not!), public health departments should play a critical role in coordinating responses to HAI outbreaks, which often involve multiple healthcare facilities in a region (across the spectrum of acute, long-term, and long-term acute care).

So when do you think a hospital should notify their state and/or local public health department? Two cases of MRSA infection in the NICU? A single serious post-operative Group A strep infection? New introduction of a carbapenemase into the ICU?

Oh, and Happy Thanksgiving!

Don't Believe the Pennsylvania Hospital-Acquired Infection Report

When reports include claims that are clearly wrong, I wouldn't trust their entirety.  That's the case with the recently released "Impact of Healthcare-associated Infections in Pennsylvania 2010."  Dan and Mike have every reason to be surprised by the very very low infection rates. I too hope that more details concerning the validation of the HAIs will be provided soon.

However, I don't think we have to wait for more details to see that the report is seriously flawed. The error is so obvious (and so frequently repeated in other reports), that it should almost make us laugh.  The serious error is that the Pennsylvania report attempts to estimate the costs and excess length of stay associated with hospital infections by including the outcomes that manifest BEFORE the infection. 

For a better explanation, here is what we said in the methods of a recent Archives of Internal Medicine paper: "Longer hospital stays and higher costs associated with HAI cases may, in part, be due to extended preinfection hospital exposure. Because extended LOS is an independent risk factor for infection, the preinfection LOS of patients with HAIs may be expected to exceed that of similar patients who did not acquire an HAI. Attributing preinfection LOS to HAIs would overstate the true costs of HAIs...A study of 490 nosocomial sepsis cases from 8 tertiary care centers found that the mean preonset LOS was approximately 40% of the total LOS for these hospitalizations." 

To quote from the PA reports methods: "The average payment reported is for the entire length of stay, and not just for the treatment related to the infection."  It's like they are FLAUNTING THEIR ERROR. When the reports states "the estimated average Medicare fee-for-service payment for hospital stays for patients who acquired an HAI was $21,378...(and) the estimated average Medicare fee-for-service payment for those without an HAI was $6,709", we know we can't trust the estimates.  In fact, if we can't trust those, why should we trust any of it?

My copy just went into the trash.

High stakes and low rates

The recent scandal surrounding a college that submitted inflated SAT scores to improve their ranking in the US News & World Report should remind us of a simple fact—the higher the stakes, the more likely cheating will occur.

Now that CMS is publishing ICU CLABSI data on their hospital compare website, and using the data in payment formulas, the stakes for hospitals could hardly be higher. And available data suggests that many hospitals stray from strict application of NHSN definitions, reporting misleadingly low CLABSI rates. We’ll soon be publishing results from a survey we did of hundreds of infectious diseases clinicians involved in CLABSI reporting at their institutions. Given patient scenarios that clearly met the NHSN definition for primary CLABSI, what percent do you think responded that they would report them as primary CLABSIs? I can’t go into detail given that this is as-yet-unpublished data, but I suspect the results will surprise even those most critical of publicly-reported HAI data. So I’m dismayed when I read quotes like “several hospitals that serve the sickest patients have been able to achieve infection rates of zero for several years”. The statement is obviously false—but even if it were true, it simply means that the hospital isn’t applying NHSN definitions correctly.

Sunshinism: Protecting Patients or Further Destroying Physician Trust?

Yesterday, Dan posted about the new rules mandating the reporting of all physician payments from drug and device manufacturers. Generally, I've been in favor of increased transparency if, as Dan said, the "information is detailed and accurate." However, I think any system like this can and will be gamed and I suspect the results will be more destructive than constructive. For example, funding for "research" is thought to be less conflicting than direct payments to physicians for giving canned talks.  However, what about "research" support that pays for each patient enrolled and what if that payment goes directly to the enrolling physicians pocket, as would be the case in private practice? Is this such a bad thing that it needs to be constrained?

I suspect there are many other examples of how we won't be able to interpret the reports that are generated from these new rules.  I even suspect that the eventual approach to determining financial conflicts will be through opening up every physician's tax return.  That way, we can look at the true financial impact to the individual.  Pharmaceutical research that goes to a university and doesn't directly increase a physician's deans-approved salary would thus not appear on a tax return.  What about physicians that own stock in Pharma?  Wouldn't that be a more important conflict? You can see where this is going.  So someday soon, all physicians will have to share their tax returns with their patients perhaps by posting them in their waiting rooms or websites.

However, do you really think this will help root out conflicts?  What are the negative externalities of such an approach?  I suspect it will root out the caring physicians who don't want to appear to be in the pocket of pharma even if they are involved in highly important clinical studies. Which gets me to why I've been moved to write this post...

In today's NYT, David Brooks and Gail Collins debate the call to release Mitt Romney's tax return.  In the column, Brooks makes some important points, which I think are worth at least pondering in regards to the new payment disclosure rules and other examples of "sunshinism."

Brooks: "...there is a misbegotten ideology haunting the land, the ideology of sunshinism. This is the belief that everything should be made public. Sunshinism is a destructive ideology. Forcing people to financially undress in public is just one of those incursions that repels decent people..."

Could these new rules further mistrust of the medical community?  Is society better off when a patient doesn't want see an ID doc because she made $5000 enrolling patients in a trial of a new antibiotic? If she won't enroll patients, who will?

Which is a greater conflict for a physician? (a) $50,000 investigator initiated grant to a University (b) $5000 direct payment for giving a canned talk (c) $5000 for enrolling patients in a trial or (d) $50,000 stock in a pharmaceutical company that won't be disclosed under the new rules?

OK, so I think I've built a solid enough straw man.

Bundle the baby

There's a new paper in Pediatrics that evaluates implementation of central line insertion and maintenance bundles across all referral NICUs (n=18) in New York state. Surveillance for infections followed NHSN methodology and evaluated a 12-month period prior to implementation of the bundles to a 10-month period after the bundles were implemented. Overall, there was a 40% reduction in CLABSI. Higher volume NICUs demonstrated lower infection rates and less variation in performance. For each standard deviation increase in maintenance checklist usage, there was a 16.5% decrease in CLABSI rate. However, the authors point out that "in light of some agencies’ considering CLABSIs to be “never events,” it is important to note that no NICU achieved an overall CLABSI rate of 0."

California's Healthcare Associated Infections Report: For what it's worth...

California has just released its first statewide report on healthcare associated infections (you can view it here). The metrics reported are healthcare associated VRE bloodstream infections per 1,000 inpatient days, healthcare associated MRSA bloodstream infections per 1,000 inpatient days, and CLABSI in ICUs per 1,000 central line days. The report has major problems as evidenced by the disclaimer on every table of rates that says that the data should not be compared between hospitals, which is generally the whole purpose of public reporting. However, since the reporting period for this report ended, the state mandated that all hospitals join NHSN, which they anticipate will improve the quality of the data reported.

Public Reporting of CLABSI: Is it a valid measure for hospital comparison?

Overview of the computerized CLABSI agorithm

In today's JAMA, Michael Lin and authors from four CDC Epicenter academic hospitals (2 in Chicago, 1 in Columbus, OH and 1 in St. Louis) compared annual IP-determined CLABSI rates in 20 ICUs during 2004-2007 with a computer-generated reference standard. The median CLABSI rate was 3.3/1000 central line days. Overall correlation with the computer algorithm was poor at 0.34 and ranged from 0.83 in one center down to 0.10 at another. Interestingly, the center with the lowest reported CLABSI rate by IP had the highest computer rate. (2.4/1000 CL-days vs. 12.6/1000 CL-days)

I have posted the schematic of the computer algorithm and also the link to the code (below).  My only methods question (at this moment) is why did they limited the analysis to yearly comparisons and not quarterly or monthly comparisons.  I would have liked to see that level of data analyzed even though it would be noisier.  It was interesting how the IP-reported rates were narrowly clustered around each other while the computer-generated rates were widely distributed.  The findings should make us pause when we consider public reporting of these rates.  If so much emphasis is being placed on CLABSI rates at the state and national level for comparison and reimbursement, there should be funded validation of the reported rates and also consideration of other measures (outcome or process) that might be more reliable. 

Lin et al JAMA November 2010
Link to computer algorithm code (looks like you might need to apply for a password)

The “vicious cycle of pseudoimprovement”

What am I talking about? Read this JAMA commentary from Drs. Muller and Detsky to find out.

Done reading? Now ask yourself these questions: is my hospital following indicator-based or evidence-based strategies to improve patient outcomes? What happens when publicly-reported indicators improve dramatically, but patient outcomes do not?

17 states report CLABSI rates. Why only 17?

The CDC just released the First State-Specific Healthcare-Associated Infections Summary Data Report, which focuses on CLABSI. HHS press release is available here. Needless to say, we are all disappointed with the rates here in Maryland given how hard the State and hospitals have worked at preventing these infections. No one seems more disappointed than Peter Pronovost. No excuses.

Did someone get fired for using NHSN or NNIS definitions to count CLABSIs?

Wow. I just read the second article that Mike posted Saturday out of the Chicago Tribune and I kinda wish I didn't. I'll paste the section that scared me a bit below, but it appears that someone might have been fired for "over-reporting HAIs", which is a bit scary to me. There is tremendous pressure to not call a BSI , a CLABSI, which I think is now part of the getting to zero culture. I wonder what percent of the way towards zero will be paved with these sorts of statements:

"Thorek's infection rate was the highest of all medical centers in Illinois. Frank Solare, Thorek's president and chief executive officer, said hospital officials have collected medical charts for the 22 infected patients and have "started an independent review … to try and understand this."

Asked why the Lakeview hospital didn't take action last year, Thorek's compliance officer Morgan Murphy said a former employee didn't alert senior management to the problem. "It wasn't making its way up the chain, unfortunately," he said.

Senior management also suspects that the employee may have counted central line infections incorrectly, inflating the hospital's numbers. "There may have been over-reporting," Murphy said."

VAP: Do you know it when you see it? (Again!)

There's a new paper on the utility (or lack thereof) of CDC's definition of ventilator-associated pneumonia. In this study 4 persons reviewed 50 cases of ventilated patients with respiratory deterioration >48 hours after intubation. Two reviewers were experienced infection preventionists who applied the CDC definition. A third IP used a modification of the CDC definition that was more quantitative, and the fourth reviewer was a physician board-certified in infectious diseases and critical care who used clinical judgment to define VAP. Using the standard definition, one IP assigned the VAP diagnosis to 11 patients and the other, 20 patients. The IP using the modified definition assigned 15 cases as VAP. The physician diagnosed VAP in 7 patients. The IPs agreed on 62% of cases (kappa=0.40). All 4 reviewers agreed on the VAP diagnosis in only 4 cases. This is not the first study to show how complex assigning the diagnosis of VAP can be.

Given that public reporting has raised the stakes to high levels, the CDC can no longer ignore this issue. Consumers cannot make choices on where to receive care if inter-hospital comparisons of infection rates are not valid. There needs to be a convening of IPs and hospital epidemiologists who use the definitions on a daily basis to thoroughly assess each of the HAI case definitions and begin to work on the development of new ones that will be fair to hospitals and helpful to consumers. Otherwise, it's garbage in, garbage out, and the entire concept of public reporting is undermined.

Hand hygiene compliance: really important but impossible to measure

This morning I ran across an article in a Canadian newspaper, The London Free Press, that listed hand hygiene compliance rates for local hospitals. This piqued my interest. While nearly all hospitals in the US attempt to measure hand hygiene compliance, driven in large part by The Joint Commission's National Patient Safety Goals (NPSG 07.07.01 in TJC-speak), I am not aware of public reporting of hand hygiene rates in the US.

For public reporting, hand hygiene compliance is a double-edged sword. Given that hand hygiene is probably the most important measure to reduce healthcare associated infections, measuring it and publicly reporting it should be a high priority. On the other hand (no pun intended), it remains largely unmeasurable. I was interested to know how many observations were performed in the hospitals reported in the London newspaper since the denominators (number of opportunities observed) were not reported in the article. So I went to the Ontario Ministry of Health Patient Safety website, which has several publicly reported metrics for their hospitals (including central line associated bloodstream infection, ventilator associated pneumonia, and surgical site infection rates). There I learned that for hospitals with 100 beds, a minimum of 200 observations is required yearly.

So let's play with some numbers. Let's say that a 500-bed hospital performed 1,000 hand hygiene observations over the course of a year. We'll assume that the mean nurse:patient ratio across the hospital is 1:5, and that on average each nurse has 10 opportunities per hour for hand hygiene. If you do the math, you'll find that for nurses only, there are roughly 9 million hand hygiene opportunities per year in this hypothetical hospital. Measuring 1,000 of those opportunities accounts for one-hundredth of one percent. My conclusion: the hand hygiene compliance rates reported are meaningless. And my advice to the savvy Canadian consumer is to focus on the outcomes measures (infection rates) that are reported.

Unfortunately, at the moment, there's no valid way to measure hand hygiene compliance. Elaine Larson wrote a great review of the literature on methodologies for measurement a few years ago in the Journal of Hospital Infection. I still think it's useful to measure hand hygiene compliance, and the more opportunities observed, the better. And feedback of the data over and over in different ways is key to reinforce the importance of hand hygiene to healthcare workers.

Long story short: inside the hospital, hand hygiene compliance rates have utility. Outside, not so much.

Seek and ye shall find....

I found this article from today’s NY Times to be very interesting, both as a critique of our ability to detect foodborne illness, and as a corollary to problems inherent in the detection and reporting of healthcare-associated infections (HAIs).

The gist of the piece is that when a state has an excellent foodborne illness surveillance and investigation system (e.g. Minnesota), that state will find a disproportionate number of foodborne illnesses and outbreaks (now you might be saying, “Duh!”).  But if one looks at the foodborne disease data without any knowledge of disparities in quality of surveillance, detection and investigation, Minnesota looks like a cesspool of foodborne disease!  In fact, Minnesota may be the safest place in the U.S. to eat, because they are more likely than any other state to quickly detect and investigate foodborne disease outbreaks.

Similarly, those hospitals that do the best surveillance are more likely to detect HAIs when they occur.  As just one example, the best way to avoid detecting Legionella infections is to fail to test for Legionella among patients with healthcare-associated pneumonia.  Because Legionella won’t grow on the routine culture media used for respiratory specimens, a concerted effort must be made to request specific testing (using enriched selective media or the urinary antigen test).

Other examples are more nuanced, but suffice to say there are also many ways to fail to detect ventilator associated pneumonias, catheter-associated bloodstream infections, and surgical site infections.

How to address this problem?  One way is to perform external validations (or audits) of each hospital’s surveillance data.  For example, the CDC is working with states to develop validation methods for data submitted to the National Healthcare Safety Network (NHSN), which is extremely important since states with mandatory reporting legislation usually utilize NHSN as the reporting mechanism.  But such validation is complicated, time-consuming and expensive. 

An alternative approach would be to directly monitor data from microbiology laboratories or pharmacies, to correlate with (and validate) reported infection rates—or even as a surrogate for those rates.  For example, if a hospital reported zero nosocomial bloodstream infections but exceeded a certain rate of blood culture positivity (for cultures drawn > 48 hours after hospital admission), that might trigger further investigation…..similarly, parameters could be established to compare reported SSI rates with post-operative antimicrobial use data.

There is no simple and inexpensive solution, but addressing this problem will become increasingly important as HAI rates become public across the country.

More on Public Reporting

A recent study in Health Affairs compared evaluations of 9 hospitals in the Boston area by 5 different web sites designed for consumers to assess quality of care. The web sites evaluated were Hospital Compare, Health Grades, Leapfrog, US News and World Report, and Massachusetts Healthcare Quality and Cost. The study investigators compared the quality ratings by each of these groups for 4 conditions (community acquired pneumonia, total hip replacement, percutaneous coronary intervention, and coronary artery bypass grafting). Interestingly, the quality assessments across the web sites were highly variable. In some cases, a hospital that was rated at the highest level of performance by one web site for a particular procedure or diagnosis was rated at the lowest level of performance by another web site. The bottom line is that until public reporting of healthcare quality data can be appropriately risk adjusted and validated, it is likely to be of little help to consumers.