Metrics, Decision Makers and Hospital Epidemiologists

This is a guest post by Silvia Munoz-Price, MD, PhD, Enterprise Epidemiologist and Professor of Medicine, Division of Infectious Diseases, Froedtert and the Medical College of Wisconsin.

During the past few weeks I have been mulling over two issues related to Quality and Infection Control that have to do with metrics, their use, and the impact that they have in our hospitals. Two separate topics but related to a certain degree.

1. My hospital has a strong Quality Department and a stronger culture of safety. For the most part we have very few serious infections and even fewer MRSA, VRE, and almost no multidrug resistant Gram negative rods. Our compliance with hand hygiene is 73% (as per >5000 observations collected over a couple of months by managers from units that are not their own and concordant readings obtained by the Infection Control Department. I am trying to tell you that 73% is probably close to accurate). From my perspective, if indeed true, this is an acceptable rate. I think we should concentrate on preserving this rate and maybe focus on areas/providers that need more attention. However, my hospital wants this number to be close to 100%. So, here are my questions: should we invest more time, effort, and money to push for higher compliance with hand hygiene? What would be the return on investment of increasing our hand hygiene compliance and how could we measure its impact? What type of infections would we prevent by increasing hand hygiene compliance? Yes, our CLABSIs and CAUTI rates could be better, but would they noticeably improve by pushing hand hygiene above 80%? (I’m intentionally not discussing C. difficile infections as these have a different etiology in my hospital). And most importantly what other interventions would we be overlooking by focusing so much on hand hygiene? Who decides where to allocate time, effort and money?

2. Over the past couple of decades I have observed a shift in Infection Control throughout the various places I have worked at from minimal interest/resources given to Infection Control, to a phase where more visibility was given to Infection Control matters --probably due to metrics--to the allocation of more interest/resources, to publicly reported metrics, to institutional notoriety and reimbursements linked to metrics, to a tunneled vision about metrics. On this topic, many years ago I observed a human behavior that since I have observed multiple times: when we started placing fluorescent powder on surfaces and giving feedback to providers on the degree of removal, cleaning rates improved (https://www.ncbi.nlm.nih.gov/pubmed/21460514). Feedback was then given weekly and in a public manner, with a substantial amount of pressure placed on EVS to improve. The numbers got much better. We eventually figured out that EVS providers bought their own ultraviolet lamps to find the fluorescent markers and spot clean them. What happened? The metric made sense at the beginning, initially it probably achieved its intended outcome (more frequent cleaning) but then a threshold was crossed, after which what mattered was the metric (as a number) and no longer the intended outcome (more frequent cleaning).  I think the same is happening to our hospitals. Metrics were beneficial initially (more resources for Infection Control, more visibility, more engagement of staff, lower infections) but I would argue that we are on the other phase of the metrics (unintended consequences).  All the pressure we are placing for lower and lower metrics (get to zero CAUTIs!) might be backfiring. We are deploying large teams and resources to decrease certain conditions which is shifting our capability to address issues that might be equally or even more important in our respective hospitals   (e.g. post craniotomy  SSIs).  So, who decides how to allocate our limited resources? I would argue that Quality is now the main decision maker and that these decisions are strongly guided by publicly reported metrics. Is this good or bad for our patients? I am not sure. Is this good for the field of Hospital Epidemiology? I don't think it is. Hospital Epidemiologists need to re-think their involvement in Quality and become their hospitals’ Chief Quality Officers. That is the only way I see that we will impart some sense to the decisions of resource allocation. However, the opportunity for our specialty to lead in this field might have already passed.     

Measurement fatigue: The backlash

Anyone who’s been in the hospital infection prevention business for any length of time is familiar with a specific form of cognitive dissonance. We believe, on the one hand, that the publicly-reported, metric-focused, pay-for-performance (PFP) environment has brought increased resources to infection prevention and resulted in a real decrease in healthcare-associated infections (HAIs); yet we believe, on the other hand, that these high stakes have led to a number of unintended adverse consequences, including gaming of HAI definitions, an unhealthy focus on measures that may not merit the resources and attention, and have engendered cynicism as it becomes apparent that PFP measures may not correlate with actual quality or value.

Over the past week, I’ve read three pieces that make me wonder if we’re reaching a tipping point, as clinicians begin to push back effectively against the proliferation of “measures”, “metrics”, “performance targets” (whatever you wish to call them), in an attempt to seek a balance between them and the words of Francis Peabody, that “the secret of the care of the patient is in caring for the patient.” I’ve pasted some key quotes below from each of these pieces.

First, an excellent opinion piece from Dr. Robert Wachter in the New York Times:

"All of this began innocently enough. But the measurement fad has spun out of control. There are so many different hospital ratings that more than 1,600 medical centers can now lay claim to being included on a “top 100,” “honor roll,” grade “A” or “best” hospitals list. Burnout rates for doctors top 50 percent, far higher than other professions. A 2013 study found that the electronic health record was a dominant culprit. Another 2013 study found that emergency room doctors clicked a mouse 4,000 times during a 10-hour shift. The computer systems have become the dark force behind quality measures….

....Our businesslike efforts to measure and improve quality are now blocking the altruism, indeed the love, that motivates people to enter the helping professions. While we’re figuring out how to get better, we need to tread more lightly in assessing the work of the professionals who practice in our most human and sacred fields."

Next, from a fascinating report of hospitalists unionizing in Oregon:

"To work in a hospital today is to be constantly preoccupied with money, and one of the more grating features as far as the Sacred Heart hospitalists are concerned has been the administration’s celebration of “skin in the game.” That means creating financial incentives for doctors to hit performance targets — like lowering patient’s length of stay and doing well on patient satisfaction surveys. The phrase entered the Sacred Heart lexicon in 2014, but the underlying concept has spread throughout the profession in recent years…. 

…the increasing focus on metrics like readmission rates and hospital-acquired infections had created more work for hospitalists, who are responsible for a lot of documentation."

And finally, from a NEJM piece on “Taylorism” in medicine:

"Instead of gaining happiness minutes, clinicians are increasingly experiencing dissatisfaction and burnout as they’re subjected to the time pressures of Taylorism and scientific management in the name of efficiency. We have watched colleagues fleeing to concierge practices, where they have control over their schedules. Others have taken early retirement, unwilling to compromise on what they believe is the time needed to deliver compassionate care. Some have moved into management or consulting positions, where they tell others how to practice while unburdening themselves of their clinical load. Just as Taylor enriched himself by consulting for companies, a growing and lucrative industry has emerged to generate and enforce metrics in medicine. By 2014, the Centers for Medicare and Medicaid Services alone had mandated the use of more than 1000 performance measures. As the Institute of Medicine recently reported, such metrics have proliferated, though many of them have little proven value."

(This last piece probably should have given a shout-out to a piece by Mike that started by similarly invoking Frederick Winslow Taylor.)

High stakes metrics and human nature

Eli didn't want to wade into the VA waiting-time crisis, so I'll just do it instead.  Because we’ve covered this ground before. Whenever a metric is tied to a high-stakes incentive or disincentive (monetary bonus or penalty, job security, etc.), gaming of that metric is inevitable. College test scores, crime rates, HAI rates, the dynamic is the same. 

So forgive me if I’m not shocked, shocked, that VA administrators used various ways to game their waiting-time metrics in response to what this NY Times report describes as “pressures to excel in the annual performance reviews used to determine raises, bonuses, promotions and other benefits.” Fudging dates, creating “ghost clinics”, keeping two sets of appointment books—all are being reported to have occurred in some facilities. If you think this is the scandal, you are missing the point. The scandal is that the VA has too few physicians to see an expanding number of veterans needing care (see Eli’s post for more on the trends contributing to primary care shortages).

So I hope lawmakers begin working to fix that problem, once they finish expressing their outrage for the cameras. The VA is an excellent healthcare system, one that matches or outperforms the private/non-VA sector in many measures of care quality and safety. I’m sad that this is getting lost in the current media frenzy—read this piece for a nice counterpoint.