Healthcare quality metrics are such a struggle. We all want metrics that best reflect our
efforts to keep patients safe at our institutions, while not penalizing
institutions who provide care for patients at higher risk for
complications. We also want the data
collection burden to be light and the outcome to be simple and easy to
understand. When comparisons are going to be made among hospitals of varying
sizes, that offer different levels of care, to populations from varying
economic and social support systems, we want known risk factors to be taken
into consideration. And not just to
avoid financial penalties at our hospitals, but also to provide better information
to patients. While I doubt that many
patients actually use the Hospital Compare data to select a facility (most “choices”
are driven by insurance coverage, geography and physician referrals), if they
did, it would be nice if the metrics actually steered them toward safer
healthcare.
And NHSN listened to these concerns, moving to risk adjusted models
and the SIR - a summary statistic that accounts for the prevalence of (a few)
known risk factors. But as the stakes
get higher, limitations to the current risk adjustment models grow increasingly
frustrating. Why can’t we make these models better?
On the other hand, the move to risk adjusted models has increased the
complexity of both understanding and communicating our outcomes, internally among
infection prevention program personnel
and hospital leadership and externally to the public and consumer
organizations. Recent work by Vineet
Chopra and his colleagues at UMichigan have been looking at how well we “experts”
even understand these metrics ourselves.
His most recent evaluation was a survey of SHEA research network
members, published in ICHE under the title “Do Experts Understand Performance
Measures? A Mixed-Methods Study of Infection Preventionists” (though 80% of
respondents were physicians). Respondents were given a table of data about 8
hypothetical hospitals and asked questions about interpreting the presented data
and about the impact changes at those facilities might (or might not) have on
the data. Of 67 respondents (only 54 of
whom answered every question, so a pretty small sample), performance was mixed. Particular difficulty was noted on questions
that involved risk adjustment, such as the impact of more G tube use at one
hospital on the calculated SIR or the impact of implementing antibiotic coated
catheters on the projected number of infections. And this from a group of primarily physician
leaders of hospital epidemiology programs, engaged in SHEA, many from academic
medical centers.
I brought the survey questions to the monthly meeting of all the
infection preventionists from across our healthcare system and I am happy to
report we did very well! We had quibbles
with how some of the questions were worded and we benefitted from being able to
talk through the questions together as we formulated our answers.
The authors concluded that limitations in understanding the risk
adjustment data may make the data ‘less actionable by end users’ and ‘..decision
makers’ trying to reduce HAIs. I’m not
sure that is true. The SIR does at least
provide a fairly simple guidepost of “numbers higher than they should be”. That should be enough to prompt action – but
sharing an SIR with leadership and program personnel to develop plans for
action requires more in depth understanding than just the SIR itself. It requires knowledge of what factors are
included in the risk adjustment model and what are not, the prevalence of all
those factors in your population, and which of those factors are
actionable/preventable. That more in
depth understanding is a bigger challenge and is harder to summarize and
communicate in a single metric - especially if you don’t fully understand it
yourself.
The other issue raised by this complexity, and our own difficulties
interpreting and explaining it, is one of trust and transparency with the other
‘end-users’: patients. While we advocate
to improve risk adjustment, to make comparisons among facilities more appropriate,
some patients and consumer groups feel that we are purposefully obscuring
actual numbers of infections in order to hide poor practices. The ‘black box’ from which the SIR emerges
can erode much needed trust.
Luckily, NHSN heard these concerns as well. Through HICPAC, two new NHSN working groups
have been formed: data and definitions
(including risk adjustment) and communication. And the communication subgroup
is co-led by Dr Vineet Chopra! That group will be discussing better ways to
communicate the complex inputs and hopefully understandable outputs both verbally
and visually. Good communication
provides much needed clarity and builds trust. I look forward to hearing about
their work.
PS I especially enjoyed reading the comments
in the supplementary material where respondents offered answers to the question
“in your opinion, what are the three biggest problems for reliability of
quality metric data at your hospital”. I
recommend them to everyone. They call out issues with risk adjustment, data
collection, definitions etc. A couple of
favorites include “some preventable infections are more preventable than others”;
“we don’t use quality metric data” ; and “gaming the system; gaming the system;
gaming the system”.
