Communicating Complexity

Healthcare quality metrics are such a struggle.  We all want metrics that best reflect our efforts to keep patients safe at our institutions, while not penalizing institutions who provide care for patients at higher risk for complications.  We also want the data collection burden to be light and the outcome to be simple and easy to understand. When comparisons are going to be made among hospitals of varying sizes, that offer different levels of care, to populations from varying economic and social support systems, we want known risk factors to be taken into consideration.  And not just to avoid financial penalties at our hospitals, but also to provide better information to patients.  While I doubt that many patients actually use the Hospital Compare data to select a facility (most “choices” are driven by insurance coverage, geography and physician referrals), if they did, it would be nice if the metrics actually steered them toward safer healthcare.

And NHSN listened to these concerns, moving to risk adjusted models and the SIR - a summary statistic that accounts for the prevalence of (a few) known risk factors.  But as the stakes get higher, limitations to the current risk adjustment models grow increasingly frustrating. Why can’t we make these models better?

On the other hand, the move to risk adjusted models has increased the complexity of both understanding and communicating our outcomes, internally among  infection prevention program personnel and hospital leadership and externally to the public and consumer organizations.  Recent work by Vineet Chopra and his colleagues at UMichigan have been looking at how well we “experts” even understand these metrics ourselves.  His most recent evaluation was a survey of SHEA research network members, published in ICHE under the title “Do Experts Understand Performance Measures? A Mixed-Methods Study of Infection Preventionists” (though 80% of respondents were physicians).   Respondents were given a table of data about 8 hypothetical hospitals and asked questions about interpreting the presented data and about the impact changes at those facilities might (or might not) have on the data.  Of 67 respondents (only 54 of whom answered every question, so a pretty small sample), performance was mixed.  Particular difficulty was noted on questions that involved risk adjustment, such as the impact of more G tube use at one hospital on the calculated SIR or the impact of implementing antibiotic coated catheters on the projected number of infections.   And this from a group of primarily physician leaders of hospital epidemiology programs, engaged in SHEA, many from academic medical centers.

I brought the survey questions to the monthly meeting of all the infection preventionists from across our healthcare system and I am happy to report we did very well!  We had quibbles with how some of the questions were worded and we benefitted from being able to talk through the questions together as we formulated our answers.

The authors concluded that limitations in understanding the risk adjustment data may make the data ‘less actionable by end users’ and ‘..decision makers’ trying to reduce HAIs.  I’m not sure that is true.  The SIR does at least provide a fairly simple guidepost of “numbers higher than they should be”.  That should be enough to prompt action – but sharing an SIR with leadership and program personnel to develop plans for action requires more in depth understanding than just the SIR itself.  It requires knowledge of what factors are included in the risk adjustment model and what are not, the prevalence of all those factors in your population, and which of those factors are actionable/preventable.  That more in depth understanding is a bigger challenge and is harder to summarize and communicate in a single metric - especially if you don’t fully understand it yourself.

The other issue raised by this complexity, and our own difficulties interpreting and explaining it, is one of trust and transparency with the other ‘end-users’: patients.  While we advocate to improve risk adjustment, to make comparisons among facilities more appropriate, some patients and consumer groups feel that we are purposefully obscuring actual numbers of infections in order to hide poor practices.  The ‘black box’ from which the SIR emerges can erode much needed trust.

Luckily, NHSN heard these concerns as well.  Through HICPAC, two new NHSN working groups have been formed:  data and definitions (including risk adjustment) and communication. And the communication subgroup is co-led by Dr Vineet Chopra! That group will be discussing better ways to communicate the complex inputs and hopefully understandable outputs both verbally and visually.  Good communication provides much needed clarity and builds trust. I look forward to hearing about their work.

PS I especially enjoyed reading the comments in the supplementary material where respondents offered answers to the question “in your opinion, what are the three biggest problems for reliability of quality metric data at your hospital”.  I recommend them to everyone. They call out issues with risk adjustment, data collection, definitions etc.  A couple of favorites include “some preventable infections are more preventable than others”; “we don’t use quality metric data” ; and “gaming the system; gaming the system; gaming the system”.