De-implementation and Noninferiority in Infection Control Studies

De-implementation or "stopping practices that lack supporting evidence" is a popular topic in infection control circles. In fact, just yesterday I read a discussion where the authors suggested we no longer need to practice hand hygiene after removing gloves when caring for patients with CDI. I guess there aren't randomized trials - you can't be serious!

Which brings me to a recent review in the NEJM by Laura Mauri and Ralph D'Agostino titled "Challenges in the Design and Interpretation of Noninferiority Trials."  This review is very well written - perhaps required reading for epidemiology students well written. In infection control, it is important to recognize that most de-implementation studies are really non-inferiority trials. For example, when we discontinue contact precautions, we are really suggesting that "stopping contact precautions" is non-inferior to continuing contact precautions in preventing MDRO transmission - of course ignoring that compliance with contact precautions is probably so poor that they are basically the same intervention!

In the contact precautions example, we would be testing whether stopping contact precautions "is not worse than the control (continuing contact precautions) by an acceptably small amount, with a given degree of confidence." The null hypothesis would be that discontinuing contact precautions leads to higher transmission of MDRO (i.e. is worse) and rejection of the null hypothesis is used to support the claim that discontinuing CP is noninferior. Here I suggest you stare at Figure 1 for a bit (probably easier to read in the paper with the description of each condition, but I have included it below anyway)

Further discussion about the design and analysis of these trials is way beyond the scope of a humble blog post; however, the authors include nice descriptions of methods for deriving noninferiority margins, the "constancy assumption" and statistical analysis approaches. But their 6th and 7th components of noninferiority trials are worth mentioning from an infection control standpoint:

6) Adequate ascertainment of outcomes: The authors write that "incomplete or inaccurate ascertainment of outcomes, as a result of loss to follow-up, treatment crossover or nonadherence, or outcomes that are difficult to measure or subjective, may cause the treatments being compared to falsely appear similar."  I would suggest that studies that seek to de-implement contact precautions that do not include admission/discharge surveillance cultures seeking to detect transmission events fail this criteria.

7) Issues with "Intention-to-Treat" in noninferiority designs: In a superiority studies (typical RCTs), intention-to-treat analysis, where anyone who receives the treatment is included even if they get one dose, is the gold standard. The authors write: "In a noninferiority study, however, if some patients did not receive the full course of the assigned treatment, an intention-to-treat analysis may produce a bias toward a false positive conclusion of noninferiority by narrowing the difference between the treatments. In some instances, a per-protocol analysis, which excludes patients who did not meet the inclusion criteria or did not receive the randomized, per-protocol assignment, may be preferable in a noninferiority trial. However, a per-protocol analysis may include fewer participants and introduce postrandomization bias. In general, both the intention-to-treat and per-protocol data sets are important. We suggest analyzing both sets and examining the results for consistency."

Just some things to think about as we read the coming wave of de-implementation studies in infection control including diagnostic stewardship.

When will the p-value finally die?

Over the past 7 years!, I've written about my concerns regarding the misuse of p-values, which I've dubbed "death by p-value." First and foremost, no one really understands what a p-value means and perhaps more importantly, a singular focus on a p-value threshold detracts from important concepts such as effect measures (e.g. hazard ratios) or the continuous nature of confidence intervals.

Recently, I've seen an increasing number of calls to hasten the death of the p-value from folks such as Ken Rothman (you might have read one of his books?) and Miguel Hernan. Professor Hernan has a request on twitter saying "statistical significance must go" where he is asking for examples of outrageous misuses of p-values. He's also suggested that "if your journal still uses 'statistical significance' in 2017, retire your statistical consultant."

Below, I've posted a letter to the editor by Professor Rothman and colleagues that he recently circulated. It has a nice brief example of how p-values are frequently misused. The key sentence, I believe, is this one: "given the expected bias toward a null result that comes from non-adherence coupled with an intent-to-treat analysis, the interpretation of the authors and editorialists is perplexing." I'll ask, are there situations such as studies of contact precautions or hand hygiene interventions where they would be analyzed using an intent-to-treat analysis and non-adherence levels might be high?

I'll leave you with this video on p-values from Carl Bergstrom at University of Washington. He and Jevin West have a course and planned book titled "Calling Bullshit: in the age of big data," which has been well received. And to connect this to MDRO, Carl is the author of a 2004 PNAS paper "Ecological theory suggests that antimicrobial cycling will not reduce antimicrobial resistance in hospitals", which is a nice example of how math models can improve our understanding of antimicrobial stewardship interventions. It's still worth reading.

Moving Beyond the 0.05 p-value

One of my common refrains in research conference and here is that the misuse of p-values has negative public health consequences, a phenomenon I call "death by p-value." Of course my level of frustration with p-values pales in comparison to what well-trained statisticians must feel. This week, the American Statistical Association Board of Directors led by Ronald Wasserstein released a Statement on Statistical Significance and P-values which include six principles on the use and interpretation of p-values. These are:

1.  P-values can indicate how incompatible the data are with a specified statistical model. 
2.  P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. 
3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. 
4. Proper inference requires full reporting and transparency. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result. 
5. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis. 

In addition to the ASA statement I highly recommend the coverage in FiveThirtyEight and Retraction Watch's interview of Professor Wasserstein. We often talk about the post-antibiotic era but even more important for public health is that researchers and journals happily embrace the post p=0.05 era.

Part 2: Influenza Vaccine Effectiveness - The Test-Negative Control Design

Before going further, I want to highlight the purpose of yesterday's and today's post. Occasionally, it is important to question standard practice in epidemiology when something doesn't make sense to you. I've experienced this first hand 3-4 times in infectious disease epidemiology and consider it an important exercise and critically important to our field. You can read this post if you want to read more about how we've questioned the ID epidemiology dogma around control-group selection, quasi-experimental design methodology and inappropriately controlling for intermediates in the causal-pathway of outcomes studies. Of course, this doesn't prove anything in this instance but rather explains my motive.

Yesterday, I posted my initial thoughts on how the CDC calculated the preliminary influenza vaccine effectiveness for the 2012-2013. I've now had several informal discussions on the phone and in person with several vaccine experts and am starting to understand how/why they conducted the study the way they did. I'm even more convinced now that design considerations (e.g. which patients are included in the cohort) are influencing the statistical analysis, which should be separate considerations. I will explain this further below.

It appears that the current standard study design in vaccine effectiveness (VE) is called the "test-negative" control design. This design is a modified case-control design where controls are required to have been tested for influenza but have been found "test negative". This appears to be a useful design when the specificity of the diagnostic test for influenza is low (Orenstein EW 2007), which is not the case in the current MMWR report. Another reason mentioned for selecting this design is that it is thought to reduce the bias, since vaccinated individuals may be more like to seek medical care vs. unvaccinated individuals. However, this benefit would apply to any patient seen in the outpatient clinic and not necessarily require diagnostic testing for influenza. 

In favor of cohort studies over case-control or test-negative designs, is the fact that "if specificity was 100%, the cohort method VE estimate was equivalent to the true VE, regardless of test sensitivity or ARs of influenza- and non-influenza-ILIs." (Orenstein EW 2007) The Orenstein paper's stated purpose was to study the impact of sensitivity and specificity of influenza testing on VE estimates across these study designs. Any benefits for the test-negative approach appear to evaporate when using a great diagnostic test, like the rt-PCR used in the MMWR report.

From what I can tell, the Orenstein paper is frequently cited to justify the test-negative design, but given current conditions, the design doesn't seem to be as useful as it once was. I also remain unconvinced that a cohort of patients already being seen in a doctor's office can tell us anything about risk factors for "medically attended" influenza. More importantly, I'm still very concerned with how the cohort was analyzed. Even if there are very good reasons to enroll a cohort of patients who presented to a clinic in order to avoid bias from vaccinated patients presenting differentially to clinics and even if all were tested for influenza, it doesn't follow that you need to use an odds-ratio approach to measure vaccine effectiveness when the relative-risk approach is more accurate. It appears that correcting one wrong (differential medical care seeking in vaccinated vs unvaccinated) is leading to a countervailing wrong when the analysis is done incorrectly. If they had just called this a cohort of tested patients or a "tested cohort", perhaps this wouldn't have happened.

Oh, and I'm still not sure why they are excluding influenza B positive patients from their VE calculation  for influenza A. Shouldn't they have to look at each A strain separately then? Well, that's another post for another day.