When will the p-value finally die?

Over the past 7 years!, I've written about my concerns regarding the misuse of p-values, which I've dubbed "death by p-value." First and foremost, no one really understands what a p-value means and perhaps more importantly, a singular focus on a p-value threshold detracts from important concepts such as effect measures (e.g. hazard ratios) or the continuous nature of confidence intervals.

Recently, I've seen an increasing number of calls to hasten the death of the p-value from folks such as Ken Rothman (you might have read one of his books?) and Miguel Hernan. Professor Hernan has a request on twitter saying "statistical significance must go" where he is asking for examples of outrageous misuses of p-values. He's also suggested that "if your journal still uses 'statistical significance' in 2017, retire your statistical consultant."

Below, I've posted a letter to the editor by Professor Rothman and colleagues that he recently circulated. It has a nice brief example of how p-values are frequently misused. The key sentence, I believe, is this one: "given the expected bias toward a null result that comes from non-adherence coupled with an intent-to-treat analysis, the interpretation of the authors and editorialists is perplexing." I'll ask, are there situations such as studies of contact precautions or hand hygiene interventions where they would be analyzed using an intent-to-treat analysis and non-adherence levels might be high?

I'll leave you with this video on p-values from Carl Bergstrom at University of Washington. He and Jevin West have a course and planned book titled "Calling Bullshit: in the age of big data," which has been well received. And to connect this to MDRO, Carl is the author of a 2004 PNAS paper "Ecological theory suggests that antimicrobial cycling will not reduce antimicrobial resistance in hospitals", which is a nice example of how math models can improve our understanding of antimicrobial stewardship interventions. It's still worth reading.

Moving Beyond the 0.05 p-value

One of my common refrains in research conference and here is that the misuse of p-values has negative public health consequences, a phenomenon I call "death by p-value." Of course my level of frustration with p-values pales in comparison to what well-trained statisticians must feel. This week, the American Statistical Association Board of Directors led by Ronald Wasserstein released a Statement on Statistical Significance and P-values which include six principles on the use and interpretation of p-values. These are:

1.  P-values can indicate how incompatible the data are with a specified statistical model. 
2.  P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. 
3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. 
4. Proper inference requires full reporting and transparency. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result. 
5. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis. 

In addition to the ASA statement I highly recommend the coverage in FiveThirtyEight and Retraction Watch's interview of Professor Wasserstein. We often talk about the post-antibiotic era but even more important for public health is that researchers and journals happily embrace the post p=0.05 era.

Tracheotomy, VAP, p-values and death

There is a new RCT just published in JAMA by a large group in Italy looking at the benefits of early (day 6-8) vs late (day 13-15) tracheotomy completed in 12 ICUs. The primary endpoint was VAP. There is also a very nice accompanying editorial. There are several interesting findings. First, patients randomized to early tracheotomy were less likely to develop VAP by day 28, 14% vs 21%, but the p-value was 0.07. Since the p value was greater than 0.05, the authors were forced to say that there was no benefit from early tracheotomy.

Interestingly they also found significantly greater vent-free days, ICU-free days, successful weaning and ICU discharges in the early tracheotomy group. There was even a trend towards higher survival in the early vs late group, HR=0.80, 95% CI 0.56-1.15. The authors and editorial do a nice job of pointing out that 31% of early and 43% of the late group didn't even receive a tracheotomy due to impending extubation or death. The editorial even makes the point that selecting an early tracheotomy is really a strategy of more trachs. The study did not assess patient comfort, which may be associated with early tracheotomy.

What is always troubling to me is that scientists, editorialists, journals and clinicians are stuck in this p-value trap. Here we have a study, a very good randomized trial, which shows likely clinically significant reductions in VAP and potentially lower mortality, but since the study was underpowered we are forced to say "no difference." I wonder if you calculated how many patients are intubated each year in the US (or Italy) and reduced VAP rates by 33%, how many VAPs would be prevented and how many deaths would be prevented? I know this study should be repeated, but will it? You have a negative JAMA study, what's the incentive? I describe this phenomenon as "Death by p-value."