Women in Healthcare Epidemiology

Dr. Janet Lane-Claypon

I'm not a big fan of National [fill-in-the-blank] Days—to me they imply we can ignore the topic on the 364 days that we aren’t supposed to celebrate it. So on National Women Physicians Day, we should vow to better recognize the huge contributions women physicians make every day. That way we might not even need a “day” in the future, and can thus focus instead on preparing for National Lima Bean Respect Day (April 20).

In the field of healthcare epidemiology and infection prevention, the list of women leaders is long--and for me to produce one would be very dangerous because I’m sure it’d be incomplete. Instead I’ll point out that the last two SHEA presidents were women, and the 2019 SHEA president will be our esteemed fellow blogger, Hilary Babcock (congrats again, Hilary!).

This also seems like a good day to point interested readers to this piece about Dr. Janet Lane-Claypon, a pioneering physician-epidemiologist who was the first to employ the now-ubiquitous cohort and case-control study designs we use so often in infection prevention. The paper was published in 2004 but I only recent stumbled on it, and found it a fascinating story about a person I clearly should have learned about during my epidemiology coursework (but didn’t!).

I recognize the irony of me posting this from a blog that has a 5:1 male:female ratio. We’ve tried over the years to recruit women to the blog, mostly unsuccessfully. One possible reason (besides the obvious—that we haven’t tried hard enough), is that women physicians put up with substantially more bullshit each day than their male counterparts, and thus have less time for blogging.

To our female readership: if you’re interested in contributing to the blog (either with periodic guest posts, or joining the group), please contact one of us. This isn’t limited to physicians: infection preventionists, non-physician epidemiologists, microbiologists, nurses…pretty much anybody with expertise and strong opinions about infection prevention and healthcare epidemiology!

-->

Influenza Vaccine Effectiveness Study Author Responds

Late last week, Dr. Jackson, one of the co-authors of the recent MMWR influenza vaccine effectiveness report, sent me an email response to my posts discussing how and why they measure influenza vaccine effectiveness the way they do. I thought that in the interest of fairness, I should post his full email, rather than pasting his response into a largely hidden comment section. I will likely post a follow-up to this at some point. Additionally, I want to thank Dr. Jackson publicly for responding in this fashion, both professionally and academically. The intent of my posts was academic, and I am pleased that he responded the way he did.

Dear Dr. Perencevich,

I am one of the co-authors on the MMWR article on influenza vaccine effectiveness, and I read your blog posts about that article. I believe I can clear up some of your questions. I tried posting this as a comment on the blog, but the website wouldn't let me, so my apologies for e-mailing you instead. Feel free to post this as comment on the website if you are able.

 (1) Regarding your first point, this study used what is known as a “test-negative” design. In the test-negative design, we enroll patients with a medically attended acute respiratory illness (MAARI). We then test these enrollees for influenza, and assess who was previously vaccinated and who was not. The test-negative design is based on an assumption that the rate of MAARI caused by pathogens other than influenza is the same in both vaccinated and unvaccinated persons. If this assumption is true, than our test-negative subjects are representative of the population from which the influenza-positive cases came, and our study does give an estimate of how well the vaccine reduces the risk of getting sick enough to visit the doctor.

 Although the paper refers to cases and controls, this is not a true case-control study, since a true case-control study requires that we know who is a case and who is not before we sample them. The design is closer to the “indirect cohort” method proposed by Claire Broome for studying pneumococcal vaccine effectiveness [NEJM 1980; 303:549-52]. The real advantage of the test-negative design is that it controls for differences in healthcare-seeking behavior. If we did the full cohort study you proposed, there would be variation among the cohort members in how often they seek healthcare, and these variations would be related both to their likelihood of being vaccinated and to their likelihood of going to the doctor if they got influenza. By only sampling people who come to the doctor, we control for those differences.

(2) Regarding your second point and third points, the relative risk (RR) is not an appropriate measure of association for this study design. The RR is (obviously) a measure of risk, which would be based on the cumulative incidence of disease in some defined cohort. In this study, we do not sample the full cohort; we simply use the influenza negative subjects to estimate the frequency of vaccination in the cohort. Using the RR in this setting would give a biased estimate of vaccine effectiveness, which is seen in the sample calculations you provided.

Sampling in a test-negative design is conceptually similar to incidence density sampling in a case-control study. When using incidence density sampling, the exposure odds ratio is a direct estimate of the incidence rate ratio and not an approximation to the RR. 

(3) Finally, regarding your final question about including the flu B cases in the estimate of VE against flu A: As mentioned above, the test-negative design assumes that the rate of non-flu MAARI is the same in vaccinated and unvaccinated persons. If we included flu B in the non-case group, we would be violating this assumption, because the vaccine does protect against B, and the rate of non-flu A MAARI would no longer be the same in vaccinated and unvaccinated persons.

 I hope this clears up your questions!
~Mike Jackson, Group Health Research Institute

How should we calculate influenza vaccine effectiveness?

You know, I probably should've just been happy with the reports that 2012-2013 influenza vaccine was 62% effective and called it a day. But this morning I read this nice report by Helen Branswell of the Canadian Press describing why vaccine-virus match isn't the only factor that impacts vaccine effectiveness and then I made the mistake of looking at the early-release CDC MMWR report more closely.  Now, I'm an ID physician epidemiologist and led the influenza response at the University of Maryland, Baltimore and UMMS during the 2009 H1N1 pandemic. Thus, I'm no stranger to reading these reports, but for some reason, today, they just didn't make sense. It occurred to me that perhaps if I'm perplexed, others might be, so I've decided to post my questions and concerns and hopefully, as I get answers, I'll post them here.

Some Background: Read the CDC MMWR report from January 11th. and focus on Table 2 (below). This table summarizes the vaccine effectiveness data for the vaccine vs. influenza A, influenza B and both.

Initial Observations: One, it appears that CDC is using a prospective cohort of 1155 sick patients who presented as outpatients for acute respiratory illness, and not a group (cohort) of all patients eligible for vaccination. Ideally, you'd want to determine the likelihood that influenza vaccine prevents clinical illness, visits to the doctor, hospital admission and mortality. The vaccine effectiveness in the MMWR report can't tell us that and I'll explain why in #1, below. Additionally, the MMWR report determines vaccine-effectiveness using a case-control method and not a cohort method. This might seem to be an esoteric point, but it could have a big influence on how effective we think a vaccine is. I will try to explain this in #2 and #3, below. Finally, I'm not sure how they decided which patients should be included in the uninfected group for their calculations. I'll explain this a bit more in #4 and show how it could bias the estimates of vaccine effectiveness.

1) Why does CDC utilize outpatient, sick controls in their estimates of vaccine efficacy?  I suspect this is an issue of expediency and cost-effectiveness. It would be more expensive to enroll 1000 patients in September and track them weekly to see if they get the vaccine and then if they develop symptoms and test them. Of course, they can't easily do randomized studies in the US or elsewhere since the vaccine is recommended for just about everybody, so randomizing to no vaccine would be unethical. Whatever the reason, selecting an entire cohort of patients, already sick enough to visit their doctor, does not tell us how effective the vaccine is in preventing illness, preventing visits to the doctor, preventing hospitalization or preventing death. The MMWR report can only tell us how effective vaccine is in preventing an influenza infection vs. another infection conditional on already being sick enough to go to the doctor's office. What does that mean for people trying to decide if they should get vaccinated?

Also, could it be that selecting this cohort biases the findings in other ways? What if vaccinated patients would be more likely to seek medical care for their symptoms? What if those that develop acute respiratory illness are different or sicker than healthy controls in a systematic way?  These could impact the measure of vaccine effectiveness.  Additionally, using outpatient, sick controls leaves out two very important groups: hospitalized patients and healthy populations that never developed an illness in the first place.  I suspect that declining funding for CDC and other groups is behind this - you get what you pay for. However, none of the reports I've read explain this limitation when reporting vaccine effectiveness. They should.

Note: I've added a second post describing a bit more why CDC selected this cohort of patients.

2) Why does the CDC measure vaccine effectiveness using odds ratios even when they have a cohort of patients?  To explain further, a case-control study would be one where they find 1000 (or any number) of influenza positive patients and then look back and see if they were vaccinated and then find another set of 1000 influenza negative controls (healthy, sick, whatever) and see if they were vaccinated. Here they identified a cohort of patients with acute respiratory illness first and then determined their influenza status and vaccine status retrospectively. Thus, this is a "retrospective" cohort study. Just because the cohort was established conditional on them having an outpatient visit for a respiratory complaint, does not invalidate that this is a cohort. This matters since they report odds ratios and not relative risks. And as a reminder, when baseline or initial risk is high, the odds ratio can over-estimate the relative risk. To find out how and why this is important read this BMJ article.

3) Did measuring vaccine effectiveness using an odds ratio (OR) method (as it appears the CDC did) vs. the relative-risk (RR) method, as normally used in cohort studies, matter? The question here is not a theoretical one, as above, but rather I'm asking if we used the exact numbers in the MMWR report but used a cohort or relative-risk method, would we get a different estimate of effectiveness?  Short answer: Yes

If we take the table showing attack rates in vaccinated vs unvaccinated for influenza A only (from Table 2 here), we get very different results based on the method used to calculate efficacy.

Using the CDC or OR method, vaccine effectiveness (VE) = (1-OR)*100 or (1-ad/bc)*100. Using that method, the calculated VE=53.4% (CDC reports 55% in their table, since they adjusted for site)

Using the RR (cohort) method, the VE = (1-RR)*100 where the RR= (a/(a+b)) / (c/(c+d)). Using that method, the calculated VE=44.1%

This is a very big difference with a 9.3% absolute reduction in effectiveness by method alone! It seems that since site level variation is not a big driver of the effectiveness, the RR approach might be more accurate. Of note, when you do the above analysis for the vaccine vs influenza A or B, the VE falls from 62% using the = OR approach to 47% using the RR approach.

*I hope someone can explain why they are analyzing cohort data using case-control methods. For more information on how I calculated these estimates, see this paper by Walter Orenstein, et al from 1985. It appears this case-control method is standard in the influenza vaccine literature.

4) Why did CDC leave the influenza B positive patients out of their calculation of the effectiveness of the vaccine versus influenza A and vice-versa? When looking at Table 2 above, one thing struck me as odd. When doing the three effectiveness calculations, they used the same control group. To me, if you don't have influenza A, you should be included in the "uninfected group" for testing the effectiveness of the vaccine against influenza A.  To see if this matters, I added in the 180 patients who were influenza B positive AND influenza A negative that the CDC left out of their calculation. Here is the new 2x2 table:

Here, if I use the CDC (case-control or odds-ratio method) I find a VE = 41% and if I use the cohort method, I find a VE = 34.4%. These results are so different from those reported in MMWR, that I'd be very interested to know why they chose to leave influenza B patients out.

OK. For influenza A, the vaccine effectiveness was reported as 55% in the MMWR report. Depending on how I calculated the vaccine effectiveness, I found that it ranged from 53.4% to 34.4%, with the more accurate estimate likely closer to 34%.  A pretty huge range, don't you think?  Perhaps these reports should calculate effectiveness in a number of different ways and provide them in a sensitivity analysis.  Better yet, we should fund prospective cohort studies that include healthy patients and measure the true effectiveness of the vaccine. Even better, a universal influenza vaccine would render this all moot, but that's in the future...

ADDENDUM:

Please see the other two posts in this thread: (1) My discussion of the test-negative design and (2) the MMWR author's explanation of why they study influenza vaccine effectiveness the way they do.

If you're criticizing methods, your methods better be strong

This morning I innocently set out to review a new study in ARIC that looks at methodological issues in C. difficile outcomes studies assessing the association between infection and excess hospital stay.  The authors make a very important point in the paper: "The studies did not collect data concerning the time of onset of CDI; therefore, it is not possible to exclude the possibility of reverse causality, in which longer lengths of hospitalisation may have increased the risk of CDI." They then go on to discuss time-dependent bias and recommend the use of multi-state models...we've made this point before.

What caused me the emotional anguish was their Table 1 that lists the 16 studies included in the review. In column two, three of the studies are listed as retrospective case-control studies...what??!!  How can you do a case-control study with an outcome being hospital length of stay?  I'm pretty sure you can't, at least not easily.

You see, a case-control study requires identifying an outcome and looking back for risk-factors associated with the outcome. For example, you could look at 50 people that died (and 100 that didn't die) and see how many had CDI, to determine if CDI was a risk-factor for death.  For length of stay, I can't even make up a good way to do a case-control study.  Would you find patients who stay >14 days and compare them to patients that stay <14 days? The three studies listed MUST have been cohort studies, so why did the authors of the ARIC paper seeking to teach us about proper methods for outcomes studies list them as case-control studies? Perhaps they pasted them from the individual studies' methods sections?

Just to confirm this, I looked at all three papers: (Ananthakrishnan (2008), Bajaj (2010), Pepin 2005).

Pepin: from the Methods  - "We compared mortality and total length of hospital stay among inpatients in whom nosocomial CDAD developed and among control subjects without CDAD."  This is a cohort study - exposed and unexposed to CDAD, looking forward to outcomes.  Just because they incorrectly use the word control, doesn't make this a case-control study. (Strike 1)

Ananthakrishnan: "Our primary case group (C difficile–IBD group) included patients who had a primary diagnosis of C difficile colitis and a secondary diagnosis of either Chrohn's Disease (CD) or Ulcerative Colitis (UC). Patients admitted to hospital with a primary diagnosis of CD or UC without a diagnosis of C. difficile colitis formed one of our comparison groups (IBD group)." Another cohort study - defined by the exposure to C. difficile and not by the outcome. (Strike 2)

Bajaj: "Among the cohort of hospitalized patients with any diagnosis of cirrhosis, co-existing diagnosis of C. difficile was associated with significantly greater in-hospital mortality."(Strike 3)

The error of incorrectly describing cohort studies as case-control studies is very common - enough to be a pandemic.  The journal CID is one of the worst offenders. One of my favorite examples is this recent VAP treatment cohort study described as a case-control study in the title!  When I've mentioned the problem to CID editors, they suggest I write a letter.  You can't write letters for each issue.

OK: Cohort - exposed/unexposed look forward in time to the outcome. Case-control - find outcomes and look back for exposures. Got it?  EOR