In its Jan. 2021 issue, the Journal of Adolescent Health tempered the conclusions of its most cited study from 2020, which originally claimed, “e-cigarettes and cigarettes are significant underlying risk factors for COVID-19.” After receiving numerous letters critical of the study’s methods, the authors finally conceded “our study does not imply causality.”
The release of the original report, by Gaiha et al., coincided with a letter by Rep. Raja Krishnamoorthi (D-IL), who used the study’s results to justify demanding “the FDA to clear the market of all e-cigarettes, temporarily, for the duration of the coronavirus.” Krishnamoorthi’s call received extensive media coverage.
Similarly, Bonnie Halpern-Felsher, one of the study’s co-authors and editor of the Journal of Adolescent Health (JAH), was quoted in stories about the study in The New York Times and other outlets, where she repeated the assertion. The New York Times, for example, wrote:
…a recent survey of more than 4,000 people ages 13 to 24 found that vaping was strongly linked to catching the coronavirus. But Bonnie Halpern-Felsher, a pediatrics researcher at Stanford University and an author on the study, said that there was probably more than biology at play.
But while it was being used by members of Congress and cited in the media, the study also raised eyebrows among researchers familiar with the evidence related to the effects of vaping. A number of experts responded swiftly to the study’s abnormal findings that lacked a credible causal theory.
What raised many questions at first was the fact that the fractions in the study’s descriptive statistics were not possible given the sample sizes, a sure sign of mathematical error. In response, JAH published four letters, all touching on different limitations of the study, that argued the errors rendered the study useless for policy suggestions.
In one of the letters, professors Konstantinos Farsalinos and Ray Niaura criticized the inconsistent results commenting. “It is not biologically plausible that e-cigarette trial or experimentation would cause effects that result in stronger predisposition to COVID-19 than current/regular use,” they wrote.
Their letter then discussed how extrapolating Gaiha et al’s descriptive statistics to the entire population during the study period would mean both young adults and teens represented 1) the majority of people who were tested for COVID-19 and 2) about half of the total positive COVID-19 cases, which are both far lower according to the Centers for Disease Control and Prevention.
Similarly, professors Brad Rodu and Nantaporn Plurphanswat pointed out in their own letter how the study’s descriptive statistics for total positive cases among “ever-users of e-cigarettes only”, “ever-dual-users of both cigarettes and e-cigarettes”, and “past 30-day dual-users” meant that there were only 5, 3, and 5 total positive COVID-19 cases in each group sample, respectively, out of about 2,184 total respondents who reported they ever tried e-cigarettes. Typically, results from such small case counts are not considered useful in the scientific community.
Meanwhile, Dr. Robeto Sussman and I found the most egregious limitation of Gaiha et al. was their failure to control for the percentage of the population in the e-cigarette user and nonuser group samples that were tested for COVID-19 in the first place. According to the study’s descriptive statistics, e-cigarette users were over three times as likely to be tested for COVID-19 as nonusers.
We suspected that this was the main reason e-cigarettes users tested positive for COVID-19 at over three times the rate of nonusers. Our letter published by JAH described a statistical “conditional-probability” identity, proving that the authors would produce biased results if they did not control for testing.
Plainly, you cannot test positive for COVID-19 if you were not tested in the first place.
Although Dr. Halpern-Felsher and six other authors (which did not include either of the other two original study authors), gave detailed justifications for why Gaiha et al. findings are still useful despite criticism from these letters, they did not respond substantively to the letter I wrote with Sussman.
In fact, the response letter gave two examples of studies that controlled for testing, hinting that Dr. Halpern-Felsher concedes that her study’s methods may depart from what is standard in the epidemiological literature. As justification for their methods, the response simply states our suggestion “is akin to ‘sampling on the outcome’ that could produce biased results. We find the suggestions from Rich and Sussman to be interesting future research ideas.”
Dr. Sussman and I specifically requested a reanalysis that controlled for testing, but we are not aware of any such reanalysis having been undertaken.
Technically, if e-cigarettes disproportionately caused COVID-19 symptoms and this led young e-cigarette users to get tested proportionately more often, Gaiha et al. could make a case for not controlling for testing. But since about half of positive COVID-19 tests come from people who are asymptomatic and the vast majority of people who are tested don’t have symptoms, this justification does not work for their study. In fact, after stating that the results could not be used to “imply causality,” the authors postulated that “removing a mask to vape or smoke could potentially increase exposure to the virus.”
We argue that the study’s correlation is more likely due to e-cigarette users being tested for COVID-19 more often than non-users in their specific sample.