Multiple studies were conducted thus far to assess the efficacy of the Pfizer’s COVID-19 vaccine. As part of the phase 3 clinical trials, Pfizer conducted a placebo-controlled, observer-blinded trial. Polack et al. (f1) (2020, hereafter Polack) report the finding of this trial. They conclude that the vaccine has 95% efficacy, relying on the respective numbers of positive PCR tests, 162 in the control (placebo) group versus 8 in the treatment (vaccinated) group. Doshi (2021) (f2) raised serious concerns about the validity of the conclusion of “95% efficacy”. The concerns point to the fact that in the FDA report on the Pfizer’s vaccine, there is a category of “suspected COVID-19”—those with COVID-19-like symptoms that were not confirmed by a PCR test. In this category, the split between vaccinated and unvaccinated is roughly equal, 1594 vaccinated versus 1816 unvaccinated. Another concerning fact according to Doshi is that many more vaccinated were excluded from the trial than placebo (311 versus 60).
These concerns, among others, emphasize the importance of the large-scale observational studies. Indeed, the massive and early vaccination campaign in Israel provides such opportunity and two related studies were conducted, Dagan et al. (2021, hereafter “Dagan”) (f3) and Haas et al. (2021, hereafter “Haas”) (f4). The latter is based on the entire Israeli population and the former on about half of the Israeli population (members of the largest HMO, Clalit Health Services) and also has a shorter study period. Notably, Dagan employs a different methodology from Hass for matching patients between the vaccinated and unvaccinated groups. Wise (2021) (f5) highlights the main finding of Dagan and their implications.
These two studies estimate the vaccine efficacy against symptomatic or asymptomatic infection (confirmed by positive PCR test) as 92% (Dagan) and 95% (Haas), starting seven days after the second dose.
Unfortunately, both papers suffer from a potential serious upward bias in their estimates, particularly with respect to asymptomatic infection and potentially also with respect to symptomatic infection with no hospitalization. The concerns stem from the fact that Israel has been mandating strict isolation on anyone who was exposed to an infected individual. The isolation is mandated to 14 days but can be shortened to only 10 days with two negative PCR tests. In contrast, vaccinated individuals have been exempted from any isolation requirement whatsoever, starting seven days after the second dose. The isolation regulations have been enforced quite strictly in Israel. In particular, during the study period in Haas there were 430,000 isolations (f6). Thus, there is a serious concern that the testing intensity of vaccinated individuals seven days after the second dose will be significantly lower compared to unvaccinated individuals. Notably, Dagan does not refer to this potential source of bias. Haas refers to the concern but mentions that only 19% of the 4.4 million PCR tests in the relevant study period were done on exempted (vaccinated) people, a fact that supports the concern above that significantly more tests were conducted among unvaccinated individuals.
The concerns regarding the potential upward bias in the efficacy of the vaccine as a result of different testing intensity are supported by additional findings. Dagan reports in Table 2 only 60% for the first seven days after the second dose with a significant increase to 92% beyond seven days after the second dose, which is exactly when testing exemption would have started. In contrast, in the Pfizer experiment, (see Figure 3 in Polack et al.) the respective increase in efficacy between the first seven days after the second dose and the beyond seven days is very small (from 90.5% to 95%). Moreover, Figure 3 in Polack shows that the vaccine reaches almost full efficacy (which is 95%) before day 14 after the first dose, which again is seemingly contradictory to the dramatic jump in the estimates of Dagan.
For some unknown reasons, Haas does not report the assessed efficacy for the first seven days after the second dose, and only reports the efficacy beyond seven days. However, in the appendix they do provide efficacy estimates for the period of 14-21 days after the first dose (appendix p. 5). Interestingly, the efficacy estimate for this period of time is only 58%, similar to the 60% estimate above by Dagan.
In conclusions, there are serious concerns, that the estimates of Dagan and Haas regarding the efficacy of the Pfizer vaccine against infection for the period from seven days after the second dose both suffer from significant upward bias. The suspected bias stems from lack of appropriate control in their analyses for the apparent differences in testing intensity between unvaccinated individuals and vaccinated individuals, particularly starting seven days after the second dose. Detail data about the tests conducted in the two populations during the respective periods studied could potentially allow reducing the bias of the estimates.
*Levi and Wohl contributed equally to this response
**We thank Ehud Qimron for helpful comments and suggestions
- 30 May 2021
- Retsef Levi* (Professor of Operations Management, Sloan School of Management, MIT) and Avi Wohl* (Professor of Finance, Coller School of Management, Tel Aviv University)
Footnotes
[1] Polack FP, Thomas SJ, Kitchin N, et al. Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine. N Engl J Med 2020; 383: 2603-15.
[2] Doshi P., Pfizer and Moderna’s “95% effective” vaccines—we need more details and the raw data, British Medical Journal 2021, available online from January-4-2021 in the Opinions section.
[3] Dagan N, Barda M, Kepten E, et al. BNT162b2 mRNA COVID-19 vaccine in a nationwide mass vaccination setting. N Engl J Med 2021; 384: 1412–23.
[4] Hass, EJ, Angulo, FJ.McLaughlin, JM. Et al. Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalizations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data, The Lancet, 2021, available online from May-5-2021.
[5] Wise, J., Covid-19: Pfizer BioNTech vaccine reduced cases by 94% in Israel, shows peer reviewed study, BMJ 2021;372:n567.
[6] According to the Israeli Ministry of Health’s data. See https://data.gov.il/dataset/covid-19.