Intended for healthcare professionals

CCBYNC Open access

Rapid response to:

Research

Effectiveness of public health measures in reducing the incidence of covid-19, SARS-CoV-2 transmission, and covid-19 mortality: systematic review and meta-analysis

BMJ 2021; 375 doi: https://doi.org/10.1136/bmj-2021-068302 (Published 18 November 2021) Cite this as: BMJ 2021;375:e068302

Linked Editorial

Public health measures for covid-19

Rapid Response:

Re: Effectiveness of public health measures in reducing the incidence of covid-19, SARS-CoV-2 transmission, and covid-19 mortality: systematic review and meta-analysis

Dear Editor

We read with interest the systematic review and meta-analysis conducted by Talic and colleagues on the effectiveness of mask-wearing, handwashing, and physical distancing on the incidence of SARS-CoV-2 infection.[1] We question the validity of the pooled estimates because of the low quality of the studies included.

For the mask-wearing analyses, one randomised controlled trial (RCT)[2] and five observational[3-7] studies were included, but all have potentially serious or critical flaws that limit the interpretation of the estimates they contribute to the meta-analysis. The limitations of the DANMASK-19 RCT[2] have been discussed elsewhere[8-10] with one of the most concerning being the use of serologic outcomes during a 1-month follow-up period, when this is not much longer than the time taken for antibodies to develop after an infection, meaning there would be considerable dilution of any effect of the mask intervention.[9] The other 5 studies included cohort,[6] case-control,[3 4] and cross-sectional[5] studies with potential for substantial reporting biases, and an ecologic study examining the effect of mask mandates (on top of existing mask use in the community) on SARS-CoV-2 incidence.[7] Ecologic data on mask effectiveness is of limited value and remain difficult to interpret given the multiple interventions simultaneously deployed, behavioural changes, and heterogenous reporting. Mask mandates are different from the intervention of mask distribution and instruction in the RCT[2] or the exposures of mask compliance in the other 4 observational studies.[3-6] Furthermore, even among the observational non-ecologic studies, the measurements of the mask compliance are different. Wang et al. assessed mask wearing at home (both index cases and contacts),[6] whereas Lio et al. and Xu et al. assessed mask wearing outside the household,[4 5] and Doung-Ngern et al. assessed mask wearing (contacts only) when in contact with the index cases regardless of setting.[3] One unfortunate observation is that the two most problematic of these observational studies (the ecologic study and the cross-sectional online survey) carried the most weight (50.2%) in the mask meta-analysis.[5 7]

While Talic et al. published a pooled mask effectiveness estimate of 53% reduction, we believe it is not a valid estimate of real-world effectiveness of masks and agree with the linked editorial by Glasziou et al.[11] that such high estimates encompass several overlapping protective behaviours and interventions. If one is to believe the 53% estimate, it would mean DANMASK-19 was not underpowered as is already widely acknowledged. Had DANMASK-192 (18% reduction in infection risk due to surgical masks) and the Bangladesh preprint mask cluster RCT[12] (9% reduction in symptomatic infection) been combined in a systematic review and meta-analysis, analyses would have resulted in an estimate of approximately 10%. Although Talic et al. mentioned the preprint RCT, they did not include it in their analyses. Evidence points toward an effect size of masks that is likely between higher estimates seen in observational studies and lower ones reported in RCTs—perhaps in the range 10%–20%.[8 9 13 14] This small-to-moderate effect of masks, particularly in households or settings with sustained stronger viral exposures, substantiates the need for a combination of non-pharmaceutical interventions (NPIs) to achieve greater prevention of SARS-CoV-2 transmission.[8 15] To complicate matters further, a single estimate of mask effectiveness is unrealistic given the heterogeneity due to setting, wearer’s adherence, type of mask, etc.[8]

The three studies on hand hygiene[3-5] were a subset of the same studies included in the mask analysis and have similar risks of reporting biases and variations in the measurement of exposures. There are 5 studies included in physical distancing analysis[3 5 6 16 17] and three of them overlap with the studies in the mask analysis.[3 5 6] The other two studies are an ecologic study[17] and a retrospective cohort study.[16] Vokó et al. examined the effects of stay-at-home orders on SARS-CoV-2 incidence and have similar problems as the ecologic study by Krishnamachari et al. The measurements of the compliance to physical distancing are different, where van den Berg et al. and Xu et al. only measured distance and Doung-Ngern et al. and Wang et al. measured both distance and contact frequency or duration.

There are several minor errors in the report which are concerning. For example, most of the “Relative risk” estimates in the facemask meta-analysis figure seem to be odds ratios based on the results in the original articles.[2-4 6] The confidence intervals in the meta-analysis Figure 5 are inconsistent with those presented in Doung-Ngern et al., Lio et al., and Wang et al.[3 4 6] We were unable to determine how relative risks (or odds ratios) of 0.77 (95% CI 0.71–0.84) for Krishnamachari et al. and 0.34 (95% CI 0.24–0.48) for Xu et al. were derived from the adjusted rate ratios and unadjusted risk ratio or adjusted odds ratio in the original articles, respectively.[5 7]

Additionally, Talic et al. assessed the risk of bias for each included study, but did not rate the quality of evidence. According to the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) framework, assessing risk of bias is a component in quality assessment for evidence.[18] Under the GRADE framework, the quality of evidence for RCTs starts at high-quality and observational studies at low-quality evidence and then the ratings could go up or down based on several factors, including risk of bias.[18] As there are potentially substantial biases in the seven observational studies in the meta-analysis, these studies are likely to be low- or very low-quality of evidence. It may therefore not be justified to include any of them in a meta-analysis.

Systematic reviews and meta-analyses on the effectiveness of various NPIs are valuable to provide real-world data evidence. However, the validity of such analyses depends on the quality of the underlying data. The most important contribution of Talic et al. was to identify the serious deficiencies in available data, sadly, more than 18 months into the pandemic. Further high-quality original studies and careful reviews that provide reliable estimates and acknowledge study limitations in the context of appropriate data interpretation are needed to avoid unrealistic expectations or overselling of the effectiveness of some NPIs.[14]

Jingyi Xiao [1], Kevin Escandón [2], Benjamin J. Cowling [1,3]

1. WHO Collaborating Centre for Infectious Disease Epidemiology and Control, School of Public Health, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong Special Administrative Region, China
2. Division of Infectious Diseases and International Medicine, University of Minnesota, Minneapolis, MN, USA
3. Laboratory of Data Discovery for Health Limited, Hong Kong Science and Technology Park, New Territories, Hong Kong Special Administrative Region, China

References
1. Talic S, Shah S, Wild H, et al. Effectiveness of public health measures in reducing the incidence of covid-19, SARS-CoV-2 transmission, and covid-19 mortality: systematic review and meta-analysis. BMJ 2021;375:e068302. doi: 10.1136/bmj-2021-068302 [published Online First: 2021/11/19]
2. Bundgaard H, Bundgaard JS, Raaschou-Pedersen DET, et al. Effectiveness of Adding a Mask Recommendation to Other Public Health Measures to Prevent SARS-CoV-2 Infection in Danish Mask Wearers : A Randomized Controlled Trial. Ann Intern Med 2021;174(3):335-43. doi: 10.7326/m20-6817 [published Online First: 2020/11/19]
3. Doung-Ngern P, Suphanchaimat R, Panjangampatthana A, et al. Case-Control Study of Use of Personal Protective Measures and Risk for SARS-CoV 2 Infection, Thailand. Emerg Infect Dis 2020;26(11):2607-16. doi: 10.3201/eid2611.203003 [published Online First: 2020/09/16]
4. Lio CF, Cheong HH, Lei CI, et al. Effectiveness of personal protective health behaviour against COVID-19. BMC Public Health 2021;21(1):827. doi: 10.1186/s12889-021-10680-5 [published Online First: 2021/05/01]
5. Xu H, Gan Y, Zheng D, et al. Relationship Between COVID-19 Infection and Risk Perception, Knowledge, Attitude, and Four Nonpharmaceutical Interventions During the Late Period of the COVID-19 Epidemic in China: Online Cross-Sectional Survey of 8158 Adults. J Med Internet Res 2020;22(11):e21372. doi: 10.2196/21372 [published Online First: 2020/10/28]
6. Wang Y, Tian H, Zhang L, et al. Reduction of secondary transmission of SARS-CoV-2 in households by face mask use, disinfection and social distancing: a cohort study in Beijing, China. BMJ Glob Health 2020;5(5) doi: 10.1136/bmjgh-2020-002794 [published Online First: 2020/05/30]
7. Krishnamachari B, Morris A, Zastrow D, et al. The role of mask mandates, stay at home orders and school closure in curbing the COVID-19 pandemic prior to vaccination. Am J Infect Control 2021;49(8):1036-42. doi: 10.1016/j.ajic.2021.02.002 [published Online First: 2021/02/13]
8. Escandón K, Rasmussen AL, Bogoch II, et al. COVID-19 false dichotomies and a comprehensive review of the evidence regarding public health, COVID-19 symptomatology, SARS-CoV-2 transmission, mask wearing, and reinfection. BMC Infectious Diseases 2021;21(1):1-47.
9. Cowling BJ, Leung GM. Face masks and COVID-19: don’t let perfect be the enemy of good. Eurosurveillance 2020;25(49):2001998.
10. Frieden TR, Cash-Goldwasser S. Of Masks and Methods. Ann Intern Med 2021;174(3):421-22. doi: 10.7326/m20-7499 [published Online First: 2020/11/19]
11. Glasziou PP, Michie S, Fretheim A. Public health measures for covid-19. BMJ 2021;375:n2729. doi: 10.1136/bmj.n2729 [published Online First: 2021/11/19]
12. Abaluck J, Kwong LH, Styczynski A, et al. The Impact of Community Masking on COVID-19: A Cluster Randomized Trial in Bangladesh 2021. Available from: https://www.poverty-action.org/publication/impact-community-masking-covi....
13. Brainard J, Jones NR, Lake IR, et al. Community use of face masks and similar barriers to prevent respiratory illness such as COVID-19: a rapid scoping review. Eurosurveillance 2020;25(49):2000725.
14. Brosseau LM, Ulrich A, Escandón K, et al. COMMENTARY: What can masks do? Part 2: What makes for a good mask study — and why most fail 2021. Available from: https://www.cidrap.umn.edu/news-perspective/2021/10/commentary-what-can-....
15. Cheng Y, Ma N, Witt C, et al. Face masks effectively limit the probability of SARS-CoV-2 transmission. Science 2021 doi: 10.1126/science.abg6296 [published Online First: 2021/05/22]
16. van den Berg P, Schechter-Perkins EM, Jack RS, et al. Effectiveness of 3 versus 6 ft of physical distancing for controlling spread of coronavirus disease 2019 among primary and secondary students and staff: A retrospective, statewide cohort study. Clin Infect Dis 2021
17. Vokó Z, Pitter JG. The effect of social distance measures on COVID-19 epidemics in Europe: an interrupted time series analysis. GeroScience 2020;42(4):1075-82.
18. Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 2011;64(4):383-94.

Competing interests: No competing interests

27 November 2021
Benjamin J Cowling
Professor
Jingyi Xiao, Kevin Escandón
The University of Hong Kong
7 Sassoon Road, Pokfulam, Hong Kong