Thread
People are sharing this thread around a lot, and I have to say that it's extremely unfair and mostly incorrect when it comes to the specifics of the Cochrane review
Of course, it's worth noting that the public statements of the lead author of the review are completely at odds with what the paper found, and the NYT opinion piece is also nonsense
That being said, if we're going to criticize research, it is EXTREMELY IMPORTANT that our critiques are accurate, otherwise we end up muddying the waters and it just becomes a nightmare for everyone involved
The mistakes in the thread are numerous. For example, this study is portrayed as one finding that masks work (the next tweet says that it SUPPORTS masks) with the implication that the Cochrane authors have done something nefarious here 
This is the table of results from Aiello et al vs the Cochrane meta-analysis. I have put red boxes around the estimate of face masks vs control which the review used
They are the same
They are the same
The thread then goes on a lengthy argument that one of the trials is totally inappropriate, insinuating that the authors have done something dodgy, before acknowledging that actually they extracted the correct numbers
The thread then implies that the authors have somehow fiddled with the weighting of the studies in their analysis
This is flatly untrue. The weights are calculated based on the inverse of the variance of each trial, which means bigger trials generally have higher weights
This is flatly untrue. The weights are calculated based on the inverse of the variance of each trial, which means bigger trials generally have higher weights
I quickly redid the analysis presented in the Cochrane review using Stata 15.1. I get precisely the same results using an inverse-variance model as the Cochrane review got in Revman. This requires no active weighting on my part 
This is a simple statistical misunderstanding. Meta-analyses produce a weighted mean average, and in this case that weighting is derived using the equation 1/variance for each study, where variance is calculated from standard error. You can redo this in Excel if you want
The thread then implies that the Cochrane reviewers have falsified the results of MacIntyre 2009
Except, the results clearly state that 14/186 vs 3/100 in the mask and control groups have lab-confirmed illness, giving a relative risk of 2.51
Except, the results clearly state that 14/186 vs 3/100 in the mask and control groups have lab-confirmed illness, giving a relative risk of 2.51
I could go on. Suffice to say that the thread appears to be wrong on virtually every major criticism, and makes a number of wildly inappropriate accusations
That being said, I ALSO THINK THE REVIEW HAS ISSUES
It is true to note that aggregating all of these incredibly disparate studies together doesn't necessarily make sense
Can we learn much from the weighted mean of masking during the Hajj in 2013 and during COVID-19 in 2021?
It is true to note that aggregating all of these incredibly disparate studies together doesn't necessarily make sense
Can we learn much from the weighted mean of masking during the Hajj in 2013 and during COVID-19 in 2021?
Moreover, if you only aggregate together studies of community masking conducted since 2020, you find a fairly strong and consistent (if modest) effect
That's an important point!
That's an important point!
In addition, many of the statements by the lead author are absolute gibberish. Most of what he's said publicly about masking is contradicted by his own paper, which is quite a remarkable thing to see
But the argument that the entire review represents some nefarious fakery is, I think, wrong, and the accusations that the authors have concocted numbers wholesale is incorrect. As far as I can tell, the actual process of the review was done to Cochrane's generally high standards
Mentions
See All
Chris Kavanagh @C_Kavanagh
ยท
Feb 28, 2023
Good balanced thread by Gideon.