At least as important as carving out the time to read new studies, it is important to develop the skills to critically appraise new studies. The last few years may not be different than any previous decade as far as the quantity of biased, half-baked, and simply fraudulent research published, but the lack of practiced skill in evaluation is on prominent display. While examples of bias across pandemic-related topics are not restricted to any single point-of-view, a recent Cochrane Review publication has yet again refreshed a persistent topic of disagreement: the efficacy of masks and other non-pharmaceutical interventions to prevent the spread of respiratory viruses.¹ In a world of mostly transient obsessions with hydroxychloroquine, ivermectin, and various and sundry vitamins, the mask provocations remain.

Use of the PICO Structure

The best approach to understanding the outcome of this Cochrane Review is first to understand the methods of a systematic review. Using the “population, intervention, comparison, and outcomes” (PICO) structure, a research question is posed. Certain types of evidence are then chosen for inclusion in the review, ranging across observational, retrospective, prospective, or randomized controlled trials (RCTs). Specific medical literature databases are searched for articles meeting criteria, and distilled down into the a relevant cohort. Finally, the results are pooled and analyzed together to generate an estimate of treatment effect, ostensibly with greater certainty and across a broader population than any individual study.

At its foundation, therefore, the inputs determine the outputs. The ultimate conclusions are shaped by rules governing which studies are included, and, by definition, excluded. With all the levers available for manipulation, it becomes clear the scales can be tilted to produce objective results aligned with a particular chosen conclusion.

Assessment of Biases

The Cochrane Review methodology does not prevent erroneous results, rather, it governs primarily the structure of the analysis and reporting. Each included study is required to undergo an assessment of biases. These biases include assessments of threats to balanced enrollment, outcomes assessment, and completeness of results. Likewise, each study included is formally tabulated to describe its relationship to the initial PICO question, and the quality of its measurement of the effects. Unfortunately, these formal processes still do not prevent inclusion and pooling of largely irrelevant results from poorly conducted trials.

For example, a Cochrane Review of parachutes to prevent death from jumping from an aircraft could be proposed. There is a great deal of observational evidence of poor outcomes absent parachute use, but little in the way of randomized-controlled trials. However, a structured search restricted to RCTs alone would uncover one such trial.² The risk of bias assessment would identify concerns regarding some elements in this trial. However, because this RCT of parachute use to prevent death failed to show a difference between groups, this would become the unavoidable output of such a Review. This is, naturally, akin to madness, and similar to the flaws misleading those who amplify this most recent review concerning masks.