Every generation of emergency physicians can tell a whiplash-inducing story of a medical reversal from their training, whether it be measuring pulmonary artery wedge pressures, the rise and fall of drotrecogin alfa, early goal-directed therapy for sepsis, or high-dose steroids for spinal cord trauma. Many of these therapies were diffused widely across practice before falling out of favor, and while each individual instance offers specific lessons, the larger question must be: why does this keep happening? 

The answer is, unfortunately, obvious, and subsequently disheartening for the future of emergency medicine (EM). There is a foundational issue with the quality of the evidence that informs clinical practice. We have seen recent articles in the Annals of Emergency Medicine describing trials in EM as “fragile,” but a new study evaluates the evidence base in its totality.¹ 

Producing what the authors term an “umbrella review,” this study sought out the top level of the evidence-based medicine pyramid for EM: systematic reviews and meta-analyses.²  The authors identified 431 eligible meta-analyses in EM in their search, comprised of 3,129 individual studies, the majority of which were randomized controlled trials. The authors rigorously evaluated each meta-analysis to determine whether there were sources of potential systematic bias affecting the reliability of the outcomes measured. 

First, each meta-analysis was evaluated for signs of publication bias. Publication bias colors the evidence base as consequence of suppression of a subset of trial results. This may happen as an inadvertent effect of academic publishing, as null results may be less appealing to medical journals and have a lower likelihood of acceptance. However, publication bias may also relate to an intentional discarding of negative results by those with professional or financial conflicts of interest. In either instance, the use techniques such as funnel plots and Egger’s regression may reveal signs of potential bias. 

The authors then exercised a more novel approach to the presence of bias: a test for excess significance. In a world where randomness ought to rule the day, it is a potential signal of threats to the internal validity of included studies if a disproportionate number reach statistical significance. Given a certain power to detect significant differences, there is an objective statistical likelihood for a pool of studies to each reach that threshold for significance based on their sample sizes. In effect, these authors have formalized “too good to be true” into an evaluation for possible bias.