In February 2022, a number of conservative news outlets reported on a working paper entitled "A Literature Review and Meta Analysis of the Effects of Lockdowns on COVID-19 Mortality" by a group of scholars at Johns Hopkins University. The paper claimed that the lockdowns implemented during the COVID-19 pandemic had little impact on preventing deaths. As this study appeared to run counter to previous reports that found lockdowns saved lives, we wanted to take a closer look at it.
Who is it from? What are the specific claims? What evidence are those claims based on?
A Working Paper, Not Peer-Reviewed Published Study
The first thing we noticed when we examined the actual study, not the media reports covering the study, was that this was a "working paper" by a group of economists, not epidemiologists. A working paper typically refers to a pre-publication study that has not yet undergone a scientific peer-review process. The authors state as much in a brief description at the top of the study:
This opening paragraph contains one other important detail. This study was not endorsed by Johns Hopkins University. While many media outlets presented this working paper as if it was a "Johns Hopkins study," this report would be more accurately described as a non-peer-reviewed working paper by three economists, one of whom is an economics professor at Johns Hopkins University.
Furthermore, the National Post noted that this paper did not come from Johns Hopkins University's Coronavirus Resource Center. Rather it comes from the Krieger School of Arts and Sciences, a branch of JHU unaffiliated with the Coronavirus Resource Center:
"Throughout the pandemic, most COVID research out of Johns Hopkins University has typically come from its Coronavirus Resource Center, an initiative run out the university's world-renowned medical school.
But the new paper, which was drafted by three economists, comes out of the university's unaffiliated Krieger School of Arts and Sciences."
Who Wrote This Paper?
This work was conducted by three economists, not epidemiologists: Jonas Herby, Lars Jonung, and Steve H. Hanke. It's worth noting that Hanke, a senior fellow at the CATO Institute, was at the center of a brief controversy in June 2020 after he erroneously claimed that Vietnam had not reported any COVID-19 data. An open letter from 285 "public health researchers and professionals and concerned citizens" to Johns Hopkins University demanded an apology from Hanke and claimed that his tweet was "more politically driven than evidence based." Hanke later deleted the tweet.
While we can't say if Hanke's political opinions influenced the conclusions of this working paper, he has repeatedly posted messages on Twitter equating lockdowns with fascism.
What is a Lockdown?
While many media reports on this working paper noted that "lockdowns only reduced COVID deaths by 0.2 per cent," this may give readers a false impression of what this working paper actually found. The common definition of a "lockdown" is a mandatory state of isolation. In terms of the pandemic, many people would take "lockdown" to refer to a requirement for people to stay inside their homes (not attending public events, school, going to restaurants, or leaving for any other non-essential reason.)
This paper, however, defines a lockdown as "the imposition of at least one compulsory, non-pharmaceutical intervention (NPI)." This means that this study interprets a mask-wearing requirement as a "lockdown," even if that requirement did not prevent a person from visiting public spaces.
Samir Bhatt, professor of statistics and public health, Imperial College London, said:
“I find this paper has flaws and needs to be interpreted very carefully ... The most inconsistent aspect is the reinterpreting of what a lockdown is. The authors define lockdown as “as the imposition of at least one compulsory, non-pharmaceutical intervention”. This would make a mask wearing policy a lockdown. For a meta-analysis using a definition that is at odds with the dictionary definition (a state of isolation or restricted access instituted as a security measure) is strange.
Professor Neil Ferguson, director of the MRC Centre for Global Infectious Disease Analysis, Jameel Institute, Imperial College London, also found this definition of a lockdown problematic:
This report on the effect of “lockdowns” does not significantly advance our understanding of the relative effectiveness of the plethora of public health measures adopted by different countries to limit COVID-19 transmission. The policies which comprised “lockdown” varied dramatically between countries, meaning defining the term is problematic. In their new report, Herby et al appear to define lockdown as imposition of one or more mandatory non-pharmaceutical interventions (NPIs); by that definition, the UK has been in permanent lockdown since 16th of March 2021, and remains in lockdown – given it remain compulsory for people with diagnosed COVID-19 to self-isolate for at least 5 days.
Which Studies Were Included? Which Excluded?
Another point of concern is that 12 of the 34 studies analyzed in this review were, themselves, working papers. The analysis of 34 included 14 in the field of economics and only one in epidemiology.
Dr. Seth Flaxman, associate professor in the Department of Computer Science, University of Oxford, said (emphasis ours):
Smoking causes cancer, the earth is round, and ordering people to stay at home (the correct definition of lockdown) decreases disease transmission. None of this is controversial among scientists. A study purporting to prove the opposite is almost certain to be fundamentally flawed.
In this case, a trio of economists have undertaken a meta-analysis of many previous studies. So far so good. But they systematically excluded from consideration any study based on the science of disease transmission, meaning that the only studies looked at in the analysis are studies using the methods of economics.
These do not include key facts about disease transmission such as: later lockdowns are less effective than earlier lockdowns, because many people are already infected; lockdowns do not immediately save lives, because there’s a lag from infection to death, so to see the effect of lockdowns on Covid deaths we need to wait about two or three weeks. (This was all known in March 2020 – we discussed it in a paper released that month, and later published in Nature. Our paper is excluded from consideration in this meta-analysis.)
It’s as if we wanted to know whether smoking causes cancer and so we asked a bunch of new smokers: did you have cancer the day before you started smoking? And what about the day after? If we did this, obviously we’d incorrectly conclude smoking is unrelated to cancer, but we’d be ignoring basic science. The science of diseases and their causes is complex, and it has a lot of surprises for us, but there are appropriate methods to study it, and inappropriate methods. This study intentionally excludes all studies rooted in epidemiology–the science of disease.”
Furthermore, nearly half of the studies analyzed (16 of 34) were published in 2020. The most recent study comes from June 2021, meaning that this meta-analysis contains little to no data related to the delta variant, and no data related to omicron.
Prof. Bhatt said:
“Two years in, it seems still to focus on the first wave of SARS-COV2 and in a very limited number of countries ... As I have mentioned it looks at a tiny slice of the pandemic, there have been many lockdowns since globally with far better data, there are many prominent studies that cover the period in question looking at infections ... The list of such studies is very long and suggests a highly incomplete meta-analysis."
The Bottom Line
The viral "Johns Hopkins study" about lockdowns was not the work of Johns Hopkins University, it was not peer-reviewed, and it was not written by epidemiologists. A number of researchers have also taken issue with the methods used in this study.
In a Feb. 4 thread on Twitter, Meyerowitz-Katz dug into the details of this study and found, among other things, that this paper was not peer-reviewed, that the analysis excluded all studies with a counter-factual model — thereby excluding nearly all epidemiological-focused papers — and that it heavily weighted studies that supported their conclusion.
Furthermore, the conclusions of this non-peer reviewed working paper run counter to published studies in academic journals that found lockdowns did prevent COVID-19 deaths. One study, for example, found that lockdown policies helped prevent millions of deaths early in the pandemic. NPR reported:
Solomon Hsiang, director of the Global Policy Lab, says these unprecedented shelter-in-place orders came at an extreme economic cost. Yet when government officials were ordering them, it was unclear exactly how significant the social benefits would be.
"The value of these studies you're seeing today is that they're demonstrating what the benefits of this policy are," Hsiang said in a press call discussing the studies. "They averted tens of millions of additional infections and millions of deaths."