
OPINION: The Pandemic of Incomplete Data
by Dave Cox
Professor Neil Ferguson of the Imperial College of London was, until very recently, an advisor to the UK government for its COVID-19 response until he resigned that position on May 5th for breaking his own lockdown recommendations to hang out with his girlfriend. He must really think those restrictions are important for everyone’s safety, right? His models at Imperial have also been the foundation for many governments around the world to make their decisions about how to handle the virus. The model used for COVID-19 was developed in 2005 and has essentially gone unchanged in the 15 years since. In addition, Ferguson is the ONLY one allowed to analyze the data, despite having a large team at his disposal at the college. While that’s not necessarily nefarious, it is incredibly arrogant at a minimum, which brings with it its own inherent issues.
How have Ferguson’s prestigious models performed in the past? (He has won awards for at least one of them). I’m glad you asked!
FERGUSON’S PREVIOUS MODELS*
YEAR | DISEASE | PREDICTED DEATHS | ACTUAL DEATHS |
2001 | Foot and Mouth Disease (UK) | 150,000 | 200 |
2002 | BSE/Mad Cow Disease (UK) | 50,000 | 177 |
2005 | Bird Flu | 200 MILLION | 282 |
2009 | Swine Flu (UK) | 65,000 | 457 |
2020 | COVID-19** | 1.5-2.0 Million (US) | 83K (May 12, 2020) |
When questioned about the disparity in numbers in his COVID-19 model, his reply on April 16, 2020 was, “I much prefer to be accused of overreacting than under-reacting. We do not have a crystal ball.” We do not have a crystal ball?? Then stop being a crystal ball salesman!!
I have also seen people defending Ferguson’s model by stating that his doomsday predictions in the examples above led to actions being taken (X), so clearly his predictions led to the actions that ultimately lessened the impact (Y). That may be the case, but it might not. It ignores the concept of Post Hoc Fallacy: Since event Y followed event X, event Y must have been caused by event X.
Modeling alone is extremely risky. It requires a lengthy list of assumptions to be built into the formulas. Simply put, assumptions create a lot of noise when trying to confidently or accurately predict the future. Even the greatest model ever developed would likely not pass a statistically based validation. Assumptions are inherently biased, not necessarily in a malicious manner, but biased nonetheless because they’re nothing more than an educated guess that forces multiple manipulations of the data processing (think: garbage in/garbage out). If you pile a stack of those into a model (and you must by nature of the process), each level of bias would add another layer of “noise” that compounds itself each time something is off in an assumption. For many years, I have developed and worked with complex statistics every day in the medical device industry, and have done the same in the automotive and defense industries as well (nearly 30 years total). Modeling can be used as a very high level “stab” at what we think might happen, but the decisions must to be made, supported, and justified, by actual proven statistical methods. Companies would never get a product to market using modeling alone. Never. To further that narrative, in today’s world, we could even argue that mass shutdowns, universal shelter in place orders and the upcoming “new normal” (whatever that is going to look like) are all “products” that have been taken to market based solely on predictive modeling.
Predictive Modeling is a still a potentially great tool when it’s used in conjunction with multiple additional tools, including a constant feedback loop to make adjustments based on real data that is coming in – a comparison of the real data to the model data to continuously improve the assumptions and make it more accurate. You also have to look at factors that models do not take into account; like psychology, unemployment, mental stress, economics, domestic violence, suicide, etc., that can’t be accounted for in a laboratory setting. This is why political leaders need to look at the entire picture and make holistic decisions that balance all facets of this very complex situation. Otherwise, wouldn’t we just have epidemiologists as our political leaders all over the world if we believe that disease is the most substantial threat to our existence? We don’t, because that would be a short-sighted and incredibly incomplete view of this crisis that would cause very real, measurable, and catastrophic collateral damage.
This is also why the, “So you think you’re smarter than an epidemiologist? You don’t have any business talking about this topic…” argument doesn’t hold water. You don’t have to be the smartest person in the room to understand that this is not a one-faceted crisis and that all aspects must be thoughtfully considered when making life and death decisions on an unprecedented scale. You need the economists, the mental health experts, the politicians (gasp!), physicians, labor leaders, small and large business owners, and more in order to understand the complete impact of these decisions. You’ve heard the old saying that “Knowledge is Power,” right? Well, willfully ignoring ALL of the knowledge as it relates to this virus is not only irresponsible, but downright dangerous. Ignoring all of the available knowledge is also choosing to live in fear rather than facing it head on and making informed decisions based on the overall calculated risk.
Stay safe. Work hard. Don’t be a dick.
*There are multiple citations available for these numbers