SPELL OUT YOUR LOSS FUNCTION.

That's an obscure and too often neglected part of statistical inference, but it matters.  It's the way the investigator tells readers whether he weights all errors of inference equally, or whether one sort of failure is less desirable than another.  Here, from an old post, is former British prime minister Tony Blair, making the case for ousting Saddam Hussein.

Can we be sure that terrorism and WMD will join together? If we are wrong, we will have destroyed a threat that, at its least is responsible for inhuman carnage and suffering. That is something I am confident history will forgive. But if our critics are wrong and we do not act, then we will have hesitated in face of this menace, when we should have given leadership. That is something history will not forgive.

Nearly twenty years on, perhaps "history" has not forgiven the failure to find those weapons, although somehow the greater error was the hope of the nation-builders that Iraq, and the Levant more generally, would embrace the institutions of the Western Enlightenment.  (Sadly, no.)

The logic, though, is the same.

The first couple of lines spell out his loss function in the case of a false positive (Saddam neither has nuclear, chemical, nor biological weapons, nor provided them to terrorists, despite what Intelligence concludes) and his argument that the consequences of removing Saddam are still to be preferred to leaving Saddam alone. The remainder of the passage spells out his loss function in the case of a false negative (Saddam deceives enough people into believing that he has no such weapons, then uses them himself or finds an agent to do so). His loss function is not symmetric with respect to false positives (small consequences from acting anyway) and false negatives (large consequence from failure to act.) No two-tailed t-test here.

Although we're not currently in a world of hypothesis testing and Type I or Type II errors, when we're modelling the effects of a pandemic, we're still in an environment of asymmetric loss functions.  Here, read off Newmark's Door, is a financial manager, Alan Beard, setting the stage.

Some of the current hysteria for the coronavirus has undoubtedly been fed by similar sophisticated healthcare computer models, most notably the Imperial College’s doomsday predictions which indicated that as many as 500,000 deaths could occur in Britain and over 2 million in the U.S. After several public policy actions had already been put into place by governments — at least in part because of computer models such as the credible scientific U.K. forecast — the projections were abruptly revised down to show fewer than 20,000 U.K. deaths were likely to occur from the virus and 200,000 in the U.S.

Those of us who use computer modeling on a daily basis to assist in our analysis know how dynamic projections can be. These types of dramatic changes are routine when there are significant changes to the inputs that go into the models. In fact, that is the reason for modeling: determining the parameters of the inputs and how they affect the underlying outcomes.

His essay focuses on an important part of extrapolation, namely recalibrating and recalculating on the basis of new evidence. You don't want programmed trading algorithms to jump out the windows (a 1987 stock market crash joke) or to shut down prosperity for fear of an ice age.

It was refreshing to have a credible scientist like Dr. Deborah Birx on the White House Coronavirus Task Force gently scold those who are promoting catastrophic outcomes from computer models when the evidence no longer supports it.

A good financial analyst knows that a computer model is dynamic and needs constant adjustments as new data becomes available. As the numbers, or data, come in differently than what was originally anticipated, models have to be changed — and sometimes even redone drastically — because the original assumptions weren’t correct or were based on limited or faulty data.

This becomes obvious when observing the predictive values of climate change models developed over the last several decades.

A few years ago a prominent science writer noted that “most temperature records show that since 1998 the [climate] models and observed average global temperatures have parted ways.  The temperature in the [climate] models continue to rise, while the real climate has refused to warm up much during the last 15 years.” One former NASA scientist goes as far as to say “global warming projections have a large element of faith programmed into them.” These are understatements when reviewing the reporting on global warming predictions over the last several decades. The “settled science” isn’t as exact as some scientists and most reporters would have us believe.

The same rush to a foregone conclusion is happening with the coronavirus. If the media had paid attention to what the scientific advisors have been saying from the beginning, they wouldn’t have been surprised by their skepticism with the computer models. In almost every press conference the doctors have acknowledged the lack of good data on this virus and that it would take time to make more accurate projections or models. Not unlike the financial models bankers work on, the coronavirus modeling needs better inputs to test the assumptions — and most probably significant adjustments to better project future outcomes.

To make dramatic policy decisions solely on the basis of very imperfect models can lead to bad life or death decisions.

As more data becomes available, the coronavirus models will be fine-tuned and will become better tools for policy makers. It will be important that — as this occurs — biases and faulty assumptions are teased out of the models to ensure their accuracy and to better inform policy makers. While this is all happening in a matter of weeks for the coronavirus, the same should apply to climate change models that have become almost sacrosanct over the last couple of decades.

Since science has now become the ultimate arbiter of how we determine most public policy debates adjudicated in the press, coronavirus forecasting is a cautionary tale to reporters who easily confuse what is science and what is conjecture.

We could start by noting that "settled science" is an oxymoron: the whole point of the scientific enterprise is the testing and destruction of hypotheses.  I really don't want to get into the weeds of journalists Pushing Narratives: we have Pajamas Media and the like for that.

Rather, I want to counsel calm in the face of those high death estimates now being revised.  Wouldn't that be consistent with inference in the face of a loss function?  Overestimate, then revise down, is probably a better place to be than underestimate, then revise up, particularly with the chattering classes making their case for Fixing It Now.

In Washington's Free Beacon, C. F. Lehman has additional methodological points that will reward careful study.  I want to highlight one sentence: by all means, go there and read the rest if it's the competing (perhaps complementary) models and recalibrations that interest you.  "Although a nation eager to return to work demands certainty, models based on uncertain data can't provide it—in fact, putting too much stock in any one of them is a certain recipe for disaster."  That, too, is a loss function argument, one that radio chatterbox Ben Shapiro echoes.  (He argues well, particularly with the written word.  I just cannot listen to his delivery for long.)

Americans are not going to stay home for months on end, and they certainly will not do so on the basis of ever-evolving models, especially as statistics roll in that look like the lower-end model estimates in terms of death and the upper-end estimates in terms of economic damage. We need transparency and honesty from our scientific experts -- we need to know what they know, what they don't and when they hope to know what they don't. We're grown-ups, and we're willing to follow their advice. But they need to start answering serious questions, or they will fall prey to the same lack of institutional faith to which all other American institutions seem deeply prone.

That is, once the epidemiologists get a handle on how contagious the virus is, how lethal it is, and how many people have already been infected, and, if currently healthy, how subject are they to a second bout, Mr Shapiro wants their advice to be of the form, "This is the kind of flareup we expect to see in currently lightly-hit areas, these are the consequences of reopening the parks and beaches, and reopening the baseball stadiums is not advisable unless ..."  They're not yet there.

While projections like these are imprecise, they're useful to policymakers and hospital leaders trying to prepare for surges. The model was designed to give hospitals an idea of how quickly they need to expand capacity and by how much.

Researchers at [the Institute for Health Metrics and Evaluation] make frequent updates to the model based on newly available data, and some of those changes have resulted in drastic shifts. The latest major update, made Sunday, shows fewer people dying over a shorter period of time, the model's lead researcher Chris Murray says.

However, he cautioned that when social distancing measures lift, outbreaks could spark up again.

"If you ease up prematurely the epidemic can rebound right back to the level we are at now in a matter of weeks," Murray says. "So the potential for rebound is enormous if we let up on social distancing."

Unfortunately, there is always a margin of error.

“No model is perfect, but most models are somewhat useful,” said John Allen Paulos, a professor of math at Temple University and author of several books about math and everyday life. “But we can’t confuse the model with reality.”

The best you can do, dear reader, is improve the precision of the mean fit, which involves reducing the margin of error.  Too much of the reporting and the briefing focuses on those solid lines: actual and projected cases.  Those clouds that suggest the upper and lower bounds on the projection matter.  When you see the clouds becoming less tall, then perhaps the policy makers will start talking more seriously about reopening life as we used to know it.