A Forecast That Knows When It's Wrong ✎

Original: "Probabilistic Recalibration of Forecasts"
arXiv:1904.02855 · 2019-04-05 · arXiv non-exclusive · independent analysis · 1 min
Researchers have found a way to correct overconfident forecasts so they honestly reflect how sure they are.

An interpreter in high-stakes negotiations sometimes starts to trust themselves too much and mistranslates words, even though they are sure they are right. Some forecasts are like that: they say "there is an 80% chance of rain," but in reality rain occurs only half the time. Researchers have figured out how to correct such forecasts and make them honest.

The new method does not require data to be independent. So it can be used for weather forecasts or other events that depend on what happened before.

Previously, it was unclear how to correct forecasts when data are dependent. Now researchers have built a flexible model that looks at how much forecasts deviate from honest behavior. If a forecast is perfectly honest, its errors are spread out evenly, like a fair die. The new method adjusts the forecast so that it resembles such an honest die.

An interpreter who has learned to double-check themselves translates accurately again. Their confidence now matches reality. Likewise, a corrected forecast becomes honest: it does not promise more than it can deliver, and it is ready to stand by its word.

The idea of using bets to test forecasts dates back to the work of Bruno de Finetti, who believed that probability can be measured through fair bets. This work takes that principle to a practical recalibration scheme.

Concepts
probabilistic forecast ensemble forecast gaussian process probabilistic model posterior distribution normalizing flows prediction bootstrap jackknife resampling Gaussian distribution gibbs sampling autoregressive model prediction error prognosis probability theory regularisation Stochastic modeling permutation test likelihood ratio test Bayesian inference