A translator who jumps at every rustle in the hall starts mixing up words by the end of the meeting. Their confidence in being right grows, while their accuracy drops. That is roughly how poorly calibrated probabilistic forecasts behave: the model claims a high probability for an event, but reality disagrees. The new work offers a way to retrain such a translator so that they pay for their own mistakes.
The authors build a flexible model that estimates the probability density of the transformed forecasts. If the model is perfectly calibrated, the transform yields a uniform distribution, like a fair die. Deviations from uniformity show where forecasts are systematically overconfident or underconfident. A key feature is that the method does not require data independence, so it applies to time series with dependencies.
The recalibrated system shows a uniform expected distribution of the transformed forecasts, a hallmark of perfect calibration. In a bet against the original system, the new system consistently wins, and the gain is predictable via entropy measures. In a laboratory experiment with a nonlinear circuit, they achieved a significant improvement in calibration over the baseline model. For seasonal El Niño forecasts, the method also proved superior, despite strong autocorrelation in the data.
Previously, it was unclear how to recalibrate probabilistic forecasts when data are not independent and identically distributed. Now there is a method that estimates the distribution of transformed forecasts via a flexible model and allows predicting the gain in a bet against the original system. This links calibration to information entropy and game theory, opening the way to new methods for evaluating forecasting systems.
A translator who has learned not to be distracted by rustles translates accurately again. Their confidence now matches reality, and they can calmly bet against their former self, knowing they will win. In the same way, a recalibrated forecast becomes honest: it does not promise more than it can deliver, and it is ready to answer for its words.
The idea of using bets to test forecasts dates back to the work of Bruno de Finetti, who believed that probability can be measured through fair bets. This work takes that principle to a practical recalibration scheme.