An orchestra of a hundred musicians plays without a conductor. As long as the score is familiar, each player confidently follows their part, and together they sound harmonious. But when the composer changes the rhythm, discord creeps in: some instruments keep playing the old tune, others try to adapt, and a few fall silent. This very disagreement among ensemble members turned out to be the most honest signal that the model has stepped outside its familiar world.
A neural network predicting El Niño faces two kinds of uncertainty. The first, aleatoric, is the randomness inherent in the system: the ocean and atmosphere are chaotic, and even a perfect model cannot state the exact water temperature two years from now. The second, epistemic, is the model's ignorance: the network has not seen such situations in its training data and therefore wavers in its estimates. Until now, it was thought that for climate forecasts, tracking the first type sufficed, since it reflects the nature of the phenomenon. But as the climate began to change, this logic broke down.
The researchers trained three ensembles, each with a hundred neural networks. The first ensemble saw only pre-industrial climate from 1850 to 1949, the second saw history up to 2024, and the third saw almost to the end of the 21st century. All three ensembles were then asked to predict El Niño 24 months ahead under climate shift. The result surprised them: when forecasts worsened, epistemic uncertainty, that is, disagreement among the networks, grew steadily. Aleatoric uncertainty, on the other hand, decreased for long horizons. A network encountering an unfamiliar situation did not admit its ignorance; instead, it became overconfident, like a musician who does not notice the orchestra is out of tune and keeps playing louder than before.
The practical takeaway is that for El Niño forecasts in a changing climate, both types of uncertainty need to be monitored. A single network can lie with a confident face, but an ensemble of a hundred networks honestly shows where it is guessing. This is similar to how an experienced listener distinguishes complex music from poor playing: if the musicians diverge, the score is unfamiliar to them. In operational forecast centers, such a signal would allow early warnings to users about forecast unreliability, rather than issuing false confidence.
An orchestra without a conductor comes full circle. When the music changes, disagreement among musicians becomes not a sign of weakness but a map of unfamiliar territory. By listening to this noise, one can discern where knowledge ends and randomness begins. For neural networks predicting climate, this distinction now carries practical weight.
The ensemble of 100 neural networks in this work is ten times larger than typical ENSO forecasting ensembles, and this size is exactly what allowed the two types of uncertainty to be reliably separated. Interestingly, for short forecasts (up to 6 months), aleatoric uncertainty does increase slightly under shift, but this signal is weaker than that of epistemic uncertainty.