Diffusion Models Learn Climate from Sparse Data ✛
Context
El Niño and La Niña govern weather over much of the planet, but detailed satellite and buoy measurements of ocean surface temperature are available only since the mid-20th century. Over that time, about seven hundred monthly mean maps have accumulated, whereas neural networks with millions of parameters typically need thousands of examples to avoid overfitting. Synthetic data generated by linear inverse models provide a known low-dimensional structure on which one can honestly test what a diffusion model has learned. Such a controlled testbed allows separating genuine understanding of climate variability from mere memorization of the training set.
Methods
The authors generated 120,000 monthly sea surface temperature fields for the tropical Pacific using linear inverse models calibrated on ERSSTv5 reanalysis data from 1948 to 2022. They used two variants: a standard Gaussian LIM and a nonlinear NG-LIM with a Yeo–Johnson transformation that better captures the asymmetry between warm and cold phases. The diffusion model was trained to predict the clean data field from a noisy version, which, according to the hypothesis, forces the network to focus on low-dimensional structure. To combat data scarcity, they pretrained the model on synthetic fields from 30 climate models of the CMIP6 project, adding an eight-dimensional embedding vector that encodes the identity of each model, and then fine-tuned only this vector on 720 real observations.
Results
When trained on the full synthetic set, the diffusion model accurately reproduces both Gaussian and non-Gaussian structure: the correlation between spatial patterns of the first ten empirical orthogonal functions exceeds 0.9, and the asymmetry maps retain the characteristic horseshoe shape with positive asymmetry in the east and negative in the west. However, ablation over the number of training samples showed that all metrics converge only at 7000 samples, whereas in ERSSTv5 there are only about 720. With such a deficit, the model begins to memorize training examples: the median distance to the nearest neighbor drops below 0.6 of the reference value. Among the three parameterizations of diffusion, only clean-data prediction passes all diagnostics, whereas noise prediction fails completely, and velocity prediction overestimates the effective dimension to 19.6% of explained variance. The strategy with pretraining on CMIP6 and fine-tuning only the embedding achieves pattern correlation above 0.85, asymmetry correlation above 0.9, and a memorization coefficient around 0.95, outperforming both standard LIM and NG-LIM on these metrics.
Implications
The work shows that diffusion models can extract physically meaningful low-dimensional structure of climate variability if data are sufficient and parameterization is chosen correctly. But the main conclusion is that observations alone are not enough: one needs an order of magnitude more data than climatologists have at their disposal. The proposed scheme with a model embedding turns the diversity of climate simulations from a source of bias into a resource that compensates for the deficit of real data. This changes the view of how generative models can be used in Earth sciences: not as a replacement for physical models, but as a tool for fishing out structure from sparse empirical samples.
Future development
A natural next step is the transition from unconditional generation, which reproduces only the distribution of states, to conditional generation, where the model generates a future field given an initial state. This would open the way to emulating El Niño dynamics and possibly to forecasting. Richer predictors will be needed: not only surface temperature but also the heat content of the subsurface ocean layer and wind stress. Including higher-resolution ensembles, such as new simulations with explicit convection, in pretraining could further improve the accuracy of reproducing distribution tails.
Impact
The results are important for climate modeling, seasonal forecasting, and risk assessment of extreme events associated with El Niño. The method can be transferred to other modes of variability, such as the North Atlantic Oscillation or the Pacific Decadal Oscillation, where data are even scarcer.
Next steps
The immediate step is to build a conditional diffusion model that, given the current state of the tropical Pacific, generates an ensemble of possible future states. Then its predictive skill must be compared with classical statistical and dynamical models on historical data.
Key open problems
The work directly addresses a fundamental problem in climatology: how to distinguish real modes of variability from artifacts of short observational records. It also contributes to the debate on which machine learning architectures can capture physically meaningful low-dimensional structure and which merely interpolate data. Finally, the proposed embedding fine-tuning method may become a general recipe for transferring knowledge between heterogeneous climate simulations and observations.
The knowledge machine's view
This section is addressed to the author of the paper and their colleagues. Neighbouring works were found by meaning-based search across our archive, not by word overlap. We do not review: we show the field around the work.
What the work does
The authors tested whether diffusion models can learn the low-dimensional structure of El Niño climate variability using synthetic data with a known structure. They found that with a sufficiently large sample, the models accurately reproduce both Gaussian and non-Gaussian structures, but on real data (about 720 months) they start to memorize examples. The solution lies in pretraining on a set of CMIP6 climate simulations followed by fine-tuning only a small embedding vector, which allows extracting more from scarce observations than classical linear models.
Where its strength is
The strength of the work is a controlled testbed on synthetic data where the true low-dimensional structure is known, allowing an honest separation of generalization from memorization. This provides a reliable basis for verifying what the diffusion model has actually learned and paves the way for application to real data.
Where it could go next
- It might be interesting to see how the approach of pretraining on CMIP6 and fine-tuning the embedding transfers to other climate variables, such as precipitation or pressure, where the variability structure is less pronounced. In our archive, there is a related paper where a neural network assembles global weather from pieces, which could inspire ideas for working with spatially inhomogeneous fields.based on: Storm Mosaic: AI Model STRATA Pieces Together Global Weather arXiv:2606.31248
- One could consider applying this to El Niño forecasting: a diffusion model trained on synthetic data could be used to generate forecast ensembles. In the archive, there is an analysis of predicting El Niño a year and a half ahead, which could provide benchmarks for comparison.based on: El Niño Can Be Predicted a Year and a Half in Advance arXiv:2601.12251
- It would be interesting to explore how diffusion models handle learning algebraic invariants in climate data, similar to how a neural network learns from an algebraic skeleton in a tropical diagram. This could offer a new way to test the physical meaningfulness of learned representations.based on: Tropical Blueprint: A Neural Network Learns from an Algebraic Skeleton arXiv:2607.00237
Worth trying
- Test replacing synthetic data with real observations from other regions with longer records for pretraining.
- Check whether adding physical constraints (e.g., energy conservation) improves generation quality on small samples.
- Use a diffusion model to generate synthetic El Niño analogs and assess their impact on training forecast models.
Why this may matter
If diffusion models can reliably extract climate variability from short observational records, this would open the door to more accurate estimates of rare events and improve our understanding of the system's long-term dynamics, which is important for adapting to climate change.
Nearby works in our archive
- El Niño Can Be Predicted a Year and a Half in Advancesimilarity 0.54
- Weather forecast on a single GPUsimilarity 0.54
- Tropical Blueprint: A Neural Network Learns from an Algebraic Skeletonsimilarity 0.50
- Storm Mosaic: AI Model STRATA Pieces Together Global Weathersimilarity 0.50
- Neural Network Charts a Precise Gravitational Map of Galaxiessimilarity 0.50
- Martian Weather, Earthly Playbooksimilarity 0.49
The archive is sparse around this work: the closest review is only 0.54, against a usual 0.64. It looks like few have drilled nearby.
Diffusion models, which now generate realistic faces and landscapes, were originally inspired by non-equilibrium thermodynamics, the science of how systems transition from order to chaos. In this work, the same principle is reversed to recover ordered climate structures from noise.
The idea of training a machine on a multitude of virtual worlds and then tuning it to a single real one echoes Ted Chiang's story "Story of Your Life," where a linguist learns an alien language by relying on many hypothetical grammars to understand one actual one.
Key numbers
- 120,000: synthetic monthly fields used for training
- 720: monthly observations in ERSSTv5 from 1948 to 2022
- 7000: samples needed for convergence of all metrics
- 10: principal components explain 90% of variability
- 0.95: asymmetry pattern correlation for the best model