Weather and climate forecasting has relied on numerical modeling of physical processes for nearly half a century. Such systems are computationally expensive and do not easily absorb new streams of data from satellites and radars. Over the past decade, vast amounts of observations have accumulated, and deep learning has already shown success in specific tasks, such as precipitation nowcasting. However, architectures based on convolutional and recurrent networks impose rigid assumptions about data structure. For a chaotic Earth system, where small changes in initial conditions can drastically alter behavior, these assumptions may not hold. Transformers can capture long-range dependencies, but their direct application to three-dimensional data runs into the quadratic complexity of attention. Earthformer stands apart from its neighbors in the archive: nearby are models for medical segmentation and weakly supervised learning, while direct competitors for Earth system forecasting are almost absent.
The authors build Earthformer as a hierarchical transformer with an encoder and a decoder. The key block is Cuboid Attention: the input tensor of dimensions time × height × width is cut into non-overlapping cubes, and within each cube self-attention is performed in parallel. Different cutting strategies (local or sparse, with shifts) yield different attention patterns. To let cubes exchange information, a small set of global vectors is introduced: they collect statistics from all cubes and distribute them back. The architecture has several hierarchy levels with downsampling and upsampling via convolutional layers and interpolation. The decoder generates the forecast non-autoregressively, directly from positional embeddings. The model is trained on synthetic datasets MovingMNIST and N-body MNIST, where digits move according to the laws of three-body gravity, and then tested on real precipitation and ocean temperature data.
On synthetic data, Earthformer with the Axial pattern and global vectors achieves an MSE of 41.79 on MovingMNIST and 14.82 on N-body MNIST, outperforming all six baseline models. On the real SEVIR dataset for precipitation forecasting, the model obtains CSI-219 = 0.1791, notably higher than Rainformer (0.0831) and UNet (0.0577). On the ICAR-ENSO dataset for El Niño forecasting, Earthformer shows a correlation C-Nino3.4-M = 0.7329, surpassing Rainformer (0.7106) and E3D-LSTM (0.7040). Global vectors provide a consistent improvement without increasing computational complexity: the number of parameters grows from 6.6 to 7.6 million, while GFLOPS remain almost unchanged. The hierarchical structure reduces MSE on MovingMNIST from 48.04 to 46.91 at the same depth. Qualitatively, the model predicts digit trajectories in N-body MNIST more accurately, whereas baseline methods blur the image or err in motion.
Earthformer shows that transformers can be adapted to three-dimensional spatiotemporal data without an explosive growth in computation. Unlike neighboring works that use convolutional or recurrent networks, here attention is restricted to cubes, and global vectors connect local blocks. This allows capturing long-range dependencies in a chaotic system while maintaining linear complexity with respect to cube size. The work sets a new benchmark for Earth system forecasting architectures: the Axial pattern with global vectors proves most effective. For data science, it exemplifies how a universal building block can be tailored to a specific problem without losing generality.
The model is currently deterministic and does not estimate uncertainty, which can lead to blurry forecasts. A probabilistic extension of Earthformer would be a natural next step. Another direction is incorporating physical knowledge: the model is purely data-driven and does not use conservation laws or dynamical equations. Hybrid approaches that combine neural networks with numerical modeling are already showing promise. Automatic search for cuboid attention patterns via Neural Architecture Search is also possible. Scaling to global, higher-resolution data is another interesting avenue.
The results directly concern meteorology and climatology: more accurate forecasts of precipitation and El Niño affect agriculture, energy, and disaster early warning. The architecture may prove useful in other domains with spatiotemporal data, such as satellite imagery analysis or medical imaging.
Immediate next steps include training Earthformer on ensemble forecasts to estimate uncertainty and adding physical constraints to the loss function. It is also worth testing the model on other climate variables, such as pressure and wind.
For Earth system forecasting, the open question remains how to account for chaos and sensitivity to initial conditions. Deterministic models tend to average over possible scenarios, losing rare extreme events. The lack of widely accepted uncertainty metrics hinders comparison of probabilistic forecasts. Earthformer does not solve these problems but provides an efficient base architecture on which probabilistic and physics-informed extensions can be built.
The N-body MNIST dataset makes digits move according to the laws of three-body gravity, a chaotic system where the slightest change in initial velocity leads to a completely different trajectory. The model must guess where the digit '0' will fly, even though the exact solution is unstable.
The idea of predicting a chaotic system from limited observations echoes Liu Cixin's 'The Three-Body Problem', where a civilization tries to compute the motion of a planet in a system of three suns. Earthformer, like the novel's protagonists, confronts fundamental unpredictability but learns to give the best possible forecast.