A student learns physics from a textbook and takes an exam. The first two problems are solved brilliantly, but on the third, they err on something not covered in the book. That's roughly what testing the atmospheric model NeuralGCM looks like, as it was run through three levels of scrutiny: cyclones, El Niño, and anomalous warming.

The model combines a physics core with machine learning that replaces expensive computations of small-scale processes. Instead of simulating every little cloud, the neural network learns from weather data from 2001–2018. Then it is asked about situations not present in that data, such as a uniform 4-degree ocean warming. This tests whether the model has grasped the physics rather than merely memorized statistics.

The model also recognizes El Niño, and under uniform warming, global temperature rises by 0.86 degrees per degree of ocean warming, which is close to classical models. However, in the upper troposphere and lower stratosphere, the model shows excessive cooling in the tropics and excessive warming in polar regions. As a result, the jet stream does not shift poleward as it does in physical models. The reason is simple: the model did not receive carbon dioxide and ozone concentrations as input, so its stratospheric response is unphysical.

The student who learned physics from a textbook passed the exam, but not with flying colors. NeuralGCM has shown that hybrid models can generalize beyond their training data, something purely neural network models could not do. But the stratospheric errors remind us: before trusting such models for climate projections, we need to understand the minimum physics they require. The exam continues.