When you look at a thermometer, you see a single number: the temperature. But behind that number hides an unimaginable crowd of molecules, each moving at its own speed. Temperature is a macrovariable, that is, a coarse-grained characteristic that stands in for many fine details. Scientists have long sought such variables to explain complex systems.
The new algorithm does this automatically. It consists of two neural networks that learn simultaneously. The first network compresses ocean temperature data into a few numbers. The second does the same with air pressure data. Then each network tries to predict the full dataset of the other from its own compressed numbers. This trick is like an interpreter who listens to a long conversation and produces a short note, from which another person can reconstruct the entire meaning.
On synthetic data with 64 dimensions, the algorithm found exactly two macrovariables, as intended. It also correctly identified which variable is the cause and which is the effect. Previously, finding such variables was a matter of a scientist's intuition. Now the neural network itself finds variables that preserve all the mutual information between two datasets. This will help extract meaningful patterns in climatology, neuroscience, and other sciences.
The algorithm discovered two known types of El Niño even though it was not told about the phenomenon.
Searching for hidden causal variables echoes Isaac Asimov's heroes trying to derive the laws of psychohistory from the chaos of galactic history.