In a tropical forest, the dry season gives way to the rainy season, and with the weather, the rules of life change: some animals go into hibernation, while others come out of hiding. Likewise, in complex systems, from climate to economics, the links between processes are not set in stone. They depend on a hidden "regime" that itself switches from time to time. For a long time, scientists either assumed these links were constant or knew in advance when and how the regime would change. The new work takes a different path: learning both at once, with no labels at all.
The method, named Regime-PCMCI, works like a patient observer who first hypothesizes which regime the system is in at each moment, then checks how well those regimes explain the data. It alternately refines two things: the partition of time into regimes and the causal links within each regime. First, the PCMCI algorithm estimates which variables influence which, using conditional independence tests. Then, for each time point, it selects the regime whose model best predicts the observed values, subject to a constraint on the total number of switches. The process repeats until convergence. To avoid getting stuck in local minima, multiple random initializations are run.
On synthetic data with two regimes, the method correctly recovered the regimes in an average of 94% of time steps for most scenarios, including changes in the sign of the link, its direction, and its lag. The hardest case was when only the strength of the link changed between regimes: accuracy dropped to 57%. For ten variables and two regimes, the method maintained high accuracy: the fraction of correctly identified links was 0.94, and regimes were recovered with an error of 11.7%. This is like an observer who, looking at the annual cycle of temperature and precipitation, figures out the existence of summer and winter without knowing about them in advance.
On real data of El Niño and Indian rainfall, the method, without external information, identified two seasons: a summer regime with a strong negative link (coefficient −0.4) and a winter regime with almost no link, consistent with climatology. This means that the warm current in the Pacific indeed suppresses rain in India in summer, but its influence vanishes in winter. The algorithm, with no hints, split the year into two seasons corresponding to the summer monsoon and winter, even though it was given only the time series of El Niño and rainfall.
Returning to the tropical forest image: we now have a tool that not only notes that in the rainy season some links strengthen and others weaken, but also finds those seasons from the data on its own. The work shows that one can simultaneously learn hidden regimes and causal structures without requiring prior knowledge of the regimes. This extends the applicability of causal analysis to a broad class of nonstationary systems, from climate to economics and neuroscience.
In the climate example, the method, without any hints, split the year into two seasons corresponding to the summer monsoon and winter, even though it was given only the time series of El Niño and rainfall.
The idea of hidden regimes governing a system's behavior evokes Clifford Simak's 'Way Station', where different states of reality replace one another, and the observer sees only their consequences.