How to Find Hidden Seasons in Data ✎

Original: "Reconstructing regime-dependent causal relationships from observational time series"
arXiv:2007.00267 · 2020-07-01 · arXiv non-exclusive · independent analysis · 2 min
The Regime-PCMCI method learns to simultaneously uncover hidden regimes of a system and the causal links within each regime.

In a tropical rainforest, the dry season gives way to the rainy season, and with the weather, the rules of life change: some animals go into hibernation, while others come out of hiding. Similarly, in complex systems, from climate to economics, the links between processes are not set in stone. They depend on a hidden "regime" that itself switches from time to time. The new work offers a way to learn both things simultaneously without any labels.

Previously, causal links were assumed constant over time, and if they changed, one needed to know in advance when and how. Now, from time series alone, one can simultaneously identify hidden regimes and the causal links within each of them.

The method works as follows: first, it hypothesizes which regime the system is in at each moment, then it checks how well these regimes explain the data. It alternately refines two things: the division of time into regimes and the causal links within each regime. On real data of El Niño and Indian rainfall, the method, without external information, identified two seasons: a summer regime with a strong negative link (coefficient −0.4) and a winter regime with almost no link. This means that the warm current in the Pacific Ocean indeed suppresses Indian rainfall in summer, but in winter its influence vanishes. The algorithm, without any hints, split the year into two seasons corresponding to the summer monsoon and winter, even though it was given only the time series of El Niño and rainfall.

Returning to the tropical rainforest image: we now have a tool that not only detects that in the rainy season some links strengthen and others weaken, but also finds these seasons from the data on its own. The work shows that one can simultaneously learn hidden regimes and causal structures without requiring prior knowledge of the regimes. This extends the applicability of causal analysis to a broad class of non-stationary systems, from climate to economics and neuroscience.

In the climate example, the method, without any hints, split the year into two seasons corresponding to the summer monsoon and winter, even though it was given only the time series of El Niño and rainfall.

The idea of hidden regimes governing a system's behavior evokes Clifford Simak's 'Way Station', where different states of reality replace one another, and the observer sees only their consequences.

Concepts
linear regression regime shift Dynamical Systems stochastic dynamical system Fermi–Dirac statistics autocorrelation function time derivative temporal entanglement correlation granger causality probabilistic model parameter estimation Jin-Timmermann model biological rhythm modeling seasonal forcing monte carlo simulation emergent complexity modeling correlation coefficient real time control