The Uneven Pulse of Time: How Irregular Measurement Rhythms Skew Conclusions and What to Do About It

Original: "Sampling rate-corrected analysis of irregularly sampled time series"
arXiv:2112.04843 · 2021-12-09 · CC BY 4.0 · 3 min
A method corrects errors in analyzing unevenly spaced time series, separating real changes from artifacts of measurement frequency.

Imagine an interpreter who gets distracted by every rustle in the hall. The more rustles, the more errors in the translation, even though the meaning of the speech hasn't changed. The same happens with data: if measurements are unevenly spaced, standard analytical tools start "hearing" changes that never occurred in the underlying process. Irregular time series are everywhere, from stock tick data to layers of stalagmite in a cave. Direct application of familiar methods is impossible, and attempts to "even out" the series by interpolation add their own distortions.

One popular way to compare segments of such a series is a similarity measure based on edit-distance (the cost of transforming one segment into another through shifts, deletions, and insertions of points). It comes from text processing and is convenient because it does not require equal lengths. But the authors of the study discovered that the transformation cost systematically increases with segment size. A longer segment appears more dissimilar to others simply because it contains more points. It's as if the interpreter confused words more often the longer the speech went on, even if the speaker was equally clear throughout.

The effect is not the same for different types of dynamics. For random noise and a sine wave, the dependence on size is stronger than for a process with memory (AR(1)).

To separate real changes from artifacts of measurement frequency, the researchers proposed a method of surrogate data with constraints on sampling frequency (SRC-surrogates). The idea is simple: generate many artificial copies of the series that preserve segment sizes and the distribution of intervals but destroy the dynamics. Then the predictability measure (DET) of the real series is compared with the distribution of DET from surrogates. If the real series falls outside this distribution, its behavior cannot be explained by uneven sampling alone.

On a synthetic series with an abrupt change in sampling frequency, the method worked: the spurious shift to higher predictability disappeared, and the linear trend built into the model was recovered. The method was then applied to real data: a stalagmite from Niue Island that grew for 1000 years (6.4–5.4 thousand years ago). Without correction, it seemed that the predictability of seasonal rainfall was increasing, but after subtracting the artifact, two periods of reduced predictability emerged. These coincide with intensified El Niño and tropical cyclones.

The stalagmite grew faster during wet periods, so its layers contained more measurement points per year. Without correction, this looked like an increase in climate predictability, but in reality it was just an artifact.

Now researchers have a tool for honest comparison of episodes with different measurement densities. This is especially important for paleoclimate, where the time step is set by natural processes, not by the experimenter's choice. The method is universal and can be applied to any edit-distance-based measure, and in the future to clustering, classification, and networks. The interpreter is no longer thrown off by rustles: they have learned to distinguish them from the speaker's voice.

The stalagmite from Niue grew faster during wet periods, so its layers contained more measurement points per year, and without correction this looked like an increase in climate predictability, when in fact it was just an artifact.

Concepts
importance sampling rmse bootstrap signal averaging jackknife resampling standard error precipitation anomaly lomb scargle periodogram regularisation linear regression Stochastic modeling sample size statistical method regime shift Monte Carlo methods statistical uncertainty survey seasonal cycle retrospective forecast cross validation