main concepts scientists sections
SimplePopularAdvanced

Adaptive Nonlinear Vector Autoregression: Robust Forecasting for Noisy Chaotic Time Series ✛

Original: "Adaptive Nonlinear Vector Autoregression: Robust Forecasting for Noisy Chaotic Time Series"
· Azimov Sherkhon, Susana Lopez-Moreno, Eric Dolores-Cuenca, Sieun Lee, Sangil Kim
arXiv:2507.08738 · 2025-07-11 · CC BY 4.0 · 5 min
Machine Learning Artificial Intelligence Dynamical Systems
A trainable neural network instead of fixed polynomials makes chaos forecasting accurate even with 30% noise.

Context

Forecasting chaotic time series is a challenge facing climatology, oceanography, and finance. Reservoir computing (RC) and its simplified variant, nonlinear vector autoregression (NVAR), have gained popularity due to fast training: instead of tuning the entire recurrent network, they fix random weights and solve only a linear problem. However, this approach has a weakness: nonlinear features are built from a rigid template (polynomial expansion in NVAR or random maps in echo state networks), which limits adaptability to noise and complex data. Around this work in our archive lie studies that also seek ways to make forecasting more reliable: some monitor heat across a room with a single thermometer, others teach neural networks to hear particle energy, and still others tame chaos with algorithms. All grapple with the common problem of extracting signal from imperfect data, but this article takes a step toward trainable nonlinearities instead of fixed ones.

Methods

The authors replaced the heart of standard NVAR—the fixed polynomial feature vector—with a small multilayer perceptron (MLP) with trainable weights. The input is a delay embedding of the original series: several consecutive values shifted in time. The MLP transforms this vector into nonlinear features, which are then combined with linear features and fed into a linear readout layer. The key difference from the classical approach: the readout matrix is not computed analytically via matrix inversion but is trained jointly with the MLP using gradient descent in an architecture with skip connections. Training proceeds in two stages: first the Adam optimizer for a fast start, then L-BFGS for fine-tuning on low-dimensional systems. Experiments were conducted on three chaotic systems: one-dimensional Mackey–Glass, three-dimensional Lorenz-63, and hundred-dimensional Lorenz-96. Gaussian noise with zero mean and standard deviation from 0 to 0.3 (that is, up to 30% of the signal amplitude) was added to the data. For a fair comparison, hyperparameters of all models were tuned via grid search on a validation set, and quality was assessed by RMSE on independent, non-overlapping forecast windows of lengths 25, 50, 75, and 100 steps.

Results

Adaptive NVAR proved more accurate than all competitors in almost all regimes. On the Mackey–Glass system without noise, its error was 3–5 times lower than that of standard NVAR: at a horizon of 100 steps, RMSE was 0.006 versus 0.029. With 30% noise, the gap persisted: 0.051 versus 0.073, while the echo state network and hybrid echo state network produced errors orders of magnitude higher (0.309 and 22.449, respectively). On the Lorenz-63 system, the adaptive model lagged behind standard NVAR only in the noiseless regime at short horizons, but with noise it moved ahead: at 30% noise and a horizon of 100 steps, its RMSE was 0.345 versus 0.773 for NVAR, 0.832 for ESN, and 1.054 for HESN. The main triumph is scalability: on the hundred-dimensional Lorenz-96 system, only adaptive NVAR could complete training and forecasting. Standard NVAR hit a memory wall (about 0.8 terabytes would be needed to store the feature matrix), and echo state networks hit a computational wall (31,104 configurations to search for ESN). The adaptive model with an MLP output layer of just 10,000 neurons (50 times smaller than the full quadratic expansion) achieved an RMSE of 0.536 at 30% noise and a horizon of 100 steps, demonstrating steady error growth without blow-ups.

Implications

This work stands apart from its archive neighbors: they mostly use fixed transformations or random reservoirs, whereas here nonlinearity is born from data through learning. This is the first test showing that a trainable MLP can replace polynomial expansion in NVAR without losing interpretability in favor of noise robustness. For chaos theory and forecasting complex systems, this means a shift from manual feature engineering to automatic pattern extraction, which is especially valuable when the physics of the process is only approximately known. The practical takeaway: gradient optimization combined with linear readout avoids the bottleneck of matrix inversion and opens the door to high-dimensional problems that were previously intractable for NVAR.

Future development

The topic could evolve toward deeper architectures: instead of a shallow MLP, convolutional or recurrent layers could capture long-range dependencies. It would be interesting to test adaptive NVAR on real geophysical data—ocean salinity, sea surface temperature, climate indices—where noise and nonstationarity are inevitable. Another direction is theoretical analysis: why do trainable features handle noise so well? Possibly the MLP acts as a filter suppressing high-frequency perturbations, but this requires rigorous proof. It is also worth investigating regularization that preserves some of the interpretability of standard NVAR, such as weight sparsity or Bayesian methods.

Impact

The method is directly applicable in oceanography and climatology for forecasting phenomena such as El Niño, ocean currents, and water temperature. In finance, it could improve prediction of volatility and price series, where noise is an inherent part of the data.

Next steps

The next step is validation on real datasets: for example, the Southern Oscillation Index or ocean temperature records from buoys. In parallel, the MLP architecture (number of layers, activations) should be optimized, and the effect of mini-batch size on training stability in high dimensions should be studied.

Key open problems

In the field of forecasting chaotic time series, an unresolved question remains: how to balance model expressiveness and robustness to noise? This work shows that trainable nonlinearities can be more reliable than fixed ones, but it does not explain why. Another open problem is choosing the optimal dimensionality of the nonlinear feature space: here the constant c was chosen empirically, but a theoretical criterion would be desirable.

The knowledge machine's view

This section is addressed to the author of the paper and their colleagues. Neighbouring works were found by meaning-based search across our archive, not by word overlap. We do not review: we show the field around the work.

What the work does

The authors replaced the fixed polynomial expansion in NVAR with a trainable multilayer perceptron that generates nonlinear features from the delayed embedding. The readout matrix and the MLP are trained jointly via gradient methods, avoiding the inversion of large matrices. On three chaotic systems (Mackey–Glass, Lorenz-63, Lorenz-96), the model showed higher accuracy and noise robustness compared to standard NVAR, echo state networks, and hybrid echo state networks, especially when noise up to 30% was added.

Where its strength is

Trainable nonlinear features instead of a rigid template allow the model to adapt to noisy and high-dimensional data while preserving forecast accuracy. Joint optimization of the MLP and the readout layer via a skip connection makes the method scalable and efficient.

Where it could go next

Worth trying

  • Try adding an attention mechanism over delayed inputs to adaptive NVAR, so the model itself identifies the most informative lags in noisy time series.
  • Test the method on real geophysical data, for example, El Niño indices, comparing with traditional reservoir approaches.
  • Investigate whether adaptive NVAR can be used for multivariate forecasting, where each variable has its own nonlinear dynamics but is coupled to others.

Why this may matter

If trainable nonlinear features prove robust on real noisy data, this could broaden the applicability of reservoir computing in climatology and oceanography, where classical NVAR often struggles due to the rigidity of the polynomial basis.

Nearby works in our archive

The archive is sparse around this work: the closest review is only 0.51, against a usual 0.64. It looks like few have drilled nearby.

Whole graph →

Storing the feature matrix for standard NVAR on the Lorenz-96 system would require about 0.8 terabytes of memory—roughly 200 full-length high-definition movies. The adaptive version got by with just tens of megabytes.

The idea of trainable nonlinearity for predicting chaos resonates with Isaac Asimov's Foundation: there, psychohistory predicted the future of large masses of people based on statistical regularities. Here, the model learns the laws of chaos from data, loosely echoing Hari Seldon's attempt to formalize the unpredictable.

Key numbers

  • noise: up to 30% of signal amplitude
  • Lorenz-96 dimensionality: 100 variables
  • feature reduction: 50-fold (10,000 instead of 500,500)
  • standard NVAR memory: about 0.8 TB
  • forecast horizon: up to 100 steps
Concepts
nonlinear laplacian spectral analysis neural network regression predictive learning neural networks reservoir computing Large Language Models linear regression probabilistic forecast Weakly Supervised Learning regularization methods nonlinear dynamics protein language model autoregressive model parameter random network regularisation transformers autoencoder robust machine learning Stochastic modeling Generative Adversarial Networks