Training dynamics of autoencoders: transient scaling and emergent concepts in the Ising model ✛ ✦
Context
Modern machine learning achieves impressive feats, yet the nonlinear methods underlying neural networks remain largely a “black box.” To shed light on learning dynamics, researchers turn to physical models. The Ising model is an ideal playground: its spin configurations are distributed according to the Boltzmann distribution, introduced by Ludwig Boltzmann, which generates complex correlations. Understanding how autoencoders learn macroscopic variables like magnetization and energy is critical for automating scientific discovery. Similar numerical simulation challenges arise in condensed matter physics, where superconductivity and Bose–Einstein condensates exhibit phase transitions governed by collective variables.
Methods
The experiment relies on numerical simulation of a two-dimensional Ising model on a 16×16 lattice. Spin configurations were generated via Monte Carlo with Glauber dynamics, proposed by Roy Glauber. The training set comprised 300,000 states at temperatures near critical. Deterministic autoencoders with ReLU activations were studied; depth (1, 4, 16 layers), width (256, 512, 1024 neurons), bottleneck size (1, 8, 64), and learning rate (10⁻⁵, 10⁻⁴, 10⁻³) were varied. Optimization used stochastic gradient descent with momentum (Adam) and cosine annealing. To analyze entropy of representations, reconstruction errors of local magnetization and energy were measured after spatial coarse-graining with kernels of different sizes. A recursive model pass was used to study the flow field in latent space.
Results
The analysis uncovered two dynamical regimes. In the early stage, magnetization dominates: the reconstruction error of large-scale averages drops, obeying the second law of thermodynamics in the sense of growing order. Later, as training progresses, the focus shifts to energy, which requires resolving fine-scale details. The transition between regimes is controlled by hyperparameters. Deep models (d=16) at medium and high learning rates get stuck in an “arrested” state, reconstructing only the dataset average. This regime resembles glass formation in nonequilibrium quantum thermodynamics and aligns with the fluctuation-dissipation theorem. Narrow models (w=256) are more stable than wide ones. In latent space, topological stability is observed: magnetization forms a one-dimensional structure before becoming perfectly accurate. Recursive dynamics show that error flow generates a common topology of trajectories across all layers, just as a magnetar possesses a global field structure. Spearman correlation confirms that energy is initially approximated via magnetization and then separated into a distinct concept. When the bottleneck is widened, the upper bound of recoverable energies shifts upward, and the trade-off between scales vanishes—akin to magnetic reconnection rearranging field topology.
Implications
The findings indicate a deep connection between neural network training and the physics of nonequilibrium phase transitions. Just as superconductors and Bose–Einstein condensates give rise to collective behavior, in autoencoders concepts emerge through a cascade of symmetry breaking, starting with global variables. The fact that learning is scale-invariant at early stages brings it closer to critical dynamics near phase transition points, described by Lev Landau. This opens the possibility of applying renormalization-group methods to analyze neural networks. Moreover, the demonstrated ability of models to autonomously extract physical concepts without prior knowledge is a step toward automating scientific discovery.
Future development
Future work involves studying more complex architectures, such as variational autoencoders and transformers, on dynamic data. It is important to uncover how the identified regimes manifest in problems with continuous fields (e.g., in fluid dynamics) and whether learning can spontaneously tune to a critical state that optimizes information processing. Extending the analysis to quantum systems described by a wavefunction will connect machine learning with quantum thermodynamics and possibly with the measurement problem.
Impact
The results will impact the interpretability of neural networks in physics, biology, and materials science, as well as the design of algorithms that purposefully exploit physical self-organization mechanisms.
Next steps
Next steps include a detailed investigation of the critical learning regime using renormalization-group methods and testing the universality hypothesis of scaling properties on other statistical mechanics models.
Key open problems
The work touches on the fundamental problem of the connection between learning and nonequilibrium statistical mechanics, as well as the question of how complex systems can generate abstract concepts—a key challenge on the path to strong artificial intelligence.
The knowledge machine's view
This section is addressed to the author of the paper and their colleagues. Neighbouring works were found by meaning-based search across our archive, not by word overlap. We do not review: we show the field around the work.
What the work does
The study traces the learning dynamics of autoencoders on Ising model configurations. By varying depth, width, and learning rate, the authors identify two regimes: first, the network captures magnetization, then energy, while deep models with high learning rates get stuck in an "arrested" state. The analysis relies on reconstruction errors after coarse-graining and recursive dynamics in the latent space.
Where its strength is
The main strength lies in detecting the transition from large-scale to small-scale variables and demonstrating how hyperparameters control this transition. This provides a concrete experimental basis for understanding the "physics" of neural network training.
Where it could go next
- The observed transition from magnetization to energy resembles hierarchical learning of patterns. In our archive, there is a related work on neural networks that learned the laws of nature: it shows how networks can learn physical symmetries and invariants. It might be interesting to see if similar invariant structures appear in your autoencoders at the level of hidden representations.based on: Neural Networks That Have Internalized the Laws of Nature arXiv:2606.21945
- Topological stability in the latent space and recursive error dynamics echo an approach where a neural network reconstructs the equation of state of neutron stars from observational data. There, too, the key role is played by how the network organizes its internal representation for efficient compression. Perhaps the error flow analysis methods from that work could help deepen understanding of your "arrested" states.based on: Listening to the Stars: AI Decoder Instantly Reconstructs the Neutron Star Equation of State arXiv:2607.12722
- The stalling of deep models at high learning rates resembles convergence problems in tasks requiring fine-tuning of representations. In our archive, there is an analysis of a neural network sculptor that reconstructs a gravitational lens from a small number of points: it discusses stabilization techniques for training under strong compression. Adapting them to your architecture might help mitigate the "glassiness."based on: Neural Network Sculptor: Reading a Gravitational Lens from Four Points arXiv:2607.22812v1
Worth trying
- Test whether spontaneous encoding of symmetries, similar to that described in 2606.21945, emerges in the hidden layers.
- Apply error flow visualization methods from 2607.12722 to analyze the topology of "arrested" states.
- Test stabilization techniques from 2607.22812v1 to prevent deep models from getting stuck.
Why this may matter
Understanding how neural networks autonomously build a hierarchy of physical concepts could bring us closer to creating algorithms capable of independently discovering the laws of nature in complex systems, from condensed matter to astrophysics.
Nearby works in our archive
- How the Brain Learns to Predict Without Cuessimilarity 0.63
- Neural Networks That Have Internalized the Laws of Naturesimilarity 0.61
- Neural Network Embedded in a Neuron Modelsimilarity 0.61
- Living Mind: Why AI Without a Body Remains Blindsimilarity 0.60
- Division That Doesn't Fragment Memorysimilarity 0.60
- Listening to the Stars: AI Decoder Instantly Reconstructs the Neutron Star Equation of Statesimilarity 0.60
- Neural Network Sculptor: Reading a Gravitational Lens from Four Pointssimilarity 0.60
The closest work in the archive sits at 0.63 by meaning — the usual level for our corpus.
Original abstract
as written by the authors arXiv:2607.10285
We study how unsupervised autoencoders trained on microscopic spin configurations from the Ising model learn macroscopic, theory-relevant variables underlying the data-generating process. We quantify learning across multiple spatial (coarse-graining) scales and reveal two distinct dynamical regimes that appear sequentially, controlled by the main hyperparameters (model depth, width, and learning rate): one in which magnetization and another in which energy is learned across scales. The first exhibits error fluctuations ordered to scale and learns global averages only; The second gradually resolves smaller scales relevant for the energy representation. Deep models trained at moderate and fast rates become arrested before reaching these regimes. We connect reconstruction errors with the latent representations using a novel analysis of self-recursive trajectories. These intrinsic dynamics are induced by prediction errors, exposing how training drives representation changes for macroscopic concepts. We utilize the intuition that learning operates as a process driven far from equilibrium by fluctuations from the training data to provide an interpretive basis grounded in both the physical world and the machine models that represent it.
The Ising model, originally proposed to describe ferromagnetism, turned out to be universal: it is applied in neuroscience, sociology, and even financial market analysis.
Key numbers
- lattice size: 16×16 (256 spins)
- critical temperature: T_c ≈ 2.269 (for infinite system)
- number of training samples: 300,000
- learning rates: 10⁻⁵, 10⁻⁴, 10⁻³
- maximum number of parameters: 33 million