Self-Organized Criticality in Deep Neural Networks ✛
Context
Near phase transitions, systems exhibit collective properties: heightened susceptibility, long-range correlations, and a wide repertoire of dynamical responses useful for computation and memory. Together with empirical signs of criticality in neural activity, this has led to the hypothesis that the brain operates near a critical state. Self-organized criticality provides a mechanism for such states to arise without external tuning. In adaptive networks, feedback between fast dynamics and slow structural adaptation can lead to self-organization. A simple local rule, inactive nodes gain connections while active ones lose them, drives global connectivity to a critical point. Similar principles operate in biologically realistic neural networks, where activity-dependent synaptic plasticity self-organizes dynamics toward criticality. In deep learning, networks are typically controlled by global optimization objectives or external architectural parameters. Directly maintaining criticality via the Jacobian requires explicit knowledge of a global network property. Adaptive self-organization offers a qualitatively different possibility: the network regulates its global state through local plasticity without measuring its distance to the critical point. This work stands apart: its nearest neighbors in the archive focus on brain dynamics, recurrent networks, and synaptic plasticity, but none have tested whether local homeostatic plasticity alone can drive deep feedforward networks to criticality.
Methods
We considered fully connected multilayer perceptrons with 9 hidden layers of 1000 or 1600 neurons each. Dynamics were described by a discrete layer-to-layer map with a scaled hyperbolic tangent activation. Weights were initialized uniformly in [−0.05, 0.05] and multiplied by a global factor m that sets the connection strength. Two measures were used for quantification: the branching parameter σ, the ratio of the average activity of the output layer to that of the input layer, and the largest finite-time Lyapunov exponent λ0, computed via the singular values of the input-output Jacobian using PyTorch's automatic differentiation. The local rule: a neuron's activity was defined as the standard deviation of its activations over a batch of MNIST images. The threshold T equals the average activity of the input layer. If activity is below threshold, the neuron is inactive and a random incoming connection is strengthened by a fixed step c; if above, it is weakened. In each iteration, 50% of neurons evaluated their activity and updated 1% of their incoming connections. When training jointly, a stochastic gradient descent step was performed first, followed by the homeostatic update. The MNIST dataset was used, with 60,000 training and 10,000 test images, a learning rate of 10⁻³, and no momentum.
Results
Networks were initialized across a wide range of connection strengths, from strongly subcritical (m=0.25, activity decays within a few layers) to supercritical (m=2.0, activity grows layer by layer). After 20 epochs of homeostatic adaptation, all networks converged to a common stationary state: activity became approximately preserved across depth, the branching parameter σ approached 1, the maximal Lyapunov exponent λ0 approached 0, and the average connection strength reached a common asymptotic value. This occurred regardless of initial conditions. When trained with gradient descent without homeostasis, the network drifted into the supercritical regime, with λ0 becoming positive. Adding homeostatic updates counteracted this drift in proportion to the step size c. For sufficiently large c, the network remained near the critical point throughout training. However, classification accuracy decreased with increasing c: pure SGD achieved the highest accuracy, while stronger homeostasis progressively degraded performance. The cosine similarity between SGD and SOC update vectors was low and negative, indicating that the two mechanisms push weights in substantially different directions in parameter space.
Implications
This work shows that a deep feedforward network can autonomously approach a critical regime through purely local, activity-dependent synaptic adaptation. It extends adaptive self-organization to criticality, previously demonstrated mainly in abstract threshold networks and models of biological neural systems, to modern deep networks. The key mechanism is negative feedback between local dynamics and the system's effective control parameter. Local neural activity acts as a sensor of the global dynamical phase: excessive activity weakens synaptic interactions, while insufficient activity strengthens them. Although the phase transition is a collective property of the entire network, local activity contains enough information about which side of the transition the system is on to return it to the critical region. Unlike neighboring works that either study biological networks or use global constraints, this is the first demonstration of purely local self-organization in deep feedforward networks.
Future development
In its current form, the SOC mechanism does not improve supervised learning but rather competes with it. The low alignment between gradient and homeostatic updates indicates that the two mechanisms optimize different objectives. A solution may lie in combining task signal and criticality within a common plasticity rule, for instance, through a global modulatory signal in conjunction with local activity. Developing such coordinated learning rules is an important direction for future work.
Impact
The results could influence the development of methods to stabilize deep network training and prevent dynamical failures such as loss of plasticity and model collapse.
Next steps
Develop plasticity rules that reconcile homeostatic regulation of criticality with optimization of the objective function, possibly via a modulatory signal.
Key open problems
This work connects self-organized criticality to the problem of stable training in deep networks: how to maintain a favorable dynamical regime without compromising task optimization. It resonates with the open question of how the brain combines multiple forms of synaptic plasticity to simultaneously regulate function and stability.
The knowledge machine's view
This section is addressed to the author of the paper and their colleagues. Neighbouring works were found by meaning-based search across our archive, not by word overlap. We do not review: we show the field around the work.
What the work does
The study shows that a local homeostatic rule (active neurons weaken their incoming connections, inactive ones strengthen them) by itself drives fully connected deep feedforward networks into a critical state, as measured by the branching parameter and the Lyapunov exponent. When trained jointly with gradient descent, homeostasis counteracts supercriticality but competes with task optimization.
Where its strength is
The strength of the work lies in demonstrating that global criticality can emerge from purely local plasticity without explicit control of global properties, and in the quantitative analysis of the interplay between homeostasis and gradient-based learning.
Where it could go next
- It might be interesting to compare the observed self-organization with mechanisms that maintain critical states in biologically realistic networks, where activity-dependent plasticity also leads to criticality.based on: How the Brain Balances on the Edge arXiv:2604.21832
- Closely related is work on how connectivity constraints (for example, the impossibility of neurons connecting to all others) affect network dynamics; this could suggest how the local rule operates in sparse architectures.based on: Why Neurons Can't Be Friends with Everyone arXiv:2603.19320
- The archive contains an analysis of how weak agents capture networks through local interactions; perhaps similar self-organization principles could help understand how homeostasis interacts with learning in more complex architectures.based on: How Weak AI Agents Take Over Networks: The Crystallization of Inequality arXiv:2607.01148
Worth trying
- Test whether self-organization to criticality persists with sparse connectivity or convolutional architectures.
- Investigate whether the conflict between homeostasis and gradient descent can be reduced, for example by applying homeostasis only to a subset of layers or with a variable step size.
- Use not only the branching parameter and the Lyapunov exponent but also other indicators, such as the distribution of activity avalanche sizes, as criticality metrics.
Why this may matter
If local plasticity can automatically keep deep networks in a critical regime, this opens the way to more robust and adaptive architectures that do not require careful initialization tuning or external control of dynamics.
Nearby works in our archive
- How the Brain Balances on the Edgesimilarity 0.53
- The Gas That Pretends to Be a Brainsimilarity 0.52
- How the Brain Learns to Predict Without Cuessimilarity 0.51
- Why Neurons Can't Be Friends with Everyonesimilarity 0.50
- The Cellular Orchestra: How Genes Learn to Play in Tune with Chaossimilarity 0.50
- Division That Doesn't Fragment Memorysimilarity 0.50
- How Weak AI Agents Take Over Networks: The Crystallization of Inequalitysimilarity 0.48
The archive is sparse around this work: the closest review is only 0.53, against a usual 0.64. It looks like few have drilled nearby.
The homeostasis rule is so simple it can be stated in one line: if a neuron is too active, weaken its incoming connections; if too passive, strengthen them. And that is enough for the network to find the critical point on its own.
Key numbers
- hidden layers: 9
- neurons per layer: 1000–1600
- homeostasis step c: 0.001
- fraction of neurons updated per iteration: 50%
- adaptation epochs: 20