Division That Doesn't Fragment Memory express
Holding in mind the exact position of a pen as it's slowly moved is a routine task for the brain. For artificial neural networks, it's nearly impossible. Standard recurrent networks like GRU and LSTM tend to break continuous motion into rigid steps: the pen doesn't glide smoothly but jumps from point to point. The network seems to remember not a line but a few discrete frames.
A group of researchers proposed a network with division — the Recurrent Divisive Normalization Network (RDNN). Its secret lies not in subtracting inhibitory signals but in dividing by them. When one neuron is active, it doesn't just silence its neighbor; it proportionally reduces the neighbor's output. This is like adjusting volume: in a noisy room, we don't subtract noise from speech; we divide: we speak louder relative to the background. This mechanism is widespread in the cerebral cortex but rarely used in artificial networks.
Mathematical analysis showed that division creates a slow manifold in the network — a smooth path along which the memory state can change without jumps. When the network is trained with backpropagation, division automatically scales the learning steps: in highly active regions, parameter updates are dampened. This compresses the effective rank of the network — the dynamics naturally squeeze into a narrow, low-dimensional channel, avoiding overfitting and chaos.
In experiments on working memory tasks, RDNN held continuous signals tens of times more accurately than standard networks. And when division was replaced with subtraction, the network immediately began to fragment the smooth signal into steps. It seems biology chose this mechanism for a reason — division turned out to be mathematically necessary for storing continuous quantities in a changing world.
Original abstract
as written by the authors arXiv:2608.01947v1
The ability to robustly maintain and update continuous variables is a hallmark of working memory. While classical continuous attractor networks suffer from severe fine-tuning fragility, standard artificial recurrent neural networks (RNNs) like GRUs and LSTMs typically fail to stably learn continuous manifolds, instead shattering the state space into discretized point attractors. To bridge this gap, we draw inspiration from divisive normalization, a canonical neural computation widely observed across cortical circuits, and propose the Recurrent Divisive Normalization Network (RDNN), a minimal and algebraically isolated model of dynamic division. Through dynamical systems analysis on canonical working memory tasks, we demonstrate that this biophysical constraint allows the network to converge to robust, high-fidelity slow manifolds. Furthermore, we analyze the gradient dynamics of divisive normalization during Backpropagation Through Time (BPTT), showing that it introduces an activity-dependent local gradient scaling. This scaling dampens parameter updates in highly active regimes, which empirically aligns with a significant self-compression of the network's effective rank, confining the recurrent dynamics to a tight, low-dimensional subspace while avoiding the optimization pathologies associated with explicit low-rank factorization. Finally, ablations demonstrate that while subtractive inhibition can maintain static memories, divisive normalization is mathematically essential to prevent manifold shattering under time-varying inputs. Our findings identify divisive normalization not merely as a biological artifact, but as a critical computational mechanism for learning high-fidelity continuous representations.
Divisive normalization in neurons was first described in the visual cortex: this is how the brain avoids being blinded by bright light: it simply lowers sensitivity, dividing rather than subtracting.