Overlapping Features Speed Up Learning express
A neural network can be compared to a dormitory where each resident gets their own room. Usually, each feature in the data occupies a separate neuron, and learning proceeds sequentially: first one feature, then another. This is slow and depends on which data you happen to get.
But if you allow several features to share a room, overlap them, the picture changes. The researchers built a simple 'teacher-student' model and derived formulas for learning without overlap. It turned out that the exponent in the learning law (how fast the error decreases over time) depends on the data statistics and the importance of the channels.
When a 'bottleneck' for overlap appears in the model, a transition to a universal law occurs: the error falls as one divided by time. This does not depend on the data or the importance of the channels. Learning speeds up by up to ten times compared to sequential feature scanning.
Such a fast pace with a universal exponent may explain why large language models, which actively use feature overlap, learn so efficiently. Perhaps it is the overlap of features that allows them to quickly grasp language patterns without getting bogged down in details.
Original abstract
as written by the authors arXiv:2602.01045
We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training without superposition, establishing that the power-law training exponent depends on both the input data statistics and channel importance. Remarkably, we discover that a superposition bottleneck induces a transition to a universal power-law exponent of $$\sim 1$$, independent of data and channel statistics. This one over time training with superposition represents an up to tenfold acceleration compared to the purely sequential learning that takes place in the absence of superposition. Our finding that superposition leads to rapid training with a data-independent power law exponent may have important implications for a wide range of neural networks that employ superposition, including production-scale large language models.
Feature overlap is similar to how humans remember faces: we do not store a separate neuron for each acquaintance, but a distributed code where the same neurons participate in different memories.
In Peter Watts's novel 'Blindsight', an alien intelligence uses distributed representations to process information at incredible speed — much like feature overlap speeds up learning.