Overlapping Features Speed Up Learning express
A neural network can be compared to a dormitory where each resident gets their own room. Usually, each feature in the data occupies a separate neuron, and learning proceeds sequentially: first one feature, then another. This is slow and depends on which data you happen to get.
But if you allow several features to share a room, overlap them, the picture changes. The researchers built a simple 'teacher-student' model and derived formulas for learning without overlap. It turned out that the exponent in the learning law (how fast the error decreases over time) depends on the data statistics and the importance of the channels.
When a 'bottleneck' for overlap appears in the model, a transition to a universal law occurs: the error falls as one divided by time. This does not depend on the data or the importance of the channels. Learning speeds up by up to ten times compared to sequential feature scanning.
Such a fast pace with a universal exponent may explain why large language models, which actively use feature overlap, learn so efficiently. Perhaps it is the overlap of features that allows them to quickly grasp language patterns without getting bogged down in details.
Feature overlap is similar to how humans remember faces: we do not store a separate neuron for each acquaintance, but a distributed code where the same neurons participate in different memories.
In Peter Watts's novel 'Blindsight', an alien intelligence uses distributed representations to process information at incredible speed — much like feature overlap speeds up learning.