A neural network learning something new is like a traveler in mountainous terrain. Each piece of learned knowledge is a depression in the landscape into which the traveler rolls and gets stuck. To escape and reach a new basin, energy is needed. But as the traveler gains experience, the walls of their pit become steeper, and each subsequent climb requires more effort.
Physicists described this process with equations typically applied to Brownian motion — the jittering of pollen in water. They represented the state of the neural network as a particle darting between two wells on an energy landscape. The noise that pushes the particle plays the role of the "learning temperature." When the temperature is high, the particle easily hops from one well to another — the network quickly masters new tasks. But at low temperature, the particle stays put — the network stubbornly clings to old knowledge.
The main pitfall of continual learning is the so-called "stability-plasticity dilemma." If the network clings too tightly to prior knowledge, it cannot absorb new information. If it forgets too easily, it loses accumulated knowledge. The authors showed that a popular method to prevent forgetting, penalizing changes to important connections, actually builds a barrier that grows with each new task. The barrier height increases linearly, while the probability of jumping over it drops exponentially. So after dozens of tasks, the network eventually freezes.
This finding offers a recipe: to keep a neural network flexible, you should not simply punish it for forgetting, but skillfully manage the "temperature"—the noise in the learning process. Sudden injections of high temperature at the right moments can help the network reconfigure without losing old skills.
Brownian motion, on which the model is based, was first observed by botanist Robert Brown in 1827 while looking at pollen in water under a microscope. He thought he was seeing a "vital force," but it turned out to be the impacts of invisible molecules.