In a nine-story building with a thousand rooms on each floor, some people whisper and others shout. Neighbors pass the message along, but if everyone speaks too softly, the message fades by the third floor. If too loudly, by the top floor it becomes an unintelligible roar. Between these extremes, there is a regime where a whisper makes it all the way through without distortion. Neuroscientists call this the critical point, and the brain seems to operate right there.
For a long time, it was thought that you could only push a network into a critical state from the outside, by tweaking global parameters like a piano tuner. But the authors of the new study ran a simple experiment: they let each neuron decide on its own whether to strengthen or weaken its connections, based only on its own activity. The rule fits in a single line: if a neuron is too active, weaken the incoming connection; if too passive, strengthen it. No thermometer, no measurements of the whole network.
The researchers took a fully connected network with nine hidden layers of a thousand neurons each and ran it from different initial states, from near silence to deafening noise. After twenty epochs of this self-tuning, all networks converged to the same regime: activity stopped fading or growing, and the average connection strength reached a common plateau. The network found the critical point on its own, just as water finds its level.
But when they added standard supervised learning to the self-tuning, competition began. Gradient descent pulls the network toward solving the task, while homeostasis pulls it toward criticality. These two forces moved the weights in different directions, and the stronger the self-regulating rule, the worse the network recognized digits. Pure gradient descent gave the highest accuracy, and increasing homeostasis progressively reduced it.
A thermostat without a thermometer works, but it doesn't yet play well with a task. That's the main challenge: combining the drive toward criticality with learning a useful skill. Perhaps the solution lies in a general plasticity rule that accounts for both local activity and a global task signal. Then the network could stay on the edge of chaos and solve practical problems at the same time, like a brain that manages to think without overheating.
The homeostasis rule is so simple it can be stated in one line: if a neuron is too active, weaken its incoming connections; if too passive, strengthen them. And that is enough for the network to find the critical point on its own.