Why neural networks are like black holes: a lesson from 'bald' models ⚡ экспресс
Researchers applied a quantum physics approach to understand the success of LoRA—a popular method for fine-tuning large language models with low-rank updates. The network parameters were represented as matrix product states, enabling the measurement of "artificial entanglement" (analogous to quantum entropy). They found that in the inner layers of LoRA updates, entropy obeys a volume law with a characteristic dip—the "entanglement valley", while in the attention layers between tokens, it follows an area law with logarithmic corrections. A striking parallel with the no-hair theorem of black holes: differences in internal structure do not manifest in the output, which explains the effectiveness of low-rank adaptation.
Fine-tuning a neural network is like trying to alter a black hole without touching its event horizon: you change only a tiny fraction of parameters. Researchers looked inside and saw stunning patterns—strong links between pieces of the model, akin to quantum entanglement. By measuring entropy (a measure of disorder), they discovered an 'entanglement valley'—a sharp drop in orderliness that depends on the training method.
In practice, this means: it doesn't matter what patterns exist inside—only the external behavior counts. So we can develop lightweight AI adaptation methods without diving into the details.
🎯 The 'no-hair' theorem: a black hole is described solely by mass, charge, and spin. All other information about consumed matter is erased.