Why neural networks are like black holes: a lesson from 'bald' models ⚡ экспресс
This work investigates the efficiency of parameter-efficient fine-tuning (PEFT) of large language models, such as LoRA, from the perspective of quantum information theory. Low-rank parameterizations correspond to low-dimensional representations as matrix product states (MPS), which allows introducing the concept of "artificial entanglement"—the entanglement entropy of neural network parameters. LoRA and full fine-tuning (FFT) are compared on LLaMA models (1B and 8B) trained on the Tulu3 and OpenThoughts3 datasets. The internal artificial entanglement in the query and value matrix updates for LoRA obeys a volume law with a suppression in the center—an "entanglement valley", sensitive to hyperparameters, and differs from FFT. The external entanglement in attention matrices, corresponding to token correlations in representation space, follows an area law with logarithmic corrections and is robust to hyperparameters and training steps. A parallel is drawn with the no-hair theorem of black holes: despite differences in internal entanglement, the attention outputs remain similar, explaining the effectiveness of low-rank updates.
Fine-tuning a neural network is like trying to alter a black hole without touching its event horizon: you change only a tiny fraction of parameters. Researchers looked inside and saw stunning patterns—strong links between pieces of the model, akin to quantum entanglement. By measuring entropy (a measure of disorder), they discovered an 'entanglement valley'—a sharp drop in orderliness that depends on the training method.
In practice, this means: it doesn't matter what patterns exist inside—only the external behavior counts. So we can develop lightweight AI adaptation methods without diving into the details.
🎯 The 'no-hair' theorem: a black hole is described solely by mass, charge, and spin. All other information about consumed matter is erased.