A large neural network like Whisper masters hundreds of languages and resembles the standard model of the universe: everything is balanced. But training on a rare language disrupts this equilibrium — it turns into a black hole, pulling in all training resources. The network starts forgetting the other languages.
The Unified Gradient Projection (UGP) method prevents this skew. It acts like a gravitational field, evenly distributing 'attention' across languages. The gradients — the directions of network tuning — don’t tilt in favor of one language, and the spacetime curvature of knowledge remains uniform. The entropy of forgetting — a measure of disorder — approaches zero.
Amazingly, the human brain solves a similar problem during sleep: as it processes daily experiences, it evenly strengthens all memories so nothing is lost. UGP does for neural networks what sleep does for humans.
as written by the authors arXiv:2607.11163
Large-scale pretrained ASR models such as Whisper exhibit strong multilingual capabilities. However, fine-tuning on low-resource languages often causes catastrophic forgetting. Although continual learning mitigates this issue, existing methods struggle to regulate cross-task interference in multilingual settings, where dominant languages bias optimization. We propose Unified Gradient Projection (UGP), which constrains parameter updates using reference gradients from language-balanced replay in a unified projection space. By equalizing per-language contributions in the projection, UGP reduces dominant-language bias and improves cross-lingual stability. We further show that combining gradient-level projection with data-level replay yields complementary gains in stability and plasticity. Across diverse low-resource language groups and model scales, UGP enables effective adaptation while substantially mitigating forgetting. On Whisper-large-v3, it achieves near-zero average forgetting.
When learning a new language, the human brain can temporarily 'switch off' an old one — this is known as language interference. Neural networks suffer from a similar effect, but unlike us, they forget forever without special techniques.
The perfect assistant C-3PO from Star Wars knows millions of languages and never mixes them up — perhaps it has a built-in version of UGP.