Why neural networks forget old knowledge when learning new things
A neural network learning something new is like a traveler in mountainous terrain. Each piece of learned knowledge is a depression in the landscape into which the traveler rolls and gets stuck. To escape and reach a new basin, energy is needed. But as the traveler gains experience, the walls of their pit become steeper, and each subsequent climb requires more effort.