Training a neural network is like sledding down a hill: you need the right speed to avoid getting stuck or flying off the track. In the popular momentum method (like inertia), a constant value of 0.9 is typically used. The authors revisited the physics of damped oscillations and proposed changing momentum during training: initially large, then decreasing according to a formula where momentum equals one minus two times the square root of the learning rate.
They tested it on a standard task: a ResNet-18 neural network learning to recognize images from the CIFAR-10 dataset. With the new momentum schedule, the model reaches 90% accuracy 2.34 times faster than with constant 0.9. However, the final accuracy is slightly lower — by 0.46 percentage points. So they applied a hybrid: critical damping until 90%, then switching to constant momentum. This eliminates the accuracy loss and retains the speedup: final accuracy is 95.45%, and it reaches 90% accuracy 2.4 times faster.
In the second part of the paper, the authors tested the idea of selecting layers for fine-tuning based on error gradients. It turned out that gradient-based selection is no better than random. But there is a useful signal: if a layer has the lowest gradient, it's definitely not worth touching — interfering with it can be very harmful. So gradients help avoid bad decisions, not find good ones.
Momentum in optimizers is named after the physical concept of moment of inertia: it helps overcome local minima, just as a sled glides over small bumps thanks to its accumulated speed.