A measure of how quickly a model's error decreases over time during training.Learning rate is a hyperparameter in machine learning that controls the step size taken when updating a model's parameters during training. It determines how quickly the model's error (loss) decreases as it iterates over data. A high learning rate makes the model learn fast but risks overshooting the optimal solution; a low learning rate makes learning slow but more stable. The learning rate is one of the most critical settings in training neural networks, as it directly influences convergence and final performance.
The concept of a learning rate emerged with the development of gradient-based optimization in the 1950s and 1960s, particularly in the context of perceptron learning and early neural networks. It became a standard term in the 1980s with backpropagation.
During training, the model computes the gradient of the loss function with respect to its parameters. The parameters are then updated by subtracting the gradient multiplied by the learning rate. This is the standard gradient descent update rule. The learning rate scales the gradient step: a larger value means larger jumps in parameter space, while a smaller value means finer adjustments. Modern optimizers like Adam adaptively adjust the learning rate per parameter, but the base learning rate still needs to be set by the practitioner.
Learning rate scheduling — adjusting the learning rate during training — is used in virtually all deep learning applications, from image recognition to natural language processing, to improve convergence and avoid local minima.
A learning rate that is too high can cause the loss to diverge to infinity, while a learning rate that is too low can get stuck in a plateau, making the training time impractically long.
| \theta_k | variable | Parameter vector at iteration k — parameter vector |
| \theta_{k+1} | variable | Updated parameter vector at iteration k+1 — parameter vector |
| \alpha_k | variable | Learning rate (step size) at iteration k — learning rate |
| f(\theta_k) | variable | Objective function evaluated at \theta_k — objective function |
| \nabla_\theta | operator | Computes the vector of partial derivatives with respect to each parameter — gradient operator |
| J_{k\alpha} | variable | parameter being updated at iteration — parameter |
| \tau_\mu | variable | iteration step index — iteration index |
| \eta | variable | learning rate, step size — learning rate |
| \epsilon | variable | error or loss function — loss function |
| \frac{\partial}{\par | operator | computes the gradient of the error with respect to the parameter — partial derivative |
