Adding a penalty term to a fit to suppress wild solutions when the data underdetermine the model.Regularisation is a technique used in fitting models to data when the problem is ill-posed or underdetermined—meaning there are too many parameters relative to the data, or the data are noisy. Without regularisation, the fit can produce wildly oscillating solutions that perfectly match the data but have no predictive power (overfitting). Regularisation adds a penalty term to the objective function (e.g., the sum of squared errors) that discourages extreme parameter values, effectively trading a small increase in bias for a large reduction in variance.
The concept dates back to the 1940s with Tikhonov's work on ill-posed problems, and was popularized in statistics by Hoerl and Kennard (1970) as ridge regression.
Instead of minimizing the usual loss L(θ) (e.g., chi-squared), you minimize L(θ) + λ R(θ), where R is a penalty (e.g., the sum of squares of parameters for ridge regression, or the sum of absolute values for lasso) and λ is a tuning parameter that controls the strength of the penalty. Larger λ forces parameters toward zero or toward each other, smoothing the solution. The choice of λ is often made by cross-validation to balance fit and simplicity.
Essential in machine learning (e.g., preventing overfitting in neural networks), in image reconstruction (e.g., CT scans), and in geophysics for inverting seismic data.
Regularisation is mathematically equivalent to assuming a prior distribution on the parameters in Bayesian inference—ridge regression corresponds to a Gaussian prior, and lasso to a Laplace prior.
