
Sampling a multidimensional posterior one coordinate at a time from its conditional distributions.Gibbs sampling is a Markov chain Monte Carlo (MCMC) algorithm for generating samples from a multivariate probability distribution when direct sampling is difficult. It works by updating one variable at a time, drawing each new value from its conditional distribution given the current values of all other variables. This approach is particularly useful when the full joint distribution is complex, but the conditional distributions are simple and easy to sample from. Gibbs sampling is a special case of the Metropolis-Hastings algorithm, but it always accepts the proposed move, making it computationally efficient in many settings.
The method is named after the physicist Josiah Willard Gibbs, but it was formalized by Stuart Geman and Donald Geman in 1984 in the context of image restoration, and later generalized by others.
The algorithm starts with an initial set of values for all variables. At each iteration, it cycles through the variables in a fixed order (or randomly). For each variable, it samples a new value from its conditional distribution, holding all other variables at their current values. These new values are immediately used in the subsequent conditional distributions. Over many iterations, the sequence of samples converges to the joint distribution, provided the chain is irreducible and aperiodic. The key is that the conditional distributions must be known and easy to sample from, which is often the case in hierarchical Bayesian models.
It is widely used in Bayesian statistics for sampling from posterior distributions, such as in latent Dirichlet allocation (LDA) for topic modeling in natural language processing, and in genetic data analysis.
Despite being named after Gibbs, the algorithm was not developed by him; the name was coined by Geman and Geman because of an analogy to Gibbs distributions in statistical mechanics.











