principle

principle of maximum entropy

The maximum entropy principle asserts that the probability distribution which best represents the current state of knowledge is the one with the largest entropy.
19 related articles · 2 who worked here

Description

In 1957, American physicist Edwin Jaynes published a paper that overturned the view of thermodynamics. He showed that equilibrium statistical physics can be derived from a single recipe: maximize Shannon entropy under the constraints imposed by known macroscopic quantities. Entropy is a measure of uncertainty, proposed by Claude Shannon in 1948 for communication theory. Jaynes noted: if we only know the average energy and nothing else, then the most humble, least presumptuous distribution is the Boltzmann distribution. The principle encompassed both quantum and classical statistics. Today it has gone far beyond physics: from machine learning and neurobiology to resource allocation in networks and the synthesis of realistic fur in animation.

History

Claude Shannon proposed entropy as a measure of information in 1948. In 1957, Edwin Jaynes, in his paper 'Information Theory and Statistical Mechanics,' demonstrated that maximizing Shannon entropy under constraints imposed by known macroscopic quantities automatically yields the distributions of statistical physics. The principle was taken up in probability theory, statistics, and computer science, as well as in signal processing and climate modeling. Independently, Leon Brillouin developed similar ideas.

How it works

The principle allows obtaining probabilistic models under incomplete information. In practice, a constrained optimization problem is solved: maximize entropy under constraints given by mean values of certain functions. It is applied in speech recognition, computer vision, tomography image reconstruction, and weather forecasting. Limitations: if constraints are incomplete or poorly formulated, the principle can generate erroneous distributions; moreover, it assumes that entropy adequately reflects uncertainty.

In practice

The maximum entropy principle reconstructs brain tomography scans from incomplete data, builds weather forecasts and climate models, synthesizes textures in computer graphics, and underlies image and video compression algorithms.

Fun fact

Thanks to the maximum entropy principle, style transfer programs turn a photo into a Van Gogh canvas. The algorithm finds the texture with the highest entropy that resembles the artist's brushstrokes and overlays this 'disorder' onto the image.

Related articles · 26

Chaos in Chemistry: How Information Loss Reveals Kolmogorov-Sinai Entropy
Write down a recipe, then check it an hour later: if the proportions are off by even a gram, the taste is gone. Chemical reactions in open systems behave much the same way: the slightest difference in initial concentrations grows over time into completely different states.
Measuring Chaos Without Burning Extra Copies
A quantum state is like mountain weather: it changes with every glance. To estimate its uncertainty, physicists make copies and take measurements. Each copy is a thermometer that breaks after use.
eess.SY2026-07-151 minarXiv:2607.14291express
The Secret of Quick Recovery: A New Method for Assessing Stability
A stable system behaves like a ball in a bowl: give it a push, and it rolls back to the bottom. The spread of possible trajectories, uncertainty, shrinks over time. But the speed depends on the slope of the bowl's walls. In complex systems, this 'bowl' has sections of varying steepness.
Signal Processing2026-07-152 minarXiv:2607.13710express
Cooking Without Delays: How Networks Achieve Ultra-Reliability
Imagine an experienced chef during rush hour: suddenly a stove fails or a key ingredient runs out. Instead of panicking, he instantly reworks the recipes — speeding up some dishes, slowing down others, reallocating resources on the fly. This principle of dynamic adaptation underpins a new generation of ultra-reliable networks.
eess.SY2026-07-151 minarXiv:2607.14245express
How Entropy Helps Robots Find Paths Faster
A search party combs a foggy field. At first, people spread out wide: a high 'temperature' of search means a great variety of attempts. Upon finding a trace, they close in. But the speed of convergence is critical — narrow the circle too early, and the best find stays aside. The coordinator assesses the spread of searchers and commands when to speed up or slow down.
Quantum Physics2026-06-272 minarXiv:2606.28660express
How Confirmation Bias Helps You Think Faster
We're used to thinking that noticing only confirmations of our guesses is a mistake. But recent calculations, inspired by wave behavior, flip that view. It turns out that the habit of seeking the familiar is the best way to reduce information noise (scientists call it entropy).
q-bio.MN2026-06-142 minarXiv:2606.15905express
Six Rules for a Living System
Life is a constant dance between order and chaos. Back in the mid-20th century, Erwin Schrödinger noted that living things feed not so much on matter as on "negative entropy"—the ability to maintain their own order by increasing disorder around them.
Information Theory2026-05-281 minarXiv:2605.30331express
The Mystery of Disorder: Why Entropy Follows Strict Rules
Entropy is a measure of disorder. Two shuffled decks of cards, dumped into one pile, won’t become more chaotic than the sum of their individual disorders. This is the rule of subadditivity: the whole is no more chaotic than its parts.
Statistical Mech2026-05-262 minarXiv:2605.27635express
How to Measure Chaos in a System Without Knowing Its Rules
A busy intersection without traffic lights. Cars drive, brake, and change lanes, and from the outside, it looks like complete chaos. But if you watch for a while, you can notice a rhythm: the flow thickens and thins. In physics, this rhythm of disorder is called entropy: it shows how far a system is from calm equilibrium, like water in a boiling kettle.
q-bio.QM2026-05-162 minarXiv:2605.16724express
How to Fish the Laws of Nature Out of Noise
A voice recorder captures a city square. You hear the hum of a crowd, footsteps, snatches of conversation, the rumble of traffic. Somewhere in that sonic stew, a street musician plays a simple tune on a flute. Hearing it isn't easy: there's just too much mixed together.
Statistical Mech2026-05-092 minarXiv:2605.08860express
A Thermostat That Likes to Glue Probabilities Together
A pot of water on the stove eventually settles down: the temperature evens out, the liquid stops boiling. Physicists call this thermal equilibrium — a state where energy is distributed in the most probable way.
Statistical Mech2026-05-092 minarXiv:2605.08967express
When Disorder Gives Birth to Order
Spread a handful of coins on the table. Each coin is a possible outcome: heads, tails, or something else. If we know how chaotic this set is (its entropy), which combinations of probabilities will we encounter more often? It turns out the answer depends on how much chaos we allow.
Quantum Physics2026-05-053 minarXiv:2605.03807express
Almost Orthogonal Worlds
Toss a dozen coins on a table. The number of heads-and-tails combinations is just over a thousand. For two dozen coins, it's already a million. The number grows as 2 to the power of the number of coins, and that's fast. But quantum systems are far richer. Each particle here is not a coin with two sides but an arrow that can point in any direction.
Consciousness as a Thermostat Glitch
A thermostat keeps a room at 20 degrees. Open a window in winter, and it kicks on the heat to restore equilibrium. The brain does something similar: it constantly predicts what will happen a moment ahead and corrects itself when wrong. That's how it learns.
Optics2026-04-172 minarXiv:2604.16692express
Molecules Fall into Line: Seeing Their True Order
A thin film for a solar cell or display is like a dense crowd in a square: each person faces a different direction, and whether light passes through the crowd or gets stuck depends on which way everyone is looking. In organic electronics, molecules play the role of passersby. Until now, instruments could only measure the crowd's overall direction — the average head tilt and its spread.
HEP Phenomenology2026-04-141 minarXiv:2604.12691express
How a quantum proton becomes a hot liquid
At the Large Hadron Collider, not only nuclei but also individual protons collide. Even in such small systems, collective behavior resembling the flow of a liquid is observed. But the proton is a quantum object; its state is described by a wave function, not by temperature and pressure.
Chemical Physics2026-03-291 minarXiv:2603.27767express
Why Water Gets Heavier at 4 Degrees
Water is the only common liquid that doesn't immediately contract when cooled. Ice melts, and water becomes denser, but only up to 4 degrees Celsius. Beyond that, with further cooling, it expands again. This quirk explains why fish survive in freezing ponds: the densest water sinks to the bottom, while a lighter cold layer stays on top and turns into ice.
Quantum Physics2026-03-251 minarXiv:2603.24522express
Quantum Leap to Calm
In the quantum world, the path to calm sometimes resembles a paradoxical race: starting further from the finish, the system reaches equilibrium faster. This effect, a quantum analog of the Mpemba phenomenon, is due to the rapid growth of entropy — a measure of disorder.
Why Neurons Can't Be Friends with Everyone
A neuron in the brain is like a person handing out business cards at a crowded conference. At first, making acquaintances is easy, but as the address book fills up, each new contact requires more effort: there's little room left. The authors based their model on this 'crowding' rule: the more incoming connections a neuron already has, the harder it is to form another one.
Quantum Physics2026-03-151 minarXiv:2603.14292express
Quantum Recipe for Perfect Mixing
Physicists have discovered exact laws by which quantum entanglement mixes in chaotic systems. In a model with rhythmic jolts — like a baker kneading dough — they studied the growth of entropy of entanglement, a measure of quantum disorder.
Chaotic Dynamics2026-03-142 minarXiv:2603.13789express
Why Some Vortices Are Predictable and Others Aren't
In a turbulent flow, like a raging river, energy constantly cascades from large whirlpools to small ones. Occasionally, particularly powerful bursts occur, resembling sudden waves crashing against the shore. Scientists asked: can we know in advance when such a "wave" will strike? It turns out the answer depends on which vortices are present in the flow at that moment.
Dynamical Systems2026-02-281 minarXiv:2603.00709express
How to Pick the Best Model Without Reading Tea Leaves
To understand a mechanism, you have to poke it. A child taking apart an alarm clock doesn't just stare at the gears: they turn the wheel and listen to how the ticking changes. Scientists do the same: instead of passive observation, they feed specially chosen signals into the system and watch the response.
Entropy as a Source of Particles: From Bits to Matter
In 1989, John Wheeler proposed a bold idea: "it from bit," meaning everything that exists arises from information. For a long time, this remained a beautiful metaphor, but now physicists have found a concrete mechanism that turns bits into particles.
The Eraser That Erases the Past: How Quantum Information Creates Irreversibility
A wave rolls onto the sand, leaving a pattern of foam and shells. A moment later it recedes, but the traces are no longer the same: the sand is smoothed, the shells are flipped over. Something similar happens in the quantum world when a system moves from a nonequilibrium state toward equilibrium.
Disentangling Complexity: How to Tell a Direct Signal from a Chorus of Voices
An orchestra without a conductor plays a complex piece. The listener hears the overall music but wants to understand who leads the melody and who merely echoes it in unison.