statistics

sigma clipping

Discarding points beyond a chosen number of standard deviations, iteratively — the common defence against outliers.
14 related articles

Description

Sigma clipping is a robust statistical technique for cleaning data by removing outliers. The idea is simple: compute the mean and standard deviation of a dataset, discard any points that lie more than a chosen number of standard deviations (the 'sigma') away from the mean, then recompute the mean and standard deviation on the remaining points and repeat. This iterative process converges to a stable set of 'good' data points, making it a standard first line of defense against spurious measurements in physics and astronomy.

History

The technique has been used informally in statistics since the early 20th century, but it became a standard tool in astronomical data reduction in the 1980s with the advent of CCD cameras and automated surveys.

How it works

Start with a dataset and a threshold, typically 2–3 sigma. Calculate the mean and standard deviation. Flag every point whose absolute deviation from the mean exceeds the threshold times the standard deviation, and remove those points. Recalculate the mean and standard deviation on the surviving points, and repeat until no more points are flagged. The procedure is sensitive to the initial estimate, so it is often run a few times with different thresholds to ensure stability.

In practice

It is used to remove cosmic ray hits, bad pixels, and other artifacts from astronomical images and spectra, as well as to clean time-series data before period-finding or model fitting.

Fun fact

Sigma clipping is so common in astronomy that it is built into nearly every major image-processing package, from IRAF to modern Python libraries like astropy.

Concept graph →

Related articles · 14

Galaxies2026-09-064 minarXiv:2609.06493
Two Windows into One House: How Fiber Aperture Reshapes the Galaxy Portrait
Two astronomical surveys look at the same galaxy through different windows. DESI's window is narrow, just 1.5 arcseconds across, and it catches only the bright core. SDSS's window is twice as wide, 3 arcseconds, and it also includes the outer spiral arms.
Quantum Physics2026-08-141 minarXiv:2608.14327express
Quantum snapshots yield a precise angle
A handful of coins drops onto a table — heads and tails land wherever they please. The same goes for an instantaneous snapshot of a quantum chain: in some places a particle is present, in others it is not. But behind the disorder lurks a precise angle. In this work, a chain of particles closed into a ring was considered. It was divided into four arcs: two were measured, two were left untouched.
Cosmology2026-07-291 minarXiv:2607.26741express
Calibrating a Cosmic Radio Receiver
The cosmic dawn is hidden behind a dense veil of time. To see beyond it, you need to catch the radio signal from clouds of neutral hydrogen — the so-called global 21 cm signal. But it is so faint that the receiver's own noise drowns it out.
Image Processing2026-07-141 minarXiv:2607.12586express
Water's Geometry Improves Medical Scans
Medical scans are maps of tissue density. The brightness of each pixel reveals what’s there, much like photometry measures light intensity. But conventional algorithms analyze pixels in isolation, producing jagged organ boundaries. A new approach compels the program to prioritize smoothness.
eess.SY2026-07-131 minarXiv:2607.10986express
Flexible Swarm: From Pursuit to Defense Without Reprogramming
A wolf pack switches from chasing prey to defending its den in an instant. Engineers have instilled this same versatility into an algorithm for a robot swarm. The key is a method proposed by John von Neumann: instead of chasing the target, the robots calculate its future moves and block escape routes, like beaters cutting off trails.
math.NA2026-07-012 minarXiv:2607.00335
Balancing on a Quantum Tightrope: How Numerical Methods Tame the Dirac Equation’s Tremors
A tightrope walker steps onto the wire. Every move is a calculation: shift your weight just right, or you’ll fall. In the world of quantum particles described by the Dirac equation, there’s a different kind of abyss — the wave function’s furious oscillations.
stat.ME2026-06-242 minarXiv:2606.26061express
A New Way to See the Hidden in Data Chaos
A lab robot counts molecules, and gets it wrong. One tube shows 0, another 12, a third 3, though in reality they all hold roughly the same amount. Such 'raw' numbers are common in biology: DNA sequencing, counting species in a water sample, gene activity. There's plenty of data, but it's sparse and more scattered than simple randomness would predict.
Molecular Ballet Without a Choreographer
A dance of hundreds of particles: some pairs spin slowly and gracefully, others twitch in a frantic rhythm. To understand the essence of the dance, you need to watch the slow movements: they are the main choreography of the system. In molecular science, such slow movements are called 'slow modes,' and they determine how a protein folds into a ball or how a drug binds to its target.
Fluid Dynamics2026-06-122 minarXiv:2606.14696express
Why Sugar in Tea Doesn't Mix the Way We Thought
A lump of sugar drops into hot tea, a spoon stirs a few times, and the sugar disappears. But how exactly does it spread through the churning liquid? Physicists have peered into this process with unprecedented detail: they simulated turbulence on a grid of 8192³ points. If you were to split every millimeter of the cup into a hundred cells, you'd get a similar resolution.
cs.PL2026-06-072 minarXiv:2606.09930
Compile Once, Differentiate Everywhere
Imagine an interpreter sitting in a meeting room, translating every word. But then they notice their translation could improve if they learned from their mistakes. They start noting which phrases caused confusion and adjust them. Within an hour, their translation is more accurate and the meeting runs smoother.
Eye Movements Reveal Autism in One in Three
Eyes follow a dot that smoothly traces a complex curve on the screen. For most people, the gaze stays close to it, but for some, it seems to slip or lag behind. Previously, such peculiarities drowned in average numbers: if you add everyone up and divide, the difference almost disappears. Instead of looking at averages, the scientists decided to examine each person individually.
Optics2026-02-141 minarXiv:2602.13714express
Light Solves Problems with Greater Precision
Light passing through a lens bends slightly. For an optical computer, this is like a warped mirror for an artist: the image is there, but the details blur. An Ising machine using light beams tackles problems like packing luggage or planning a courier's route.
Materials2026-01-242 minarXiv:2601.17262express
Crystal Map: How to Sort Through Millions of Images
An electron microscope scans a sample and records how electrons scatter off atoms at each point. The result is a giant stack of images — one for every point. These images encode information about how atoms are arranged: in neat rows or with a tilt, the orientation of the crystal lattice, and whether there are internal strains.
math.NA2024-07-152 minarXiv:2407.10931
The Climate Clockwork: New Math Captures Seasonal Rhythms
Every winter, the Pacific Ocean warms slightly off the coast of Peru, and every summer it cools. This rhythm is set by the gears of the climate machine, but some turn fast, others slow, and still others speed up or stall. Linear inverse models have helped us glimpse these gears in data for decades, but they were blind to seasonal changes in pace.