
A sequence of data processing steps used to convert raw observations into scientific products.A pipeline is a sequence of data processing steps that transforms raw observations into final scientific products. In modern physics and astronomy, raw data from telescopes, detectors, or simulations is rarely usable directly; it must be calibrated, cleaned, filtered, and analyzed step by step. The concept is central to fields like radio astronomy, where signals from thousands of antennas are combined and processed, or in computational biology, where protein folding predictions require multiple stages of refinement.
The term 'pipeline' in computing dates back to the 1960s, but its application to scientific data processing became widespread with the rise of large-scale experiments in the 1980s and 1990s, such as those in particle physics and astronomy.
A pipeline typically starts with data acquisition, followed by preprocessing (e.g., removing noise or artifacts), then core analysis (e.g., fitting models or extracting features), and finally post-processing (e.g., visualization or statistical validation). Each step takes the output of the previous one, and the entire chain is often automated. In practice, pipelines are modular: individual steps can be swapped or tuned without redesigning the whole system, which is why they are used in everything from speech compression to neural network training.
Pipelines are used in radio telescope arrays like LOFAR or ALMA to convert raw voltage data into sky images, and in genomics to turn sequencing reads into annotated genomes.
In the LOFAR radio telescope, a pipeline processes data from 32 antennas in real time, effectively creating a virtual telescope with a resolution comparable to a single dish kilometers wide.






