A process of choosing a subset of relevant features for use in model construction.Feature selection is the practice of deciding which of the available input variables a model should actually use. In any real dataset, some columns carry genuine signal about the outcome, some are redundant with each other, and some are pure noise. Keeping everything tends to make a model slower, harder to interpret, and — counterintuitively — less accurate on new data, because the model can latch onto accidental patterns that do not generalize. Feature selection trims the input down to a smaller, more informative set.
It matters because the cost of a model is often dominated by its inputs. In medicine, a diagnostic rule based on three cheap measurements is far more usable than one based on three hundred, even if both perform similarly. In genomics, where a study may start with a million genetic markers and only a handful matter, selection is not a convenience but a necessity.
The concept sits alongside feature extraction, which creates new variables by combining old ones (as principal component analysis does). Selection keeps the original variables and discards some; extraction replaces them. Both aim at the same goal: a representation of the data that is compact without throwing away what predicts the outcome.
Methods fall into three broad families. Filter methods score each variable on its own — by correlation with the target, for instance — and keep the top-ranked ones; they are fast but blind to how variables interact. Wrapper methods repeatedly train the model on different subsets and keep whichever subset performs best under cross-validation; they capture interactions but are computationally expensive. Embedded methods build selection into training itself: lasso regression, for example, shrinks the coefficients of useless variables all the way to zero, effectively deleting them.
In every case the evaluation must be done on data the selection did not see. If you choose features using the whole dataset and then report accuracy on that same dataset, the result is optimistic, sometimes wildly so.
Used throughout machine learning and statistics — in biomarker discovery, text classification, sensor-network design, and anywhere a model must run under constraints on input count or interpretability.
Selecting features using the full dataset before cross-validating can make a model built on pure noise appear to predict the outcome, a failure mode known as selection bias or data leakage.
No articles yet