
Attention mechanism is a computational process that dynamically focuses on relevant parts of input data, used in gaze coordination, sensory processing, and motor control.The attention mechanism is a computational technique that allows a model to dynamically weigh the importance of different parts of the input data. Unlike fixed convolutions, attention adapts to each specific example: the word "crane" (water tap or construction machine) will get different surroundings depending on the context. Like a director moving the spotlight from one actor to another, attention redirects light according to the mise-en-scène.
The concept of attention in AI goes back to ideas from the 1960s about visual search. In neural networks, the first implementations were alignment systems for machine translation (Bahdanau et al., 2014). Then the mechanism became modular with "Attention is All You Need" (2017), where it was transformed into a self-contained building block — self-attention in the transformer.
Three matrices are computed: queries Q, keys K, values V (linear projections of the inputs). For each query, dot products are calculated with all keys, scaled by √d_k, and transformed via softmax into a probability distribution. These probabilities are used as weights for summing the values. Multi-head attention repeats the procedure several times with different projections, capturing various types of relationships.
Smart cameras that track motion, and translation systems that grasp the meaning of a long phrase, work exactly like that.
The human visual system also has an analogue — selective attention: the brain amplifies the signal from the desired object and suppresses the rest.
| Q | variable | Query matrix — query matrix |
| K | variable | Key matrix — key matrix |
| V | variable | Value matrix — value matrix |
| d_k | variable | Dimension of keys — dimension |
| softmax | operator | Normalizes rows to probability distributions — softmax function |
| ⊤ | operator | Matrix transpose — transpose operator |
| √ | operator | Square root — square root |
















