concept

attention mechanism

Attention mechanism is a computational process that dynamically focuses on relevant parts of input data, used in gaze coordination, sensory processing, and motor control.
28 related articles · 1 formulas

Description

The attention mechanism is a computational technique that allows a model to dynamically weigh the importance of different parts of the input data. Unlike fixed convolutions, attention adapts to each specific example: the word "crane" (water tap or construction machine) will get different surroundings depending on the context. Like a director moving the spotlight from one actor to another, attention redirects light according to the mise-en-scène.

History

The concept of attention in AI goes back to ideas from the 1960s about visual search. In neural networks, the first implementations were alignment systems for machine translation (Bahdanau et al., 2014). Then the mechanism became modular with "Attention is All You Need" (2017), where it was transformed into a self-contained building block — self-attention in the transformer.

How it works

Three matrices are computed: queries Q, keys K, values V (linear projections of the inputs). For each query, dot products are calculated with all keys, scaled by √d_k, and transformed via softmax into a probability distribution. These probabilities are used as weights for summing the values. Multi-head attention repeats the procedure several times with different projections, capturing various types of relationships.

In practice

Smart cameras that track motion, and translation systems that grasp the meaning of a long phrase, work exactly like that.

Fun fact

The human visual system also has an analogue — selective attention: the brain amplifies the signal from the desired object and suppresses the rest.

Concept graph →

Formulas

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$
Attention weights are computed from queries and keys, scaled by dimension, then applied to values.
What is in the formula
QvariableQuery matrix — query matrix
KvariableKey matrix — key matrix
VvariableValue matrix — value matrix
d_kvariableDimension of keys — dimension
softmaxoperatorNormalizes rows to probability distributions — softmax function
⊤operatorMatrix transpose — transpose operator
√operatorSquare root — square root
w_t(u,v)=\max(0,\frac{(q_t^v)^\topk_t^u}{\sqrt{r}}z_t^u) · 2607.01148

Related articles · 38

The Piston That Waits for Emptiness
A piston in a cylinder with gas usually moves when the gas pushes on it. But there is a piston that moves when the space in front of it is empty. Every five seconds, it checks the space near its wall, and if no one is there, it takes a step inward, compressing the gas without direct mechanical force.
Physics and Society2026-08-142 minarXiv:2608.14398express
When being together is a choice
Two people agree to meet, but each has their own things to do. One hesitates: what if the other doesn't come? And decides not to go. The other thinks the same. The meeting doesn't happen. That's the simple mechanism the authors dissect: a joint endeavor requires everyone to invest time, but the payoff depends on whether the partner participates. In the model, each person keeps a to-do list ordered by importance.
Biological Physics2026-08-081 minarXiv:2608.08096express
The Ear as a Living Amplifier
Sound enters the ear and hits a thin membrane inside the cochlea, the spiral organ in the inner ear. This membrane doesn't just vibrate like a string; it actively pushes the vibrations when they are weak and stops when they are strong. That's how the ear hears both a whisper and a loud conversation. Scientists built a model where each piece of the membrane is a tiny pendulum with a spring.
Movement Alters Decisions
A person flips a coin to make a decision. They stand still — the coin lands heads. They take a few steps, and flip again. It would seem walking shouldn't affect the coin. But what if the walk itself changes how often they decide to simply repeat the previous toss? That's exactly what researchers tested.
Statistical Mech2026-08-072 minarXiv:2608.07465express
Entanglement Takes the Slow Lane
Hot water sometimes freezes faster than warm water: this odd fact is known as the Mpemba effect. Now something similar has been found in the quantum world: a state with weak entanglement (a special quantum connection between particles) can beat a more entangled one to the finish line.
Machine Learning2026-08-022 minarXiv:2608.01481express
Reading Thoughts from Magnetic Fields
A magnetoencephalograph, a helmet-like device with hundreds of sensors, picks up the weak magnetic fields generated by active neurons. When a person listens to speech, these signals contain imprints of the heard words. But extracting them is tricky: signals from thousands of sources merge into a tangled mess.
Machine Learning2026-08-011 minarXiv:2608.00776express
Chemistry Looks at Molecules Through the Eyes of a Computer
Predicting the yield of a chemical reaction is a bit like trying to guess the harvest knowing only the soil composition. Traditionally, chemists have relied on long columns of numbers, quantum descriptors, that describe electron energies but say nothing about molecular shape.
Quantum Physics2026-07-252 minarXiv:2607.23179express
A Braid of Light Cools a Mirror
A laser beam can do more than heat things up: it can also calm motion. When light enters a mirror trap, it exerts pressure on the mirrors, and if you tune the beam's color just right, that pressure becomes a gentle braking force. This is how tiny mechanical vibrations are cooled to nearly absolute zero.
How the Brain Conducts Attention: A Meditation Model
The meditating brain is an orchestra, and attention is the conductor. When you're focused on your breath, all instruments play in harmony. But suddenly, a thought-cello plays a false note: that's mind wandering. An experienced conductor doesn't panic: they instantly catch the dissonance and gently guide the orchestra back to the main melody — the breath.
eess.AS2026-07-151 minarXiv:2607.14310express
Cooking a Living Voice: The Russian Dataset That Teaches Assistants to Feel
Voice assistants often sound like a chef reading a recipe from a card with zero appetite. The new Russian dialogue archive, “Dialogues,” flips the script: three studio actors, like master chefs, blended 20 hours of live conversation with twelve emotional “spices.” Each line got tagged for style and emotion — creating a complete book of flavors.
Machine Learning2026-07-141 minarXiv:2607.12771express
AI Masters the Cookbook of Chemical Reactions
Cooking a complex dish requires a clear recipe: first chop, then sauté, add spices at just the right moment. Chemical transformations also follow step-by-step recipes — mechanisms where atoms of carbon and hydrogen rearrange in a strict order. Previously, neural networks often invented nonexistent steps, like a whimsical cook swapping salt for sugar.
cs.AR2026-07-141 minarXiv:2607.13106express
Teaching Cameras to See Faster and Cheaper: On-Chip Neuromorphic Video Processing
Modern machine vision systems are gaining autonomy: engineers have managed to implement an image recognition algorithm directly on a miniature chip, freeing the camera from having to send data to a bulky computer.
Actions are recognized by posture, not by trajectory
Researchers took 16 everyday activities — from washing dishes to squatting — and broke the movements down into elementary components, much like a spectroscope splits light into a spectrum. Comparing three approaches, they found an unexpected key to recognition: a single averaged snapshot is enough. Only nine joints out of dozens carried the crucial information — the rest hardly mattered.
Electron Jazz: How Neural Network Self-Attention Unveils a Quantum Symphony
The quantum many-body world is a jazz ensemble, where each electron improvises, continuously listening to all the others, and the overall wave function is born from this tense dialogue.
Medical Physics2026-06-072 minarXiv:2606.08618express
How to Press a Coil to the Head and Not Miss
A flashlight shines through skin and bone to gently switch on neurons at the right spot in the brain. That is how transcranial magnetic stimulation works — a coil on the head creates brief magnetic pulses that trigger electrical signals in cells. But if you tilt the flashlight or move it away, the light goes astray.
When the Mind Honestly Says 'I Don't Know'
A person looks at a door lock and sees a keyhole. They know that a key is inserted and turned. But if instead of a keyhole there's a touch panel, the familiar frame breaks. The person doesn't just make a mistake: they realize that the usual approach won't work and can say, "There's no place for a key here; a different approach is needed." Such an honest stop with an explanation is rare for machines.
Quantum Physics2026-05-201 minarXiv:2605.21206express
How Motion Teaches a Detector to Distinguish Directions
An ordinary photometry detector can't tell a left photon from a right one. But let it start moving, and everything changes. Imagine a rower moving between two flutists playing the same note. His ear is perfectly tuned to that note and ignores all others. As he moves, the pitch from the flutist ahead sounds higher, and from the one behind sounds lower.
Statistical Mech2026-05-092 minarXiv:2605.09148express
How to Tune a Molecular Motor Without Losing Your Mind
Imagine a factory where hundreds of tiny assembly lines run nonstop. Each line is a molecular motor that converts chemical fuel into motion. To make the motor go faster or in the right direction, an engineer tweaks the settings — the shape of parts, the strength of interactions.
q-bio.QM2026-04-272 minarXiv:2604.24365express
Stepping to the Metronome: How Rhythm Breaks the Memory of Walking
The room is quiet, just footsteps and an internal count: “one-two, one-two.” After a minute, the gait stiffens, legs move like pistons. Forget the count and just walk — the body relaxes, the step becomes smooth. The difference between these two states is not just a feeling. It's written in numbers, and those numbers tell the story of how the brain controls movement.
The Brain's Conductor: How the Parietal Region Switches Modes
The brain constantly juggles tasks: it reacts to a sharp sound, solves an equation, or drifts into daydreams. Scientists have long suspected that switches between different "scenarios" of neural network activity underlie this, but extracting them from the chaos of electrical signals was difficult.
Computer Vision2026-04-251 minarXiv:2604.23137express
Two Eyes Are Better Than One: How a Neural Network Learns to See Tumors
A doctor studies a brain scan: somewhere in the tangle of tissue, a tumor is hiding. One glance picks out small spots and edges, another takes in the overall shape and location. A person combines both, but across hundreds of scans that gets tedious. A new program does the same thing automatically.
Signal Processing2026-04-041 minarXiv:2604.24765express
The Brain Translator: How Different People Hear the Same Signal
A person looks at flashing letters on a screen, and a computer guesses which one they chose by a burst of brain activity. It's like an interpreter who catches not words but the brain's electrical signals. But different people's brains 'speak' with an accent: one person's signal is bright and clear, another's is quiet and smeared.
Disordered Systems2026-03-151 minarXiv:2603.14534express
A Powder That Remembers Motion
A handful of sand on a table is just a pile. But if you shake the grains just right, the pile starts behaving like a memory: it remembers which motion follows which. In the new experiment, particles were moved cyclically, and after training, they reproduced the desired pattern even without a full command. The secret lies in the contacts between particles.
HEP Experiment2026-03-021 minarXiv:2603.02369express
The Detector That Never Tires
A particle detector at the Large Hadron Collider runs for years without stopping. Its sensors produce mountains of data every second, and among it all, you need to spot a fault quickly, like overheating or a broken channel. In the past, simple rules did this, but they often made mistakes.
Machine Learning2026-03-011 minarXiv:2603.02273express
Alzheimer's Genes: A Fresh Look Through Attention
Alzheimer's disease is like a tangled ball of yarn: thousands of genes are intertwined, and it's hard to tell which ones snap first. Previously, scientists looked for important genes by simply counting how many connections each node in the network had. But that approach is like judging a person by the number of acquaintances they have — the most sociable isn't always the most influential.
Dynamical Systems2026-02-281 minarXiv:2603.00709express
How to Pick the Best Model Without Reading Tea Leaves
To understand a mechanism, you have to poke it. A child taking apart an alarm clock doesn't just stare at the gears: they turn the wheel and listen to how the ticking changes. Scientists do the same: instead of passive observation, they feed specially chosen signals into the system and watch the response.
Machine Learning2026-02-282 minarXiv:2603.00786express
How the Brain Rewires Itself in Alzheimer's Disease
The brain isn't one big computer: it's more like an orchestra, where different groups of musicians play their parts. Neural networks for memory, attention, or emotions are constantly talking to each other. In Alzheimer's disease, this orchestra starts to go off-key: some groups fall silent, others play out of sync.
Machine Learning2026-02-212 minarXiv:2602.18679express
How a Neural Network Predicts the Unseen
Turbulent water flowing through a pipe and the swing of a pendulum may seem worlds apart. But they share something fundamental: both follow their own laws of motion, which can be described by numbers. A neural network trained on just one such 'dance of numbers' suddenly starts to understand another — without a single extra example.
physics.data-an2026-02-211 minarXiv:2602.18949express
A Forecast That Remembers the Cycle
A solar panel on a roof produces little current in the morning, a lot at noon, and stays silent at night. Wind doesn't blow randomly either: it's often stronger in the morning and evening than during the day. Energy providers need to know in advance how much energy will be available in an hour or three, so they can switch on backup plants or store electricity in batteries in time.
q-bio.GN2026-02-192 minarXiv:2602.17532express
What Smart Cells Aren't Saying: Attention Without Payoff
Inside a cell, work is in full swing: thousands of genes switch on and off like lights on a huge display. Neural networks trained to read this display have learned to guess what will happen if you 'turn off' one gene or add a drug. But how exactly they do it is a mystery.
Too-Bright Flash Blinds the Camera
A camera that captures not light but a stream of electrons reflected off a crystal is a hybrid pixel detector. It can count every electron that hits it, generating almost no noise of its own. In normal mode, when the electron beam flows continuously, the detector works perfectly.
The Gaze's Signature: How Eyes and Hand Agree on Timing
When the hand draws a line, the gaze almost always runs ahead, like a scout peeking around the corner before the column turns. But how far ahead? It turns out that each person has their own distance, and it does not change from attempt to attempt. In the experiment, professional calligraphers and ordinary people traced a contour on a screen while instruments recorded eye and pen movements.
Memory Under Load: How Anxiety Steers Recollections
You're trying not to think about the argument you had this morning, but vivid images keep popping into your head. At the same time, you're preparing a report and holding numbers in your mind. It doesn't go well: memory interferes with work, and work prevents forgetting. Scientists set out to see how the brain copes with this dual load in people with varying levels of anxiety.
How a Pause Becomes a Reward
A pea-sized brain region brings a rat to a halt mid-step. A laser beam hits light-sensitive cells, and the animal stops as if it hit an invisible wall. This is the rostral pedunculopontine nucleus — a cluster of neurons in the brainstem that controls movement. The authors built a low-cost setup with a camera and a mini-computer.
Memory in Optomechanics: A Laser Conductor Teaches Light and Sound to Remember
In a dark room, a resonator catches light the way a spiderweb catches dew drops. A laser pulse bursts in, and the mechanical membrane starts to vibrate in sync, leaving a trace in the very fabric of the system. That is how memory is born: not in a silicon cell, but in the dance of photons and phonons.
IAFormer: Sparse Attention for Collision Analysis
At the Large Hadron Collider, millions of jets are born every second: streams of particles flying out from a single quark or gluon. Telling a jet from a top quark apart from a jet from an ordinary quark is a task on which the hunt for new physics depends.
SFANet: a translator that hears both words and intonation
A translator at negotiations picks up not only the words but also the tone in which they are spoken. If they focus only on vocabulary, they might miss sarcasm or fatigue in the voice. If only on intonation, they could mistranslate the details of the contract.
Attention Cubes for Weather Forecasting
A meteorologist looks at a map where clouds gather in flocks, like birds before a storm. Each spot on the radar could be a future downpour or just noise. Earth system forecasting is like trying to assemble a puzzle with thousands of pieces that keep changing shape.