Doubt is Given Away by Silence: AI Reads the Pauses express
A new system analyzes a short interview to recognize human doubt by combining text, voice, and facial expressions. But the main key to uncertainty turned out to be the dark matter of speech — pauses and filler words that standard speech systems erase like a black hole. It is these gaps, usually considered junk, that provided the most independent nonverbal signal.
Although language remains the strongest channel, the silence between words betrays doubt more accurately than intonation. Researchers also noticed that attempts to account for contradictory signals — for example, a smile with a sad voice — only confused the system. Simple threshold adjustment and breaking the signal into components allowed bypassing the standard model, accurately detecting doubt even on unfamiliar data. This approach will be useful in support services and emotional state assessment.
Original abstract
as written by the authors arXiv:2607.11120
We address ambivalence and hesitancy (A/H) recognition in the ABAW 2026 BAH Challenge: given a short interview video, predict whether the person shows signs of A/H. Our system combines affect-specialised text, audio, and visual representations with a small set of readable linguistic hesitation cues, fused by a reliability gate we call Affective Marker Fusion (AMF), and finished with a simple AP-weighted ensemble at a fixed decision threshold. We also introduce \emph{ASR-erased time}: speech recognisers delete fillers and hesitation pauses from the transcript, but the chunk timestamps keep the time those events took, and sixteen features built from these gaps form the strongest and most independent non-verbal channel we measured (AP $$0.718$$, correlation $$0.11$$--$$0.36$$ with all other members). Across controlled experiments we find three things: cross-modal conflict design does not reliably help on BAH; language is by far the strongest channel while affect-specialised audio is a useful second; and calibration matters more than architecture. Fitting ensemble weights and a threshold on the small validation split overfits: it scores $$0.741$$ macro-F1 on validation but only $$0.690$$ on the untouched test set. AP-weighting at a fixed threshold instead reaches $$\mathbf{0.731}$$ on test.
About 40% of spontaneous speech consists of pauses and fillers, which speech systems discard, losing valuable clues about uncertainty.