Neural Network Explanations: Where the Model Looks for a Pulse and Whether to Trust It ✛
A treasure map drawn by a neural network promises to show where the gold is buried. But if you take that map to another island, the marks may point to nothing. That is exactly what happened with the explanations from a model that estimates pulse from facial video.
The RhythmFormer neural network picks up subtle color changes in the skin caused by heartbeats. To understand what it looks at, the researchers applied four explanation methods and measured two quantities: skin coverage (the fraction of model attention that lands on the face) and fidelity score SaCo (how well the importance ranking of regions matches their actual influence on the prediction).
However, within a single participant, neither skin coverage nor SaCo correlated with pulse error: 186 of 252 coefficients fell below |ρ|=0.10. The treasure map did not show where the gold was buried for a specific person.
But across eight scenarios, skin coverage from the Beyond Intuition method moved together with estimation quality: ρ=−0.43 for error, +0.57 for waveform correlation, and +0.43 for signal-to-noise ratio. Explanations carry additional information, but they do not replace accuracy metrics.
This work is the first to systematically test the transfer of explanations across datasets and their connection to model reliability. The results will help avoid false confidence in the behavior of neural networks that monitor health from facial videos.
Under the dimmest lighting (40 lux), the Beyond Intuition model stopped concentrating on the skin, and its explanations became worse than random. Yet motion, which degraded pulse accuracy more severely, did not cause such a breakdown in explanations.
The idea of reading a pulse from the face without contact echoes the medical scanners of science fiction, such as the tricorder from Star Trek, which remotely assesses vital signs. The question of whether to trust AI explanations is also raised in Westworld, where the behavior of artificial intelligence remains opaque to its creators.