Show the same photo of a cat to two different neural networks, and they often reach a similar conclusion: "cat." But ask them for more detail, and their answers may diverge. One model focuses on the whiskers, another on the coat color, a third on the pose. So even within the "visual" world of neural networks, there's no complete unity.
Researchers devised a way to measure this unity for each image individually. They took several visual neural networks with different training tasks and compared their internal representations. For some images, the models are nearly unanimous; for others, they diverge significantly. Then they showed these images to language models that describe images in words. It turned out: when visual models agree with each other, their collective opinion aligns better with the language description. The difference can be twofold.
This result suggests that agreement among neural networks is not a coincidence. It reflects how "understandable" the image turned out to be. The clearer its structure, the easier it is for both vision and language to find common ground. This approach helps reveal where neural networks converge with human perception and where they diverge.
as written by the authors arXiv:2604.21836
Neural networks exhibit a remarkable degree of representational convergence across diverse architectures, training objectives, and even data modalities. This convergence is predictive of alignment with brain representation. A recent hypothesis suggests this arises from learning the underlying structure in the environment in similar ways. However, it is unclear how individual stimuli elicit convergent representations across networks. An image can be perceived in multiple ways and expressed differently using words. Here, we introduce a methodology based on the Generalized Procrustes Algorithm to measure intra-modal representational convergence at the single-stimulus level. We applied this to vision models with distinct training objectives, selecting stimuli based on their degree of alignment (intra-modal dispersion). Crucially, we found that this intra-modal dispersion strongly modulates alignment between vision and language models (cross-modal convergence). Specifically, stimuli with low intra-modal dispersion (high agreement among vision models) elicited significantly higher cross-modal alignment than those with high dispersion, by up to a factor of two (e.g., in pairings of DINOv2 with language models). This effect was robust to stimulus selection criteria and generalized across different pairings of vision and language models. Measuring convergence at the single-stimulus level provides a path toward understanding the sources of convergence and divergence across modalities, and between neural networks and human neural representations.
Agreement among visual neural networks for the same image can vary by a factor of two, as if one photo is "more understandable" than another for artificial intelligence.