main concepts scientists sections
SimplePopularAdvanced

Anatomy Without a Compass: How Medical Models See Space ✛

Original: "Do Medical Vision Models Reason About Anatomy? Probing the Spatial Inductive Biases of Learned Visual Representations"
· Naren Akash, Neeraja Ramanan
arXiv:2608.28092v1 · 2026-08-28 · CC BY-SA 4.0 · 4 min
Image Processing Artificial Intelligence Computer Vision Machine Learning
Medical encoders know where organs usually sit, but they can't compare them in a specific patient.

A radiologist looks at an abdominal scan and sees not just organs but their spatial relationships: which kidney is larger, where the tumor lies relative to a vessel. It's like reading a map, where it's not enough to know where a city usually stands; you also need to measure the distance between two points on the ground. Researchers from IIIT Hyderabad and AIG Hospitals built SPAR-Bench, eight probes that test whether medical vision models can perform such reasoning. It turns out that models answer questions about typical organ positions well, but when the task requires comparison within a specific scan, for example, which kidney is larger, accuracy drops to chance.

SPAR-Bench is built on the WORD dataset, which contains 150 abdominal CT scans with annotations for 16 organs. From the slices, they cropped tiles containing individual organs, removed the context, and assembled three levels of tasks. The first level is localization: the model must predict the coordinates of the organ's center. The second is order restoration: tiles are shuffled, and the model must put them back in place. The third level includes six types of questions about a whole slice: is the liver to the left of the spleen, is the scan flipped, which kidney is larger, what fraction of the area does an organ occupy, which distance is greater, and how many tumors are visible.

They tested five configurations: ResNet-50 and ViT-B/32 with MLP or Transformer heads, plus a combined variant, as well as three medical foundation models: RadDINO, BiomedCLIP, and SAM-Med2D, both frozen and fine-tuned. To control for learned answers, some questions were asked on mirror-flipped slices, and then all models were tested on the external TotalSegmentator set and compared with four open multimodal models.

The main result is that bilateral symmetry remained at chance level across all 13 configurations, when transferred to external data, and for multimodal models. Accuracy never exceeded 36% against a chance of 33%. In contrast, relative organ positions within the WORD set reached 99%, but when transferred to TotalSegmentator, it dropped to 50–70%, and for one model to 50.03±0.04, which is indistinguishable from randomness. This means that high scores reflected memorization of typical anatomy, not computation from the scan.

Another finding: the way features are read changes the conclusions. Frozen BiomedCLIP with an MLP head restored tile order in only 0.7% of cases, but with a Transformer head on the same features, it did so in 67.8%. That is, pooling hides information that exists in the tokens. Fine-tuning RadDINO improved order restoration (from 22% to 69% for MLP) but worsened localization (from 9.88 to 11.46 pixels of error), showing a trade-off between abilities. Multimodal models answered at chance level on questions that encoders solved almost perfectly, and vice versa, with a spread of 75 percentage points on organ size.

This work stands apart from its neighbors: they evaluate either the final diagnosis or feature preservation, while here the focus is on functional spatial reasoning in an isolated encoder. This is the first attempt to separate "knowing the map" from "measuring on the ground." The conclusion that pooling hides relational information changes how we read scores from frozen models: a low score with an MLP head does not mean the ability is absent. And the failure on symmetry points to a fundamental gap: models do not compare paired structures, which is critical for screening, for example, renal artery stenosis.

The takeaway brings us back to a map without a compass. The model remembers where organs usually are, like a student who has memorized an atlas, but cannot measure the distance between two points on a specific scan. This finding changes how we evaluate medical models: instead of the final diagnosis, we should test the intermediate spatial judgments that constitute the real work of a radiologist.

Whole graph →

The WORD dataset has 150 CT scans, but only 20 volumes with LiTS annotations were used for counting tumors, a rare case where the task is complicated by a lack of data, not an excess.

Concepts
computed tomography medical image segmentation medical image analysis brain scan neural networks in spatial mapping tumor segmentation mri machine phantom computer vision spatial resolution visual cortex tomograph medicine parietal region labeled data confusion matrix support vector machine protein language model frame self-supervised learning