Anatomy Without a Map: How Medical Models See Space ✛
Context
Interpreting CT scans is a chain of spatial judgments: find an organ, measure it, compare it with its paired counterpart, assess the distance to a neighboring structure. In the clinic, an error at any step can change the diagnosis. But current models are evaluated on the final answer, sick or healthy, not on how they arrived at it. Surrounding this work are studies that either analyze how features are preserved in network layers or build multimodal systems and blame errors on the encoder or the language model. The authors took a step aside: they isolated the visual encoder and asked it directly what it knows about space.
Methods
The authors built SPAR-Bench on the WORD dataset, 150 abdominal CT scans with 16 organs annotated. From the slices, they extracted tiles containing individual organs, removed context, and assembled three levels of tasks. The first level is localization: the model must predict the coordinates of the organ's center. The second is order restoration: tiles are shuffled, and the model must put them back in place. The third consists of six types of questions about a whole slice: is the liver to the left of the spleen, is the image flipped, which kidney is larger, what fraction of the area does an organ occupy, which distance is greater, and how many tumors are visible. They tested five configurations, ResNet-50 and ViT-B/32 with MLP or Transformer heads, plus a combined variant, as well as three medical foundation models: RadDINO, BiomedCLIP, and SAM-Med2D, both frozen and fine-tuned. To control for memorized answers, some questions were asked on mirrored slices, and then all models were tested on the external TotalSegmentator set and compared with four open multimodal models.
Results
The main result: bilateral symmetry remained at chance level across all 13 configurations, when transferred to external data, and in multimodal models. Accuracy never exceeded 36% against a 33% chance level. At the same time, relative organ position within WORD reached 99%, but when transferred to TotalSegmentator it dropped to 50–70%, and for one model to 50.03±0.04, indistinguishable from randomness. This means that high scores reflected memorization of typical anatomy, not computation from the scan. Another finding: the way features are read changes the conclusions. Frozen BiomedCLIP with an MLP head restored tile order in only 0.7% of cases, while with a Transformer head on the same features it achieved 67.8%. In other words, pooling hides information that exists in the tokens. Fine-tuning RadDINO improved order restoration (from 22% to 69% for MLP) but worsened localization (from 9.88 to 11.46 pixels of error), showing a trade-off between abilities. Multimodal models answered at chance level on questions that encoders solved almost perfectly, and vice versa: on organ size, the spread among them was 75 percentage points.
Implications
This work stands apart from its neighbors: they evaluate either the final diagnosis or feature preservation, whereas here functional spatial reasoning is tested in an isolated encoder. This is the first attempt to separate 'knowing the map' from 'measuring on the ground'. The finding that pooling hides relational information changes how we read scores from frozen models: a low score with an MLP head does not mean the ability is absent. And the failure on symmetry points to a fundamental gap: models do not compare paired structures, which is critical for screening, for example, renal artery stenosis.
Future development
The topic could evolve toward three-dimensional probes: 2D slices cannot test relationships that require volume, such as organ nesting or vessel adjacency. Other body regions and modalities (MRI, X-ray) will provide new tests. A separate question is why multimodal systems lose the encoder's abilities: experiments isolating the projection layer and the language model are needed.
Impact
The results will affect the evaluation of medical vision models and the design of computer-aided diagnosis systems, where spatial comparisons are routine.
Next steps
The immediate next steps are to check whether the symmetry task is solved when the organ mask itself is provided, to separate annotation noise from the model's limit, and to test the probe on three-dimensional data.
Key open problems
For medical imaging, this connects to a long-standing problem: models learn from annotations that encode typical anatomy, but they do not distinguish 'knowing where an organ usually is' from 'seeing where it is now'. The work shows that standard transfer learning metrics mask this gap, so we need probe tasks that cannot be solved by memory.
The knowledge machine's view
This section is addressed to the author of the paper and their colleagues. Neighbouring works were found by meaning-based search across our archive, not by word overlap. We do not review: we show the field around the work.
What the work does
The authors built SPAR-Bench on the WORD dataset: they cut out organ tiles, removed context, and assembled tasks at three levels: localization, order recovery, and whole-slice questions. They tested 13 encoder configurations and multimodal models, including transfer to TotalSegmentator. The main result: bilateral symmetry remained at the level of random guessing, while high scores for relative position reflected memorization of typical anatomy rather than computation from the image.
Where its strength is
The work isolates the visual encoder and asks it directly about spatial judgments, which makes it possible to separate memorization of anatomical templates from genuine spatial reasoning. This provides a clear tool for diagnosing weaknesses in medical models.
Where it could go next
- It might be interesting to see how models handle organ size and shape estimation, taking the approach from the work on growing organoids: there, measuring dynamics required special tuning for continuous changes, whereas in SPAR-Bench the tasks are static.based on: How a Computer Learned to Measure Growing Organoids arXiv:2605.03053
- A related avenue is training with gaps: if the model does not see the full context, it learns to reconstruct what is missing. One could test whether such training improves spatial judgments on context-free tiles.based on: Doctors Don't See Everything: A Model Learns from Gaps arXiv:2604.04175
- There is a paper in the archive on how combining two modalities improves tumor visibility. It might be worth checking whether a multimodal representation (text plus image) gives a boost on SPAR-Bench tasks, using not a ready-made multimodal model but a specially trained projector.based on: Two Eyes Are Better Than One: How a Neural Network Learns to See Tumors arXiv:2604.23137
Worth trying
- Test whether the symmetry failure persists when the model is given not cropped tiles but the full slice with an organ mask.
- Fine-tune the encoder with a contrastive objective on pairs of mirror-reflected and original slices to encourage orientation invariance.
- Use augmentations that distort organ shapes to separate learned typical anatomy from computation based on the specific image.
Why this may matter
If models cannot tell left from right at the level of individual organs, this calls into question their use in tasks where laterality is critical, such as surgical planning or asymmetry assessment. Understanding which architectural changes fix such errors could directly affect the safety of medical AI.
Nearby works in our archive
- How a Computer Learned to Measure Growing Organoidssimilarity 0.53
- Doctors Don't See Everything: A Model Learns from Gapssimilarity 0.52
- How a Bladder Turns into a Pearsimilarity 0.52
- Two Eyes Are Better Than One: How a Neural Network Learns to See Tumorssimilarity 0.52
- How a Tomograph Gathers Shadows to See the Invisiblesimilarity 0.51
- Your Smartphone Instead of an X-Ray: How a Photo of Your Body Predicts Healthsimilarity 0.51
The archive is sparse around this work: the closest review is only 0.53, against a usual 0.64. It looks like few have drilled nearby.
The WORD dataset has 150 CT scans, but only 20 volumes with LiTS annotations were used for counting tumors, a rare case where the task is complicated by a lack of data, not an excess.
Key numbers
- maximum symmetry accuracy: 36.4%
- drop in relative position upon transfer: from 99% to 50%
- jump in order restoration when changing head: from 0.7% to 67.8%
- best localization error: 8.95 pixels
- spread among multimodal models on organ size: 75 percentage points