A Map Without a Compass: Why Medical Models Confuse Left and Right ✛
A radiologist looks at an abdominal scan and sees not just organs, but their spatial relationships: which kidney is larger, where the tumor is relative to a vessel. This is like reading a map, where it's important not only to know where a city usually stands, but also to be able to measure the distance between two points on the ground. Researchers from IIIT Hyderabad and AIG Hospitals built SPAR-Bench, eight probes that test whether medical vision models can perform such reasoning. It turns out that models answer questions about typical organ positions well, but as soon as the task requires comparison in a specific scan—for example, which kidney is larger—accuracy drops to random guessing.
At the core of SPAR-Bench is the WORD dataset, which contains 150 abdominal CT scans with 16 organs annotated. From the slices, they cropped tiles with individual organs, removed context, and assembled three levels of tasks. The first level is localization: the model must predict the coordinates of the organ's center. The second is order restoration: the tiles are shuffled, and the model must put them back in place. The third consists of six types of questions about a whole slice: is the liver to the left of the spleen, is the image rotated, which kidney is larger, what fraction of the area does an organ occupy, which distance is greater, and how many tumors are visible.
The conclusion brings us back to the map without a compass. The model remembers where organs usually are, like a student who has memorized an atlas, but cannot measure the distance between two points on a specific scan. This finding changes the rules for evaluating medical models: instead of the final diagnosis, we should test intermediate spatial judgments, which constitute the real work of a radiologist.
The WORD dataset has 150 CT scans, but only 20 volumes with LiTS annotations were used for counting tumors, a rare case where the task is complicated by a lack of data, not an excess.