How AI Learns Not to Peek at the Cheat Sheet express
The AI diagnostician learns like a med student before an exam. You have an X-ray and patient complaints. But sometimes, a ready-made doctor's report pops up in the training materials — the full answer. Then the neural network memorizes it, and test accuracy jumps to 98%. However, that's cheating: real skill isn't developed.
Without hints, accuracy from symptoms alone reaches 75%. Adding the image — 78%. But leaking even a couple of words from the report can double the results, creating a 'look-ahead bias' — the AI seems like a genius, but only within the training set. Honest models master flexible data fusion, filtering out cheat sheets.
Original abstract
as written by the authors arXiv:2607.13800
Chest radiograph datasets often combine multiple images with Clinical Indication, Findings, and Impression, although these inputs are produced at different stages of care. We evaluated 15,000 ReXGradient-160K studies with two readable images and five CheXbert-derived report observations. Frozen DenseNet-121 and Bio+ClinicalBERT encoders were used to compare image-only, Indication-only, fixed-order multimodal, random-swap, DeepSets, and SectionGuard-MI models. Findings and Impression were evaluated only as post-hoc leakage controls. Models were trained with five seeds, and public-test uncertainty was estimated with 2,000 patient-cluster bootstrap replicates. Under U-Ones, macro AUROC was 0.643 for the primary image, 0.694 for two images, 0.749 for Indication, and 0.780 for ordinary two-image-plus-Indication fusion. SectionGuard-MI achieved AUROC 0.783 and AUPRC 0.260. Relative to ordinary fusion, its paired AUROC difference was 0.0031 (95% CI, -0.0042 to 0.0104; adjusted p=0.374), while its AUPRC difference was 0.0289 (95% CI, 0.0095 to 0.0413; adjusted p=0.004). DeepSets had the highest prospective AUROC point estimate (0.787), and random-swap fusion had the highest prospective AUPRC point estimate (0.265) with better calibration than SectionGuard-MI. Full report text alone reached AUROC 0.979 and AUPRC 0.836; AUROC remained above 0.973 after exact or expanded masking. These results show that prospective Indication is strongly associated with report-derived targets, permutation-aware fusion is competitive, and post-hoc report text creates substantial report-label circularity.
Even a small leak of information from a future report into training data can double an AI's accuracy — this is called the 'look-ahead bias'.