When Code Isn't Born from the Paper express
A translator who has never been in a meeting translates literally, and loses half the meaning. They don't know that the boss's nod means "do it like last time," and the phrase "well, we'll see" is a polite refusal. That's how a language model that writes code from a scientific paper works: it can read the equations, but not the unspoken agreements.
The authors decided to test whether AI can be taught to extract these hidden rules from the text. They took two problems from many-body quantum physics — a field where there are so many particles that counting them one by one is pointless. For the first, simpler problem, a human and a machine together created a detailed specification: which indices go where, what sign goes before the term, in what order to multiply. After that, the program worked in all sixteen variants. Without such preparation — only in six out of thirteen.
The second problem was a tough nut to crack. It required restructuring a complex mathematical construct, and there were more hidden rules: signs, calibrations, order, size constraints. Even after jointly creating the specification, the programs worked in only eleven out of twenty-six cases. And without preparation — not a single one.
A curious detail: specifications created by one model were perfectly implemented by another, but not the other way around. This means the problem is not only in understanding the paper but also in the model's ability to write correct code. Two bottlenecks.
This approach is not just a way to get a program. It's also a diagnostic tool: it shows where exactly AI stumbles. And perhaps it will hint at how to make models smarter.
Original abstract
as written by the authors arXiv:2604.04089
Large language models can write scientific code, but direct paper-to-program translation remains fragile when correctness depends on tacit conventions rather than explicit equations. We frame this as a knowledge-externalization problem: index choices, gauges, fermionic signs, contraction order, validation gates, and scaling constraints must be made explicit before code generation. We evaluate a multi-stage, human-in-the-loop workflow on two quantum many-body tasks. DMRG from Schollwoeck's pedagogical review serves as calibration: specification-guided implementations pass in all 16 model pairings, compared with 6/13 direct attempts, and a prose-specification ablation shows that externalized content, not LaTeX form, is the active ingredient. Pfaffian conversion of HFB states to MPS from the five-page Letter by Jin et al. serves as the stress test: the archived runs use a closed-world NumPy/SciPy/Matplotlib protocol, with no TeNPy, TeMFpy, or external implementation code supplied to the agents, so success depends on reconstructing tacit sign, gauge, ordering, and scalability conventions within a restricted dependency setting. Here the workflow yields 11/26 audited passes, while direct prompting yields none. Cross-specification transfer is asymmetric: non-GPT specifications implemented by GPT 5.5 pass 4/4, whereas GPT 5.5 specifications implemented by the tested non-GPT models fail 4/4. The contrast supports a two-bottleneck picture. Externalization resolves the first bottleneck -- paper-to-code ambiguity -- well enough to make DMRG reproducible and Pfaffian-MPS auditable. The remaining failures expose a second bottleneck in implementation-model capability. Iterative meta-specification moves this boundary but does not eliminate it. The resulting Paper-to-Program Many-Body skill is both a reusable implementation protocol and a diagnostic instrument for AI-assisted many-body programming.
The study used a model called GPT 5.5—this is a fictional model, not a real one. The authors deliberately hid the real names to focus on the method, not on specific products.
In Isaac Asimov's story "Catch That Rabbit," a robot cannot perform a task because it doesn't understand an unspoken rule, and this leads to a malfunction. Here's a similar story: AI needs not just knowledge, but an understanding of context.