Imagine an interpreter sitting in a meeting room, translating every word. But then they notice their translation could improve if they learned from their mistakes. They start noting which phrases caused confusion and adjust them. Within an hour, their translation is more accurate and the meeting runs smoother. That's the idea here: a Scheme interpreter, compiled once, can run any program given to it as data while also computing gradients to improve those programs.
Previously, every scientific model had to be manually rewritten in a differentiable form to apply gradient-based optimization. That's like hiring a new translator for every new text. Now the translator itself becomes trainable: it doesn't just translate, it understands how to tweak the text to make it better. The authors built a compiler that translates Scheme source code into an intermediate representation called ComputeGraph, then into PyTorch or JAX modules. The key feature is that the interpreter itself becomes differentiable, not individual programs.
Inside the interpreter, all values are represented as 14-dimensional vectors: a 10-dimensional one-hot type tag and a 4-dimensional payload. The heap is implemented as a write-once dictionary, which preserves gradient correctness. Tail calls use a trampoline, a loop that replaces deep recursion and prevents stack growth. As a result, gradients computed through the compiled interpreter match direct compilation to within 7×10⁻⁷ in loss on 171 (program, initial value) pairs.
On real battery degradation data, joint search of structure and parameters recovered the knee-shaped degradation curve from a smooth initial guess and improved extrapolation on held-out data. In an inverse modeling task for El Niño, the differentiable interpreter optimized a Kalman filter likelihood with 611 parameters, where gradient-free search failed. Batching provided up to 875× throughput speedup.
The interpreter now doesn't just translate, it learns from every word. It can automatically calibrate parameters of models generated by language models in scientific and engineering tasks. It's like having a translator who, given a draft from an author, polishes it to perfection without requiring a rewrite from scratch. Returning to the beginning: a translator who learns from every word turns an hour-long meeting into minutes and makes science faster and more precise.
The Scheme interpreter written in Scheme itself is only 288 lines of code. Yet it supports closures, recursion, lists, and tail calls, sufficient for self-application.
The idea of a program that can modify and improve itself through gradients echoes the concept of Gödel machines by Jürgen Schmidhuber, described in his work on universal self-improvement.