The boundary between program execution and gradient-based optimization has long limited the use of code as trainable scientific models. Typically, each model had to be manually rewritten in a differentiable framework. This work stands alongside research on differentiable interpreters and neurosymbolic programming, but differs in that the interpreter itself becomes differentiable, not individual programs. This paves the way for automatic calibration of any programs, such as those generated by language models.
The authors developed a compiler that translates Scheme source code into an intermediate representation called ComputeGraph, and then into PyTorch or JAX modules. The key feature is a differentiable metacircular interpreter: a Scheme interpreter written in Scheme itself, compiled once, which can then execute arbitrary programs passed as data. All values are represented as 14-dimensional vectors: a 10-dimensional one-hot type tag and a 4-dimensional payload. The heap is implemented as a single-assignment dictionary, preserving gradient correctness. Tail calls are handled via a trampoline.
Gradients computed through the compiled interpreter match direct compilation to within 7×10⁻⁷ in loss on 171 (program, initial value) pairs. On real battery degradation data, joint structure and parameter search recovered the knee-shaped degradation structure from a smooth initial approximation and improved extrapolation on held-out data. In an inverse modeling task for El Niño, the differentiable interpreter optimized the likelihood function of a Kalman filter with 611 parameters, where gradient-free search failed. Batching provided up to 875× throughput speedup.
This work differs from neighboring approaches in that the interpreter itself becomes differentiable, not individual programs. This removes the need to rewrite each new model in a differentiable form. Compared to language-level automatic differentiation (as in JAX or Zygote) and manually crafted differentiable interpreters (like TerpreT), it achieves a combination of self-application, reverse mode, and compilation. This extends symbolic regression and neurosymbolic search from closed-form expressions to executable stateful programs.
Possible directions include porting the interpreter to JAX by rewriting the trampoline as a fixed-length lax.scan with masking. Also promising is the use of MLIR and Enzyme to reduce overhead on single computations. Research on smoothing branches (smooth interpretation or DiscoGrad) could address the problem of zero gradients for parameters that only affect branch conditions. More broadly, this opens the path to gradient-based self-improvement of programs.
The method could impact automated scientific modeling, where language models generate hypotheses as code, and a differentiable interpreter calibrates their parameters. It is also useful for symbolic regression and neurosymbolic program synthesis tasks.
Next steps include implementing a differentiable interpreter in JAX to leverage jit compilation and vmap. It is also worth investigating scaling to larger program spaces and testing robustness under model misspecification.
This work directly relates to the unsolved problem of automatic program synthesis: how to efficiently search discrete structure spaces using gradient signals. It shows that differentiating the interpreter solves the parameter calibration problem, but not the discrete structure search problem, which remains challenging.
This section is addressed to the author of the paper and their colleagues. Neighbouring works were found by meaning-based search across our archive, not by word overlap. We do not review: we show the field around the work.
What the work does
This work presents a compiler that translates a subset of Scheme into differentiable computation graphs. The key idea is to compile the interpreter itself rather than individual programs, which allows any program fed as data to automatically receive gradient backpropagation. The authors prove the correctness of the gradients almost everywhere and demonstrate applications for joint search of program structure and parameters on real scientific tasks.
Where its strength is
The strength of the work lies in the self-applicable differentiable interpreter, which removes the need to manually rewrite every model in a differentiable form. This paves the way for automatic calibration of any program, including those generated by language models.
Where it could go next
Worth trying
Why this may matter
The work extends automatic differentiation to the level of executable stateful programs, which could accelerate scientific modeling and make it more flexible, especially when combined with code generation by language models.
Nearby works in our archive
The archive is sparse around this work: the closest review is only 0.50, against a usual 0.64. It looks like few have drilled nearby.
The Scheme interpreter written in Scheme itself is only 288 lines of code. Yet it supports closures, recursion, lists, and tail calls, sufficient for self-application.
The idea of a program that can modify and improve itself through gradients echoes the concept of Gödel machines by Jürgen Schmidhuber, described in his work on universal self-improvement.