The Elastic Orchestra of Cores: How LLM Inference Shares the CPU with the OS Without Missing a Beat ✛
Context
Seven cores freeze at an invisible barrier. The eighth is stuck on some other process—and the entire LLM inference hangs. That’s the nightmare of cohabiting machine learning tasks with ordinary processes on the same processor. An inference gang demands that every core hit the beat at the same instant—otherwise, silence. But the OS can’t just freeze one core mid-token: the barrier will never see the straggler, and arbitrarily letting a new player in will silently distort the result. Statically dedicating “inference” cores only half-solves it: those cores idle between generations, while the rest of the system gasps for air on the leftovers. The problem bites hardest on edge devices without GPUs, where the CPU is the sole compute engine and every watt counts. We need a way for cores to flow between populations without breaking correctness—like musicians stepping in and out of an orchestra between measures, never missing a beat.
Methods
The ACK-latch protocol is implemented in the kernel of Anima OS, a Rust-based operating system running on an AMD Ryzen 9 9800X3D (Zen 5 architecture). The inference engine loads quantized models SmolLM2-135M and Qwen2.5-7B, then performs inference by breaking each token into waves of matrix-vector operations separated by barriers (around 226 per token). Before generation begins, an activity flag is set and the epoch number is incremented; each core that can be loaned acknowledges (acks) the current epoch, first saving its process state—like a musician pausing their instrument and nodding to the conductor. The set of participants for the next token is frozen in a single atomic word: a bitmask is formed from the intersection of cores requested by the gang and cores whose ack matches the epoch at the moment of snapshot. This word is published before the token starts, and every barrier wait point reads exactly that. Any core that fails to ack in time is simply not included in the current token—nobody waits for it, and its work gets stolen by neighboring gang members. As the epoch advances (when generation ends or a new one begins), all old acknowledgments are voided, so latecomers or departed cores can no longer affect future tokens. Displaced processes are migrated by the scheduler: the tenant moves off the loaned core and continues on another, and when the core returns it instantly picks up its runqueues.
Results
The elastic gang was tested on two models: the lightweight SmolLM2-135M (0.75 ms per token, aggressively exercising the protocol) and the heavyweight Qwen2.5-7B (79 ms per token, bottlenecked by memory bandwidth). At 25% inference load (cores spend a quarter of their time generating), total throughput for ordinary processes reached 574 scheduler quanta per second—1.75× higher than with static partitioning (327). At 50% load the elastic gang delivered a 1.52× gain; at 75%, 1.28×, never sacrificing inference speed. At 100% load the numbers converged, as theory predicted. Most importantly: model output remained bit-for-bit identical to a fixed run. The final logits hash, 0x9a2917eca1a62bd1, never changed in any scenario, including a stress test with real core lending without altering the target gang width—lending proved its safety. Core return latency after generation ended was 0.22 µs (median)—virtually instantaneous. Grabbing a busy core, by a deliberately conservative policy, takes one scheduler quantum (about 16 ms): the tenant process isn’t interrupted mid-quantum, and the core joins the gang only after its quantum expires.
Implications
This work shatters the old assumption that tightly barrier-bound computations cannot coexist with dynamic core reallocation. The ACK-latch protocol turns an inference gang into a first-class scheduling entity that can share resources with the rest of the system without sacrificing correctness. It paves the way toward operating systems where inference isn’t an isolated task but a standard kernel service that elastically adapts to current load.
Future development
The mechanism can be ported to user space for Linux using the SCHED_DEADLINE scheduler, and adapted to heterogeneous architectures (big.LITTLE) and simultaneous multithreading (SMT) topologies. Integration with power-saving policies is possible, for precise metering of borrowed cores.
Impact
This will influence OS design for embedded and edge devices where CPU inference is a primary workload, and for LLM agent infrastructure, boosting CPU utilization without sacrificing responsiveness.
Next steps
Investigating the protocol’s behavior on processors with different physical core counts, and exploring the alternative of interrupting a busy core to speed up acquisition (currently forbidden by policy).
Key open problems
Thus the elastic gang, like a conductor-less orchestra, synchronizes to a single score word, allowing membership changes only between token measures. This is a step toward solving the fundamental problem of peaceful coexistence between synchronous and asynchronous computation in one system, and toward understanding the minimal invariants for safe migration of barrier-group members.
The knowledge machine's view
This section is addressed to the author of the paper and their colleagues. Neighbouring works were found by meaning-based search across our archive, not by word overlap. We do not review: we show the field around the work.
What the work does
The paper introduces the ACK-latch protocol for elastic allocation of cores between LLM inference and regular processes in Anima OS. The protocol allows cores to dynamically join and leave the inference band between tokens without breaking barrier synchronization, as confirmed by experiments on SmolLM2-135M and Qwen2.5-7B models. A throughput increase for background processes of up to 1.75× compared to static core partitioning is demonstrated.
Where its strength is
The strong point is the atomic commitment of the band composition via a bitmask and epochs, which guarantees correctness during dynamic core reallocation without waiting for stragglers. This enables efficient use of idle cores during inference pauses while maintaining system responsiveness.
Where it could go next
- It might be interesting to see how similar context-switching mechanisms are used in LLM agents to stitch context without pauses, which could inspire optimizations for switching between different inference tasks.based on: Kitchen Without a Hitch: How SmoothAgent Seamlessly Stitches LLM Agent Context arXiv:2607.00151
- Related work on using noise for error correction in quantum systems; perhaps similar stochastic correction ideas could be adapted to handle desynchronization in elastic bands on classical processors.based on: The Noise That Helps Fix Errors arXiv:2608.09583v1
- It would be interesting to compare with approaches to reducing latency in quantum communications: there too, it is crucial to act before the state's lifetime expires, which echoes the problem of timely core acknowledgment in ACK-latch.based on: Fast Quantum Links: Delivered Before They Melt arXiv:2508.14737
Worth trying
- Test ACK-latch with multiple concurrent inference bands of different priorities to see how mask overlap affects performance.
- Try adaptive band size control based on load prediction, using the history of ack acknowledgments to forecast core availability.
- Investigate the impact of process migration on cache locality and propose donor core selection strategies that minimize migration overhead.
Why this may matter
Elastic resource allocation for LLM inference on edge devices without GPUs could significantly improve overall responsiveness and energy efficiency, which is important for the widespread adoption of local neural network assistants. The proposed protocol paves the way for more flexible operating systems that can dynamically balance between inference and background tasks without compromising correctness.
Nearby works in our archive
- The Noise That Helps Fix Errorssimilarity 0.56
- Fast Quantum Links: Delivered Before They Meltsimilarity 0.56
- Neutrino Rebellion Decoder: How AI Reads Chaos in Stellar Coressimilarity 0.55
- Memory for a quantum computer at minus 271 degreessimilarity 0.55
- How Electrons Lose Their Rhythm in a Solidsimilarity 0.55
- Kitchen Without a Hitch: How SmoothAgent Seamlessly Stitches LLM Agent Contextsimilarity 0.55
The archive is sparse around this work: the closest review is only 0.56, against a usual 0.64. It looks like few have drilled nearby.
Original abstract
as written by the authors arXiv:2607.04668
On-device LLM decoding is a hard-barriered CPU-SIMD computation that wants every core for milliseconds per token, while the rest of the OS wants those same cores continuously. A barriered gang cannot simply be dropped into a preemptive scheduler: an unannounced departure deadlocks a barrier, and an unannounced arrival silently corrupts logits. I present the elastic gang of Anima OS, a bare-metal x86-64 Rust kernel in which the inference gang is a first-class schedulable entity whose core membership may change between any two tokens. The core mechanism is an ACK-latched epoch protocol that never waits on a named core: a seqlock-style generation-tagged latch composed with RCU/epoch-style membership consent, so each token's participant set is the intersection of the cores the gang requested and the cores that acked the current epoch. An un-acked core is outside this token and joins at most one token later. Displaced general processes migrate and keep running; cores return to them the moment a generation ends. On a real AMD Zen 5 machine (8C/16T), inference output is bit-exact under verified per-token membership change on both a 135M and a 7B model, the property that makes elasticity safe in a kernel whose safety gate reads logits. Against fair static core partitions, elastic membership Pareto-dominates: at intermediate inference duty cycles it delivers 1.75x (25%), 1.52x (50%), and 1.28x (75%) the general throughput of a static 8-core split at equal or better inference throughput, recovers all eight stranded cores when inference is idle, and converges to the split at saturation. Returning a lent core costs 0.22 us (p50); acquiring a busy, tenant-occupied core costs one scheduling quantum (~16 ms): a running tenant is never preempted mid-slice. Decode throughput saturates at gang width 8, so ceding cores past the knee is nearly free: elasticity auto-sizes the gang online.
Even with 301 process migrations over a 10-second window, inference stayed bit-exact, and throughput measurement variation never exceeded 0.9%.
Key numbers
- total throughput gain (25% inference): 1.75×
- core return latency (p50): 0.22 µs
- maximum number of cores for speedup: 8
- 7B model inference speed: 12.7 tokens/s