A single recorded GPU sequence (a CUDA Graph) cut a 7B model's draft latency from 112ms to 25ms

A two-node cluster splits an open-source 7-billion-parameter model across two Google Cloud regions (Iowa and Oregon), with a small Ohio relay between them. The ~86 ms public-internet round trip was not the bottleneck. The drafter was.

ShardFlow, a distributed inference framework posted to r/MachineLearning by its author and tagged as a project promotion, runs a 0.5-billion-parameter neural drafter (a small model that proposes candidate tokens) alongside the 7-billion-parameter target. Speculative decoding, the trick of letting the drafter guess multiple tokens that the target then verifies, was already reframing the WAN round-trip as a per-round cost: tokens per round climbed from 1.00 to 4.07 at K=8. The remaining cost was Python. The drafter's forward pass was launching roughly 1500 small CUDA kernel calls per round.

Capturing that forward pass as a single CUDA Graph (a recorded sequence of GPU operations replayable with one driver call) dropped draft latency from 112 ms to 25 ms. Throughput on the 7-billion-parameter model moved from 4.92 tokens/second (no drafter) to 14.3 tokens/second (eager drafter) to 28.10 tokens/second peak and 20.31 tokens/second average. A 14-billion-parameter variant at NF4 4-bit quantization reached 14.43 tokens/second on the same two-node setup. The drafter's accept rate held at 65.0% peak and 42.9% average.

The numbers are self-reported on the project's GitHub README and the author's Reddit post. No independent reproduction exists on identical hardware, and the T4 nodes sit on Kaggle's time-sliced free tier, so 28.10 tokens/second is a peak, not a service-level objective. What generalizes: on small-batch distributed inference, the host-side launch path often dominates per-round cost before the network does, and a single CUDA Graph capture of a small drafter is one named fix.