I run my own inference server as my daily driver. My coding agent talks to it all day through the OpenAI-compatible API, same as any client would. On 2026-08-04 it locked into a loop: the same tool call, roughly ten identical cycles, with an instruction in its own context telling it not to repeat commands it had already run. It repeated them anyway.
The model was fine. The sampler kernels were fine. The bug was two lines of serde, and my entire test battery was structurally incapable of seeing it. This post is about the two bugs found that day (a third arrived a day later — see the postscript), but mostly about that last part.
Bug 1: omitted temperature meant greedy
Both request structs in the server carried this:
#[serde(default)]
temperature: f32, // -> 0.0serde(default) on an f32 gives you 0.0, and the sampler’s is_greedy() is temperature <= 0.0. So every client that omitted temperature — which OpenAI documents as defaulting to 1.0 — got deterministic argmax instead.
I confirmed this against my actual client config, not by inference: the agent’s models.json has no temperature key anywhere. It never sends one. Deterministic argmax plus a repeating agentic context equals the same tool call forever. That is the loop, exactly.
The fix is one serde attribute: #[serde(default = "default_temperature")] returning 1.0. An explicit "temperature": 0 still means greedy — that stays a caller decision, not an accident of deserialization. While in there I audited the neighboring defaults against OpenAI semantics: top_p already defaulted to 1.0 correctly, and top_k/min_p are not OpenAI parameters at all — zero is their correct disable value. A unit test now pins all four corners (omitted and explicit-0, on both the chat and completions surfaces) through to the sampler config. That fix is commit 8032ab01.
Shipped it, felt good for about half an hour.
Bug 2: the loop survived the fix
Fixing the temperature was not sufficient. Both structs also carried:
#[serde(default)]
seed: u64, // -> 0Zero is a perfectly valid fixed seed. So a temperature-1.0 request that omits seed replays one single sampled stream forever. Same context in, same tokens out. The loop survives the temperature fix untouched.
I found this by driving the live pre-fix server, not by reading code, and the differential is in the commit body with raw per-run JSON committed alongside:
- temperature omitted, 4 runs: byte-identical 4/4 — the original loop
- temperature 1.0 + explicit seed 0: byte-identical 4/4
- temperature 1.0 + seeds 1/2/3/4: outputs differ
- temperature 1.0, seed omitted, 4 runs: byte-identical 4/4
The third line proves the sampler is fine. The fourth proves the omitted seed is what pins it. Same bug class as the first one, same file, same mistake: #[serde(default)] on a field where zero is a meaningful value, not “unset.”
The fix makes the type say what the protocol means: seed: Option<u64>, resolved via seed.unwrap_or_else(fresh_seed). fresh_seed() mixes the nanosecond clock with a process-lifetime atomic counter through SplitMix64’s finalizer, so two requests arriving in the same nanosecond tick still get distinct streams, and it never returns 0. An explicit seed — including an explicit 0 — is honored exactly, which matters because every determinism gate in the repo sends seed: 0 deliberately.
One real cost, documented rather than hidden: the sampled draft CUDA graph bakes the seed into its capture key, so a seed-omitting request that resumes a parked speculative session pays one graph recapture on its first burst. Once per resumed request, bounded, and a client that wants both the parked graph and reproducibility supplies an explicit seed.
The part that actually bothers me
Neither bug is embarrassing on its own. #[serde(default)] on a numeric field is idiomatic Rust; it just happens to be wrong whenever zero is meaningful, and in a sampling API zero is meaningful twice over — temperature 0 is greedy, seed 0 is a fixed stream.
What bothers me is why no test caught it. This repo is built around determinism gates: byte-identity checks, golden-token comparisons, seeded reruns. Dozens of them. Every single one runs temperature: 0 — explicitly, because determinism gates need determinism. And temperature 0 routes around the sampler chain entirely. The greedy path never reads the seed at all.
So the entire argmax gate battery, the thing I point at when I say the server is tested, exercised exactly the path where both bugs are invisible. A broken sampled path could not fail a single gate, because no gate ever took the sampled path. The tests were not weak; they were aimed somewhere else. The default request shape — what every OpenAI SDK client sends when it sends nothing — was the one shape nothing tested.
The uncomfortable general form: your most rigorous tests are pinned to the configurations that make rigor easy. Determinism is easy to assert at temperature 0, so that is where all the assertions live, and the sampled path ships on vibes. It took a client that sends the default shape all day — my own agent — to hit it within hours.
The gate that fills the hole
The fix for a class of invisible bug is a gate that can see it. The existing sample-check binary oracles each sampling primitive in isolation — Gumbel perturbation, softmax gather, residual sampling, the filter kernels. All useful, all passing, all blind to composition: a speculative decoder can pass every isolation arm and still emit the wrong distribution if the primitives are composed wrong.
The new arm runs the real device primitives in the production order — draft proposal, accept test, residual on reject — and checks the composed output distribution against the CPU softmax of the target p. The Leviathan/Chen result says the accept-walk output must equal p exactly, for any draft distribution q, so the check is against ground truth, not against a tolerance-fudged reference. 20,000 draws, L-inf on the empirical PMF plus total-variation distance, with a non-degenerate-acceptance guard so the arm cannot go vacuous. The rationale is written at the top of sample_check.rs so the next reader knows why arm 6 exists.
A gate that has never failed is not evidence, so I broke the composition twice on purpose. Inverting the accept test: total variation 0.0184 to 0.8826, caught. Dropping the residual and sampling from p alone on reject — the classic mistake: acceptance rate unchanged at 0.114, every kernel individually correct, all five isolation arms green, and only the distribution check fails (TV 0.0184 to 0.0881). That second control is the whole argument. The bug class it represents is invisible to everything else in the tree.
Verification on the config that found the bugs
The final battery ran on my exact daily-driver serve config — same model, same context and session limits — on a fresh port so the live server stayed untouched. Receipts, raw JSON and server logs included, in research/sampledspec-20260804, merged as c716954b:
- No loop: the agent’s exact request shape (temperature and seed omitted), 4 runs, 4 distinct outputs. Pre-fix, the same shape was 3/3 identical.
- Greedy unchanged: explicit temperature 0, seed 0, 3 runs, byte-identical.
- Seed semantics: explicit seed 4242 reproduces across runs; omitted seed varies.
And the fix does not cost the speculative decode win. Sampled requests still engage spec — 266 spec bursts in the verification log, cumulative acceptance 0.59 — and the new default lands at 73.92 tok/s median (N=5), 1.72x plain sampled decode. It sits 16% below greedy-spec’s 88.28, which is the honest cost of rejection-sampling verification versus argmax verification, not a regression: the pre-fix alternative was not greedy-spec at 88 tok/s, it was greedy-spec stuck in a loop. One protocol caveat for the 1.72x: the plain-sampled arm required a server restart, so it is a cross-run comparison — but its clocks ran higher than the spec arms’, so the ratio is conservative. The sampled-versus-greedy comparison was interleaved in one process and is drift-clean. Details are the full commit message of 95df4f3c.
Known gaps, named rather than hidden: the composition gate closes the math, not the end-to-end path — no serve-level script yet runs a full sampled spec decode, and the temp-to-0 continuity gate exists only at kernel level. Those are on the board.
What I keep
Two bugs, one class: zero is a value, not “unset.” When a protocol says “omitted means X,” the type should be Option, and the default should be spelled out where a reader can see it — #[serde(default)] on a numeric field is a decision disguised as a shrug.
And the meta-lesson, which I think generalizes past this server: audit your test battery for the configurations it never runs, because that is where the default user lives. My gates were thorough and pointed at the wrong path for an entire class of client — the class that includes every OpenAI SDK with default settings. The best bug report I have received so far came from my own agent, going in circles, doing exactly what I told it to.
Postscript, 2026-08-05
The class was not done with me. A day after this post’s battery went green, dogfooding turned up a third bug — same lesson, third coat: a meaningful zero is not “unset,” and this time the zero was token id 0. Any client that set top_p or min_p got stray ! characters injected into output — !bash, grep -!q — because in the sampled-spec full-accept path the truncation filter read its stats from a neighboring column. The foreign row-max mis-scaled the exponent, every candidate failed the threshold, the whole row masked out, and the argmax fell through to the smallest-index tie-break: token id 0, which renders as !. Fragility scaled with the threshold — min_p: 0.05 hit a 100% id-0 rate — while plain top_k was clean, and memra’s own default, which runs no truncation filter, is structurally immune. That immunity is exactly why my daily driver never showed it, and why the bug shipped for a window.
The honest part: this post named the gap the bug lived in. The composition gate above closes the math, not the end-to-end path, and the serve-level battery was still greedy-only — structurally blind to a sampled-truncation bug for the same reason it was blind to the first two. The fix is commit d1dc79b8, and the hole is closed by a differential serve-smoke matrix over the truncation filters, proven in both directions: three failures on the pre-fix binary, zero on the post-fix — serve-smoke-prefix-TEETH.log vs serve-smoke-fixed.log. The isolation matrix that pinned the mechanism is in posthoc-lsampler.txt; the fix lane’s full receipts live in research/sampfix-20260805/. Three bugs now, one class, all found by using the thing.