I run my own inference engine as my daily driver, and my coding agent talks to it all day through the OpenAI-compatible API. It carried a symptom no benchmark chart shows: the longer a conversation ran, the longer each reply took to start. Ten turns into a real session, time-to-first-token sat at eleven to fourteen seconds. Nothing in the metrics looked wrong — every request was a legitimate cold prime, metered and served correctly. That was the problem.
The rewrite that defeats every prefix cache
Prefix caching — every engine’s version of it — rests on one assumption: turn N+1 extends turn N. memra had two reuse tiers built on that assumption, a token-prefix probe and a text-prefix probe, and both require the new prompt to extend what was cached.
Real agent clients do not extend. They rewrite. Mine strips <think> blocks out of prior assistant turns before re-sending the history — a sensible client-side economy; the reasoning tokens are spent, why resend them. But it means turn N’s prompt is no longer an extension of turn N−1’s committed text. Both probes miss on every turn, the parked session — about 4 GB of KV and recurrent state — is discarded, and the whole growing conversation re-primes from byte zero, slower every turn.
From inside the server the symptom is invisible, because every decision is individually correct. The cache was not broken; it was unreachable.
Identity nominates, bytes decide
The tempting fix is to trust the session id: the client says this is conversation X, so resume conversation X. That is not a fix; it is a cache-poisoning surface. An id is a claim, and if a claim alone can attach a request to parked state, a wrong claim attaches you to someone else’s.
So the decision is split in two. Identity nominates a candidate. Bytes decide.
Nomination has two tiers. The explicit tier accepts three spellings — a session_id body field, OpenAI’s user field, or an x-session-id header. The body beats the header, and a request that names a conversation never matches a parked session that did not, or vice versa: affinity declines rather than guesses.
The implicit tier handles clients that name nothing, which includes mine. It builds a structural fingerprint: split the token stream at the tokenizer’s control tokens — exactly what a chat template emits at turn boundaries — and hash, per segment, only its first and last eight tokens, never its interior. The rewrite class that motivated all of this mutates segment interiors (a stripped <think> block is deleted text inside a turn), so interior-blind hashes are invariant under it. The fingerprint is a chain, one hash per segment, and identity is a prefix relation over the chain with a nomination bar of three shared leading segments — three, because a bare system prompt is byte-identical across every fresh conversation from the same client and would cross-link unrelated ones. Markerless raw prompts can never clear the bar, so non-chat callers keep the plain prefix probes untouched.
And nomination is all it is. The resume happens only if the incoming prompt reproduces the parked session’s committed tokens exactly, up to its checkpoint, with a non-empty suffix left to prime. Any divergence inside that range means the caches hold state for tokens this request does not have — and the hybrid model’s recurrent state is mutated in place, with no per-position index to truncate, so no suffix priming repairs a mismatch. Divergence is a full re-prime, always. The worst a fingerprint collision can cost is one wasted comparison, never a wrong resume.
The checkpoint that sat one token too far
Fully wired, affinity fired zero times on my workload. The decline diagnostic named the reason in one line:
spec-affinity: declined (history diverged at 12233 of checkpoint 12234)Diverging one token below the boundary is not a history rewrite. The checkpoint capture sat after the engine’s init feed, so the boundary included the first generated token — which, on a reasoning model, is the first thing inside the <think> block my client strips. Affinity declined 100% of the time while looking, from the outside, like a safe correctness decline. The fix moved the capture to before the init feed and pinned the invariant with a debug assertion. Immediately after, turn 2 dropped from 16.4 s to 6.5 s and reported cached_tokens went from 0 to 12233. The failure mode is instructive: bytes decided, correctly, against a boundary off by one. In a byte-exact system an off-by-one does not degrade output — it silently turns the feature off.
The measurement
Rig and protocol in full, because every number below is conditional on them: local RTX 5090 Laptop (24 GB), my daily-driver serve config verbatim, one recorded 25-turn transcript replayed by both arms so every turn’s prompt is byte-identical across them, N=3 interleaved reps per arm (on, off, on, off — never all of one arm then the other), the GPU lock held with no other tenant, and a thermal ramp from 61 C idle to 85 C steady state spread across both arms by the interleave. Per-turn median over the three reps, never a mean over turns — turn 0 is a cold prime in both arms and an aggregate would hide the shape. The harness fails if no turn rewrote its history, so a run cannot claim a regime it did not reproduce. TTFT is streamed: the clock stops on the first SSE chunk carrying text.
The comparison is our own engine with the feature off (MEMRA_AFFINITY=0). Not a competitor — the same binary, one flag.
| turn class | affinity on | affinity off | ratio |
|---|---|---|---|
| turn 0 — cold prime | 9.882 s | 9.962 s | 1.01x |
| turns 1, 23 — pure extension | 0.590 / 0.544 s | 0.591 / 0.541 s | ~1.00x |
| turns 2–22, 24 — rewritten history | 0.525–0.645 s | 11.28–14.03 s | 20–24x |
Sum of per-turn medians across all 25 turns: 23.1 s against 287.2 s, 12.4x. End-to-end wall clock is 5.30x — necessarily smaller, because decode time is identical in both arms: affinity removes re-priming, not generation.
The claim I am making is not any of those multiples. It is the flat line. With affinity on, TTFT was 0.525 s at 13.1k prompt tokens and 0.548 s at 14.6k; over the same span the off arm went from 11.89 s to 13.36 s. TTFT stopped scaling with conversation length. That is the property a daily driver needs, and it is why “20–24x” appears here only with the turn class attached. Turn 0 is 1.01x because there is nothing to resume, and the pure-extension turns are 1.00x because the old prefix probe already served both arms. A bare “24x faster” would be disproved by your first cold request.
The residual half-second is not prefill that scales — at turn 5 the prompt delta is roughly 85 tokens. It is a fixed per-turn floor: rewind, delta prime, one decode step.
What this does not fix
That floor is the honest edge of the result, and it belongs in the same breath. 0.53 s is the same number that loses cold time-to-first-token to llama.cpp in the same day’s head-to-head on the same rig: 0.19 s against our 0.53 s, same model artifact, each engine on its owner’s daily flags (research/memra-vs-llama-daily-20260805/RESULTS.md). Session affinity removed the re-prime that made turn 20 worse than turn 2. It did not lower the floor, and the floor is what a cold request feels. No interactive-latency claim follows from this post; the cold-TTFT gap is open, its causes named in that receipt.
Three turns are not byte-identical, and the cause is not affinity
The byte-identity check across arms came back 22 of 25 turns identical. Turns 2, 3, and 24 differ — the same three turns in every rep, at divergence depths of 2236, 882, and 137 characters. The affinity arm against itself is deterministic: two runs, same flag, 25 of 25 identical output hashes.
Chasing the three led somewhere more general. With affinity off and per-turn cache salts so nothing resumes at all, changing only the prefill chunk size changes greedy output: chunk 2048 versus 64 diverges on a 149-token prompt, 2048 versus 32 on a 97-token one. A different chunk split changes the reduction order in the prefill GEMMs, perturbs logits in their last bits, and flips a near-tie argmax. Every resume necessarily re-chunks the prefill — it primes from the rewind boundary instead of from zero — so every reuse tier inherits this, including the prefix tiers that predate affinity.
Two consequences, stated rather than buried. First, resumed-equals-cold is not a property this engine currently has on any reuse tier, and no gate pretends otherwise: the permanent gate this lane added (serve-smoke check 10) asserts what affinity owns — determinism of the resume path across servers, plus proof the resume actually fired — not byte-equality against a cold run. That naive assertion was written first, failed, and isolation showed why. Second, every exactness statement about this engine is scoped to one configuration; the prefill chunk size is a documented machine-config knob, so two rigs with different values already produce different greedy text on the same prompt. Making chunked prefill reduction-order-stable is an open engine question, on the board, not this lane’s.
The rollback seam shipped broken
One more find, kept because this post should admit it: MEMRA_REUSE_POOL=0, the flag that disables the reuse pool entirely, panicked the worker thread on the first session retire — an index-zero removal from an empty list under a cap-zero loop. Verified present on the base binary before this work, then fixed at both park sites while building this lane’s control arm. The rollback seam for the new feature itself is MEMRA_AFFINITY=0, the A/B arm every number above was measured against.
Receipts
The full gate battery ran green on the final code — kernel checks, prime gate 8/8, argmax parity on the 31B and 12B, verify-gate at K=7 on both depths, spec self-consistency 64/64 streams, serve-smoke including the new check 10, and the server test suite, 0 failed. Raw rows for every run in this post — per-turn JSONL plus the raw server logs, where the resume decisions actually live — are in research/session-affinity-20260805/, analysis in its RESULTS.md; the chunk-order reproducer, chunk-order-probe.py, runs in under two minutes on a 9B. The user-facing contract is in docs/SERVING.md under “Session affinity”.
If your client rewrites its history — strips reasoning, compacts old turns, re-renders templates — the flat line is the number to hold us to: turn 20 should answer like turn 2. Your first turn will still be slower than llama.cpp’s. Both belong on the record.