On 2026-08-04 a 27B model decoded at 170.5 tok/s for a single request, through an OpenAI-compatible chat endpoint, on one workstation card — an RTX PRO 6000 Blackwell 96GB (RunPod secure instance, 188 SM, 600 W, driver 610.43.02, 1.79 TB/s-class GDDR7). Same card, same model, plain decode through the same endpoint: 75.9 tok/s. The 2.2x in between is MTP speculative decoding, and every accepted token was gated identical to what plain decode would have produced. Raw rows: pro6000wk-runpod.jsonl.
This post is about where that speedup comes from, where it stops — in the same breath, because the stop matters as much as the win — and what single-stream speed on cheap silicon does to the economics of serving.
The gate comes first
Speculative decoding drafts K tokens cheaply, then verifies them against the target model in one batched pass. Done right, it changes nothing about the output — the target model still decides every token; the draft only decides how many you get per verify pass. “Done right” is a gate, not an adjective: on this card, run-spec at K=1/2/3 passed token-identical against plain decode on both production artifacts (NVFP4+MTP and Q8_0), and every spec rep in the benchmark battery — including all 10/10 best-K confirmation cells — passed self-consistency. A mismatch voids every number after it; that rule is why the numbers below are worth reading.
The number and its conditions
The measured ladder, N=5 medians, arms interleaved within every rep, same session, 2026-08-04 (receipt: pro6000wk-runpod.jsonl):
- Bare engine (CLI, no server): plain decode 86.8 tok/s; spec at its best K=3, 186.7 tok/s.
- Through the serve surface (streaming SSE, chat template, per-token detokenization, session overhead): plain 75.9 tok/s (−12.6% serve tax), spec at K=3 170.5 tok/s (−8.6%). The serve overhead also shifts the spec optimum deeper — K=5 reaches 172.5 through the server where the bare engine peaked at K=3.
- First token: 0.182 s cold on a 512-token-class prompt (prefix-cache miss forced via cache_salt, N=5), 3 ms warm (N=3).
The gains track acceptance — the fraction of drafted tokens the target model agrees with. On the Q8_0 artifact the gate battery’s acceptance ladder ran 77.8/75.0/71.4% at K=1/2/3 (82.2/108.1/127.9 tok/s), with best-K landing at K=4, 143.5 tok/s (N=5 median confirm). That acceptance trace, to the decimal, reproduced a Q8_0 session on a different die (a desktop 5090 on vast.ai) — deterministic drafting means the acceptance ladder is a property of the model pair, not the box. Acceptance also moves by prompt class: at K=3 on the NVFP4 artifact, 0.683/0.675/0.622 across the three tracked prompt classes, and throughput moves with it (182.7/191.0/165.3 tok/s). Speculation is not a flat multiplier; it is a bet on how predictable the next few tokens are, and code-shaped or structured output pays better than free prose.
Speculation you don’t have to configure
The 170.5 number uses a draft artifact built for the target. The cheaper question: what do you get from a checkpoint that ships its own draft head?
Qwen3.6-27B-FP8 — the official checkpoint, 29 GB — includes mtp.safetensors. The loader picks it up and the server spec-bursts out of the box, no drafter to download, no flag to set. Measured 2026-08-04 on a vast.ai 2x RTX 5090 box (streaming SSE, greedy, N=5 medians, fresh cache_salt per request so nothing hits the prefix cache): plain 48.99 tok/s, embedded-MTP spec 128.06 tok/s — 2.61x. Attaching a trimmed-vocabulary drafter built from the same model lifts it to 136.75 tok/s, 2.79x. Receipt: fp8ship-20260804/official/RESULTS.md.
The takeaway is that the 2.6x is the default path now: model vendors are shipping draft heads inside checkpoints, and an engine that loads them turns every such checkpoint into a spec-decode deployment with zero extra artifacts.
Keeping the draft legal under json_schema
Structured output is the workload where single-stream speed matters most — agent loops emitting tool calls and schema-constrained JSON — and it is exactly where naive speculation wastes work. Under a grammar, the drafter can propose tokens the schema forbids; verification truncates the burst at the first illegal token and the rest of the draft is discarded.
The fix is to mask the drafter’s sampling with the grammar’s legal set, so proposals are legal by construction; verify-side truncation stays as the correctness backstop. Measured on an RTX 5090 with a 9B target, 2026-08-04 (draft-mask-20260804/RESULTS.md): on a tight unbounded schema the truncation fired in 28 of 30 spec rounds without masking, 0 of 30 with it; on the bounded tight schema, acceptance rose from 0.561 to 0.651 and throughput from 216.6 to 227.5 tok/s (+5.0%, N=3 medians, interleaved). The cost is one grammar-matcher clone per spec round at 0.002 ms — below the measurement floor. And the exactness bar held: with masking on versus off, the final emitted stream is byte-identical on every gated cell, greedy and sampled, because the mask changes what gets proposed, never what gets emitted.
Where it stops
Speculative decoding is a single-stream and low-concurrency win, and the crossover is early. On the PRO 6000, the spec serve lane is flat in concurrency — 171.3/169.3/171.2 tok/s at c=1/2/4, because verification serializes — while plain batched decode scales: 75.7/147.9/270.3 tok/s at the same c=1/2/4. Plain overtakes somewhere between c=2 and c=4, the same crossover shape measured on a laptop board; 188 SMs did not move it. Batched serving then saturates at c=8: 420.6 tok/s aggregate, and pushing to c=16/32 holds 421.9/423.0 while p50 latency doubles per step — that is queueing, not throughput.
So the honest deployment picture on one 96 GB card is two regimes: spec decode for latency-bound single streams at 170.5 tok/s, plain batching for throughput at 420.6 tok/s aggregate at c=8. An engine that runs both behind one endpoint — and lane admission that keeps them from eating each other — is the actual product decision; the spec-decode number alone is half of it.
The economics, carefully
Now the frame the title promises. Everything in this section comes from an internal buy-side study dated 2026-08-02 (hw-buy-20260802/REPORT.md, model code in tco-model.py). Market prices in it were fetched live that day with sources inline, but the $/token outputs are an internal estimate — a 3-year TCO model with stated assumptions (energy at $0.18/kWh, resale curves from an executed-price tape, one serving rate projected from a measured half-bandwidth board and flagged as such). They are not market prices, and I want them argued with, not quoted.
With that caveat: a 2x RTX 5090 box (used cards, ~$8.4k all-in) models out at $0.47 per million output tokens at 30% utilization on the 30–40B model class. A used H100 NVL at $30.3k models out at $2.86/Mtok on the same class — TCO-negative at 30% utilization against that market’s ~$1.20/M-out blended revenue, a projected −$11.8k over three years. The mechanism, which is measured rather than modeled: small-model serving is latency- and scheduler-bound, not bandwidth-bound, so the H100’s HBM advantage does not convert into small-model tokens. Its silicon is priced for a workload this SKU does not have.
The card under this post’s headline number sits between those poles: the RTX PRO 6000 Blackwell 96GB listed at $13,250 new on NVIDIA’s marketplace and $9.5–11k refurbished as of 2026-08-02. Workstation silicon, consumer-class pricing per TB/s, no datacenter-driver licensing asterisk — and it just served a 27B at 170.5 tok/s single-stream.
For the regression-tracking view against the incumbent: on the RTX 5090 laptop rig, the 9B MTP spec path measured 2.30x/1.74x/1.59x llama.cpp across short/medium/long-agentic prompt classes — same machine, same prompts, same-session interleaved medians, memra at naked defaults versus llama.cpp at its swept best (2026-08-02; protocol in docs/COMPETITOR-SETUP.md, full boards in docs/PERFORMANCE.md). That table is maintained as a regression suite, not a scoreboard: cells below the bar are tracked openly, and best-vs-best is the only comparison posture that means anything.
What this changes
The standard economics of hard-latency serving assume datacenter silicon: if a workload needs a fast single stream with a real p95 guarantee, you rent the expensive card. The receipts above say that for the 9–35B class, the fast single stream now lives on workstation- and consumer-priced cards, not $30k datacenter silicon — 170.5 tok/s of gated, token-identical 27B output from one workstation card, 2.61x out of the box from a checkpoint’s own draft head, and drafts that stay legal under json_schema where agent workloads actually run.
Where it stops is stated above and it is structural: past c=4, speculation loses to batching, and the aggregate ceiling on this card is 420.6 tok/s at c=8 — the knee. But interactive agent traffic — the workload people pay hard-guarantee latency for — is mostly single streams that arrive one at a time. Consumer-priced silicon serving that traffic at these speeds, with the internal math putting owned small boxes at $0.47/Mtok while the small-SKU H100 models out negative, moves a real class of workloads from “rent a datacenter card” to “own the box.” That is the economics shift, and every number behind it links to its receipt.