Constrained decoding

/v1/chat/completions honors response_format {"type":"json_object"} and {"type":"json_schema", ...} as real constrained decoding, not post-hoc validation: the schema compiles to an llguidance grammar, and each step's token bitset masks disallowed tokens on device before sampling. Constrained sessions ride the same device-sample, CUDA-graph, and speculative paths as unconstrained ones — no path is lost to being constrained.

r = client.chat.completions.create(
    model="qwen3.6-27b",
    messages=[{"role": "user", "content": "reply as JSON"}],
    response_format={"type": "json_object"},
)

r = client.chat.completions.create(
    model="qwen3.6-27b",
    messages=[{"role": "user", "content": "extract the invoice fields"}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "invoice",
            "schema": {
                "type": "object",
                "properties": {
                    "vendor":   {"type": "string", "maxLength": 120},
                    "total":    {"type": "number"},
                    "currency": {"type": "string", "enum": ["USD", "EUR", "ILS"]},
                },
                "required": ["vendor", "total", "currency"],
                "additionalProperties": False,
            },
        },
    },
)

Cost

Plain constrained-greedy runs at 99.4% of unconstrained speed (123.7 vs 124.4 tok/s, N=3 same-session); per-step grammar compute is 0.006–0.007 ms. The spec lane pays more under tight grammars, and the cost there is draft acceptance, not mask overhead — draft-side masking recovers a measured part of it.

Draft-side masking

The speculative drafter is masked too: a constrained session clones its grammar matcher once per spec round and bans illegal tokens in the draft head's own logits, so proposals are legal by construction. The verify-side truncation backstop stays as the correctness backstop but stops firing — grammar cuts went to 0/N on every measured cell. On a bounded tight schema this is +5.0% throughput (216.6 → 227.5 tok/s, N=3 interleaved medians). Unconstrained traffic is unaffected.

Bound your schema

Prefer bounded schemas: maxLength on strings, enum where the value set is known, additionalProperties: false, required listing every field. Two reasons:

• Tighter grammars give the drafter less room to propose tokens the grammar rejects, which is where constrained throughput is won.
• An unbounded schema that lets the model degenerate into arbitrary whitespace against a token cap has a measured draft-chain-shape-dependent tail (floating-point summation order flips argmax at near-ties in that tail). It predates constrained decoding and is not a masking property. Bound the schema and it goes away.

Interaction with reasoning ("no-think")

Constrained requests force the chat template's no-think switch: a grammar that masks from token 0 can never close an open <think> tail. If a model's template opens a think tail and has no enable_thinking switch, a constrained request is a loud 400 — never silently wrong output.

Exactness

Device masking is gated byte-identical to a host reference oracle; spec-constrained is byte-identical to plain-constrained; graphed is byte-identical to eager; draft-masking on is byte-identical to off (greedy and seeded-sampled, 7 cells). Unknown response_format types are loud 400s. /v1/completions (non-chat) carries no response_format.