Caching

A cross-request prefix cache serves repeated prompt prefixes without recomputing them. Entries are keyed by the exact token-id prefix, per model, and evicted LRU under a byte budget. Entries are reusable: a hit deep-copies the cached state into the new session, so one shared system prompt serves any number of concurrent sessions.

The learning sequence for a shared-prefix pattern: request 1 seeds its full prompt, request 2 primes from the longest common prefix and inserts the boundary entry, request 3 and onward hit.

Exactness

A cached hit is bit-identical to the run that computed the prefix — gated 16/16 partial-prefix and 16/16 full-prefix cached-vs-fresh greedy identity across depths. Caching is a cost optimization, not an output change.

Isolation: your tenant, then cache_salt

Every cross-request reuse tier keys on (model, tenant, cache namespace), not model alone. The first level is automatic: every API key gets its own tenant namespace, so cross-key isolation is built in and gated — the same tenant shares hits, a different tenant always misses, and tenant A cannot probe whether tenant B's prompt is cached. The second level is yours: the optional cache_salt string field on /v1/completions and /v1/chat/completions — the vLLM cache_salt design, an OpenAI-compatible extension — sub-namespaces within your key. Requests only share cached prefixes with requests carrying the same salt, in either direction, so cached_tokens can only ever reflect the caller's own namespace's history.

When you must salt: any deployment where multiple end-users share your one API key — a gateway, a marketplace listing, a multi-tenant app — must set a per-end-user or per-session salt. No salt means your key's default shared namespace: fine when the key serves one tenant, a cross-end-user timing side channel inside your own account otherwise. Isolation across API keys holds either way.

# gateway multiplexing many end-users through one API key:
# set a per-end-user salt or their caches are mutually visible
r = client.chat.completions.create(
    model="qwen3.6-27b",
    messages=[
        {"role": "system", "content": SHARED_SYSTEM_PROMPT},
        {"role": "user", "content": user_msg},
    ],
    extra_body={"cache_salt": f"user-{user_id}"},
)

print(r.usage.prompt_tokens_details.cached_tokens)  # tokens served from cache

The LRU byte budget stays global across namespaces — VRAM is one resource; only visibility is namespaced.

Accounting

Every response shape carries OpenAI-schema usage with the worker-truth split: usage.prompt_tokens, completion_tokens, total_tokens, and usage.prompt_tokens_details.cached_tokens — the tokens served from any cache tier rather than recomputed. Tools requests cache like any other: the cache keys on the rendered prompt's token ids, so a repeated tools block is a cacheable prefix.

What a hit is worth

Cached prefill costs approximately nothing to serve: warm TTFT is 3 ms against 0.182 s cold. If your workload repeats a long system prompt, the first request pays the prefill and every subsequent request starts nearly instantly — and cached input is billed at a discount (see pricing).