tiyuvta

Errors & limits

Honest 400s

Parameters the engine cannot honor semantically are rejected with a 400 naming the parameter — never silently ignored. A silently dropped parameter means your requests do something other than what you asked for; a loud 400 means you find out in development, not in production output.

parameter behavior
logit_bias, logprobs / top_logprobs, n != 1, best_of != 1, unknown response_format types 400, parameter named in param
tool_choice: "required" and named-function forms 400 ("auto" and "none" are supported)
stream_options accepted and ignored — cosmetic, no semantic effect
user, session_id, x-session-id header accepted as session-affinity nomination — identity only nominates a candidate parked session; the bytes decide the resume (see Caching)

Error body shape

Error bodies are the OpenAI error object. Mid-stream worker errors arrive as a final data: error chunk followed by [DONE] — never a named SSE event your parser might not handle. 

{
  "error": {
    "message": "logit_bias is not supported",
    "type": "invalid_request_error",
    "param": "logit_bias",
    "code": null
  }
}

Every completion and stream chunk carries id, created, and system_fingerprint; the id echoes as the x-request-id response header — quote it when reporting an issue. SSE keep-alive comments flow every 5 s, so a long prefill never looks like a dead stream.

Rate-limit headers

Both completion routes return X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset with concurrency-slot semantics: a request's slot is held for the entire life of its SSE stream and released when the stream completes, so Remaining is truthful while streams are in flight — not a number that was accurate only at admission. Rate-limited requests carry 429 + Retry-After.

Graceful drain

Deploys drain, they don't drop. On SIGTERM:

  • /health flips to "draining".
  • New completion requests get 503 + Retry-After — retry against the fleet and you land on a live replica.
  • In-flight requests, streams included, run to [DONE] within the drain deadline (default 30 s).
  • The process exits 0.

Live receipt: a 1024-token stream completed mid-drain. If you see a 503 + Retry-After, it is a deploy in progress, not an outage — retry after the header's delay.

Disconnects

A hung-up client's session retires at the next scheduling tick and is billed to the abort point; queued requests from dead clients never reach the GPU. Closing the connection is a safe way to cancel.

receipts