Streaming

Set "stream": true on any gateway request. The response is a server-sent events stream in the ingress format’s native protocol — the same events the format’s official SDKs already parse.

Per-format protocols

data: {"id":"gen-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""}}]}

data: {"id":"gen-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"}}]}

data: {"id":"gen-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"gen-…","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":7,"total_tokens":19}}

data: [DONE]

event: response.created
data: {"type":"response.created","response":{"id":"gen-…","status":"in_progress",…}}

event: response.output_text.delta
data: {"type":"response.output_text.delta","delta":"Hello"}

event: response.completed
data: {"type":"response.completed","response":{"id":"gen-…","status":"completed","usage":{"input_tokens":12,"output_tokens":7,…}}}

event: message_start
data: {"type":"message_start","message":{"id":"gen-…","role":"assistant","content":[],…}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":7}}

event: message_stop
data: {"type":"message_stop"}

hi.tool_event — server-tool lifecycle frames (opt-in)

When a server tool runs during a stream, the default stream stays byte-identical to a request with no tools — the tool’s work is invisible on the wire and you just get the final answer. Opt in to lifecycle frames to watch each tool call as it happens:

{ "plugins": [{ "id": "web_search", "events": true }] }

…or opt in to every tool at once with a top-level flag:

{ "hi_tool_events": true }

Either opt-in makes the gateway emit a compact start → result (or start → error) pair around each tool execution. The frame body:

{
  "object": "hi.tool_event",
  "tool": "web_search",
  "phase": "start",
  "id": "call_srv1",
  "payload": { "query": "hyperinfer" }
}

• object — always "hi.tool_event".
• tool — which server tool: web_search, web_fetch, datetime, or image.
• phase — "start" (before execution), "result" (success), or "error".
• id — the tool-call id; a start and its result/error share it so you can fold them client-side.
• payload — a compact, tool-specific object (the query, the result count, the cost). Heavy content (full page text, image bytes) is never in the payload — it goes to the model, not the stream.

Per-format wire shape

data: {"object":"hi.tool_event","tool":"web_search","phase":"start","id":"call_srv1","payload":{"query":"hyperinfer"}}

data: {"object":"hi.tool_event","tool":"web_search","phase":"result","id":"call_srv1","payload":{"query":"hyperinfer","resultCount":1,"costMicro":1798}}

event: hi_tool_event
data: {"type":"hi_tool_event","object":"hi.tool_event","tool":"web_search","phase":"start","id":"call_srv1","payload":{"query":"hyperinfer"}}

event: hi_tool_event
data: {"type":"hi_tool_event","object":"hi.tool_event","tool":"web_search","phase":"result","id":"call_srv1","payload":{"query":"hyperinfer","resultCount":1,"costMicro":1798}}

Default stream is unchanged

If you do not opt in, no hi.tool_event frames are emitted and the stream is byte-for-byte identical to a request without server tools — existing SDKs and proxies are unaffected. The opt-in is strictly additive.

Usage in streams — always

Token usage is reported on every request, streamed or not. For streaming, the final usage chunk is always emitted — you do not need to send stream_options: {"include_usage": true} (Chat Completions ingress accepts it for compatibility; the behavior is always on). Usage includes prompt, completion, and reasoning tokens, with cached vs. uncached prompt tokens broken out — see Usage & Credits.

Keep-alives, timeouts, aborts

• SSE keep-alive comments every 15 seconds, so load balancers and proxies never idle-close a healthy stream.
• Time to first token up to 5 minutes; 120 seconds inter-chunk idle; 60 minutes absolute stream cap. Details in Limits & Timeouts.
• If your client aborts mid-stream, the tokens generated up to that point are still metered, billed, and recorded — the partial usage settles when the stream closes.