Streaming LLMs in Production: The Edge Cases That Break Your App

Streaming looks easy. You add stream=True to the API call, iterate over chunks, and append text to the UI. It works in development. Then you ship it and the first production incident arrives: a mobile user's stream hangs mid-sentence, a tool call arrives corrupted, reconnection after a network blip restarts the entire generation from scratch, and your 30-second read timeout is killing requests that should take four minutes.

None of these are edge cases. They're the default behavior of streaming under real network conditions. Here's the complete set of failure modes and how to handle each.

Why streaming breaks differently than batch

A batch (non-streaming) LLM call is a standard HTTP request-response cycle. The server buffers the entire response, sets Content-Length, and sends it when generation completes. Every proxy, CDN, and load balancer in the world handles this correctly. Streaming inverts this: the server sends a response body that grows over time, and the client is expected to process it incrementally before the response is complete.

LLM streaming uses Server-Sent Events (SSE), a simple text protocol over an HTTP response with Content-Type: text/event-stream. Each chunk is a frame:

data: {"type":"content_block_delta","delta":{"text":"The capital"}}\n\n
data: {"type":"content_block_delta","delta":{"text":" of France"}}\n\n
data: {"type":"content_block_delta","delta":{"text":" is Paris."}}\n\n
data: [DONE]\n\n

The double newline terminates each event. data: [DONE] is the stream terminator — when the client sees this, generation is complete. The protocol is intentionally simple, which is part of the problem: it has no built-in flow control, no acknowledgment, and no fragmentation handling.

The immediate failure mode in production is proxy buffering. Nginx, by default, buffers upstream responses before forwarding them to clients. When proxy_buffering is on (the default), your SSE stream is collected into memory and forwarded in large batches — or worse, held until the upstream closes the connection, which defeats the entire purpose of streaming. The fix:

# nginx — disable buffering for SSE endpoints
location /api/stream {
    proxy_pass http://backend;
    proxy_buffering off;
    proxy_cache off;
    proxy_set_header Connection '';
    proxy_http_version 1.1;
    chunked_transfer_encoding on;
    X-Accel-Buffering: no;
}

AWS ALB has a 60-second idle timeout by default. A streaming response that produces a chunk every 30 seconds will hit this on long generations — the load balancer closes the connection mid-stream. Set the idle timeout to 300+ seconds for endpoints that serve long streams, or emit a comment frame (: keepalive\n\n) every 15 seconds to keep the connection alive. Cloudflare has a 100-second response timeout on its free and Pro plans that cannot be extended — if you're behind Cloudflare and generating long responses, you need Argo or Enterprise, or you need to restructure the response.

Gzip compression interacts badly with streaming. Gzip buffers data to find compression opportunities — the typical minimum flush size is 8KB. If your average chunk is 40 bytes of JSON, gzip won't flush until it has accumulated 200 chunks. Disable gzip for SSE endpoints, or configure minimum flush to 1 byte (which defeats the compression anyway). The correct approach is to not compress streaming responses.

The backpressure problem

In a streaming pipeline, the producer (the LLM inference server) generates tokens at a rate that may differ from the rate at which the consumer (your client) can process them. When the consumer is faster than the producer — a desktop browser reading at 1000 tokens/s from a model generating at 80 tokens/s — no problem exists. The client waits for each chunk. When the producer is faster than the consumer, the server-side TCP buffer fills up.

A consumer on a slow mobile network may receive data at 500 bytes/s. Your inference server is generating at 80 tokens/s, which at roughly 4 bytes/token is 320 bytes/s — slower than even a 3G connection. But the consumer isn't always the network. A client doing heavy DOM manipulation on each token, or a server-side proxy accumulating and re-streaming, can fall behind the producer even on a fast connection.

When the consumer falls behind, the TCP receive window closes, which signals the producer to stop sending. On the inference server side, the response write blocks. Depending on the server implementation, this either causes the goroutine/thread to hang waiting for the write to unblock, or the server detects a stalled connection and closes it. The client sees a partial stream followed by a connection reset.

The correct pattern on the client side is to read with a per-chunk timeout, detecting stalls before they cascade into a connection reset:

async function consumeStream(response, onChunk, stallTimeoutMs = 10000) {
  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = '';

  while (true) {
    // Race the read against a stall timeout
    const readPromise = reader.read();
    const timeoutPromise = new Promise((_, reject) =>
      setTimeout(() => reject(new Error('Stream stalled')), stallTimeoutMs)
    );

    let done, value;
    try {
      ({ done, value } = await Promise.race([readPromise, timeoutPromise]));
    } catch (err) {
      // Stall detected — cancel the reader and surface the error
      await reader.cancel();
      throw err;
    }

    if (done) break;

    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split('\n');
    buffer = lines.pop(); // keep incomplete line

    for (const line of lines) {
      if (line.startsWith('data: ')) {
        const data = line.slice(6).trim();
        if (data === '[DONE]') return;
        try {
          onChunk(JSON.parse(data));
        } catch {
          // Malformed chunk — log and continue
        }
      }
    }
  }
}

The stall timeout should be distinct from the overall read timeout. You want to detect "no bytes received for N seconds" (stall) separately from "total request taking too long" (overall timeout). A 10-second stall timeout catches hung connections without killing legitimately slow generations.

Incomplete JSON from tool calls

Tool call arguments arrive as a JSON string that is itself streamed in fragments. This is the most commonly mishandled streaming edge case. A single SSE event might contain:

data: {"type":"tool_use","name":"get_weather","input":"{\"loc"}

That's a valid SSE frame, but the input field is a truncated JSON string. The next frame continues it:

data: {"type":"tool_use_delta","input":"ation\": \"Paris\", \"unit\": \"celsius\"}"}

Naive JSON.parse on either frame throws. The fix is to accumulate the tool call input across frames before attempting to parse:

import json

class ToolCallAccumulator:
    def __init__(self):
        self.calls = {}  # index -> {name, input_chunks}

    def feed(self, event: dict):
        etype = event.get("type")

        if etype == "tool_use":
            idx = event["index"]
            self.calls[idx] = {
                "name": event["name"],
                "input_chunks": [event.get("input", "")]
            }

        elif etype == "tool_use_delta":
            idx = event["index"]
            if idx in self.calls:
                self.calls[idx]["input_chunks"].append(
                    event.get("input", "")
                )

    def finalize(self) -> list[dict]:
        results = []
        for idx, call in self.calls.items():
            raw = "".join(call["input_chunks"])
            try:
                args = json.loads(raw)
            except json.JSONDecodeError as e:
                raise ValueError(
                    f"Tool call '{call['name']}' produced invalid JSON: {e}\nRaw: {raw!r}"
                )
            results.append({"name": call["name"], "arguments": args})
        return results

For cases where you need to act on tool call arguments before the stream finishes — for example, to start fetching data in parallel — you need progressive (incremental) JSON parsing. Python's ijson library parses a JSON stream token by token, emitting events as structure becomes available:

import ijson
import io

def stream_parse_arguments(chunks):
    """Parse tool call arguments progressively as chunks arrive."""
    buf = io.BytesIO()
    parser = ijson.parse(buf)

    for chunk in chunks:
        buf.write(chunk.encode())
        buf.seek(0)
        try:
            for prefix, event, value in parser:
                # Act on partial structure as it arrives
                if (prefix, event) == ('location', 'string'):
                    yield ('location', value)
        except ijson.IncompleteJSONError:
            pass  # More data needed — continue

In JavaScript, the equivalent is a streaming JSON parser like @streamparser/json or building a simple state machine for the specific schema you're parsing. For most tool call use cases — where you need the complete arguments before dispatch — accumulate-then-parse is simpler and correct.

SSE reconnection semantics

The SSE specification includes a built-in reconnection mechanism via the Last-Event-ID header. Servers can assign an ID to each event:

id: chunk-47
data: {"type":"content_block_delta","delta":{"text":" surprising"}}\n\n

When the client reconnects after a dropped connection, it sends the last-seen ID as a request header:

GET /api/stream HTTP/1.1
Last-Event-ID: chunk-47

The server is then supposed to resume the stream from that offset. The problem: most LLM streaming implementations don't implement this correctly. The common mistake is to restart generation from the beginning and skip the first N events — which sends the same input to the model and gets a different output, because LLM generation is stochastic. The user sees the text change on reconnect.

The correct implementation requires storing the accumulated response server-side and replaying from an offset on reconnect — not restarting generation:

import asyncio
from dataclasses import dataclass, field

@dataclass
class StreamSession:
    session_id: str
    chunks: list[str] = field(default_factory=list)
    complete: bool = False

# In-memory store; use Redis in production with a TTL
sessions: dict[str, StreamSession] = {}

async def stream_with_replay(request, model_client):
    session_id = request.query.get("session_id")
    last_event_id = request.headers.get("Last-Event-ID")

    if session_id and session_id in sessions:
        session = sessions[session_id]
        start_idx = int(last_event_id) + 1 if last_event_id else 0

        # Replay stored chunks from offset
        for i, chunk in enumerate(session.chunks[start_idx:], start=start_idx):
            yield f"id: {i}\ndata: {chunk}\n\n"

        if session.complete:
            yield "data: [DONE]\n\n"
            return

        # Continue live stream from where we left off
        start_idx = len(session.chunks)
    else:
        session = StreamSession(session_id=session_id or generate_id())
        sessions[session.session_id] = session
        start_idx = 0

    # Stream new generation, storing every chunk
    async for chunk in model_client.stream(request.prompt):
        session.chunks.append(chunk)
        idx = len(session.chunks) - 1
        yield f"id: {idx}\ndata: {chunk}\n\n"

    session.complete = True
    yield "data: [DONE]\n\n"

This matters most for mobile clients where network transitions (WiFi to LTE) cause connection drops mid-stream. Without proper replay, a 2000-token response that drops at token 1500 forces the user to wait for the full generation again — and gets different text the second time. With replay, the reconnect picks up at token 1501 instantly from cache, and the user sees continuity.

Session storage has a natural TTL: keep chunks for the duration of a session plus a grace period (5 minutes is typically enough for reconnect). After that, the session is expired and a reconnect triggers a fresh generation with an appropriate error to the client.

The first-token latency trap

Streaming creates a perceptual illusion: the interface feels fast because text appears immediately after the first token. But "feels fast" and "is fast" are different measurements. Time To First Token (TTFT) — the interval between sending the request and receiving the first meaningful chunk — can be 2–5 seconds on cold inference infrastructure, during peak load, or with a long system prompt that requires significant prefill computation.

The trap is that TTFT is invisible in your throughput metrics. If you're measuring "time to stream completion," a response with a 4-second TTFT and 1-second generation time shows up identically to a response with 1-second TTFT and 4-second generation time. But the user experience is completely different. The first user sees a blank UI for 4 seconds before text appears. The second sees text almost immediately.

Measure TTFT explicitly, separated from throughput:

import time
import httpx

async def call_with_ttft_measurement(client, payload):
    request_start = time.perf_counter()
    first_token_time = None
    chunks_received = 0

    async with client.stream("POST", "/v1/messages", json=payload) as response:
        async for line in response.aiter_lines():
            if not line.startswith("data: "):
                continue
            data = line[6:]
            if data == "[DONE]":
                break

            event = json.loads(data)
            if event.get("type") == "content_block_delta":
                if first_token_time is None:
                    first_token_time = time.perf_counter()
                    ttft = first_token_time - request_start
                    metrics.record("llm.ttft_seconds", ttft)
                chunks_received += 1

    total_time = time.perf_counter() - request_start
    throughput_time = total_time - (first_token_time - request_start)
    metrics.record("llm.throughput_seconds", throughput_time)
    metrics.record("llm.chunks_received", chunks_received)

On the UI side, the correct UX pattern is to decouple the loading state from the streaming state. Show a spinner until TTFT, then transition to the streaming render. A spinner for 3 seconds feels acceptable. A blank text box with a cursor for 3 seconds feels broken, even if the total time is identical.

async function streamWithTTFT(prompt, onFirstToken, onChunk, onDone) {
  const startTime = performance.now();
  let firstTokenReceived = false;

  const response = await fetch('/api/stream', {
    method: 'POST',
    body: JSON.stringify({ prompt }),
  });

  // Show spinner immediately (caller renders spinner on button click)
  // Switch to streaming render on first token

  await consumeStream(response, (chunk) => {
    if (!firstTokenReceived) {
      firstTokenReceived = true;
      const ttft = performance.now() - startTime;
      console.log(`TTFT: ${ttft.toFixed(0)}ms`);
      onFirstToken(); // caller hides spinner, shows text area
    }
    onChunk(chunk);
  });

  onDone();
}

TTFT degrades predictably with prompt length. Prefill (processing the input tokens) is typically 2–4× faster than generation, but on a 10k-token system prompt the prefill alone takes 1–2 seconds before the model generates a single output token. If your TTFT is high, the first place to look is prompt length — not generation speed.

Timeout mismatches

The most common streaming failure in production is a read timeout misconfiguration. The default read timeout in most HTTP clients — Python's requests, Node's fetch, axios, Go's http.Client — is between 30 and 60 seconds. A model generating 10,000 tokens at 40 tokens/second takes 250 seconds. The client drops the connection at second 30, the user sees an error, and your logs show a "timeout" with no indication that the issue is configuration rather than infrastructure.

The correct timeout strategy has two separate controls — connection timeout and read timeout — which most HTTP clients expose independently:

import httpx

# Correct: tight connection timeout, generous (or disabled) read timeout
client = httpx.AsyncClient(
    timeout=httpx.Timeout(
        connect=5.0,   # 5s to establish TCP + TLS
        read=300.0,    # 5 minutes for the response body
        write=10.0,    # 10s to send the request body
        pool=5.0,      # 5s to acquire a connection from the pool
    )
)

# Wrong: single timeout applies to everything including response body
client = httpx.AsyncClient(timeout=30.0)  # kills long streams

Read timeout and stall detection are different things. A 300-second read timeout means "if no bytes arrive for 300 seconds, give up." That's too long to wait for a hung connection. The correct architecture is: set read timeout to your maximum expected generation time (300–600 seconds), and separately implement stall detection (no bytes for 10 seconds triggers an error) at the application layer. This way you distinguish "legitimately slow generation" from "connection is dead but TCP hasn't timed out yet."

import asyncio
import httpx

async def stream_with_stall_detection(
    client: httpx.AsyncClient,
    payload: dict,
    stall_timeout: float = 10.0,
):
    async with client.stream("POST", "/v1/messages", json=payload) as response:
        async for line in response.aiter_lines():
            # aiter_lines doesn't expose per-chunk timing directly;
            # wrap with asyncio.wait_for for stall detection
            try:
                line = await asyncio.wait_for(
                    anext(response.aiter_lines()),
                    timeout=stall_timeout
                )
            except asyncio.TimeoutError:
                raise StreamStallError(
                    f"No data received for {stall_timeout}s — "
                    "connection may be dead"
                )

            if line.startswith("data: "):
                yield line[6:]

For gateways that proxy LLM streams to downstream clients, the timeout problem compounds. Your gateway receives a stream from the model API on one connection and forwards it to the user on another. Each connection has its own timeout. If the upstream stream takes 200 seconds and your downstream client has a 60-second read timeout, the client drops at 60 seconds while your gateway continues consuming the upstream. You now have a zombie stream consuming inference resources for a request that already failed from the user's perspective. Implement bidirectional liveness checking: if the downstream connection drops, cancel the upstream stream immediately.

Putting it together

Each of these failure modes is independent but they compound. A mobile client with a 30-second read timeout on a slow connection hits backpressure, drops the connection at second 30, attempts to reconnect using Last-Event-ID which your server doesn't implement, restarts generation, gets different text, and the tool call that was mid-stream arrives as unparseable JSON. The user sees corrupted output and a loading spinner that never resolves.

The practical implementation checklist:

• Disable proxy buffering (proxy_buffering off) and gzip for all SSE endpoints
• Set ALB/nginx idle timeout to 300+ seconds on streaming endpoints, or emit keepalive comment frames every 15 seconds
• Implement stall detection at the client with a per-chunk timeout separate from the overall read timeout
• Accumulate tool call arguments by index before parsing — never parse streaming fragments directly
• Assign id: to every SSE event, store chunks server-side with a TTL, and replay from offset on reconnect
• Measure TTFT separately from throughput; use a spinner until first token, then switch to streaming render
• Set connection timeout to 5s and read timeout to 300–600s; implement application-layer stall detection at 10s
• In gateway/proxy setups: cancel upstream streams immediately when downstream clients disconnect

The streaming implementation in llm-gateway handles all of these: buffering-safe SSE forwarding, stall detection, chunk storage for reconnect replay, TTFT instrumentation, and bidirectional connection lifecycle management.

Streaming LLMs is not hard to implement in a demo. It's hard to implement correctly under production conditions: intermittent networks, variable client hardware, proxy infrastructure that wasn't designed for long-lived HTTP responses, and the specific quirks of tool call serialization. The patterns above are each independently necessary — there's no single fix that covers all six failure modes. Ship them as a unit or you'll encounter the ones you skipped in your next 2am incident.