The Hidden Tax of LLM API Calls: Rate Limits, Retries, and Cost Leaks

You integrate an LLM API. The first thousand calls work. You put it in production. Within a week you're debugging a mix of 429s during traffic spikes, a background job that burned $800 overnight because the prompt was larger than you thought, and an on-call alert about P99 latency jumping to 12 seconds on cold start.

None of this is surprising once you've seen it. All of it is avoidable. But the gap between "it works in development" and "it works reliably at 10k calls per day" requires engineering that most LLM integration guides never cover.

The four failure modes every LLM integration hits

In rough order of when you encounter them:

1. Rate limit 429s: you hit the provider's token-per-minute or requests-per-minute ceiling. Traffic spikes, batch jobs, and concurrent user sessions all cause this. Naive retry logic makes it dramatically worse.
2. Transient 5xx errors: provider-side infrastructure errors. These are usually short-lived (seconds to a few minutes) and require retry with backoff. The mistake is treating them the same as 429s — they're not.
3. Cost overruns: token counts are larger than expected (long conversation histories, verbose system prompts, RAG context stuffing). You discover this on your billing dashboard, not in your error logs.
4. Cold-start latency spikes: the first call to a low-traffic endpoint, or calls after a period of inactivity, takes 3–5× longer than steady-state. Users see timeouts; your p50 looks fine; your p99 looks like an incident.

Each failure mode has a different root cause and a different fix. The most common mistake is applying one strategy — usually "retry with sleep" — to all four and wondering why things aren't getting better.

Token bucket rate limiting: how the ceiling works

Provider rate limits are implemented as token buckets. The bucket has a capacity (maximum burst) and a refill rate (tokens per minute). Each API call consumes from the bucket proportionally to the tokens used. When the bucket is empty, the API returns 429.

Bucket capacity: 100,000 tokens
Refill rate: 100,000 tokens/minute (1,667 tokens/second)

Steady state (1 call/sec, avg 500 tokens):
  Consumption: 500 tokens/sec
  Refill:     1,667 tokens/sec
  Net:        +1,167 tokens/sec → bucket stays full ✓

Burst (50 calls/sec, avg 2,000 tokens):
  Consumption: 100,000 tokens/sec
  Refill:      1,667 tokens/sec
  Net:        -98,333 tokens/sec → bucket empties in ~1 second ✗
  Then:       429s until bucket refills (~60 seconds)

The trap: when you hit a 429 and sleep for a fixed interval (say, 5 seconds), then retry — your retry fires at the same time as every other thread that also slept 5 seconds. You've synchronized your retries into a thundering herd. The next wave of requests hits the API simultaneously, depletes the bucket again immediately, and you get another 429. You can cycle through this pattern for minutes, making the outage worse with every retry attempt.

Exponential backoff with full jitter: the only retry strategy that doesn't make 429s worse

The fix is exponential backoff with full jitter. The backoff time grows exponentially with each retry attempt, and the jitter randomizes the actual wait time within that window. This de-synchronizes your retries so they don't all fire at once.

import random
import time

def backoff_with_jitter(
    attempt: int,
    base_delay: float = 1.0,
    max_delay: float = 60.0,
    multiplier: float = 2.0,
) -> float:
    """Full jitter: uniform random in [0, cap]."""
    cap = min(max_delay, base_delay * (multiplier ** attempt))
    return random.uniform(0, cap)

def call_with_retry(fn, max_attempts: int = 5):
    for attempt in range(max_attempts):
        try:
            return fn()
        except RateLimitError:
            if attempt == max_attempts - 1:
                raise
            wait = backoff_with_jitter(attempt)
            time.sleep(wait)
        except TransientError:
            if attempt == max_attempts - 1:
                raise
            # Shorter backoff for 5xx — these usually clear fast
            wait = backoff_with_jitter(attempt, base_delay=0.5, max_delay=10.0)
            time.sleep(wait)

Full jitter (uniform random over [0, cap]) outperforms equal jitter (cap/2 ± random) and decorrelated jitter in simulations of thundering herd scenarios. The math: if N threads all receive 429 at time T, full jitter distributes their retries uniformly across [0, cap]. Equal jitter clusters them in the upper half. Full jitter minimizes the probability that any two threads retry simultaneously.

Critical distinction: 429 and 5xx are different errors with different backoff parameters. A 429 means the rate limit bucket is empty — it will refill at the provider's refill rate, which is predictable. A 5xx means a transient infrastructure failure — it usually resolves in seconds. Use a longer backoff ceiling for 429s (up to 60 seconds) and a shorter one for 5xx (up to 10 seconds). Conflating them causes you to either wait too long for 5xx errors or not long enough for rate limits.

Prompt caching economics: the breakeven formula

Prompt caching stores a prefix of your prompt on the provider's infrastructure so that subsequent calls with the same prefix don't re-process those tokens. The pricing model: cache writes cost slightly more than a normal input token; cache reads cost significantly less.

The write cost is 25% higher than the base input rate. The read cost is 10% of the base input rate — a 10× reduction. Whether caching is worth it depends entirely on how many times a cached prefix is reused before the cache expires.

Breakeven formula: let W be the write cost per token, R the read cost per token, I the base input cost per token, and N the number of calls that reuse the same prefix.

# Total cost with caching (1 write + N-1 reads):
cost_cached   = W + (N - 1) * R

# Total cost without caching (N full input processes):
cost_uncached = N * I

# Caching saves money when:
# W + (N-1)*R < N*I
# Solving for N:
# N > (W - R) / (I - R)

def breakeven_calls(write_cost, read_cost, input_cost):
    return (write_cost - read_cost) / (input_cost - read_cost)

# Claude Sonnet 4.6 (per-token costs, scaled to same units):
# W=3.75, R=0.30, I=3.00 (per million tokens)
breakeven_calls(3.75, 0.30, 3.00)  # → 1.27

For Sonnet 4.6, caching breaks even at 1.27 reuses. Any prompt prefix that's used more than twice pays for itself. The write cost premium is small relative to the read savings, so the math strongly favors caching whenever the same prefix appears in more than 2 calls.

When it doesn't help: if your prompts are highly dynamic (user-specific context, per-request variable data early in the prompt) and the cacheable prefix is short, the write cost overhead may not be recovered. Cache hit rate — the fraction of calls that read from cache rather than writing — is the metric to watch. If hit rate is below 50%, revisit your prompt structure before assuming caching is working.

Concrete numbers: a 10k-call/day pipeline

Assume a pipeline running 10,000 calls per day on Sonnet 4.6. Each call has a 2,000-token system prompt (same across all calls) plus 500 tokens of variable user context, and produces 300 tokens of output.

The input cost drops from $75/day to $21.60/day — a 71% reduction — just by caching the static system prompt. Output tokens are unchanged. Combined daily cost goes from $120 to $66.60, a 44% reduction on total API spend.

What caching doesn't save: output tokens (no caching equivalent exists), variable input tokens that change per request, and the first call of each cache epoch (the cache miss that triggers the write). On a high-volume pipeline, the write cost for that first call is negligible. On a pipeline with many short-lived unique prefixes, it can dominate.

Cold-start latency and what to do about it

LLM inference on hosted infrastructure has a warm/cold distinction analogous to serverless functions. When a model instance has been handling traffic recently, your request is routed to an already-warm instance. When traffic has been low or the first request comes in after a long idle, the instance may need to load model weights into GPU memory — a process that adds 2–5 seconds of latency before the first token is generated.

Cold starts are invisible in your p50 metrics and catastrophic in your p99. Typical mitigation strategies:

• Keep-alive pings: send a minimal request every 30–60 seconds to a low-traffic endpoint to prevent instances from going cold. Cost: nearly zero. Effectiveness: high for endpoints with predictable low traffic.
• Provisioned throughput: most enterprise tiers offer a dedicated capacity option that eliminates cold starts entirely. Expensive if you're underutilizing it; cost-effective if your p99 latency requirements are strict.
• Timeout budgets: set your timeout at 2× your p95 latency, not 2× your p50. If p50 is 1.2s and p95 is 3.5s, a 2.5s timeout will time out 5% of your requests. Use p99 as your baseline when cold starts are a real possibility.
• Async fallback: for workloads that can tolerate async processing, route cold-start requests to a queue rather than failing synchronously. Return a job ID; poll or webhook for the result.

Putting it together: what a production LLM gateway needs

The engineering required to handle these four failure modes in a single place — rather than scattered across every integration point in your codebase — is the reason a gateway layer exists. A production gateway needs:

• Per-model rate limit tracking: maintain token bucket state per model and per tenant. Block requests before they reach the API when the bucket is nearly depleted — a proactive 429 at your gateway is cheaper than one from the provider, because yours can apply priority queuing.
• Retry logic with correct backoff: separate retry policies for 429 vs. 5xx, with full jitter, and Retry-After header parsing.
• Token counting before dispatch: count input tokens before making the API call. Flag requests that exceed a per-call budget threshold. This is the only way to catch cost overruns before they appear on the billing dashboard.
• Prompt caching coordination: detect when a prompt's static prefix is cacheable and emit the cache control markers the provider API requires. Track cache hit rate as a first-class metric.
• Cost attribution: tag every call with the caller, model, prompt template ID, and token counts (input/output/cache-read/cache-write). Aggregate by dimension. The first time you see a per-user cost breakdown, you will find the user generating 40% of your token spend.

# Example gateway call with cost attribution
response = gateway.complete(
    model="claude-sonnet-4-6",
    messages=messages,
    system=SYSTEM_PROMPT,  # gateway handles cache-control markers
    metadata={
        "caller": "document-qa",
        "user_id": user_id,
        "template": "rag-synthesis-v2",
    },
    budget_tokens=4096,    # hard cap — raises before API call if exceeded
    priority="high",       # affects queue position under rate limiting
)

The open-source implementation of this gateway — with token bucket rate limiting, full-jitter backoff, prompt cache coordination, and cost attribution — is in llm-gateway.

The meta-point: LLM API reliability engineering is infrastructure work, not application work. The same patterns that make distributed systems reliable — circuit breakers, rate limiting, retry budgets, cost attribution — apply here. The failure modes are different in detail but identical in structure. Engineers who've operated microservices at scale will recognize most of this. Engineers who haven't will learn it the hard way on their billing dashboard at 2am.

Build the gateway layer before you need it. Retrofitting retry logic into 15 different service integrations after the first production incident is the more expensive path.