Production ML, evaluation, calibration, and agentic systems. Things I've found surprising, broken, or underappreciated.
The real decision tree for fine-tuning vs. prompting: do you have labeled data, is the task format-sensitive or knowledge-sensitive, have you hit the cost crossover, and can you build the eval harness? With the breakeven formula, LoRA vs. RAG tradeoffs, and the data requirements teams consistently underestimate.
SSE framing, proxy buffering, backpressure on slow clients, incomplete JSON from tool calls, reconnection replay with Last-Event-ID, TTFT measurement, and the timeout misconfiguration that kills every long generation. The complete set of streaming failure modes and how to handle each.
Every LLM integration hits the same four failure modes: 429s, transient 5xx errors, cost overruns, and cold-start latency spikes. Here's the engineering — token bucket mechanics, full-jitter backoff, prompt caching economics, and a concrete cost model — needed to handle them at scale.
Cosine similarity between embeddings measures topical overlap, not answer relevance. The metrics that actually characterize retrieval quality — Precision@K, Recall@K, NDCG, MRR — and how to use hybrid retrieval plus cross-encoder reranking to improve them.
Most deployed risk models output scores that are monotonically correct but probabilistically wrong. This is almost always fine for ranking — and almost always wrong for threshold decisions, cost modeling, and downstream systems that interpret 0.8 as "80% chance of fraud."