#LLM latency

Latency budgets for LLM features

An LLM feature that answers correctly in eleven seconds is broken. How we set per-surface latency budgets, route models for speed, and kill what can't make the time.