#evals
7 pieces filed under “evals”.
Human-in-the-loop: designing the review queue
Human-in-the-loop AI fails at the queue, not the model. Triage design, keyboard-first reviewer ergonomics, feedback loops into evals, and honest staffing maths.
Swapping the model without breaking the product
Model migrations without breaking the product: shadow runs, eval-gated cutovers, tone and format regression diffs, and rollback plans for behaviour you can't unit test.
The model deprecation playbook nobody writes until it's 3am
Your LLM provider just announced a deprecation window. How to migrate without panic: abstraction layers, eval-gated cutovers, shadow traffic and honest customer comms.
Evals-first development: the build order that actually works
Write the golden set before the prompt, version prompts like code, and gate merges on eval scores. The workflow that makes LLM features shippable.
Evals are the new unit tests: a practical guide for product teams
Golden sets, rubric graders, regression gates and CI plumbing: how product teams build LLM evaluations that catch real regressions without a research team.
Feedback loops users will actually feed
Thumbs-down rates near zero tell you nothing. Designing AI feedback signals that collect usable data, close the loop visibly, and track volume as a health metric.
Prompt libraries are a design system too
Prompts scattered through a codebase rot like untracked CSS. How we build prompt libraries: structure, versioning, test coverage, and a shared vocabulary.