Build a representative golden set
Collect real examples with expected outcomes or review criteria. Include easy cases, edge cases and known failures. A tiny hand-picked demo set overestimates performance.
Separate dimensions
Score factual correctness, completeness, format, safety and latency separately when they matter. A single average can hide serious failures.
Use humans where judgment matters
Automated judges are useful for scale but can be biased toward certain styles or models. Calibrate them against human-reviewed examples and keep human review for ambiguous or high-impact cases.
Compare before and after
Every change should be measured against a baseline. Record the model version, prompt, retrieval configuration and dataset so the result can be reproduced.
Connect evals to production
Offline quality is only one layer. Monitor cost, latency, retries, user corrections and incident rates. The best release is the one that improves the real system, not only the benchmark.