Define the task distribution
Do not test models on a single impressive prompt. Collect representative examples: easy, normal, difficult and failure-prone cases. Your real distribution may be dominated by routine work where a smaller model is more economical.
Measure what matters to the user
Choose two to five criteria such as factual accuracy, coding correctness, format compliance, time to first token, total latency and cost. Weight them explicitly. A model that is “best” on a public benchmark may still lose on your weighted objective.
Use current source data
Context windows, prices, model IDs and availability change. Treat them as volatile facts with a source and review date. Lumaria keeps these fields separate from evergreen explanation for exactly this reason.
Test routing before standardizing one model
Many systems benefit from routing: simple requests go to a cheaper fast model; complex or high-risk cases escalate. Routing can reduce cost without forcing every request through the most capable model.
Re-run evals after changes
A new model, prompt or retrieval strategy is a deployment change. Re-run the same eval set, compare failures and watch production metrics. Model selection is an ongoing operating process, not a one-time ranking.