Benchmarks answer someone else’s question
Leaderboards rank models on standardised tasks that resemble nobody’s production workflow. They are useful for tracking the frontier and close to useless for choosing between two credible candidates for your specific feature.
The substitute is not opinion — it is an evaluation built from your own inputs, which lets you rank models on the only task that matters.
1. What does failure cost here?
A drafting assistant whose output a human edits can tolerate a weaker model. A step that classifies something irreversible cannot. Match capability to consequence rather than buying the strongest option for everything.
2. What is the latency budget?
A model in an interactive path competes with the user’s patience; one in a background queue competes with nothing. Interactive features often justify a faster, smaller model plus better retrieval over a slower, stronger one.
The strongest model that arrives too late is worse than the adequate one that arrives now.
3. What does the volume do to the bill?
Per-call costs that look trivial in testing become the dominant line at production volume. Estimate from real traffic, not from the demo, and check what happens if the feature succeeds and usage triples.
4. How hard is it to change your mind?
Keep the model behind an interface so swapping it is a configuration change, not a refactor. Models improve and prices fall on a cadence measured in months; a design that assumes one specific model will be expensive to unwind exactly when it is worth unwinding.
This matters more than the initial choice. Teams who can re-run their eval and switch in an afternoon end up on better models continuously; teams who can’t stay on whatever they picked first.