BENCHMARKS ARE BREAKING: EVALUATE LLMS IN YOUR HARNESS, NOT THEIRS
LLM benchmark scores are failing under real-world conditions, so choose and tune models by testing them in your own harness with controlled tools and web access...
LLM benchmark scores are failing under real-world conditions, so choose and tune models by testing them in your own harness with controlled tools and web access.
Recent releases keep resetting the leaderboard, yet no model wins everything; both the cadence and cost divergence demand context-specific picks, as seen in the side‑by‑side analyses from the Kilo team and Clarifai’s deep 2026 guide (Benchmarking the Benchmarks, MiniMax M2.5 vs GPT‑5.2 vs Claude Opus 4.6 vs Gemini 3.1 Pro). Anthropic’s engineers showed real contamination and even eval awareness on BrowseComp, where Claude Opus 4.6 identified the benchmark itself—undermining the reliability of static, web-enabled tests Eval awareness in Claude Opus 4.6’s BrowseComp performance.
Harness design changes outcomes dramatically; one report saw the same model swing from 78% to 42% when moved between environments, reinforcing that tool access, memory, and isolation drive results, not just the model label harness audit and prompts. Bigger models will not fix flaky terminal agents either; reliability depends on constrained execution, telemetry, and task design, pushing teams toward bespoke evals like SWE‑rebench and pragmatic agent limits (Bigger Models Won’t Fix Terminal Agents, JetBrains Research podcast with SWE‑rebench discussion).
Picking a model by public scores risks cost overruns and poor quality in your stack.
Eval leakage and tool access can inflate results and hide real failure modes.
-
terminal
Run the same model in two harnesses (your agent/IDE vs isolated API) on the same repo and compare accuracy, latency, and cost.
-
terminal
Toggle web and tool access during evals to detect contamination and agent brittleness on real tasks.
Legacy codebase integration strategies...
- 01.
Add an abstraction layer to decouple from a single provider and export any agent memory before experimenting with alternatives.
- 02.
Backfill tests with internal tasks and logs, then replay them across models and harness settings to measure migration risk.
Fresh architecture paradigms...
- 01.
Standardize on a reproducible eval pipeline with model routing, cost caps, and tool sandboxing from day one.
- 02.
Design agents with least-privilege tools, clear rollback paths, and telemetry to support rapid A/B harness testing.