Key takeaways
- A model's benchmark score reflects the harness around it as much as the weights — run your own eval.
- Agents can execute a tool before refusing, so govern the action path, not just the model output.
- Cheap models with heavy retry loops can cost more than premium models that answer correctly once.
- Design harnesses so models stay swappable; licensing and openness protect that optionality.
- Everyday agent incidents fall on you, not regulators — build action-level guardrails now.
Here is a question that trips up most procurement teams: how can the same AI modele score eight points apart on the same benchmark, run by two credible labs, in the same week? No one swapped the weights. No one retrained anything. What changed was the harness — the orchestration layer that turns a raw language model into a working agent. Understanding that gap is now one of the most commercially important skills in AI procurement, and almost nobody talks about it.
The model is not the product
When Mistral opened the preview of Mistral Large 4 — a 1,050-billion-parameter model marketed as the most capable open model outside China — the headline number was raw capability. When Aleph Alpha shipped Kolibri under Apache 2.0 with fully downloadable weights, the headline was openness. And when Google opened Gemini 4 Argon to developers worldwide (before announcing its API price would double on 1 December), the headline was cost per million tokens.
All three framings miss the same thing. A benchmark score is not a property of the model alone. It is a property of the model plus everything wrapped around it: the system prompt, the tool definitions, the retry logic, how many reasoning steps the orchestrator allows, how it parses outputs, and how it handles tool failures. That wrapper is the agent harness. Change it, and the same model behaves like a different one.
Where the eight points actually come from
In practice the spread comes from a handful of unglamorous engineering choices:
- Prompt scaffolding. A richer system prompt and few-shot examples can lift a model several points before you ever touch the weights.
- Tool access. A model that can call a calculator, a code sandbox or a search tool solves problems that the bare model cannot. The capability lives in the harness, not the parameters.
- Retry and verification loops. Letting the agent check its own work, or re-run on failure, trades tokens for accuracy — and inflates scores at real cost.
- Output parsing. Sloppy parsing marks correct answers wrong. A better extractor recovers points that were always there.
None of this is cheating. It is exactly what happens when you move a model from a demo into production. But it means a vendor leaderboard tells you very little about how a model will perform inside your harness.
The harness is also where the risk lives
This is where it stops being a benchmarking curiosity and becomes a governance problem. A recent preprint showed that an AI agent can execute a tool before it even formulates a refusal — the model's text says "I won't do that," while the orchestrator has already fired the action. The same study documented 253 verified bugs across 118 MCP servers. The lesson is blunt: the model's output is not the thing that acts. The harness acts.
If you govern only what the model says, you are auditing the wrong layer. Guardrails have to sit on the action path — the tool calls, the API executions, the writes to your systems — independently of what the model claims it is doing. For any business deploying agents against real data or real money, action-level controls are not optional.
What this means for your buying decisions
The policy backdrop reinforces the point. Anthropic is circulating a proposal that would only halt a model for imminent risk of mass casualties or a billion dollars in damage — a very high bar, which means the everyday agent incidents (the ones in the tens of thousands of euros range) fall squarely on you, not the regulator. Meanwhile Beijing's new export-declaration regime explicitly leaves API calls and open-source weights out of scope, so the model you integrate today may sit on shifting legal ground tomorrow.
At Evocative we tell clients to invert the usual order of questions:
- Benchmark the harness, not the model. Run your own evaluation with your tools, your prompts and your data. Vendor numbers are a floor, not a forecast.
- Separate output from action. Put verification, rate limits and human approval on the tool-call path. Assume the model can act against its stated intent.
- Price the loops, not the tokens. A cheap model that needs five retries can cost more than a premium one that gets it right once — especially with Gemini-style price changes looming.
- Keep the model swappable. Design the harness so you can replace Mistral with Kolibri with Gemini without rewriting your logic. Openness and licensing matter precisely because they protect that optionality.
The uncomfortable, useful truth is that most of the value — and most of the risk — in an AI system is not in the modele you chose. It is in the harness you built around it. Choose the model carefully, but engineer the harness like it's the product. Because it is.
Sources
- Mistral Large 4 : ce que vaut le « meilleur modèle ouvert hors Chine » — Actu IA
- Kolibri d'Aleph Alpha : ce que la licence Apache 2.0 garantit — Actu IA
- Un agent IA peut exécuter l'outil que son modèle dit refuser — Actu IA
- L’API Gemini 4 Argon doublera ses tarifs le 1er décembre — Actu IA
- Anthropic propose de ne bloquer un modèle qu'en cas de risque imminent de plus de 50 morts ou blessés graves, ou 1 Md$ — Actu IA
- Pékin veut restreindre l'export de 23 technologies d'IA, sans toucher à l'API de DeepSeek et Qwen — Actu IA
