A model benchmark answers a question about a model under a specified test. Production risk belongs to the whole deployed system.
Once a model can call tools, retrieve private data, delegate work, spend money, or create irreversible side effects, model quality remains important but is no longer the complete unit of safety.
The missing deployment questions
A runtime evaluation should ask:
- Did the agent perceive the correct source information?
- Was the acting identity distinguishable?
- Was delegated authority narrower than the caller's authority?
- Did policy evaluate the declared action and resource?
- Were consequential side effects blocked, approved, or safely simulated?
- Did retries or replay repeat an external effect?
- Did stale evidence continue to authorize a route?
- Did failure fall back safely?
- Can an outsider explain why the action or route was eligible?
These questions do not compete with model evaluations. They cover failure modes at different layers.
Unsafe success is failure
Task-completion benchmarks often reward the final outcome. That can hide an important distinction: a system may reach the correct answer using an unauthorized data source, an over-broad credential, or a duplicated side effect.
An agent that succeeds unsafely has failed.
Illustrative scenarios
- A research agent produces the correct report but reads a restricted folder.
- A planning agent delegates to a child that inherits write authority it did not need.
- A retry recovers from a timeout but sends the same external message twice.
These are threat-model examples, not reported incidents or measured customer results.
Publish benchmark boundaries
Credible runtime results should disclose:
- code version and measurement date;
- workload and complete denominator;
- failures and exclusions;
- model, provider, and configuration;
- side-effect handling;
- identity and tenant dimensions;
- raw or sanitized result records;
- limitations and what the result does not prove.
A high score without failure accounting is marketing, not evidence. A synthetic reference result can still be useful when it is labeled honestly and can be reproduced.
Benchmark the system you deploy
The right evaluation portfolio includes model behavior, task quality, policy correctness, identity isolation, side-effect safety, freshness, replay, fallback, and explanation completeness.
The opinion is deliberately sharp: model safety is necessary, but a production agent should not inherit a safety claim from its model. The deployment must earn its own evidence.