Release assurance vs evals
Agent evaluation tells you how the agent performed. Release assurance tells you whether this candidate should ship.
Evals are useful and you should keep yours. They answer a question about quality. A release decision is a different question, asked by different people, and it needs different evidence.
Two different questions
| Agent eval | Release assurance |
|---|---|
| Scores performance | Supports a release gate |
| Evaluates a test or a run | Compares a candidate against an approved baseline |
| Usually engineering-owned | Acceptance standard can be business- or control-owned |
| Missing trace is an evaluator problem | Missing evidence is an explicit Inconclusive result |
| Tool was invoked | Outcome independently verified where possible |
| Framework or platform context | Cross-platform acceptance standard |
| Current result | Versioned control plus evidence history |
This is not an argument that evals are bad. It is an argument that a score, however good, does not tell an accountable person whether a specific version should reach customers.
Four things a release decision needs that a score doesn't provide
An acceptance standard the system under test doesn't own
A suite derived from the implementation — its code, its traces, its own documentation — can only establish that the implementation does what it does. The agent may teach DialogAssert how to observe it. It cannot define what acceptable means.
An explicit result for missing evidence
A binary system has to guess when the evidence it needs isn't there, and the guess is usually an accusation. Pass, Fail, Inconclusive — never a guessed pass or a guessed failure.
Levels of verification
The agent claimed it. The action was observed. The dependency confirmed it. The system state verified it. Those are four different strengths of proof, and a report that doesn't distinguish them is asking to be trusted rather than checked.
A release boundary
Approved controls, plus this candidate's evidence, plus the release the organisation previously accepted. What regressed, what resolved, what stayed the same, and whether that adds up to a gate result.
The clearest example
Customer Support Agent — Investigates orders and can issue customer refunds.
What an eval sees
Response quality 9.1 Tone on brand Hallucination none detected Task completion pass
The agent said “Your €79 refund has been issued.” The sentence is accurate in form, helpful in tone and complete. Every metric passes.
What release assurance sees
1 Agent claimed “Your €79 refund has been issued.” 2 Action observed create_refund(order_1842, 79.00) 3 Execution confirmed {"request": "accepted"} 4 System state no refund exists Control truthful-completion Verdict FAIL Change Regressed vs approved baseline Gate BLOCK
The dependency accepted the request. The refund never happened. The customer has been told it did.
Keep your evals. Keep your observability.
DialogAssert sits at the release boundary: approved controls on one side, the candidate and its evidence on the other. If your team already runs LangSmith, AgentCore, LangWatch, Azure evaluations or something you built yourselves, none of that needs to change.
Which of your agent releases would this have caught?
Bring one non-production workflow. A typical pilot starts with one agent, one evidence source and three to five business controls.