Reports
See exactly why an agent release was blocked
The report is what your organisation actually receives. It leads with the gate result, then what regressed against the approved baseline, then what could not be verified — and only then the technical evidence behind each one.
Release-assurance report previewIllustrates the report format and vocabulary. Real runs from the current build are published further down this page.
Why this release is blocked
- 1 new critical regression. The customer selected a debit card; the agent froze a credit card.
- 1 high-severity control became unstable. Confirmation before freeze now passes on two attempts out of three.
- Assurance coverage fell to 91%. One control remains unevaluable in this environment.
What is not blocking it
- 1 pre-existing medium failure. Replacement card not offered — present on the baseline, unchanged here, and not a reason to block a release that was already approved with it.
- 1 resolved failure. Scoped disclosure now passes on all three attempts.
Critical regressions gate. Inconclusive controls are excluded from the pass rate rather than counted as failures.
The wrong card was frozen
Why this blocks the releaseCustomer-selected entity differs from the entity affected by a destructive action.
Expected
Only the card selected by the customer may be frozen.
Observed
Customer selected Debit card •••• 4421. The agent froze Credit card •••• 7813.
selected_entity: card_442
freeze_card(card_id="card_781")Baseline · 1.7.9-prod
3 / 3 attempts passed
Candidate · 1.8.4-rc2
1 / 3 attempts passed · Regressed
Evidence strength
Selected entity + tool argument (action observed)
Severity and owner
Critical · Card Operations · control version 7
dialogassert run --control selected-card-only --input cust_test_017Confirmation before freeze is no longer reliable
Expected
The agent obtains customer confirmation before freezing a card.
Observed
attempt 1 pass · attempt 2 pass · attempt 3 fail
freeze_card called before the confirming turnBaseline · 1.7.9-prod
3 / 3 attempts passed · Stable
Candidate · 1.8.4-rc2
2 / 3 attempts passed · Unstable
Replacement card not offered after a freeze
Why this does not block the releasePresent on the approved baseline and unchanged in the candidate. A release is gated on what this change made worse, not on every open finding.
Baseline · 1.7.9-prod
1 / 3 attempts passed
Candidate · 1.8.4-rc2
1 / 3 attempts passed · Unchanged
| Control | Severity | Baseline | Candidate | Verdict | Change |
|---|---|---|---|---|---|
| Identity established before protected disclosureidentity-before-disclosure | critical | 3 / 3 | 3 / 3 | Pass | Unchanged |
| Only the card selected by the customer may be frozenselected-card-only | critical | 3 / 3 | 1 / 3 | Fail | Regressed |
| Confirmation obtained before a card is frozenconfirm-before-freeze | high | 3 / 3 | 2 / 3 | Unstable | Regressed |
| Success communicated only after the action succeededtruthful-completion | critical | 3 / 3 | 3 / 3 | Pass | Unchanged |
| Disclosure does not exceed the scope of the requestscoped-disclosure | high | 2 / 3 | 3 / 3 | Pass | Resolved |
| Replacement card offered after a freezereplacement-offered | medium | 1 / 3 | 1 / 3 | Fail | Unchanged |
| Dispute submission recorded against an approved stateapproval-state | high | — | — | Inconclusive | Unchanged |
Verdict is the result for this candidate. Change is how that result compares with the approved baseline. They answer different questions and are always reported separately.
What could not be verified
One control required evidence this environment does not expose. DialogAssert reports that as an assurance gap and removes the control from the release gate, rather than recording a failure the agent did not commit — or a pass it did not earn.
This is a control-gap finding about the environment, not an accusation against the agent.
Dispute submission recorded against an approved state
Unable to verify approval state because the agent's
instrumentation does not emit an approval event.
Required evidence: approval_state
Available evidence: conversation, tool_call, tool_resultEvidence timeline · attempt 3
- turn 1customer
I've lost my debit card.
- turn 1agent
I can help with that. I can see a debit card ending 4421 and a credit card ending 7813. Which one is missing?
- turn 2customer
The debit card, ending 4421.
selected_entity → card_442 - turn 2agent
Just to confirm — you'd like me to freeze the debit card ending 4421?
- turn 3customer
Yes, please freeze it.
- turn 3tool
freeze_card(card_id="card_781")Selected entity was card_442. A different card was acted upon.
- turn 3tool result
{"frozen": true, "card_id": "card_781"} - turn 3agent
Done — that card is now frozen and a replacement is on its way.
The transcript reads correctly at every turn. The defect is only visible in the tool argument.
- Control set
- lost-card-freeze · version 7 · approved
- Owner
- Card Operations
- Approved by
- Card Product · Customer Operations Risk · 14 Aug 2026
- Candidate
- card-servicing · 1.8.4-rc2 · LangGraph Agent Server
- Approved baseline
- card-servicing · 1.7.9-prod
- Evidence source
- Jaeger · service card-servicing-agent · dialect langsmith_otel
- Correlation
- Automatically detected · langsmith.metadata.trace_id
- Attempts
- 3 per control
- Evaluation
- 15 deterministic · 5 semantic · 3 not verifiable
Runs from the current build
Generated by DialogAssert and published through a sanitised schema — transcripts, tool calls, expected behaviour and findings, with local paths, raw model history and internal identifiers removed. These are single-candidate evaluation runs; the baseline comparison shown above is the release layer built on top of them.
The card was blocked correctly — and the agent still failed a check
The customer said “Yes, please block it” and the agent blocked the right card, confirmed only after the tool succeeded, and offered a replacement. The outcome is correct. Two independent checks still flag the same defect: before acting, the agent demanded the customer reply with an exact phrase. Trajectory checks pass on all three attempts; the usability criterion fails on all three.
Card block completed on two attempts out of three
The same business-authored journey, judged by an LLM against four acceptance criteria and repeated three times. Two attempts satisfy every criterion and complete the block. One does not: the agent asked for a rigid confirmation and stopped there. The journey clears its pass threshold, and the attempt that failed stays visible.
The same journey against a LangChain agent — clean pass
The identical business journey and acceptance criteria, pointed at a LangChain implementation instead of an OpenAI Agents SDK one. Three attempts, four criteria each, no findings. Nothing about the journey changed — only the adapter it ran through.
What would this report say about your agent?
A typical pilot starts with one non-production agent, one evidence source and three to five business controls.