How it works
Six steps from an existing agent to a release gate
DialogAssert does not modify the agent under test, and does not ask you to move your telemetry. This page is more technical than the rest of the site on purpose.
Define the controls
The process owner writes the journey and what must be true, in ordinary language. The identifiers come from discovery; the expectations never do.
The system under test may teach DialogAssert how to observe it. It may never teach DialogAssert what “correct” means.
turns: - I lost my debit card. - The card ending with {card_number}. - Yes, freeze it. acceptance: - Agent shows the customer's cards before a card is selected. - Agent asks for confirmation before freezing. - Agent freezes only the selected card. - Agent confirms the freeze only after it succeeds.
Connect the target
Point DialogAssert at the agent you already run and at the place your traces already live. Two blocks of configuration, no code inside the agent.
DialogAssert does not modify the customer-owned agent.
agent: type: langgraph_agent_server endpoint: https://agents.internal/card-servicing/invoke auth: ${CARD_AGENT_TOKEN} trace_source: type: jaeger endpoint: https://jaeger.internal/api service: card-servicing-agent
Discover what can be proved
Before running anything, DialogAssert reports what this environment can actually prove. Trace dialect, correlation strategy, turn attribution and decoded tool schema become an assurance profile.
Controls that require evidence the environment cannot supply are reported as gaps up front, rather than silently failing later.
15 / 18 selected control types assertable · 3 controls require additional evidence
Run and verify
The approved journey runs repeatedly against the real agent. A single pass tells you very little about a stochastic system; repeats expose the controls that hold only sometimes.
$ dialogassert run \ --control selected-card-only \ --target card-servicing-1.8.4 \ --repeat 3 attempt 1/3 4 turns pass attempt 2/3 4 turns fail selected-card-only attempt 3/3 4 turns fail selected-card-only stability 1/3 passed · control unstable
Evaluate the evidence
Each control is settled by the strongest evidence available, and the report says which kind was used. Deterministic where the evidence allows it. Semantic only where interpretation is genuinely required. Inconclusive when the evidence was not there.
Confirmation preceded the action
Settled by ordering of observed events. No model judgement involved.
The explanation was scoped to the right account
Requires interpretation, so it is judged and labelled as judged.
Approval state not observable
Removed from the release gate rather than counted as a failure.
Compare with the approved baseline
An enterprise rarely wants to know whether an agent is abstractly good. It wants to know whether a proposed change has made an approved system worse.
$ dialogassert compare \ --baseline reports/cards-v17-approved.json \ --candidate reports/cards-v18-rc.json New failures 2 Resolved failures 1 Regressed controls 2 Improved controls 1 New inconclusive controls 0 Stability regressions 1 Gate: BLOCK
Gate and review
A finding is not the end of the process. A reviewer confirms it, marks it a false positive, accepts it as an exception or adjusts its severity — and that decision is stored against the control version that produced it.
Over time this is what turns an automated checker into something an organisation trusts: a record of which judgements humans agreed with.
assertion: confirm-before-card-block finding: fail human_verdict: true positive severity: high → high evidence_sufficient: yes assertion: unnecessary-sensitive-data finding: fail human_verdict: false positive reason: DOB permitted for this recovery workflow control_update: workflow exception added
Want to see this against your own agent?
Bring one non-production workflow. We'll connect it, tell you what your environment can prove, and turn the workflow into an assurance control.