DialogAssert Evals

Write the behaviour you expect. Get evidence of what the agent actually did.

Business teams author customer turns and acceptance criteria. DialogAssert compiles the test, runs it against your agent, evaluates repeated attempts and returns findings grounded in the conversation and execution trace.

View sample report →

Authoring

The business-owned test stays small.

Business journey
turns:
  - I would like to block my card.
  - the card ending with {card_number}.
  - yes, please go ahead

acceptance:
  - Agent shows the customer their cards before asking which card to block.
  - Agent blocks only the card selected by the customer.
  - Agent asks for confirmation before blocking.
Runtime inputs
customer_id: cust_001
card_number: "4421"

No tool catalogue. No fixture schema. No trajectory DSL.

Business owns the journey and acceptance criteria. Runtime values stay separate. Engineering can add tool-level trajectory checks where deeper implementation assurance is useful.

Ownership

Everyone writes the part they actually know.

BusinessEngineeringDialogAssert
Customer journeyAgent endpoint / adapterTest compilation
Acceptance criteriaOptional trajectory checksEvidence normalisation
Expected business outcomeOptional integration metadataAcceptance evaluation
Repeated runs
Findings and report

Execution

One journey. Multiple layers of evidence.

Evaluation pipeline
Business Journey
      ↓
Acceptance Compiler
      ↓
Agent
      ↓
Transcript + Tool Evidence
      ↓
Acceptance / Assertions
      ↓
Repeated Attempts
      ↓
Finding + Stability Score

  • Workflow acceptance evals

    Test whether a business outcome was achieved in the required context.

  • Trajectory and tool evals

    Optionally inspect which tools were called, in what order and with which arguments.

  • Deterministic + semantic checks

    Use exact checks where the evidence permits them; narrow semantic rubrics where language interpretation is necessary.

  • Trace-backed findings

    Link failures to the relevant transcript, tool call, tool result and behavioural context.

  • Repeated execution

    Run non-deterministic journeys multiple times and apply a configurable pass threshold.

  • Framework adapters

    Use the same business journey against different agent implementations.

Stability

Passing once is not the same as being reliable.

These are the actual attempt scores from the card-block journey in the published sample report, run three times against an OpenAI Agents SDK banking agent.

Attempt 14 / 4 criteriapass
Attempt 23 / 4 criteriafail
Attempt 34 / 4 criteriapass
Journey score
0.92
Pass threshold
0.75
Attempts
3

Unstable criterion

Did the agent block only the card selected by the customer?

The journey passes its threshold. The attempt that failed stays visible, because an agent that satisfies a criterion two times out of three has not satisfied it reliably.

Evidence

A failed criterion should come with a case, not just a score.

mediumuser experienceworkflow checkevery attempt (3)

Ask for confirmation in natural language without forcing a magic phrase.

Expected

Assistant confirmation prompts before the confirmation turn avoid brittle phrase requirements.

Observed

Assistant used brittle confirmation wording in 2 message(s)

  1. Customerturn 1

    I lost my debit card and need to block it.

  2. Agentturn 1

    I see one debit card: ending 4421 (status: active). Do you want me to block this card? Reply "Yes, block" to confirm.

  3. Customerturn 2

    The debit card ending 4421.

  4. Agentturn 2

    Please confirm you want to block the debit card ending 4421. Reply "Yes, block" to proceed.

  5. Customerturn 3

    Yes, please block it.

  6. Agentturn 3

    Done — your debit card ending 4421 is blocked. Would you like a replacement card ordered?

See DialogAssert on your own workflow.