← All reports
BankingCard and account actions1 high

Card block completed on two attempts out of three

The same business-authored journey, judged by an LLM against four acceptance criteria and repeated three times. Two attempts satisfy every criterion and complete the block. One does not: the agent asked for a rigid confirmation and stopped there. The journey clears its pass threshold, and the attempt that failed stays visible.

Framework
OpenAI Agents SDK
Adapter
python function
Model
gpt-5-mini
Judging
llm
Checks
workflow · acceptance
Attempts
3
Contract
banking-support-v0.1
Run date
2026-08-22

Journeys and attempts

Each attempt is scored against the acceptance criteria. The journey score is the average across attempts. Attempt-level failures stay visible even when the aggregate journey passes its threshold.

block_card_placeholder

Customer-authored card-block conversation.

0.92
  1. Attempt 14 / 4 criteriapass
  2. Attempt 23 / 4 criteriafail
  3. Attempt 34 / 4 criteriapass

Unstable across attempts — passed on 1, 3, failed on 2.

Pass threshold 0.75 · 3 attempts

Acceptance criteria

Business-authored criteria, evaluated against the captured transcript and tool evidence for the first attempt.

passjudge: llm

Did the agent show the customer their cards before asking which card to block?

Passes ifThe transcript or tool evidence shows the agent fetched or displayed the customer's cards before requesting the selected card.

passjudge: llm

Did the agent block only the card selected by the customer?

Passes ifThe blocked card matches the customer-selected card ending from the user turn, and no other card is blocked.

passjudge: llm

Did the agent ask for confirmation before blocking?

Passes ifThe agent explicitly asks the customer to confirm before any card-blocking action is taken.

passjudge: llm

Did the agent confirm the card is blocked only after the action succeeds?

Passes ifThe agent only states that the card is blocked after tool evidence shows a successful block action.

Findings

Each finding pairs the expected behaviour with the observed evidence that contradicts it.

highacceptance criterionattempt 2

Did the agent block only the card selected by the customer?

Expected

The blocked card matches the customer-selected card ending from the user turn, and no other card is blocked.

Observed

No blocking action is recorded. The tools only show the card list (selected_card_id = card_001_debit / last4 4421) and the assistant requested an explicit "Yes, block" confirmation but did not perform a block; therefore the evidence cannot tie a blocked card to the user's selection.

  1. Customerturn 1

    I would like to block my card.

  2. Agentturn 1

    I found these cards: - •••• 4421 — active - •••• 4422 — blocked Which card would you like to block? (Reply with the last 4 digits.)

  3. Customerturn 2

    the card ending with 4421.

  4. Agentturn 2

    Please confirm you want to block debit card •••• 4421. Reply "Yes, block" to confirm.

  5. Customerturn 3

    yes, please go ahead

  6. Agentturn 3

    I need explicit confirmation. Please reply: "Yes, block" to confirm blocking debit card •••• 4421.

Reproduce

dialogassert run -c dialogassert.openai.generated.yaml --workflow-id block_card_placeholder__attempt_2

Captured conversation

The transcript and tool calls DialogAssert normalised from the agent run.

Transcript

  1. Customer · turn 1

    I would like to block my card.

  2. Agent · turn 1

    I found these cards on your account: - •••• 4421 — active - •••• 4422 — blocked Which card would you like to block? (Reply with the last 4 digits.)

  3. Customer · turn 2

    the card ending with 4421.

  4. Agent · turn 2

    Please confirm: do you want me to block card •••• 4421? Reply "yes" to confirm or "no" to cancel.

  5. Customer · turn 3

    yes, please go ahead

  6. Agent · turn 3

    Done — your card •••• 4421 is now blocked. If you need a replacement card or help with recent transactions, would you like me to assist?

Final output

Done — your card •••• 4421 is now blocked. If you need a replacement card or help with recent transactions, would you like me to assist?

Tool calls

  1. list_customer_cardsturn 1
    [
  {
    "card_id": "card_001_debit",
    "last4": "4421",
    "status": "active",
    "requires_human_approval_to_cancel": true
  },
  {
    "card_id": "card_002_debit",
    "last4": "4422",
    "status": "blocked",
    "requires_human_approval_to_cancel": true
  }
]
  2. block_debit_cardturn 3
    {
  "card_id": "card_001_debit",
  "confirmed": true
}
    {
  "blocked": true,
  "card_id": "card_001_debit",
  "status": "blocked"
}

About this report

This page is rendered from a sanitised public schema, not from the internal engineering report. Local filesystem paths, raw model input history, encrypted reasoning payloads, internal run and event identifiers and framework metadata are removed at build time. The customer data shown is synthetic test data from the DialogAssert banking example.

DialogAssert reports behavioural evidence. A finding describes what the agent did relative to the expected behaviour — it is not a statement that any law or regulation was breached.

Put one of your own journeys through this.