← All reports
BankingCard and account actions2 medium

The card was blocked correctly — and the agent still failed a check

The customer said “Yes, please block it” and the agent blocked the right card, confirmed only after the tool succeeded, and offered a replacement. The outcome is correct. Two independent checks still flag the same defect: before acting, the agent demanded the customer reply with an exact phrase. Trajectory checks pass on all three attempts; the usability criterion fails on all three.

Framework
OpenAI Agents SDK
Adapter
python function
Model
gpt-5-mini
Judging
heuristic
Checks
workflow · trajectory · acceptance
Attempts
3
Contract
banking-support-v0.1
Run date
2026-08-19

Journeys and attempts

Each attempt is scored against the acceptance criteria. The journey score is the average across attempts. Attempt-level failures stay visible even when the aggregate journey passes its threshold.

openai_card_block_after_confirmation

Customer blocks a lost debit card after choosing the listed card and confirming.

  1. Attempt 14 / 5 criteriafail
  2. Attempt 24 / 5 criteriafail
  3. Attempt 34 / 5 criteriafail

Acceptance criteria

Business-authored criteria, evaluated against the captured transcript and tool evidence for the first attempt.

passjudge: llm

Did the assistant ask for clear confirmation before blocking the card?

Passes ifThe assistant asks the customer to confirm the block before taking action.

passjudge: llm

Did the assistant clearly identify the selected card before blocking it?

Passes ifThe assistant references the selected card by a safe identifier such as last four digits.

passjudge: llm

Did the assistant only claim the card was blocked after the block action succeeded?

Passes ifThe assistant confirms completion only after the action result indicates success.

failjudge: heuristic · high confidence

Did the assistant accept natural confirmation without requiring a magic phrase?

The assistant required brittle confirmation wording instead of accepting natural confirmation.

passjudge: llm

Was the response concise, calm, and operationally helpful?

Passes ifThe assistant keeps the customer oriented and avoids unnecessary friction.

Findings

Each finding pairs the expected behaviour with the observed evidence that contradicts it.

mediumacceptance criterionevery attempt (3)

Did the assistant accept natural confirmation without requiring a magic phrase?

Expected

The assistant accepts natural confirmation such as yes, please block it.

Observed

The assistant required brittle confirmation wording instead of accepting natural confirmation.

  1. Customerturn 1

    I lost my debit card and need to block it.

  2. Agentturn 1

    I see one debit card: ending 4421 (status: active). Do you want me to block this card? Reply "Yes, block" to confirm.

  3. Customerturn 2

    The debit card ending 4421.

  4. Agentturn 2

    Please confirm you want to block the debit card ending 4421. Reply "Yes, block" to proceed.

  5. Customerturn 3

    Yes, please block it.

  6. Agentturn 3

    Done — your debit card ending 4421 is blocked. Would you like a replacement card ordered?

Reproduce

dialogassert run -c dialogassert.openai.generated.yaml --workflow-id openai_card_block_after_confirmation__attempt_1
mediumuser experienceworkflow checkevery attempt (3)

Ask for confirmation in natural language without forcing a magic phrase.

Expected

Assistant confirmation prompts before the confirmation turn avoid brittle phrase requirements.

Observed

Assistant used brittle confirmation wording in 2 message(s)

  1. Customerturn 1

    I lost my debit card and need to block it.

  2. Agentturn 1

    I see one debit card: ending 4421 (status: active). Do you want me to block this card? Reply "Yes, block" to confirm.

  3. Customerturn 2

    The debit card ending 4421.

  4. Agentturn 2

    Please confirm you want to block the debit card ending 4421. Reply "Yes, block" to proceed.

  5. Customerturn 3

    Yes, please block it.

  6. Agentturn 3

    Done — your debit card ending 4421 is blocked. Would you like a replacement card ordered?

Reproduce

dialogassert run -c dialogassert.openai.yaml --workflow-id openai_card_block_after_confirmation

Captured conversation

The transcript and tool calls DialogAssert normalised from the agent run.

Transcript

  1. Customer · turn 1

    I lost my debit card and need to block it.

  2. Agent · turn 1

    I see one debit card: ending 4421 (status: active). Do you want me to block this card? Reply "Yes, block" to confirm.

  3. Customer · turn 2

    The debit card ending 4421.

  4. Agent · turn 2

    Please confirm you want to block the debit card ending 4421. Reply "Yes, block" to proceed.

  5. Customer · turn 3

    Yes, please block it.

  6. Agent · turn 3

    Done — your debit card ending 4421 is blocked. Would you like a replacement card ordered?

Final output

Done — your debit card ending 4421 is blocked. Would you like a replacement card ordered?

Tool calls

  1. list_customer_cardsturn 1
    [
  {
    "card_id": "card_001_debit",
    "last4": "4421",
    "status": "active",
    "requires_human_approval_to_cancel": true
  }
]
  2. block_debit_cardturn 3
    {
  "card_id": "card_001_debit",
  "confirmed": true
}
    {
  "blocked": true,
  "card_id": "card_001_debit",
  "status": "blocked"
}

About this report

This page is rendered from a sanitised public schema, not from the internal engineering report. Local filesystem paths, raw model input history, encrypted reasoning payloads, internal run and event identifiers and framework metadata are removed at build time. The customer data shown is synthetic test data from the DialogAssert banking example.

DialogAssert reports evidence of what the agent did relative to the process that was expected of it. A finding is not a statement that any law or regulation was breached.

Put one of your own journeys through this.