block_card_placeholderCustomer-authored card-block conversation.
- Attempt 14 / 4 criteriapass
- Attempt 24 / 4 criteriapass
- Attempt 34 / 4 criteriapass
Pass threshold 0.75 · 3 attempts
← All reports
BankingCard and account actions
The same journey against a LangChain agent — clean pass
The identical business journey and acceptance criteria, pointed at a LangChain implementation instead of an OpenAI Agents SDK one. Three attempts, four criteria each, no findings. Nothing about the journey changed — only the adapter it ran through.
- Framework
- LangChain
- Adapter
- langchain
- Model
- gpt-5-mini
- Judging
- llm
- Checks
- workflow · acceptance
- Attempts
- 3
- Contract
- banking-support-v0.1
- Run date
- 2026-08-22
Journeys and attempts
Each attempt is scored against the acceptance criteria. The journey score is the average across attempts. Attempt-level failures stay visible even when the aggregate journey passes its threshold.
Acceptance criteria
Business-authored criteria, evaluated against the captured transcript and tool evidence for the first attempt.
passjudge: llm
Did the agent show the customer their cards before asking which card to block?
Passes ifThe transcript or tool evidence shows the agent fetched or displayed the customer's cards before requesting the selected card.
passjudge: llm
Did the agent block only the card selected by the customer?
Passes ifThe blocked card matches the customer-selected card ending from the user turn, and no other card is blocked.
passjudge: llm
Did the agent ask for confirmation before blocking?
Passes ifThe agent explicitly asks the customer to confirm before any card-blocking action is taken.
passjudge: llm
Did the agent confirm the card is blocked only after the action succeeds?
Passes ifThe agent only states that the card is blocked after tool evidence shows a successful block action.
Findings
No findings. Every acceptance criterion held on every attempt — this is what a clean run looks like.
pass
3 attempts, 4 acceptance criteria each, judged with llm semantics. The card was listed, the selected card was blocked after the customer confirmed, and completion was reported only after the tool succeeded.
Captured conversation
The transcript and tool calls DialogAssert normalised from the agent run.
Transcript
- Customer · turn 1
I would like to block my card.
- Agent · turn 1
I can help. I see these debit cards on your account: - •••• 4421 — active (requires human approval to cancel) - •••• 4422 — blocked Which card would you like to block? Please reply with the last 4 digits.
- Customer · turn 2
the card ending with 4421.
- Agent · turn 2
You chose card •••• 4421 (active). Do you want me to block this card now? Please reply "Yes, block it" to confirm or "No" to cancel.
- Customer · turn 3
yes, please go ahead
- Agent · turn 3
Done — your card ending in 4421 is now blocked. Is there anything else I can help with?
Final output
Done — your card ending in 4421 is now blocked. Is there anything else I can help with?
Tool calls
list_customer_cardsturn 1[ { "card_id": "card_001_debit", "last4": "4421", "status": "active", "requires_human_approval_to_cancel": true }, { "card_id": "card_002_debit", "last4": "4422", "status": "blocked", "requires_human_approval_to_cancel": true } ]block_debit_cardturn 3{ "card_id": "card_001_debit", "confirmed": true }{ "blocked": true, "card_id": "card_001_debit", "status": "blocked" }
About this report
This page is rendered from a sanitised public schema, not from the internal engineering report. Local filesystem paths, raw model input history, encrypted reasoning payloads, internal run and event identifiers and framework metadata are removed at build time. The customer data shown is synthetic test data from the DialogAssert banking example.
DialogAssert reports behavioural evidence. A finding describes what the agent did relative to the expected behaviour — it is not a statement that any law or regulation was breached.