Overview

DialogAssert is a behavioural QA runner for AI agents.

Generic LLM eval tools can judge prompts and outputs. DialogAssert focuses on agent behaviour inside a business process.

Questions DialogAssert is designed to answer

• Did the agent ask for confirmation before taking a high-impact action?
• Did the agent use the account, card or transaction selected by the customer?
• Did the agent expose unrelated data?
• Did the agent retrieve or use sensitive data before it was needed?
• Did the agent complete the intended workflow after the customer confirmed?
• Is the behaviour stable across repeated runs of a non-deterministic agent?

Current MVP scope

• Business-authored journeys with turns and acceptance.
• Separate runtime inputs for placeholders such as customer_id and card_number.
• Real-agent runs through OpenAI Agents SDK, LangChain, HTTP or local Python adapters.
• Acceptance criteria evaluated with heuristic or LLM semantic judging.
• Optional generated workflow, trajectory and rubric files.
• Optional engineering-authored trajectory checks.
• Optional deterministic assertions.
• Repeated test runs for non-deterministic agents.
• Aggregate pass thresholds such as repeat: 3 and pass_threshold: 0.75.
• JSON and HTML reports.
• Replay of saved evidence after changing assertions or rubrics.

MVP boundaries

• Customer-owned agents should not be modified for DialogAssert.
• Business users should not have to write fixtures, tool catalogues, trajectory files or risk metadata.
• Every placeholder used in a business journey must be supplied through an inputs file.
• Trajectory tests are optional and can run with or without workflow tests.
• DialogAssert should not infer private tool order, backend schema or domain behaviour by reading customer agent source code.