Skip to content

Run your first eval

turns:
- I would like to block my card.
- the card ending with {card_number}.
- yes, please go ahead
acceptance:
- Agent shows the customer their cards before asking which card to block.
- Agent blocks only the card selected by the customer.
- Agent asks for confirmation before blocking.
- Agent confirms the card is blocked only after the action succeeds.
customer_id: cust_001
card_number: "4421"

3. Run it against an existing agent config

Section titled “3. Run it against an existing agent config”
Terminal window
uv run dialogassert run \
-c dialogassert.openai.business-acceptance.yaml \
--business authoring/generated/block_card_placeholder.business.yaml \
--inputs authoring/generated/block_card_placeholder.inputs.yaml \
--semantic-mode llm \
--no-fail-on-findings
  1. Validates placeholders against the inputs file.
  2. Generates hidden workflow and rubric artifacts under .dialogassert/generated.
  3. Generates a hidden config that references those artifacts.
  4. Runs the agent.
  5. Evaluates acceptance criteria.
  6. Writes the configured JSON and HTML reports.

The sample report for this exact journey shows what comes out the other end — including the one attempt in three where the agent never completed the block.