Repeated runs and thresholds

AI agents are non-deterministic. DialogAssert supports repeated attempts and aggregate pass thresholds.

evaluation:
  repeat: 3
  pass_threshold: 0.75

Scoring rule

attempt score = passed acceptance criteria / total acceptance criteria
journey score = average attempt score across attempts
journey passes when journey score >= pass_threshold

Attempt-level failures remain visible even when the aggregate journey passes.

Why this matters

In the published card-block acceptance report, the journey scores 0.92 against a threshold of 0.75 — a pass. One of the three attempts still failed: the agent demanded a rigid confirmation phrase and never blocked the card. A single run would have reported either a clean pass or an alarming failure, depending on which attempt you happened to get.

Override the repeat count for one run:

uv run dialogassert run \
  -c dialogassert.openai.generated.yaml \
  --workflow-id block_card_placeholder \
  --repeats 3