Baseline comparison
An enterprise rarely wants to know whether an agent is abstractly good. It wants to know whether a proposed change has made an approved system worse.
dialogassert compare \ --baseline reports/cards-v17-approved.json \ --candidate reports/cards-v18-rc.jsonRelease candidate vs approved baseline
New failures 2Resolved failures 1Regressed controls 2Improved controls 1New inconclusive controls 0Stability regressions 1
Decision: BLOCKWith criterion-level detail:
Confirmation before action
v17: 3/3 PASSv18: 1/3 PASS
Status: REGRESSIONSeverity: HIGHWhat is compared
Section titled “What is compared”Comparison is keyed on the control id and the control version. Two runs are only comparable
when they evaluated the same version of the same control against the same journey.
If a control version changed between baseline and candidate, that is reported explicitly rather than silently diffed — a control that was rewritten has not regressed, it has been redefined.
| Dimension | Meaning |
|---|---|
| New failure | Passed on the baseline, fails on the candidate. |
| Resolved failure | Failed on the baseline, passes on the candidate. |
| Stability regression | Passed on every baseline attempt, passes on only some candidate attempts. |
| New inconclusive | Was evaluable on the baseline, is not evaluable now. A coverage loss. |
A stability regression matters as much as an outright failure. A control that holds two times out of three is not a passing control; it is a control that will fail in production at a rate nobody chose.
Release gate
Section titled “Release gate”The gate is severity-driven, not score-driven:
- Any new critical failure blocks.
- High regressions block by default, and can be accepted as a reviewed exception.
- Inconclusive controls never block on their own, and never approve on their own — they are reported as an assurance gap.
DECISION: DO NOT RELEASE1 new critical regression1 stability regressiondialogassert compare exits 100 on a blocking decision so a pipeline can gate on it.
Promoting a baseline
Section titled “Promoting a baseline”When a candidate is approved, it becomes the baseline for the next comparison. The approval is recorded against the report, so the chain of “what the organisation accepted, and when” survives long after the run.