Model regression testing

When to use this

• Your provider rotates model snapshots without changing the API model name.
• Your automation runs unattended for hours or days at a time.
• You have an evidence brief from a previous selection round and want to keep it honest.

Test setup

• Freeze a canary suite of 10–25 prompts that exercises the behaviours your application depends on.
• Store reference outputs from the snapshot you originally selected — keep them out of source control if they contain real data.
• Schedule the canary suite to run on a regular cadence (every release, every snapshot bump, or every 24 hours).
• Wire alerting that fires when canary pass rate drops below your floor.

Minimum test set

• 10–25 representative prompts you also ran in the original selection.
• At least one prompt per behaviour your application depends on (recall, format, refusal, modality).

Prompt variants

Vary one dimension at a time so you can attribute behaviour.

• Same prompt, same parameters, two snapshots — reference vs current.
• Same prompt with and without retrieved context to separate model drift from retrieval drift.

Observations to record

Record observations, not scores. The evidence brief stays auditable.

• Pass / fail per prompt against reference output (or rubric, if exact match is not appropriate).
• Latency drift over time.
• Cost drift over time (input/output token shifts can move cost without changing pricing).
• Lifecycle field changes against /reverification.

Failure modes to watch

• Pass rate drops sharply on one prompt category — likely targeted regression.
• Pass rate drops gradually across categories — likely a generic snapshot drift.
• Cost climbs while pass rate stays flat — verbose-output drift.
• The provider deprecates the snapshot mid-run — confirm with the catalogue's lifecycle field.

Stop conditions

Knowing when to stop is part of the test.

• Stop and re-evaluate if pass rate drops below your floor on any release.
• Stop and re-verify if the catalogue's lifecycle field shifts to deprecated for the snapshot under test.

Outputs

• A Markdown evidence delta listing the prompts that regressed.
• An updated decision brief if the regression triggers a re-selection.

Weak test vs stronger test

Observation rubric

Observations, not scores. Each row names what to look at and what to record — there is no aggregate number.

Record this in your brief

Brief content that keeps the playbook output reusable for a reviewer.

• Embed the canary set + reference rubric so the regression record is self-contained.
• Pair drift observations with the snapshot ID under test.
• List the pass-rate floor and the action taken on breaches.
• Cross-reference any failing canary with /reverification before treating it as a vendor allegation.