Automation risk checklist
> A pre-launch risk checklist for automations that depend on a model. Pair with the automation workflow testing playbook.
Pipeline scope
- Input source (what triggers the automation):
- Model step (which candidate, which snapshot, which region):
- Downstream parser (deterministic or model-assisted):
- Output destination (write target, who reads it):
- Idempotency guarantees per step:
Failure surface
- What happens on a model timeout?
- What happens on a schema validation failure?
- What happens on an unexpected refusal?
- What happens on a downstream parser failure?
- What is the maximum acceptable retry count?
Observability
- Are inputs, outputs, and errors logged with correlation IDs?
- Is the canary suite scheduled and alerting wired?
- Is the catalogue's reverification queue subscribed for the model under test?
- Where do operators see drift first?
Guardrails
- Are PII / sensitive data flows minimised on the input side?
- Are outputs reviewed by a human before they go to a customer-facing surface?
- Are there hard rate limits, hard cost limits, and a kill switch?
- Is there a rollback path if the snapshot rotates?
Approval
- Reviewer sign-offs required for launch:
- Reviewer sign-offs required for snapshot promotion:
- Reviewer sign-offs required for prompt edits: