Production / On-call runbook
Incident Response
Protect the user journey first, contain before diagnosing, and recover in stages. Severity is decided by impact, never by guesses about provider internals.
First decision
Make the severity call
Severity is set by what users cannot do — never by assumptions about a provider’s internals. Choose the level, then follow the response it prescribes.
- Impact
- A journey is degraded, or one cohort is broken while others work.
- Response
- Page the on-call engineer. Freeze releases for the affected surface.
- Communication
- Update every hour.
Runbook
Work the timeline
Turn off optional memory calls, lower concurrency, and stop automated retries from amplifying load.
Do not raise timeouts or retry limits to “push through”.
Escalation
Assemble a support-ready evidence pack
When you escalate through your support channel, send facts rather than raw user data. This is usually enough to start an investigation without a second round of questions.
impact_started 2026-08-01T14:20Z // UTC affected_scope one project, web release web-2026.08.1 symptom query latency p95 2.4s, 429 share 11% requests aggregate counts by status class request_ids small sample from response headers containment optional recall disabled, concurrency halved next_update 2026-08-01T15:00Z
Keep end-user content out of the pack. Identifiers, aggregates, and timestamps are what make an incident reproducible.