Incident response is the operational discipline of detecting, diagnosing, mitigating, and communicating about production outages and degradations.
Incident response is the coordinated set of practices for handling production outages, degradations, and security events — from detection through resolution to post-incident review. Google's SRE book codified much of what modern teams practice: severity levels, an incident commander role, communication cadence, and blameless postmortems. The point is not to prevent all incidents (impossible) but to make them shorter, less severe in customer impact, and generative of real improvements.
A typical 4-level scale: SEV1 — full outage or data loss, all-hands, immediate communication to customers. SEV2 — major functionality broken for a significant customer segment; wake people up but not the whole team. SEV3 — degraded but functional; handle in business hours. SEV4 — minor bug or single-customer issue; normal work queue. The critical part isn't the scale but the shared calibration: every engineer should be able to look at a bug and agree on the SEV. Recalibrate quarterly by reviewing recent incidents against the definitions.
During a real incident, one person is the incident commander (IC). The IC doesn't do the debugging — they coordinate. They open the incident channel, assign roles (investigator, communicator, scribe), keep a running timeline, decide when to escalate, and make the mitigation-vs-root-cause tradeoff calls. The debugging engineer needs cognitive space to think; the IC absorbs the interruptions. Even a 3-engineer team benefits from designating an IC during incidents — the alternative is everyone trying to debug and coordinate simultaneously, which produces neither.
The single most common incident-response mistake is trying to find and fix the root cause during the incident. Correct order: stop the bleeding (revert, disable feature flag, failover, scale up), verify customer impact is reduced, then investigate root cause on a normal timeline. 'Rolling back a deploy without understanding why it broke' is almost always the right call during an incident; understanding can wait an hour, customer impact can't. This is where blameless culture matters — engineers who fear being blamed for reverts will delay them.
Internal: incident channel updates every 15-30 minutes for SEV1/SEV2, even if the update is 'still investigating, no ETA.' Silence during an active incident generates more noise (people pinging for updates) than proactive updates. External: customer-facing status page updates within 15 minutes of detection for anything customer-visible; initial post can be 'we're investigating,' but the acknowledgement is what matters. Overcommunicating during incidents is nearly impossible; undercommunicating is the default failure mode.
Every SEV1 and SEV2 gets a written postmortem within 5 business days. The document should be blameless (focus on systems and processes, not individuals), specific about the timeline (with timestamps from the incident channel), and action-item-generating. Action items should have owners and deadlines; a postmortem with no owned actions is a document, not a process improvement. Share postmortems widely internally — the learning value only compounds if the whole engineering org sees them.
Investor directory · Fundraising library · Articles A–Z · Company funding database