Incident response

When an Incident is declared, we follow a specific process in order to ensure that it is resolved quickly. This section describes our incident response process, major roles and what to expect after an incident has been mitigated. We continually refine this process, taking lessons from our own experience and other experts. [1][2][3][4].

In Beta!

We are currently working out our Incident Response process. The content on this page might change over time, and we welcome suggested changes and pull requests!

• Things to know before an incident
  • What is an incident
  • When is the incident response process triggered
  • The incident team
    • Roles and responsibilities
    • Accountability
• Incident response process
  • Incident sources
  • Steps
    • 1. Validate that we are dealing with an outage
    • 2. Officially mark the beginning of the incident
    • 3. Try resolving the issue
    • 4. Get all hands on deck
    • 5. Communicate our status every few hours
    • 6. Make sure the incident is resolved
    • 7. Communicate when the incident is resolved
    • 8. Take follow-up actions
  • Handing off Incident Responder status
• After the incident
  • 1. Create post-incident action items
    • Responsibility
  • 2. Create an Incident Report
    • Responsibility
    • Steps
    • Tips for writing an incident timeline
  • Key terms

---

[1]

The PagerDuty Incident Response Guide describes a similar process with different roles.

[2]

The Google SRE Incident response guide has a wealth of information about incident response and distributed SRE teams.

[3]

This ACM blog post describes the complexity of coordinating across a team of distributed responders during an incident, and notes a places where Incident Commander roles may actually hinder responsiveness. It is a good lesson in the complexity of incidents with distributed teams!

[4]

The WikiMedia Clinic Duty process also inspired our process here, and is a great overall workflow around distributed SRE.