Incident Response Playbooks for Small Teams: What to Do When Things Break

When the Pager Goes Off at 2 AM: Why Incident Response Needs a Playbook Before the Incident

Small teams face an uncomfortable truth about incident response: when something breaks badly, the people who know the system best are also the people who are most panicked. Adrenaline degrades decision-making. Without a documented playbook, you end up with five engineers in a Slack channel each doing something different, someone accidentally restarting a healthy service, and nobody updating the status page for 45 minutes. This guide gives small teams a practical incident response framework — not the NIST 800-61 theoretical version, but the one that actually works when you’re three people trying to restore a production database at midnight.

The Core Problem With Ad-Hoc Incident Response

Most small teams handle incidents the same way: someone notices something is broken, posts in Slack, people start investigating, and eventually the problem gets fixed. This works — until it doesn’t. The scenarios where ad-hoc response fails catastrophically include:

• The one person who knows the system is unavailable
• Multiple people make conflicting changes simultaneously
• The incident spans multiple services and no one owns coordination
• The same type of incident has happened three times because there was no post-incident review

A playbook does not replace expertise. It creates a scaffold that lets experienced people apply their expertise under stress, while also giving less experienced team members a role that does not require deep system knowledge — like managing communications or logging the timeline.

Incident Severity Levels: Keep It Simple

Before writing any playbooks, define your severity levels. Four levels is usually the right number for a small team. The exact thresholds matter less than writing them down and using them consistently.

SEV-1: Production is completely down or data is being corrupted. All hands. Wake people up.
SEV-2: Major functionality is broken or performance is severely degraded. On-call handles it, escalate if no progress in 30 minutes.
SEV-3: Non-critical feature broken or significant performance issue for a subset of users. Fix during business hours.
SEV-4: Minor issue, cosmetic problem, or single-user report. Log it, prioritize it, fix it in the normal workflow.

Write these definitions into your runbook wiki, your on-call documentation, and your alerting rules. When PagerDuty fires a SEV-1 alert, the engineer who gets paged should not have to think about whether to call their manager — the severity level already tells them.

Roles in an Incident

Even a three-person team needs role clarity during an incident. The same person should not be simultaneously diagnosing the problem, making changes, and updating customers. Define these roles explicitly:

Incident Commander (IC)

The IC owns the incident process, not the technical solution. Their job is to coordinate, not to fix. They declare severity, assign roles, set investigation timeboxes, make escalation decisions, and ensure the status page is updated. Critically, the IC should be the person in the room who is least deep in the technical details — this is counterintuitive but important. Technical depth creates tunnel vision. The IC needs system-level visibility.

Technical Lead

The technical lead owns the diagnosis and remediation. They drive the investigation, propose solutions, and execute (or supervise) changes to production. During a SEV-1, they should not be writing status page updates or answering customer DMs — that cognitive load belongs to the IC.

Scribe

The scribe maintains a real-time timeline of the incident. Every observation, every change made to production, every hypothesis tested goes into the log with a timestamp. This role sounds unimportant until you’re trying to write a post-incident review and realize nobody remembers the sequence of events. The scribe’s log is also invaluable if the incident runs long and you need to hand off to a fresh team.

# Example incident log format (keep this in a shared doc or incident channel)

## Incident #2026-03-15 — Database Connection Exhaustion
**SEV-1 declared:** 02:14 UTC
**IC:** Sarah
**Tech Lead:** Marcus
**Scribe:** Dev (async, logging from alerts)

### Timeline
02:14 - PagerDuty fires: "Database connection pool exhausted" on prod-db-01
02:15 - Sarah declares SEV-1, creates #inc-2026-03-15 channel
02:16 - Marcus joins. Status page updated: "Investigating elevated error rates"
02:19 - Marcus: `SHOW PROCESSLIST` shows 487 connections, max is 500. 
         Most are in "Sleep" state, held by app-server-03
02:22 - Hypothesis: app-server-03 has a connection leak. 
         Check deploy log — new version deployed at 01:55 UTC
02:24 - Marcus: restarted app-server-03 to drain its connections
02:26 - Connection count drops to 180. Error rate normalizes.
02:28 - Status page updated: "Issue identified and resolved. Monitoring."
02:45 - No recurrence. SEV-1 downgraded to SEV-3. Post-mortem scheduled.

Playbook Structure: What Every Runbook Needs

Each playbook should cover a specific failure scenario. The goal is not to document every possible cause — it’s to document the diagnostic decision tree for the most common causes, and the remediation steps for each.

Required Sections

A good runbook has six sections. Keep each one concise. A runbook that takes 20 minutes to read is not useful during an incident.

1. Trigger Conditions: What alert or symptom leads someone to this runbook? Be specific. “High error rate” is not specific enough. “5xx error rate above 1% for more than 3 minutes on the API service” is a trigger condition.

2. Severity Assessment: Questions to ask to determine the right severity level. Include specific metrics thresholds.

3. Diagnostic Steps: A structured set of commands or checks to run in a logical order. Include the actual commands — not “check the logs” but the exact log query.

# Diagnostic commands for high 5xx rate on API service

# 1. Check current error rate by endpoint
kubectl logs -n production -l app=api-service --since=10m | \
  grep '"status":5' | \
  jq -r '.path' | \
  sort | uniq -c | sort -rn | head -20

# 2. Check recent deployments
kubectl rollout history deployment/api-service -n production

# 3. Check database connection health
kubectl exec -n production deploy/api-service -- \
  curl -s localhost:8080/health/db | jq .

# 4. Check upstream dependencies
kubectl exec -n production deploy/api-service -- \
  curl -s localhost:8080/health/dependencies | jq .

# 5. Check recent error patterns in APM
# Navigate to: https://apm.internal/services/api-service/errors?timeRange=15m

4. Remediation Steps: Specific actions for each identified cause, with rollback steps for each action. Never document a remediation action without also documenting how to undo it.

5. Escalation Criteria: When to escalate, who to call, and how. Include phone numbers, not just Slack handles — Slack might be down.

6. Post-Incident Actions: What follow-up actions are required after this type of incident. Link to the post-mortem template.

The Playbooks Your Team Needs First

Start with the five scenarios that have happened most frequently or would cause the most damage. Common first playbooks for web application teams:

• Database unavailable / connection exhaustion
• Deployment rollback procedure
• SSL certificate expiry (yes, it still happens)
• DDoS or traffic spike causing service degradation
• Data breach suspicion or confirmed unauthorized access

The breach playbook deserves special attention. This is the scenario where having a documented procedure matters most, because it involves legal obligations (breach notification laws), coordination with non-technical stakeholders, and preservation of evidence that can be destroyed by well-intentioned remediation actions.

# Breach response checklist — First 30 minutes
# DO NOT wipe or restart any systems until forensics decision is made

[ ] Declare severity and create incident channel
[ ] Notify: Engineering lead, Legal/Compliance, CEO if data confirmed affected
[ ] Isolate (not terminate) suspected compromised instances:
    aws ec2 create-security-group --group-name ISOLATED-INCIDENT-2026-03-XX
    aws ec2 modify-instance-attribute --instance-id i-XXXX \
      --groups sg-ISOLATED-ID
[ ] Capture memory dump if possible (evidence preservation)
[ ] Take EBS snapshot of affected volumes before any changes
[ ] Enable enhanced logging if not already active:
    aws cloudtrail start-logging --name prod-trail
[ ] Check CloudTrail for the 24 hours preceding discovery
[ ] Do NOT notify customers or regulators yet — confirm scope first
[ ] Document every action in incident log with timestamps

Communication Templates

During a SEV-1, writing a status page update from scratch costs precious cognitive load. Prepare templates in advance:

# Template: Initial Status Page Update (SEV-1)
Title: Investigating [service name] issues
Body: We are aware of issues affecting [feature/service]. 
Our team is actively investigating. We will provide an update in [15/30] minutes.
Status: Investigating

# Template: Update with Progress
Title: [Service name] — identified cause, implementing fix
Body: We have identified the root cause as [brief description]. 
We are implementing a fix and expect resolution within [timeframe]. 
Affected users: [scope]. [X]% of requests are currently failing.
Status: Identified

# Template: Resolution
Title: [Service name] — resolved
Body: The issue affecting [service] has been resolved as of [time UTC]. 
[Brief description of what was fixed]. We will publish a post-incident 
review within 48 hours. We apologize for the disruption.
Status: Resolved

Post-Incident Review: The Part Most Teams Skip

The incident timeline ends when the service is restored. The incident process ends when you have taken steps to prevent a recurrence or reduce its impact. A post-incident review is not a blame session — it is a structured analysis of why the system (technical and human) allowed the incident to happen.

A blameless post-mortem asks: “What in our systems, processes, or tooling created conditions where this failure was possible?” not “Who made the mistake?” The distinction matters enormously for team culture. Engineers who fear blame will hide information during incidents. Engineers who trust the post-mortem process will share everything they know.

Schedule the post-mortem within 48 hours while memory is fresh. Assign action items with due dates and owners. Track whether those action items get completed — the most common failure in incident management is identifying the right fix and then never implementing it.

Tooling Recommendations

You do not need expensive tools to run effective incident response. A small team can operate well with:

• PagerDuty or Opsgenie: On-call scheduling, alert routing, and escalation policies
• Statuspage.io or Instatus: Customer-facing status communication
• A dedicated incident Slack channel pattern: #inc-YYYY-MM-DD-brief-description
• Confluence, Notion, or a plain Git repo: Runbook storage — the format matters less than the practice of keeping it current
• Grafana + AlertManager: Metrics-based alerting that fires your PagerDuty integration

The highest-leverage investment is not a tool — it is a practice: running a 30-minute tabletop exercise once per quarter where you walk through a hypothetical incident using your playbooks. You will find gaps, outdated commands, and missing escalation contacts. Better to find them during a tabletop than at 2 AM.

Conclusion

A small team that writes and maintains incident playbooks will outperform a larger team that relies on tribal knowledge — every time. The playbook does not need to be perfect. It needs to exist, be findable during an incident, and be updated after every incident that reveals a gap. Start with one playbook for your most common failure mode, drill it in a tabletop, and build from there.

The goal is not to eliminate incidents. Systems fail. The goal is to fail with less chaos, recover faster, and build institutional knowledge that makes the next incident less severe than the last.

Key Takeaways

• Define four severity levels with clear thresholds. Severity determines who gets paged and what escalation path to follow.
• During incidents, separate the roles of Incident Commander, Technical Lead, and Scribe — even on a small team.
• Every playbook must include exact diagnostic commands, rollback steps, escalation criteria, and post-incident follow-up actions.
• Breach response playbooks require special care: preserve evidence before remediating, and isolate rather than terminate compromised systems.
• The post-mortem process ends when action items are completed, not when the document is written.