# Incident Response Procedures

Structured procedures for responding to infrastructure incidents at different severity levels.

## Incident Severity Levels

| Level | Definition | Response Time | Examples |
| ------| - ----------| - --| - ---------- |
| **Critical** | Complete service outage, data loss risk | < 15 minutes | All services down, database corruption |
| **High** | Partial outage, severe performance degradation | < 1 hour | One region down, 50% latency increase |
| **Medium** | Limited functionality, moderate impact | < 4 hours | Feature broken, 10% error rate |
| **Low** | Minor issues, minimal user impact | < 8 hours | UI glitch, non-critical metric alert |

## Critical Incident Response

### Initial Triage (First 5 Minutes)

```nushell
def triage-critical-incident [--alert: record] {
    # 1. Acknowledge incident
    let incident_id = (provisioning incident create \
        --severity critical \
        --alert-id $alert.id)

    print $"Incident created: ($incident_id)"

    # 2. Page on-call engineer immediately
    provisioning pagerduty trigger \
        --incident-id $incident_id \
        --severity critical \
        --service $alert.service

    # 3. Open war room
    provisioning war-room create \
        --incident-id $incident_id \
        --channel "slack:#critical-incidents"

    # 4. Gather initial context
    let context = {
        service: $alert.service,
        error_rate: (provisioning metrics get --metric error_rate),
        latency_p99: (provisioning metrics get --metric latency_p99),
        affected_regions: (provisioning analyze --detect-regions),
        recent_deployments: (provisioning deployments list --limit 5)
    }

    # 5. Attempt auto-remediation
    print "Attempting automatic remediation..."
    let auto_result = (provisioning auto-remediate --incident-id $incident_id)

    if $auto_result.success {
        print "✓ Auto-remediation successful"
        return 0
    }

    # 6. Manual investigation needed
    print "⚠️  Auto-remediation failed, human investigation required"
    return 1
}
```

### Investigation (5-30 Minutes)

```nushell
def investigate-incident [--incident-id: string] {
    print $"Investigating incident: ($incident-id)..."

    # 1. Collect logs from affected period
    let logs = (
        provisioning logs query \
            --filter "service=api-backend AND level=ERROR" \
            --start-time "15 minutes ago" \
            --limit 1000
    )

    # 2. Check recent changes
    let changes = (
        provisioning changes list \
            --limit 10 \
            --before "15 minutes ago"
    )

    print "Recent changes:"
    $changes | each { | change |
        print $"  - ($change.timestamp): ($change.description)"
    }

    # 3. Analyze metrics around incident time
    let metrics = (
        provisioning metrics timeline \
            --window "1h" \
            --metrics ["cpu", "memory", "network", "error_rate", "latency"]
    )

    # 4. Check dependencies
    let dependency_health = (
        provisioning dependencies health-check \
            --service "api-backend"
    )

    # 5. Generate initial findings
    let findings = {
        root_cause_hypothesis: "Database connection pool exhaustion",
        affected_systems: ["api-backend", "web-app"],
        data_loss_risk: false,
        estimated_impact: "10% of users",
        recommended_action: "Restart connection pool service"
    }

    print ""
    print "Investigation Findings:"
    print $"Root cause hypothesis: ($findings.root_cause_hypothesis)"
    print $"Recommended action: ($findings.recommended_action)"

    return $findings
}
```

### Mitigation (During Incident)

```nushell
def mitigate-incident [--action: string] {
    match $action {
        "scale-up" => {
            print "Scaling up affected service..."
            provisioning scale --service api-backend --target-replicas 20
            sleep 30s
        },

        "failover" => {
            print "Initiating failover..."
            provisioning failover \
                --service api-backend \
                --target-region us-west-2
            sleep 60s
        },

        "rollback" => {
            print "Rolling back recent deployment..."
            provisioning deployment rollback --to-previous
            sleep 60s
        },

        "drain-traffic" => {
            print "Draining traffic from problematic service..."
            provisioning lb drain --service api-backend --duration 5m
        },

        _ => {
            print "Unknown mitigation action"
            return 1
        }
    }

    # Verify mitigation worked
    let health = (provisioning health check api-backend)
    if $health.status == "healthy" {
        print "✓ Mitigation successful"
        return 0
    } else {
        print "✗ Mitigation did not resolve issue"
        return 1
    }
}
```

## Communication Template

### Initial Notification (< 5 minutes)

```text
INCIDENT ALERT: Critical - API Service Down

Status: INVESTIGATING
Severity: CRITICAL
Start Time: 2025-01-16 14:23 UTC
Duration: < 5 minutes

Impact:
- Service: API Backend
- Affected users: ~10% (customers in us-east-1)
- Error rate: 95% (normal: 0.1%)

Initial findings:
- Investigating database connection issues
- Automatic remediation attempted without success

Next update in 5 minutes
War room:  [https://zoom.us/...](https://zoom.us/...)
```

### Status Updates (Every 15 Minutes)

```text
INCIDENT UPDATE #2: Investigation Ongoing

Status: INVESTIGATING
Severity: CRITICAL
Duration: 15 minutes
Estimated Time to Resolution: 10 minutes

Current findings:
- Root cause: Database connection pool exhaustion
- 5 minute spike in connection requests
- Likely triggered by recent deployment

Actions underway:
- Scaling database connection pool
- Analyzing recent code changes
- Preparing rollback if needed

Next update in 5 minutes
```

### Resolution Notification

```text
INCIDENT RESOLVED

Status: RESOLVED
Severity: CRITICAL
Total Duration: 23 minutes
Detection Time: 3 minutes
MTTR: 20 minutes

Root cause: Database connection leak in recent API deployment

Remediation: Rolled back API deployment to previous stable version

Impact:
- 23 minutes of degraded service
- ~10% of users affected
- No data loss

Post-incident: Full RCA scheduled for 2025-01-17 10:00 UTC

Apologies for the disruption. Thank you for your patience.
```

## Post-Incident (After Resolution)

### Root Cause Analysis

```nushell
def conduct-rca [--incident-id: string] {
    print "Conducting Root Cause Analysis..."

    # 1. Detailed timeline
    let timeline = (
        provisioning incident timeline --incident-id $incident-id
    )

    # 2. What happened
    let what_happened = "API deployment at 14:20 UTC introduced database connection leak"

    # 3. Why it happened
    let why = {
        code_review: "Code change not caught in review",
        testing: "Load testing did not expose connection leak",
        deployment: "Canary deployment percentage too high (50%)"
    }

    # 4. Contributing factors
    let contributing_factors = [
        "No connection pool monitoring",
        "No automated connection leak detection",
        "Insufficient load in test environment"
    ]

    # 5. Preventive measures
    let preventive = [
        "Add connection pool metrics to monitoring",
        "Implement connection leak detector",
        "Increase canary percentage gradually (5% → 25% → 100%)",
        "Mandatory load testing for all API deployments"
    ]

    # 6. Generate RCA report
    provisioning rca report create \
        --incident-id $incident-id \
        --what $what_happened \
        --why ($why | to json) \
        --contributing-factors $contributing_factors \
        --preventive-measures $preventive
}
```

### Postmortem Meeting

- **When**: Within 48 hours of resolution
- **Duration**: 60 minutes
- **Attendees**: Engineers, on-calls, managers
- **Agenda**:
  1. Timeline review (5 min)
  2. Root cause discussion (10 min)
  3. What went well (5 min)
  4. What could improve (10 min)
  5. Action items (10 min)
  6. Follow-up (5 min)

## Escalation Procedures

### Escalation Criteria

```nickel
{
  escalation = [
    {
      condition = "MTTR exceeds 15 minutes",
      escalate_to = "Engineering Manager",
      action = "Request additional resources"
    },
    {
      condition = "Data loss confirmed",
      escalate_to = "VP Engineering + Legal",
      action = "Initiate incident response protocol"
    },
    {
      condition = "Incident ongoing for 30 minutes",
      escalate_to = "CTO",
      action = "Executive notification"
    },
    {
      condition = "Multiple critical incidents in 24h",
      escalate_to = "CEO + Board",
      action = "Emergency infrastructure review"
    }
  ]
}
```

## Incident Trends Analysis

```nushell
def analyze-incident-trends [--period-days: int = 30] {
    let incidents = (
        provisioning incidents list \
            --after $"($period-days) days ago"
    )

    let by_service = ($incidents | group-by service | each { | group |
        {
            service: $group.0.service,
            count: ($group | length),
            avg_mttr: (($group | map mttr | math avg) / 1000)
        }
    })

    let critical_count = ($incidents | where severity == "critical" | length)

    print $"Incidents last ($period-days) days: ($incidents | length)"
    print $"Critical incidents: ($critical_count)"
    print ""
    print "By service:"
    $by_service | sort-by count --reverse | table
}
```

## See Also

- [Automated Incident Response](../architecture/adr-010-incident-response-automation.md)
- [Monitoring Guide](./monitoring.md)
- [SLO and Error Budgets](../architecture/adr-009-slo-error-budgets.md)
