# Incident Response Runbooks

**Step-by-step procedures for responding to common platform incidents**

**Version**: 1.0.0
**Last Updated**: 2026-01-05
**Target Audience**: On-Call Engineers, Platform Operators
**Status**: Production Ready

---

## Quick Navigation

- [Service Down](#service-down)
- [High Error Rate](#high-error-rate)
- [Memory Exhaustion](#memory-exhaustion)
- [Disk Space Critical](#disk-space-critical)
- [Database Connection Failure](#database-connection-failure)
- [Queue Backlog](#queue-backlog)
- [Authentication Failure](#authentication-failure)
- [Data Corruption](#data-corruption)
- [Network Latency](#network-latency)
- [Full Cluster Failure](#full-cluster-failure)

---

## Service Down

**Alert**: `ServiceDown` (severity: critical)

**Impact**: Service unavailable, users cannot access functionality

**Detection**: Service unreachable for 5+ minutes

### Phase 1: Immediate Response (0-2 minutes)

**1.1 Acknowledge alert**
```bash
# In PagerDuty/Slack
# React with 🚨 to acknowledge you're investigating
```

**1.2 Verify service is actually down**
```bash
# Check if service is running
pgrep -a service-name
ps aux | grep service-name | grep -v grep

# Check if port is listening
ss -tlnp | grep PORT_NUMBER
lsof -i :PORT_NUMBER

# Test HTTP endpoint
curl -v http://localhost:PORT/health
curl -v http://SERVICE_HOSTNAME:PORT/health
```

**1.3 Check recent events**
```bash
# View last 20 log lines
tail -20 /var/log/provisioning/service-name.log

# Check for crash
grep -i "panic\|fatal\|error" /var/log/provisioning/service-name.log | tail -10

# Check systemd status (if using systemd)
systemctl status provisioning-service-name
journalctl -u provisioning-service-name -n 50
```

### Phase 2: Diagnosis (2-5 minutes)

**2.1 Determine if service crashed or hung**
```bash
# Check process exists
pgrep service-name && echo "Process running" || echo "Process crashed"

# If running but not responding, check for deadlock/hang
strace -p PID 2>&1 | head -20

# Check CPU usage
top -p PID -n 1
ps aux | grep service-name | grep -v grep | awk '{print $3, $6}'
```

**2.2 Check dependencies**
```bash
# Test database connectivity (if applicable)
curl http://surrealdb:8000/health
etcdctl --endpoints=http://etcd:2379 endpoint health

# Test upstream service connectivity
curl http://vault:8200/health
curl http://registry:8081/health

# Test DNS resolution
nslookup service-name.internal
dig service-name.internal
```

**2.3 Review configuration**
```bash
# Check schema exists and is valid
nickel typecheck provisioning/schemas/platform/schemas/service.ncl

# Check defaults are valid
nickel typecheck provisioning/schemas/platform/defaults/service-defaults.ncl

# Check environment variables
env | grep -E "SERVICE_|VAULT_|REGISTRY_"
```

### Phase 3: Remediation (5-10 minutes)

**3.1 Attempt restart**
```bash
# Stop service gracefully
pkill -SIGTERM service-name
sleep 5

# Verify it stopped
pgrep service-name || echo "Stopped successfully"

# Clear any lock files
rm -f /var/run/service-name.lock
rm -f /tmp/service-name.*

# Start service with verbose logging
RUST_LOG=debug cargo run --release -p service-name 2>&1 | tee /tmp/service-startup.log &

# Wait for startup
sleep 10

# Verify startup
curl http://localhost:PORT/health
```

**3.2 If restart fails**
```bash
# Check full error output
RUST_LOG=trace cargo run -p service-name 2>&1 | head -100

# Check if port is in use by another process
lsof -i :PORT
netstat -tlnp | grep PORT

# Kill process using port
pkill -9 -f "cargo run.*service-name"

# Try on different port (temporary)
export SERVICE_PORT=9999
cargo run --release -p service-name &
curl http://localhost:9999/health
```

**3.3 If still failing**
```bash
# Restore schemas from backup
git checkout HEAD~1 -- provisioning/schemas/platform/schemas/service.ncl
git checkout HEAD~1 -- provisioning/schemas/platform/defaults/service-defaults.ncl

# Re-generate runtime config
./provisioning/.typedialog/platform/scripts/generate-configs.nu service solo

# Try with solo mode (if available)
export SERVICE_MODE=solo
cargo run --release -p service-name &
sleep 5
curl http://localhost:PORT/health

# If that works, mode-specific defaults are broken
# Escalate to team for debugging
```

### Phase 4: Verification (10-15 minutes)

**4.1 Confirm service is healthy**
```bash
# Health check
curl -s http://localhost:PORT/health | jq .

# Test basic operation
curl -X GET http://localhost:PORT/api/v1/status

# Check metrics
curl -s http://localhost:PORT/metrics | head -20

# Monitor logs for errors
tail -f /var/log/provisioning/service-name.log
```

**4.2 Check downstream services**
```bash
# If this service is dependency for others, verify they recovered
for port in 8200 8081 8083 8082 9090 8080; do
  curl -s http://localhost:$port/health && echo "✓ Port $port" || echo "✗ Port $port"
done

# Check orchestrator is accepting tasks
curl -s http://localhost:9090/api/v1/status | jq .
```

**4.3 Update incident status**
```bash
# In PagerDuty/Slack
# Post: "✓ Service recovered at HH:MM, investigating root cause"
# Start war room if needed
```

### Phase 5: Root Cause Analysis (During + After Incident)

**5.1 Collect data**
```bash
# Full logs from incident window
grep "2026-01-05 14:" /var/log/provisioning/service-name.log > /tmp/incident-logs.txt

# Service metrics during incident
curl -s 'http://localhost:9090/api/v1/query_range?query=up{job="service"}' | jq .

# Dependent services status
curl -s http://localhost:9090/api/v1/services | jq .

# System resources at time of incident
uptime
free -h
df -h
```

**5.2 Investigate common causes**
```bash
# OOM (Out of Memory)
dmesg | grep -i "killed\|oom" | tail -5
grep "memory" /var/log/provisioning/service-name.log

# Disk full
df -h | grep -E "9[0-9]%|100%"
du -sh /var/lib/provisioning/*

# File descriptor exhaustion
lsof -p PID | wc -l
cat /proc/sys/fs/file-max

# Goroutine/thread leak
curl -s http://localhost:PORT/debug/pprof/goroutine

# Dependency timeout
grep -i "timeout\|connection reset" /var/log/provisioning/service-name.log
```

### Phase 6: Post-Incident

**6.1 Document incident**
```bash
# Create incident summary
cat > /tmp/incident-summary.md << EOF
## Incident: SERVICE-NAME Down
- Start time: 2026-01-05 14:30 UTC
- Duration: 15 minutes
- Root cause: [TO BE DETERMINED]
- Impact: [SERVICE FUNCTIONALITY UNAVAILABLE]
- Resolution: [RESTART]
- Prevention: [TBD]
EOF
```

**6.2 Implement prevention**
```bash
# Add monitoring/alerting if missing
# Implement auto-restart if applicable
# Update runbooks based on findings
# Schedule follow-up action items
```

---

## High Error Rate

**Alert**: `HighErrorRate` (severity: warning)

**Impact**: Service returning 5xx errors, degraded functionality

**Detection**: Error rate above 5% for 5+ minutes

### Phase 1: Immediate Response (0-2 minutes)

**1.1 Acknowledge alert and gather metrics**
```bash
# Get current error rate
curl -s 'http://localhost:9090/api/v1/query?query=rate(http_requests_total{status=~"5.."}[5m])' | jq .

# Get affected endpoints
curl -s 'http://localhost:9090/api/v1/query?query=rate(http_requests_total{status="500"}[5m]) by (handler)' | jq .

# Get error rate per service
curl -s 'http://localhost:9090/api/v1/query?query=rate(http_requests_total{status=~"5.."}[5m]) by (job)' | jq .
```

**1.2 Check service health**
```bash
# Check all services
for port in 8200 8081 8083 8082 9090 8080; do
  echo "=== Port $port ==="
  curl -s http://localhost:$port/health | jq .status 2>/dev/null || echo "DOWN"
done

# Get recent logs
tail -50 /var/log/provisioning/affected-service.log | grep -i error
```

### Phase 2: Diagnosis (2-5 minutes)

**2.1 Identify error pattern**
```bash
# Get detailed error messages
curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.alertname=="HighErrorRate")'

# Check service logs for panic
grep "panic\|fatal\|ERROR" /var/log/provisioning/affected-service.log | tail -20

# Get stack traces if available
tail -100 /var/log/provisioning/affected-service.log | grep -A 20 "panic"
```

**2.2 Check dependencies**
```bash
# Test database
curl http://surrealdb:8000/health
surreal sql --endpoint http://surrealdb:8000 --username root --password root --query "SELECT count(*) FROM services"

# Test cache
curl http://cache:6379/health

# Test external services
curl http://vault:8200/health
curl http://registry:8081/health
```

**2.3 Check for resource issues**
```bash
# Memory
free -h
ps aux | grep service | awk '{print $6}' | head -5

# CPU
top -bn1 | head -15

# Disk
df -h

# Network connections
ss -s
netstat -an | grep ESTABLISHED | wc -l
```

### Phase 3: Remediation (5-10 minutes)

**3.1 If dependency is unhealthy**
```bash
# Restart database
systemctl restart surrealdb

# Verify connectivity restored
curl http://surrealdb:8000/health

# Reload affected service
pkill -SIGTERM affected-service
sleep 2
cargo run --release -p affected-service &
```

**3.2 If resource exhausted**
```bash
# Scale up if possible
export SERVICE_WORKERS=16  # Increase workers
pkill -SIGTERM affected-service
sleep 2
cargo run --release -p affected-service &

# Or reduce load if under DoS
# Contact network team to implement rate limiting
```

**3.3 If code issue (deployment-related)**
```bash
# Rollback to previous version
git checkout HEAD~1
cargo build --release -p affected-service

# Stop current version
pkill -SIGTERM affected-service

# Start previous version
export SERVICE_VERSION=previous
cargo run --release -p affected-service &

# Verify error rate drops
sleep 30
curl -s 'http://localhost:9090/api/v1/query?query=rate(http_requests_total{status=~"5.."}[5m])' | jq .
```

### Phase 4: Verification (10-15 minutes)

**4.1 Confirm error rate recovered**
```bash
# Check current error rate
curl -s 'http://localhost:9090/api/v1/query?query=rate(http_requests_total{status=~"5.."}[5m])' | jq .

# Should be < 0.05 (5%)

# Check alert status
curl -s http://localhost:9093/api/v1/alerts | jq '.data.alerts[] | select(.labels.alertname=="HighErrorRate")'

# Should be resolved or not present
```

**4.2 Monitor for recurrence**
```bash
# Set up dashboard to watch for 15 minutes
watch -n 10 'curl -s "http://localhost:9090/api/v1/query?query=rate(http_requests_total{status=~\"5..\"}[5m])" | jq .'

# Or use Grafana dashboard
# Navigate to Platform Overview → Error Rate
```

---

## Memory Exhaustion

**Alert**: `HighMemoryUsage` (severity: warning)

**Impact**: Service slowdown, potential crashes, cascading failures

**Detection**: Memory usage > 90% for 5+ minutes

### Phase 1: Immediate Response (0-2 minutes)

**1.1 Verify memory issue**
```bash
# Check system memory
free -h
# Look for very low "available"

# Check service memory
ps aux | grep service-name | grep -v grep | awk '{printf "Service: %d MB\n", $6/1024}'

# Find top memory consumers
ps aux --sort=-%mem | head -10
```

**1.2 Check for memory leak**
```bash
# Monitor memory growth over time
for i in {1..5}; do
  ps aux | grep service-name | grep -v grep | awk '{printf "%s: %d MB\n", $2, $6/1024}'
  sleep 30
done

# If increasing rapidly, memory leak suspected
```

### Phase 2: Diagnosis (2-5 minutes)

**2.1 Identify leak source**
```bash
# Check goroutine count (Rust threads)
curl -s http://localhost:PORT/debug/pprof/goroutine | head -20

# Check open file descriptors
lsof -p PID | wc -l
cat /proc/sys/fs/file-max

# Check memory allocations
curl -s http://localhost:PORT/debug/pprof/heap | head -50
```

**2.2 Check for unbounded growth**
```bash
# Cache size
curl -s http://localhost:PORT/metrics | grep "cache.*bytes"

# Connection pool
curl -s http://localhost:PORT/metrics | grep "connection.*total"

# Queue depth
curl -s http://localhost:PORT/metrics | grep "queue.*depth"
```

### Phase 3: Remediation (5-10 minutes)

**3.1 Immediate: Restart service**
```bash
# Graceful shutdown (allows cleanup)
pkill -SIGTERM service-name
sleep 30

# Verify stopped
pgrep service-name || echo "Stopped"

# Remove any memory-mapped files
rm -f /dev/shm/service-name.*

# Restart with reduced worker count (less memory)
export SERVICE_WORKERS=2
cargo run --release -p service-name &

# Monitor memory
watch -n 5 'ps aux | grep service-name | grep -v grep | awk "{printf \"Memory: %d MB\\n\", \$6/1024}"'
```

**3.2 If memory still grows**
```bash
# Stop service
pkill -9 service-name

# Clear any resource leaks from previous runs
ps aux | grep service-name | grep -v grep | awk '{print $2}' | xargs kill -9

# Check for zombie processes
ps aux | grep defunct

# Review logs for memory-related errors
grep -i "oom\|memory\|alloc" /var/log/provisioning/service-name.log

# Escalate to development team
```

**3.3 Temporary mitigation**
```bash
# Enable memory limits via systemd
cat > /etc/systemd/system/service-name.service.d/memory-limit.conf << EOF
[Service]
MemoryLimit=2G
MemoryMax=2.5G
EOF

systemctl daemon-reload
systemctl restart service-name
```

### Phase 4: Verification (10-15 minutes)

**4.1 Confirm memory stabilized**
```bash
# Monitor for 10 minutes
for i in {1..20}; do
  echo "=== $(date) ==="
  ps aux | grep service-name | grep -v grep | awk '{printf "Memory: %d MB, CPU: %s%%\n", $6/1024, $3}'
  sleep 30
done

# Memory should stabilize, not keep growing
```

**4.2 Post-incident actions**
```bash
# Enable memory profiling
RUST_LOG=debug MALLOC_TRACE=/tmp/mem-trace.log cargo run -p service-name

# Analyze heap dump
heapdump analyze /tmp/heap.bin

# File ticket for development team to investigate
```

---

## Disk Space Critical

**Alert**: `HighDiskUsage` (severity: warning)

**Impact**: Service failures, data corruption, cascading failures

**Detection**: Free disk space < 10%

### Phase 1: Immediate Response (0-2 minutes)

**1.1 Verify disk space issue**
```bash
# Check disk usage
df -h
df -h /var/lib/provisioning

# Show large files
du -sh /var/lib/provisioning/* | sort -rh | head -10

# Find large directories
find /var/lib/provisioning -type d -exec du -sh {} + | sort -rh | head -10
```

**1.2 Identify culprit**
```bash
# Check logs size
du -sh /var/log/provisioning/

# Check database size
du -sh /var/lib/provisioning/surrealdb*
du -sh /var/lib/provisioning/etcd*

# Check temporary files
du -sh /tmp/provisioning-*
du -sh /dev/shm/
```

### Phase 2: Remediation (2-10 minutes)

**2.1 Clean up temporary files**
```bash
# Remove old logs (keep 7 days)
find /var/log/provisioning -name "*.log" -mtime +7 -delete

# Remove old core dumps
find /var/lib/provisioning -name "core.*" -delete

# Clean temporary files
rm -rf /tmp/provisioning-*
rm -rf /tmp/service-*.lock
```

**2.2 Archive old metrics (if Prometheus)**
```bash
# Archive old Prometheus data
cd /var/lib/prometheus
tar -czf /archive/prometheus-old-$(date +%s).tar.gz snapshots/
rm -rf snapshots/

# Reduce retention (temporary)
# Edit prometheus.yml: --storage.tsdb.retention.time=3d
```

**2.3 Rotate logs**
```bash
# Manual log rotation
logrotate -f /etc/logrotate.d/provisioning

# Or manually
gzip /var/log/provisioning/*.log
mv /var/log/provisioning/*.log.gz /archive/
```

**2.4 Check disk usage again**
```bash
# Verify freed space
df -h

# Should now have > 20% free space
# If not, escalate (may need storage expansion)
```

### Phase 3: Permanent Fix

**3.1 Enable automatic log rotation**
```bash
# Create logrotate config
cat > /etc/logrotate.d/provisioning << EOF
/var/log/provisioning/*.log {
    daily
    rotate 7
    compress
    delaycompress
    notifempty
    create 0640 provisioning provisioning
    sharedscripts
    postrotate
        systemctl reload provisioning-services
    endscript
}
EOF

logrotate -f /etc/logrotate.d/provisioning
```

**3.2 Implement metrics retention policy**
```bash
# Update Prometheus config
cat >> /etc/prometheus/prometheus.yml << EOF
global:
  scrape_interval: 30s
  retention: 15d           # Keep 15 days of metrics
  retention_size: 100 GB    # Or max 100 GB
EOF

systemctl restart prometheus
```

**3.3 Plan for growth**
```bash
# Calculate daily growth
du -sh /var/lib/provisioning
# Check again tomorrow

# If still growing fast, may need:
# - Storage expansion
# - Additional nodes
# - Compression enabled
```

---

## Database Connection Failure

**Alert**: `DatabaseConnectionError` (severity: critical)

**Impact**: Services cannot persist/retrieve data, functionality broken

**Detection**: Connection errors > 10 in 5 minutes

### Phase 1: Immediate Response (0-2 minutes)

**1.1 Verify database is running**
```bash
# Check SurrealDB (if multiuser/enterprise)
curl -s http://surrealdb:8000/health
curl -s https://surrealdb:8000/health  # Try HTTPS if HTTP fails

# Check Etcd (if enterprise)
etcdctl --endpoints=http://etcd:2379 endpoint health

# Check database process
pgrep surrealdb || echo "SurrealDB not running"
pgrep etcd || echo "Etcd not running"
```

**1.2 Test connectivity**
```bash
# From affected service
curl -v http://surrealdb:8000/health
telnet surrealdb 8000

# Check DNS
nslookup surrealdb
dig surrealdb

# Check network
ping -c 3 surrealdb
traceroute surrealdb
```

**1.3 Get error details**
```bash
# Check affected service logs
tail -50 /var/log/provisioning/affected-service.log | grep -i "database\|connection"

# Get specific error
grep "connection refused\|connection reset\|timeout" /var/log/provisioning/affected-service.log
```

### Phase 2: Diagnosis (2-5 minutes)

**2.1 If database is down**
```bash
# Check why database crashed
journalctl -u surrealdb -n 100
tail -100 /var/log/surrealdb.log | grep -i error

# Check system resources
free -h
df -h
ps aux | grep surrealdb | grep -v grep

# Check for crashes
dmesg | tail -20
```

**2.2 If database is running but unreachable**
```bash
# Check database is listening
ss -tlnp | grep surrealdb
lsof -i :8000

# Check firewall
iptables -L | grep 8000
# or
firewall-cmd --list-all | grep 8000

# Check database configuration
cat /etc/surrealdb/config.toml | head -20
```

**2.3 If connection pool exhausted**
```bash
# Count active connections
curl -s http://surrealdb:8000/api/v1/stats | jq .connections

# Check service connection limits
ps aux | grep affected-service | awk '{print $2}' | xargs lsof -p | grep -c ESTABLISHED

# Check ulimit
ulimit -n
```

### Phase 3: Remediation (5-10 minutes)

**3.1 If database crashed - Restart**
```bash
# Restart database
systemctl restart surrealdb

# Wait for startup
sleep 10

# Verify it's running
curl http://surrealdb:8000/health

# Check logs for startup errors
journalctl -u surrealdb -n 20
```

**3.2 If database won't start**
```bash
# Check storage corruption
ls -la /var/lib/surrealdb/

# Try recovery
surrealdb --recovery

# Or restore from backup
tar -xzf /backup/surrealdb-latest.tar.gz -C /var/lib/

# Start fresh (last resort, loses data)
rm -rf /var/lib/surrealdb
systemctl restart surrealdb
```

**3.3 If connection pool exhausted**
```bash
# Reduce service worker count (uses fewer connections)
export SERVICE_WORKERS=2
pkill affected-service
sleep 2
cargo run --release -p affected-service &

# Or increase database connection pool
# Edit database config: max_connections = 200
systemctl restart surrealdb
```

**3.4 Restart affected service to clear connections**
```bash
# Graceful restart clears connection pool
pkill -SIGTERM affected-service
sleep 5

# Verify connections closed
lsof -p PID | wc -l

# Start service
cargo run --release -p affected-service &
sleep 5

# Verify connection established
curl http://localhost:PORT/health
```

### Phase 4: Verification (10-15 minutes)

**4.1 Confirm connectivity restored**
```bash
# Test database access
curl -s http://surrealdb:8000/health | jq .

# Test affected service
curl -s http://localhost:PORT/health | jq .

# Run query
surreal sql --endpoint http://surrealdb:8000 --username root --password root \
  --query "SELECT * FROM services LIMIT 1"

# Check affected service logs
tail -20 /var/log/provisioning/affected-service.log | grep -v "debug\|info"
```

**4.2 Monitor for recurrence**
```bash
# Watch for connection errors
watch -n 5 'curl -s "http://localhost:9090/api/v1/query?query=increase(database_connection_errors_total[5m])" | jq .'

# Should remain at 0
```

---

## Queue Backlog

**Alert**: `QueueBacklog` (severity: warning)

**Impact**: Tasks delayed, orchestrator falling behind

**Detection**: Queue depth > 1000 tasks for 5+ minutes

### Phase 1: Immediate Response (0-2 minutes)

**1.1 Check queue status**
```bash
# Get queue depth
curl -s http://localhost:9090/api/v1/queue/status | jq .

# Get queue trend
curl -s 'http://localhost:9090/api/v1/query?query=orchestrator_queue_depth[5m]' | jq .

# Get per-service queue depth
curl -s 'http://localhost:9090/api/v1/query?query=orchestrator_queue_depth by (service)' | jq .
```

**1.2 Check processing rate**
```bash
# Tasks processed per minute
curl -s 'http://localhost:9090/api/v1/query?query=rate(orchestrator_tasks_total[5m])' | jq .

# Task success rate
curl -s 'http://localhost:9090/api/v1/query?query=rate(orchestrator_tasks_successful_total[5m]) / rate(orchestrator_tasks_total[5m])' | jq .

# Task failure rate
curl -s 'http://localhost:9090/api/v1/query?query=rate(orchestrator_tasks_failed_total[5m])' | jq .
```

**1.3 Get task samples**
```bash
# Get recent tasks
curl -s http://localhost:9090/api/v1/tasks?limit=20 | jq .

# Get stuck tasks (pending > 5 minutes)
curl -s http://localhost:9090/api/v1/tasks?status=pending | jq '.[] | select(.duration_seconds > 300)'

# Get failed tasks
curl -s http://localhost:9090/api/v1/tasks?status=failed&limit=10 | jq .
```

### Phase 2: Diagnosis (2-5 minutes)

**2.1 Identify processing bottleneck**
```bash
# Check orchestrator CPU
top -p ORCHESTRATOR_PID -n 1 | grep orchestrator

# Check orchestrator memory
ps aux | grep orchestrator | grep -v grep | awk '{printf "Memory: %d MB\n", $6/1024}'

# Check orchestrator logs for errors
tail -50 /var/log/provisioning/orchestrator.log | grep -i error
```

**2.2 Check if workers are healthy**
```bash
# Get worker status
curl -s http://localhost:9090/api/v1/workers | jq .

# Check specific worker health
curl -s http://localhost:9090/api/v1/workers?id=WORKER_ID | jq .

# Check worker logs
tail -20 /var/log/provisioning/orchestrator.log | grep -i "worker"
```

**2.3 Check for deadlock**
```bash
# Get task dependencies
curl -s http://localhost:9090/api/v1/tasks/TASK_ID | jq '.dependencies'

# Look for circular dependencies
curl -s http://localhost:9090/api/v1/queue/analysis | jq '.deadlocks'

# Check for blocked tasks
curl -s http://localhost:9090/api/v1/tasks?status=blocked | jq length
```

### Phase 3: Remediation (5-10 minutes)

**3.1 Scale up workers**
```bash
# Increase worker count
export ORCHESTRATOR_WORKERS=16
pkill -SIGTERM orchestrator
sleep 2

# Build and restart
cargo build --release -p orchestrator
cargo run --release -p orchestrator &

# Monitor queue depth
watch -n 5 'curl -s http://localhost:9090/api/v1/queue/status | jq .depth'
```

**3.2 Skip failing tasks (if safe)**
```bash
# Get failed task
curl -s http://localhost:9090/api/v1/tasks?status=failed&limit=1 | jq '.[] | .id'

# Manually fail task to unblock queue
curl -X POST http://localhost:9090/api/v1/tasks/TASK_ID/skip \
  -H "Content-Type: application/json" \
  -d '{"reason": "operator override - unblock queue"}'

# Monitor queue depth
curl -s http://localhost:9090/api/v1/queue/status | jq .depth
```

**3.3 Drain queue safely**
```bash
# Get queue depth before
curl -s http://localhost:9090/api/v1/queue/status | jq .depth

# Process queue aggressively
export ORCHESTRATOR_BATCH_SIZE=100
pkill -SIGTERM orchestrator
sleep 2
cargo run --release -p orchestrator &

# Monitor progress
for i in {1..30}; do
  echo "$(date): $(curl -s http://localhost:9090/api/v1/queue/status | jq .depth) tasks"
  sleep 10
done
```

**3.4 Clear stuck tasks (last resort)**
```bash
# Get list of stuck tasks
STUCK_TASKS=$(curl -s http://localhost:9090/api/v1/tasks?status=pending | \
  jq -r '.[] | select(.duration_seconds > 900) | .id')

# Fail all stuck tasks
for task_id in $STUCK_TASKS; do
  curl -X POST http://localhost:9090/api/v1/tasks/$task_id/fail \
    -H "Content-Type: application/json" \
    -d '{"reason": "operator override - task stuck > 15 min"}'
done

# Verify queue depth decreased
curl -s http://localhost:9090/api/v1/queue/status | jq .depth
```

### Phase 4: Verification (10-15 minutes)

**4.1 Confirm queue depth decreasing**
```bash
# Check trend
curl -s 'http://localhost:9090/api/v1/query_range?query=orchestrator_queue_depth&start=10m&step=1m' | jq .

# Should show downward trend

# Monitor real-time
watch -n 5 'curl -s http://localhost:9090/api/v1/queue/status | jq ".depth"'

# Should reach 0 or acceptable level
```

**4.2 Check task completion**
```bash
# Success rate
curl -s 'http://localhost:9090/api/v1/query?query=rate(orchestrator_tasks_successful_total[5m])' | jq .

# Failure rate
curl -s 'http://localhost:9090/api/v1/query?query=rate(orchestrator_tasks_failed_total[5m])' | jq .

# Processing time
curl -s 'http://localhost:9090/api/v1/query?query=histogram_quantile(0.95, rate(orchestrator_task_duration_seconds_bucket[5m]))' | jq .
```

---

## Authentication Failure

**Alert**: `RegistryAuthError` (severity: warning)

**Impact**: Extension registry unavailable, deployment failures

**Detection**: Auth failures > 5 in 5 minutes

### Phase 1: Immediate Response (0-2 minutes)

**1.1 Check registry status**
```bash
# Registry health
curl -s http://localhost:8081/health | jq .

# Check auth endpoint
curl -X POST http://localhost:8081/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "test", "password": "test"}'

# Check logs
tail -30 /var/log/provisioning/extension-registry.log | grep -i auth
```

**1.2 Check auth backend**
```bash
# If using Gitea
curl -s http://gitea:3000/health

# If using local auth
curl -s http://localhost:8081/api/auth/status

# Get auth errors
grep "auth.*error\|authentication.*failed" /var/log/provisioning/extension-registry.log
```

### Phase 2: Diagnosis (2-5 minutes)

**2.1 Check auth configuration**
```bash
# View registry schema
nickel typecheck provisioning/schemas/platform/schemas/extension-registry.ncl

# Check defaults contain auth config
cat provisioning/schemas/platform/defaults/extension-registry-defaults.ncl | grep -A 5 "auth\|gitea"

# Verify mode-specific settings
cat provisioning/schemas/platform/defaults/deployment/*-defaults.ncl | grep -A 5 "extension.registry"
```

**2.2 Check credential validity**
```bash
# Test Gitea login if applicable
curl -X POST http://gitea:3000/api/v1/user/login \
  -H "Content-Type: application/json" \
  -d '{"login_name": "admin", "password": "PASSWORD"}'

# Test OCI registry auth if applicable
docker login registry.internal
```

**2.3 Check network connectivity**
```bash
# Test connection to auth backend
curl -v http://gitea:3000/health
curl -v http://ldap:389/health

# Check DNS
nslookup gitea
dig gitea
```

### Phase 3: Remediation (5-10 minutes)

**3.1 If credential expired**
```bash
# Generate new token
curl -X POST http://gitea:3000/api/v1/users/admin/tokens \
  -H "Authorization: token OLD_TOKEN"

# Update registry defaults
vim provisioning/schemas/platform/defaults/extension-registry-defaults.ncl
# Update: auth_token = "NEW_TOKEN", then re-generate

# Re-generate runtime config
./provisioning/.typedialog/platform/scripts/generate-configs.nu extension-registry solo

# Restart registry
pkill -SIGTERM extension-registry
sleep 2
cargo run --release -p extension-registry &
```

**3.2 If auth backend down**
```bash
# Check if Gitea is running
pgrep gitea || echo "Gitea not running"
systemctl status gitea

# Restart Gitea
systemctl restart gitea
sleep 10

# Verify it's up
curl http://gitea:3000/health

# Restart registry to reconnect
pkill -SIGTERM extension-registry
sleep 2
cargo run --release -p extension-registry &
```

**3.3 If auth service misconfigured**
```bash
# Check auth configuration
cat /etc/gitea/app.ini | grep -A 5 "\[auth"

# Verify LDAP/auth settings if applicable
ldapwhoami -H ldap://ldap:389 -x

# Restart auth service
systemctl restart gitea-auth

# Restart registry
pkill extension-registry
cargo run --release -p extension-registry &
```

### Phase 4: Verification (10-15 minutes)

**4.1 Confirm auth working**
```bash
# Test login
curl -X POST http://localhost:8081/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "PASSWORD"}'

# Should return 200 with token

# Test token usage
TOKEN="..."
curl -H "Authorization: Bearer $TOKEN" http://localhost:8081/api/extensions

# Should return 200
```

**4.2 Check alert cleared**
```bash
# Verify no more auth errors
curl -s 'http://localhost:9090/api/v1/query?query=rate(registry_auth_failures_total[5m])' | jq .

# Should be 0

# Check alert status
curl -s http://localhost:9093/api/v1/alerts | jq '.data.alerts[] | select(.labels.alertname=="RegistryAuthError")'

# Should be empty or resolved
```

---

## Data Corruption

**Alert**: None (detected through manual inspection or consistency checks)

**Impact**: Severe, data loss, system unreliable

**Detection**: Failed checksums, database validation errors

### Phase 1: Immediate Response (0-2 minutes)

**1.1 Isolate affected service**
```bash
# Stop affected service to prevent further corruption
pkill -SIGTERM affected-service
sleep 5
pgrep affected-service || echo "Stopped"

# Alert team immediately
# Post to incident channel: "DATA CORRUPTION DETECTED - isolating affected-service"
```

**1.2 Assess scope**
```bash
# What data is affected
curl -s http://localhost:9090/api/v1/health | jq '.data_integrity'

# Check database
surreal sql --endpoint http://surrealdb:8000 --username root --password root \
  --query "SELECT count(*) FROM CORRUPTED_TABLE"

# Check for error patterns
grep -i "corrupt\|checksum\|validation.*failed" /var/log/provisioning/*.log
```

### Phase 2: Diagnosis (2-10 minutes)

**2.1 Determine extent of corruption**
```bash
# Run integrity check
curl -X POST http://localhost:9090/api/v1/admin/integrity-check

# Get results
curl -s http://localhost:9090/api/v1/admin/integrity-check/status | jq .

# Check what failed
curl -s http://localhost:9090/api/v1/admin/integrity-check/errors | jq .
```

**2.2 Locate backup**
```bash
# Find most recent backup
ls -lrt /backup/surrealdb*.sql | tail -5
ls -lrt /backup/provisioning*.tar.gz | tail -5

# Verify backup integrity
file /backup/surrealdb-latest.sql
tar -tzf /backup/provisioning-latest.tar.gz | head -5
```

**2.3 Determine restore point**
```bash
# When was corruption detected
grep "corrupt\|checksum" /var/log/provisioning/*.log | head -1

# Work back to find clean backup
# Usually safe to restore from backup 1-2 hours before detection
```

### Phase 3: Remediation (10-30 minutes)

**3.1 Restore from backup**
```bash
# Stop all services using the data
pkill -SIGTERM -f "cargo run --release"
sleep 5

# Create backup of corrupted data (for forensics)
tar -czf /archive/corrupted-data-$(date +%s).tar.gz /var/lib/provisioning/

# Restore clean data
tar -xzf /backup/provisioning-2026-01-05-0800.tar.gz -C /var/lib/

# Verify restore
ls -la /var/lib/provisioning/
```

**3.2 If database restore needed**
```bash
# Backup current corrupted database
mv /var/lib/surrealdb /var/lib/surrealdb.corrupted

# Create fresh database directory
mkdir -p /var/lib/surrealdb
chmod 755 /var/lib/surrealdb

# Restore from backup
surreal import --endpoint http://surrealdb:8000 \
  --username root --password root < /backup/surrealdb-2026-01-05-0800.sql

# Verify restore
surreal sql --endpoint http://surrealdb:8000 --username root --password root \
  --query "SELECT count(*) FROM services"
```

**3.3 Restart services and verify**
```bash
# Start services one by one
cargo run --release -p vault-service &
sleep 5
curl http://localhost:8200/health

cargo run --release -p extension-registry &
sleep 5
curl http://localhost:8081/health

# And others...

# Run integrity check again
curl -X POST http://localhost:9090/api/v1/admin/integrity-check
```

### Phase 4: Post-Incident (After Incident)

**4.1 Root cause analysis**
```bash
# What caused corruption
# Check logs for:
# - Crashes during writes
# - Disk full errors
# - Database errors
# - Network interruptions

grep -B 5 -A 5 "corrupt" /var/log/provisioning/*.log > /tmp/corruption-analysis.txt
```

**4.2 Prevent recurrence**
```bash
# Implement integrity checks
# Add scheduled checksum validation
# Add crash consistency checks
# Implement write-ahead logging (WAL)
# Add monitoring for anomalies
```

**4.3 Document incident**
```bash
# Create detailed incident report
cat > /tmp/data-corruption-incident.md << EOF
## Data Corruption Incident

**Detection**: [WHEN AND HOW]
**Root Cause**: [TBD]
**Impact**: [DATA LOST/MODIFIED]
**Duration**: [TIME]
**Recovery**: [WHAT WAS RESTORED]
**Prevention**: [PLANNED FIXES]
EOF
```

---

## Full Cluster Failure

**Alert**: Multiple critical alerts firing simultaneously

**Impact**: Complete platform unavailable, business-critical

**Detection**: Multiple services down, cascading failures

### Phase 1: Immediate Response (0-5 minutes)

**1.1 Assess situation**
```bash
# Check all services
for port in 8200 8081 8083 8082 9090 8080; do
  curl -s http://localhost:$port/health || echo "Port $port DOWN"
done

# Check infrastructure
curl http://surrealdb:8000/health
etcdctl --endpoints=http://etcd:2379 endpoint health

# Check system
free -h
df -h
ps aux | head -20
```

**1.2 Declare incident**
```bash
# Create war room
# Notify all stakeholders
# Post: "INCIDENT: Platform unavailable - full cluster failure - details TBD"
# Start recording
# Get additional on-call people
```

**1.3 Check recent changes**
```bash
# What changed?
git log --oneline | head -5

# Recent deploys
git log --oneline | grep "deploy\|release" | head -3

# System updates
journalctl --since "30 min ago" | grep -i "update\|upgrade\|restart"
```

### Phase 2: Diagnosis (5-15 minutes)

**2.1 Identify common cause**
```bash
# All services crashed?
pgrep -a cargo | grep "release -p"

# Database down?
curl http://surrealdb:8000/health
ps aux | grep surrealdb

# Network issue?
ping -c 5 8.8.8.8
ss -s
netstat -an | grep ESTABLISHED | wc -l

# Disk full?
df -h

# OOM?
dmesg | tail -20 | grep -i "killed\|oom"
```

**2.2 Check shared infrastructure**
```bash
# Load balancer status
curl http://loadbalancer:8080/health

# DNS
nslookup surrealdb
dig services.internal

# Network
traceroute surrealdb
iptables -L | head -20

# Firewall
systemctl status firewalld
```

**2.3 Determine if issue is infrastructure or application**
```bash
# Infrastructure signs:
# - No network connectivity
# - DNS failures
# - All services unable to start
# - System running out of resources

# Application signs:
# - Services start but crash
# - Database errors
# - Dependency failures
# - Configuration issues
```

### Phase 3: Recovery (15-60 minutes)

**3.1 Infrastructure restart (if infrastructure issue)**
```bash
# Restart network
systemctl restart networking

# Restart DNS
systemctl restart systemd-resolved
systemctl restart coredns

# Restart load balancer
systemctl restart haproxy

# Test connectivity
curl http://8.8.8.8
nslookup google.com
```

**3.2 Restart all services (methodical order)**
```bash
# Stop everything
pkill -9 -f "cargo run"

# Clean up resources
rm -f /var/run/provisioning/*.lock
rm -f /tmp/provisioning-*

# Restart infrastructure first
systemctl restart surrealdb
sleep 10
curl http://surrealdb:8000/health

systemctl restart etcd
sleep 10
etcdctl --endpoints=http://etcd:2379 endpoint health

# Then restart services in dependency order
cargo run --release -p vault-service &
sleep 5
curl http://localhost:8200/health

cargo run --release -p extension-registry &
sleep 5
curl http://localhost:8081/health

# Continue with others...
```

**3.3 Monitor recovery**
```bash
# Watch startup
watch -n 5 'for port in 8200 8081 8083 8082 9090 8080; do
  curl -s http://localhost:$port/health | jq -r .status && echo "Port $port: OK" || echo "Port $port: DOWN"
done'

# Watch logs
tail -f /var/log/provisioning/*.log | grep -v "debug"

# Watch resources
watch -n 5 'free -h && echo "---" && df -h'
```

### Phase 4: Verification (After Recovery)

**4.1 System fully recovered**
```bash
# All services responding
for port in 8200 8081 8083 8082 9090 8080; do
  status=$(curl -s http://localhost:$port/health | jq -r .status 2>/dev/null)
  if [ "$status" = "ok" ]; then
    echo "✓ Port $port: OK"
  else
    echo "✗ Port $port: FAILED"
  fi
done

# No errors in logs
tail -100 /var/log/provisioning/*.log | grep -i "error\|warn" | wc -l

# System resources normal
free -h
df -h
```

**4.2 Data integrity verified**
```bash
# Run integrity check
curl -X POST http://localhost:9090/api/v1/admin/integrity-check

# Verify no corruption
curl -s http://localhost:9090/api/v1/admin/integrity-check/status | jq '.corruption_detected'

# Should be false
```

**4.3 Close incident**
```bash
# Resolve in incident tracker
# Post: "✓ RESOLVED at HH:MM UTC - full cluster recovered"

# Schedule post-incident review
# Collect logs and metrics
# Document timeline
```

---

## Post-Incident Actions (All Incidents)

### Immediate (Within 1 hour)

1. Document incident start/end times
2. Collect logs and metrics
3. Note impact and scope
4. List immediate actions taken
5. Identify if data was lost

### Short-term (Within 24 hours)

1. Send incident notification to stakeholders
2. Conduct war room debriefing
3. Identify root cause
4. Create action items to prevent recurrence
5. Update runbooks based on learnings

### Long-term (Within 1 week)

1. Complete RCA (Root Cause Analysis)
2. Implement fixes to prevent recurrence
3. Update monitoring and alerting
4. Schedule postmortem review
5. Update documentation

---

## Quick Reference: Critical Numbers

```bash
Service Availability: Should be 99.9% or better
Error Rate: Should be < 0.1% (< 1 error per 1000 requests)
P95 Latency: Should be < 500 ms
P99 Latency: Should be < 1000 ms
Queue Depth: Should stay < 100 tasks
Memory Usage: Should stay < 80% of limit
Disk Usage: Should stay < 80% full
Database Connection Time: Should be < 100 ms
```

---

## Resources & Escalation

**On-Call**: Check PagerDuty for current on-call rotation
**Slack**: #platform-incidents for incident discussion
**Runbook Updates**: After each incident, update relevant runbook
**Documentation**: Keep these runbooks updated as platform evolves

---

**Last Updated**: 2026-01-05
**Version**: 1.0.0
**Status**: Production Ready ✅
