# Incident Response Runbooks\n\n**Step-by-step procedures for responding to common platform incidents**\n\n**Version**: 1.0.0\n**Last Updated**: 2026-01-05\n**Target Audience**: On-Call Engineers, Platform Operators\n**Status**: Production Ready\n\n---\n\n## Quick Navigation\n\n- [Service Down](#service-down)\n- [High Error Rate](#high-error-rate)\n- [Memory Exhaustion](#memory-exhaustion)\n- [Disk Space Critical](#disk-space-critical)\n- [Database Connection Failure](#database-connection-failure)\n- [Queue Backlog](#queue-backlog)\n- [Authentication Failure](#authentication-failure)\n- [Data Corruption](#data-corruption)\n- [Full Cluster Failure](#full-cluster-failure)\n\n---\n\n## Service Down\n\n**Alert**: `ServiceDown` (severity: critical)\n\n**Impact**: Service unavailable, users cannot access functionality\n\n**Detection**: Service unreachable for 5+ minutes\n\n### Phase 1: Immediate Response (0-2 minutes)\n\n**1.1 Acknowledge alert**\n```\n# In PagerDuty/Slack\n# React with 🚨 to acknowledge you're investigating\n```\n\n**1.2 Verify service is actually down**\n```\n# Check if service is running\npgrep -a service-name\nps aux | grep service-name | grep -v grep\n\n# Check if port is listening\nss -tlnp | grep PORT_NUMBER\nlsof -i :PORT_NUMBER\n\n# Test HTTP endpoint\ncurl -v http://localhost:PORT/health\ncurl -v http://SERVICE_HOSTNAME:PORT/health\n```\n\n**1.3 Check recent events**\n```\n# View last 20 log lines\ntail -20 /var/log/provisioning/service-name.log\n\n# Check for crash\ngrep -i "panic\|fatal\|error" /var/log/provisioning/service-name.log | tail -10\n\n# Check systemd status (if using systemd)\nsystemctl status provisioning-service-name\njournalctl -u provisioning-service-name -n 50\n```\n\n### Phase 2: Diagnosis (2-5 minutes)\n\n**2.1 Determine if service crashed or hung**\n```\n# Check process exists\npgrep service-name && echo "Process running" || echo "Process crashed"\n\n# If running but not responding, check for deadlock/hang\nstrace -p PID 2>&1 | head -20\n\n# Check CPU usage\ntop -p PID -n 1\nps aux | grep service-name | grep -v grep | awk '{print $3, $6}'\n```\n\n**2.2 Check dependencies**\n```\n# Test database connectivity (if applicable)\ncurl http://surrealdb:8000/health\netcdctl --endpoints=http://etcd:2379 endpoint health\n\n# Test upstream service connectivity\ncurl http://vault:8200/health\ncurl http://registry:8081/health\n\n# Test DNS resolution\nnslookup service-name.internal\ndig service-name.internal\n```\n\n**2.3 Review configuration**\n```\n# Check schema exists and is valid\nnickel typecheck provisioning/schemas/platform/schemas/service.ncl\n\n# Check defaults are valid\nnickel typecheck provisioning/schemas/platform/defaults/service-defaults.ncl\n\n# Check environment variables\nenv | grep -E "SERVICE_|VAULT_|REGISTRY_"\n```\n\n### Phase 3: Remediation (5-10 minutes)\n\n**3.1 Attempt restart**\n```\n# Stop service gracefully\npkill -SIGTERM service-name\nsleep 5\n\n# Verify it stopped\npgrep service-name || echo "Stopped successfully"\n\n# Clear any lock files\nrm -f /var/run/service-name.lock\nrm -f /tmp/service-name.*\n\n# Start service with verbose logging\nRUST_LOG=debug cargo run --release -p service-name 2>&1 | tee /tmp/service-startup.log &\n\n# Wait for startup\nsleep 10\n\n# Verify startup\ncurl http://localhost:PORT/health\n```\n\n**3.2 If restart fails**\n```\n# Check full error output\nRUST_LOG=trace cargo run -p service-name 2>&1 | head -100\n\n# Check if port is in use by another process\nlsof -i :PORT\nnetstat -tlnp | grep PORT\n\n# Kill process using port\npkill -9 -f "cargo run.*service-name"\n\n# Try on different port (temporary)\nexport SERVICE_PORT=9999\ncargo run --release -p service-name &\ncurl http://localhost:9999/health\n```\n\n**3.3 If still failing**\n```\n# Restore schemas from backup\ngit checkout HEAD~1 -- provisioning/schemas/platform/schemas/service.ncl\ngit checkout HEAD~1 -- provisioning/schemas/platform/defaults/service-defaults.ncl\n\n# Re-generate runtime config\n./provisioning/.typedialog/platform/scripts/generate-configs.nu service solo\n\n# Try with solo mode (if available)\nexport SERVICE_MODE=solo\ncargo run --release -p service-name &\nsleep 5\ncurl http://localhost:PORT/health\n\n# If that works, mode-specific defaults are broken\n# Escalate to team for debugging\n```\n\n### Phase 4: Verification (10-15 minutes)\n\n**4.1 Confirm service is healthy**\n```\n# Health check\ncurl -s http://localhost:PORT/health | jq .\n\n# Test basic operation\ncurl -X GET http://localhost:PORT/api/v1/status\n\n# Check metrics\ncurl -s http://localhost:PORT/metrics | head -20\n\n# Monitor logs for errors\ntail -f /var/log/provisioning/service-name.log\n```\n\n**4.2 Check downstream services**\n```\n# If this service is dependency for others, verify they recovered\nfor port in 8200 8081 8083 8082 9090 8080; do\n  curl -s http://localhost:$port/health && echo "✓ Port $port" || echo "✗ Port $port"\ndone\n\n# Check orchestrator is accepting tasks\ncurl -s http://localhost:9090/api/v1/status | jq .\n```\n\n**4.3 Update incident status**\n```\n# In PagerDuty/Slack\n# Post: "✓ Service recovered at HH:MM, investigating root cause"\n# Start war room if needed\n```\n\n### Phase 5: Root Cause Analysis (During + After Incident)\n\n**5.1 Collect data**\n```\n# Full logs from incident window\ngrep "2026-01-05 14:" /var/log/provisioning/service-name.log > /tmp/incident-logs.txt\n\n# Service metrics during incident\ncurl -s 'http://localhost:9090/api/v1/query_range?query=up{job="service"}' | jq .\n\n# Dependent services status\ncurl -s http://localhost:9090/api/v1/services | jq .\n\n# System resources at time of incident\nuptime\nfree -h\ndf -h\n```\n\n**5.2 Investigate common causes**\n```\n# OOM (Out of Memory)\ndmesg | grep -i "killed\|oom" | tail -5\ngrep "memory" /var/log/provisioning/service-name.log\n\n# Disk full\ndf -h | grep -E "9[0-9]%|100%"\ndu -sh /var/lib/provisioning/*\n\n# File descriptor exhaustion\nlsof -p PID | wc -l\ncat /proc/sys/fs/file-max\n\n# Goroutine/thread leak\ncurl -s http://localhost:PORT/debug/pprof/goroutine\n\n# Dependency timeout\ngrep -i "timeout\|connection reset" /var/log/provisioning/service-name.log\n```\n\n### Phase 6: Post-Incident\n\n**6.1 Document incident**\n```\n# Create incident summary\ncat > /tmp/incident-summary.md << EOF\n## Incident: SERVICE-NAME Down\n- Start time: 2026-01-05 14:30 UTC\n- Duration: 15 minutes\n- Root cause: [TO BE DETERMINED]\n- Impact: [SERVICE FUNCTIONALITY UNAVAILABLE]\n- Resolution: [RESTART]\n- Prevention: [TBD]\nEOF\n```\n\n**6.2 Implement prevention**\n```\n# Add monitoring/alerting if missing\n# Implement auto-restart if applicable\n# Update runbooks based on findings\n# Schedule follow-up action items\n```\n\n---\n\n## High Error Rate\n\n**Alert**: `HighErrorRate` (severity: warning)\n\n**Impact**: Service returning 5xx errors, degraded functionality\n\n**Detection**: Error rate above 5% for 5+ minutes\n\n### Phase 1: Immediate Response (0-2 minutes)\n\n**1.1 Acknowledge alert and gather metrics**\n```\n# Get current error rate\ncurl -s 'http://localhost:9090/api/v1/query?query=rate(http_requests_total{status=~"5.."}[5m])' | jq .\n\n# Get affected endpoints\ncurl -s 'http://localhost:9090/api/v1/query?query=rate(http_requests_total{status="500"}[5m]) by (handler)' | jq .\n\n# Get error rate per service\ncurl -s 'http://localhost:9090/api/v1/query?query=rate(http_requests_total{status=~"5.."}[5m]) by (job)' | jq .\n```\n\n**1.2 Check service health**\n```\n# Check all services\nfor port in 8200 8081 8083 8082 9090 8080; do\n  echo "=== Port $port ==="\n  curl -s http://localhost:$port/health | jq .status 2>/dev/null || echo "DOWN"\ndone\n\n# Get recent logs\ntail -50 /var/log/provisioning/affected-service.log | grep -i error\n```\n\n### Phase 2: Diagnosis (2-5 minutes)\n\n**2.1 Identify error pattern**\n```\n# Get detailed error messages\ncurl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.alertname=="HighErrorRate")'\n\n# Check service logs for panic\ngrep "panic\|fatal\|ERROR" /var/log/provisioning/affected-service.log | tail -20\n\n# Get stack traces if available\ntail -100 /var/log/provisioning/affected-service.log | grep -A 20 "panic"\n```\n\n**2.2 Check dependencies**\n```\n# Test database\ncurl http://surrealdb:8000/health\nsurreal sql --endpoint http://surrealdb:8000 --username root --password root --query "SELECT count(*) FROM services"\n\n# Test cache\ncurl http://cache:6379/health\n\n# Test external services\ncurl http://vault:8200/health\ncurl http://registry:8081/health\n```\n\n**2.3 Check for resource issues**\n```\n# Memory\nfree -h\nps aux | grep service | awk '{print $6}' | head -5\n\n# CPU\ntop -bn1 | head -15\n\n# Disk\ndf -h\n\n# Network connections\nss -s\nnetstat -an | grep ESTABLISHED | wc -l\n```\n\n### Phase 3: Remediation (5-10 minutes)\n\n**3.1 If dependency is unhealthy**\n```\n# Restart database\nsystemctl restart surrealdb\n\n# Verify connectivity restored\ncurl http://surrealdb:8000/health\n\n# Reload affected service\npkill -SIGTERM affected-service\nsleep 2\ncargo run --release -p affected-service &\n```\n\n**3.2 If resource exhausted**\n```\n# Scale up if possible\nexport SERVICE_WORKERS=16  # Increase workers\npkill -SIGTERM affected-service\nsleep 2\ncargo run --release -p affected-service &\n\n# Or reduce load if under DoS\n# Contact network team to implement rate limiting\n```\n\n**3.3 If code issue (deployment-related)**\n```\n# Rollback to previous version\ngit checkout HEAD~1\ncargo build --release -p affected-service\n\n# Stop current version\npkill -SIGTERM affected-service\n\n# Start previous version\nexport SERVICE_VERSION=previous\ncargo run --release -p affected-service &\n\n# Verify error rate drops\nsleep 30\ncurl -s 'http://localhost:9090/api/v1/query?query=rate(http_requests_total{status=~"5.."}[5m])' | jq .\n```\n\n### Phase 4: Verification (10-15 minutes)\n\n**4.1 Confirm error rate recovered**\n```\n# Check current error rate\ncurl -s 'http://localhost:9090/api/v1/query?query=rate(http_requests_total{status=~"5.."}[5m])' | jq .\n\n# Should be < 0.05 (5%)\n\n# Check alert status\ncurl -s http://localhost:9093/api/v1/alerts | jq '.data.alerts[] | select(.labels.alertname=="HighErrorRate")'\n\n# Should be resolved or not present\n```\n\n**4.2 Monitor for recurrence**\n```\n# Set up dashboard to watch for 15 minutes\nwatch -n 10 'curl -s "http://localhost:9090/api/v1/query?query=rate(http_requests_total{status=~\"5..\"}[5m])" | jq .'\n\n# Or use Grafana dashboard\n# Navigate to Platform Overview → Error Rate\n```\n\n---\n\n## Memory Exhaustion\n\n**Alert**: `HighMemoryUsage` (severity: warning)\n\n**Impact**: Service slowdown, potential crashes, cascading failures\n\n**Detection**: Memory usage > 90% for 5+ minutes\n\n### Phase 1: Immediate Response (0-2 minutes)\n\n**1.1 Verify memory issue**\n```\n# Check system memory\nfree -h\n# Look for very low "available"\n\n# Check service memory\nps aux | grep service-name | grep -v grep | awk '{printf "Service: %d MB\n", $6/1024}'\n\n# Find top memory consumers\nps aux --sort=-%mem | head -10\n```\n\n**1.2 Check for memory leak**\n```\n# Monitor memory growth over time\nfor i in {1..5}; do\n  ps aux | grep service-name | grep -v grep | awk '{printf "%s: %d MB\n", $2, $6/1024}'\n  sleep 30\ndone\n\n# If increasing rapidly, memory leak suspected\n```\n\n### Phase 2: Diagnosis (2-5 minutes)\n\n**2.1 Identify leak source**\n```\n# Check goroutine count (Rust threads)\ncurl -s http://localhost:PORT/debug/pprof/goroutine | head -20\n\n# Check open file descriptors\nlsof -p PID | wc -l\ncat /proc/sys/fs/file-max\n\n# Check memory allocations\ncurl -s http://localhost:PORT/debug/pprof/heap | head -50\n```\n\n**2.2 Check for unbounded growth**\n```\n# Cache size\ncurl -s http://localhost:PORT/metrics | grep "cache.*bytes"\n\n# Connection pool\ncurl -s http://localhost:PORT/metrics | grep "connection.*total"\n\n# Queue depth\ncurl -s http://localhost:PORT/metrics | grep "queue.*depth"\n```\n\n### Phase 3: Remediation (5-10 minutes)\n\n**3.1 Immediate: Restart service**\n```\n# Graceful shutdown (allows cleanup)\npkill -SIGTERM service-name\nsleep 30\n\n# Verify stopped\npgrep service-name || echo "Stopped"\n\n# Remove any memory-mapped files\nrm -f /dev/shm/service-name.*\n\n# Restart with reduced worker count (less memory)\nexport SERVICE_WORKERS=2\ncargo run --release -p service-name &\n\n# Monitor memory\nwatch -n 5 'ps aux | grep service-name | grep -v grep | awk "{printf \"Memory: %d MB\\n\", \$6/1024}"'\n```\n\n**3.2 If memory still grows**\n```\n# Stop service\npkill -9 service-name\n\n# Clear any resource leaks from previous runs\nps aux | grep service-name | grep -v grep | awk '{print $2}' | xargs kill -9\n\n# Check for zombie processes\nps aux | grep defunct\n\n# Review logs for memory-related errors\ngrep -i "oom\|memory\|alloc" /var/log/provisioning/service-name.log\n\n# Escalate to development team\n```\n\n**3.3 Temporary mitigation**\n```\n# Enable memory limits via systemd\ncat > /etc/systemd/system/service-name.service.d/memory-limit.conf << EOF\n[Service]\nMemoryLimit=2G\nMemoryMax=2.5G\nEOF\n\nsystemctl daemon-reload\nsystemctl restart service-name\n```\n\n### Phase 4: Verification (10-15 minutes)\n\n**4.1 Confirm memory stabilized**\n```\n# Monitor for 10 minutes\nfor i in {1..20}; do\n  echo "=== $(date) ==="\n  ps aux | grep service-name | grep -v grep | awk '{printf "Memory: %d MB, CPU: %s%%\n", $6/1024, $3}'\n  sleep 30\ndone\n\n# Memory should stabilize, not keep growing\n```\n\n**4.2 Post-incident actions**\n```\n# Enable memory profiling\nRUST_LOG=debug MALLOC_TRACE=/tmp/mem-trace.log cargo run -p service-name\n\n# Analyze heap dump\nheapdump analyze /tmp/heap.bin\n\n# File ticket for development team to investigate\n```\n\n---\n\n## Disk Space Critical\n\n**Alert**: `HighDiskUsage` (severity: warning)\n\n**Impact**: Service failures, data corruption, cascading failures\n\n**Detection**: Free disk space < 10%\n\n### Phase 1: Immediate Response (0-2 minutes)\n\n**1.1 Verify disk space issue**\n```\n# Check disk usage\ndf -h\ndf -h /var/lib/provisioning\n\n# Show large files\ndu -sh /var/lib/provisioning/* | sort -rh | head -10\n\n# Find large directories\nfind /var/lib/provisioning -type d -exec du -sh {} + | sort -rh | head -10\n```\n\n**1.2 Identify culprit**\n```\n# Check logs size\ndu -sh /var/log/provisioning/\n\n# Check database size\ndu -sh /var/lib/provisioning/surrealdb*\ndu -sh /var/lib/provisioning/etcd*\n\n# Check temporary files\ndu -sh /tmp/provisioning-*\ndu -sh /dev/shm/\n```\n\n### Phase 2: Remediation (2-10 minutes)\n\n**2.1 Clean up temporary files**\n```\n# Remove old logs (keep 7 days)\nfind /var/log/provisioning -name "*.log" -mtime +7 -delete\n\n# Remove old core dumps\nfind /var/lib/provisioning -name "core.*" -delete\n\n# Clean temporary files\nrm -rf /tmp/provisioning-*\nrm -rf /tmp/service-*.lock\n```\n\n**2.2 Archive old metrics (if Prometheus)**\n```\n# Archive old Prometheus data\ncd /var/lib/prometheus\ntar -czf /archive/prometheus-old-$(date +%s).tar.gz snapshots/\nrm -rf snapshots/\n\n# Reduce retention (temporary)\n# Edit prometheus.yml: --storage.tsdb.retention.time=3d\n```\n\n**2.3 Rotate logs**\n```\n# Manual log rotation\nlogrotate -f /etc/logrotate.d/provisioning\n\n# Or manually\ngzip /var/log/provisioning/*.log\nmv /var/log/provisioning/*.log.gz /archive/\n```\n\n**2.4 Check disk usage again**\n```\n# Verify freed space\ndf -h\n\n# Should now have > 20% free space\n# If not, escalate (may need storage expansion)\n```\n\n### Phase 3: Permanent Fix\n\n**3.1 Enable automatic log rotation**\n```\n# Create logrotate config\ncat > /etc/logrotate.d/provisioning << EOF\n/var/log/provisioning/*.log {\n    daily\n    rotate 7\n    compress\n    delaycompress\n    notifempty\n    create 0640 provisioning provisioning\n    sharedscripts\n    postrotate\n        systemctl reload provisioning-services\n    endscript\n}\nEOF\n\nlogrotate -f /etc/logrotate.d/provisioning\n```\n\n**3.2 Implement metrics retention policy**\n```\n# Update Prometheus config\ncat >> /etc/prometheus/prometheus.yml << EOF\nglobal:\n  scrape_interval: 30s\n  retention: 15d           # Keep 15 days of metrics\n  retention_size: 100 GB    # Or max 100 GB\nEOF\n\nsystemctl restart prometheus\n```\n\n**3.3 Plan for growth**\n```\n# Calculate daily growth\ndu -sh /var/lib/provisioning\n# Check again tomorrow\n\n# If still growing fast, may need:\n# - Storage expansion\n# - Additional nodes\n# - Compression enabled\n```\n\n---\n\n## Database Connection Failure\n\n**Alert**: `DatabaseConnectionError` (severity: critical)\n\n**Impact**: Services cannot persist/retrieve data, functionality broken\n\n**Detection**: Connection errors > 10 in 5 minutes\n\n### Phase 1: Immediate Response (0-2 minutes)\n\n**1.1 Verify database is running**\n```\n# Check SurrealDB (if multiuser/enterprise)\ncurl -s http://surrealdb:8000/health\ncurl -s https://surrealdb:8000/health  # Try HTTPS if HTTP fails\n\n# Check Etcd (if enterprise)\netcdctl --endpoints=http://etcd:2379 endpoint health\n\n# Check database process\npgrep surrealdb || echo "SurrealDB not running"\npgrep etcd || echo "Etcd not running"\n```\n\n**1.2 Test connectivity**\n```\n# From affected service\ncurl -v http://surrealdb:8000/health\ntelnet surrealdb 8000\n\n# Check DNS\nnslookup surrealdb\ndig surrealdb\n\n# Check network\nping -c 3 surrealdb\ntraceroute surrealdb\n```\n\n**1.3 Get error details**\n```\n# Check affected service logs\ntail -50 /var/log/provisioning/affected-service.log | grep -i "database\|connection"\n\n# Get specific error\ngrep "connection refused\|connection reset\|timeout" /var/log/provisioning/affected-service.log\n```\n\n### Phase 2: Diagnosis (2-5 minutes)\n\n**2.1 If database is down**\n```\n# Check why database crashed\njournalctl -u surrealdb -n 100\ntail -100 /var/log/surrealdb.log | grep -i error\n\n# Check system resources\nfree -h\ndf -h\nps aux | grep surrealdb | grep -v grep\n\n# Check for crashes\ndmesg | tail -20\n```\n\n**2.2 If database is running but unreachable**\n```\n# Check database is listening\nss -tlnp | grep surrealdb\nlsof -i :8000\n\n# Check firewall\niptables -L | grep 8000\n# or\nfirewall-cmd --list-all | grep 8000\n\n# Check database configuration\ncat /etc/surrealdb/config.toml | head -20\n```\n\n**2.3 If connection pool exhausted**\n```\n# Count active connections\ncurl -s http://surrealdb:8000/api/v1/stats | jq .connections\n\n# Check service connection limits\nps aux | grep affected-service | awk '{print $2}' | xargs lsof -p | grep -c ESTABLISHED\n\n# Check ulimit\nulimit -n\n```\n\n### Phase 3: Remediation (5-10 minutes)\n\n**3.1 If database crashed - Restart**\n```\n# Restart database\nsystemctl restart surrealdb\n\n# Wait for startup\nsleep 10\n\n# Verify it's running\ncurl http://surrealdb:8000/health\n\n# Check logs for startup errors\njournalctl -u surrealdb -n 20\n```\n\n**3.2 If database won't start**\n```\n# Check storage corruption\nls -la /var/lib/surrealdb/\n\n# Try recovery\nsurrealdb --recovery\n\n# Or restore from backup\ntar -xzf /backup/surrealdb-latest.tar.gz -C /var/lib/\n\n# Start fresh (last resort, loses data)\nrm -rf /var/lib/surrealdb\nsystemctl restart surrealdb\n```\n\n**3.3 If connection pool exhausted**\n```\n# Reduce service worker count (uses fewer connections)\nexport SERVICE_WORKERS=2\npkill affected-service\nsleep 2\ncargo run --release -p affected-service &\n\n# Or increase database connection pool\n# Edit database config: max_connections = 200\nsystemctl restart surrealdb\n```\n\n**3.4 Restart affected service to clear connections**\n```\n# Graceful restart clears connection pool\npkill -SIGTERM affected-service\nsleep 5\n\n# Verify connections closed\nlsof -p PID | wc -l\n\n# Start service\ncargo run --release -p affected-service &\nsleep 5\n\n# Verify connection established\ncurl http://localhost:PORT/health\n```\n\n### Phase 4: Verification (10-15 minutes)\n\n**4.1 Confirm connectivity restored**\n```\n# Test database access\ncurl -s http://surrealdb:8000/health | jq .\n\n# Test affected service\ncurl -s http://localhost:PORT/health | jq .\n\n# Run query\nsurreal sql --endpoint http://surrealdb:8000 --username root --password root \\n  --query "SELECT * FROM services LIMIT 1"\n\n# Check affected service logs\ntail -20 /var/log/provisioning/affected-service.log | grep -v "debug\|info"\n```\n\n**4.2 Monitor for recurrence**\n```\n# Watch for connection errors\nwatch -n 5 'curl -s "http://localhost:9090/api/v1/query?query=increase(database_connection_errors_total[5m])" | jq .'\n\n# Should remain at 0\n```\n\n---\n\n## Queue Backlog\n\n**Alert**: `QueueBacklog` (severity: warning)\n\n**Impact**: Tasks delayed, orchestrator falling behind\n\n**Detection**: Queue depth > 1000 tasks for 5+ minutes\n\n### Phase 1: Immediate Response (0-2 minutes)\n\n**1.1 Check queue status**\n```\n# Get queue depth\ncurl -s http://localhost:9090/api/v1/queue/status | jq .\n\n# Get queue trend\ncurl -s 'http://localhost:9090/api/v1/query?query=orchestrator_queue_depth[5m]' | jq .\n\n# Get per-service queue depth\ncurl -s 'http://localhost:9090/api/v1/query?query=orchestrator_queue_depth by (service)' | jq .\n```\n\n**1.2 Check processing rate**\n```\n# Tasks processed per minute\ncurl -s 'http://localhost:9090/api/v1/query?query=rate(orchestrator_tasks_total[5m])' | jq .\n\n# Task success rate\ncurl -s 'http://localhost:9090/api/v1/query?query=rate(orchestrator_tasks_successful_total[5m]) / rate(orchestrator_tasks_total[5m])' | jq .\n\n# Task failure rate\ncurl -s 'http://localhost:9090/api/v1/query?query=rate(orchestrator_tasks_failed_total[5m])' | jq .\n```\n\n**1.3 Get task samples**\n```\n# Get recent tasks\ncurl -s http://localhost:9090/api/v1/tasks?limit=20 | jq .\n\n# Get stuck tasks (pending > 5 minutes)\ncurl -s http://localhost:9090/api/v1/tasks?status=pending | jq '.[] | select(.duration_seconds > 300)'\n\n# Get failed tasks\ncurl -s http://localhost:9090/api/v1/tasks?status=failed&limit=10 | jq .\n```\n\n### Phase 2: Diagnosis (2-5 minutes)\n\n**2.1 Identify processing bottleneck**\n```\n# Check orchestrator CPU\ntop -p ORCHESTRATOR_PID -n 1 | grep orchestrator\n\n# Check orchestrator memory\nps aux | grep orchestrator | grep -v grep | awk '{printf "Memory: %d MB\n", $6/1024}'\n\n# Check orchestrator logs for errors\ntail -50 /var/log/provisioning/orchestrator.log | grep -i error\n```\n\n**2.2 Check if workers are healthy**\n```\n# Get worker status\ncurl -s http://localhost:9090/api/v1/workers | jq .\n\n# Check specific worker health\ncurl -s http://localhost:9090/api/v1/workers?id=WORKER_ID | jq .\n\n# Check worker logs\ntail -20 /var/log/provisioning/orchestrator.log | grep -i "worker"\n```\n\n**2.3 Check for deadlock**\n```\n# Get task dependencies\ncurl -s http://localhost:9090/api/v1/tasks/TASK_ID | jq '.dependencies'\n\n# Look for circular dependencies\ncurl -s http://localhost:9090/api/v1/queue/analysis | jq '.deadlocks'\n\n# Check for blocked tasks\ncurl -s http://localhost:9090/api/v1/tasks?status=blocked | jq length\n```\n\n### Phase 3: Remediation (5-10 minutes)\n\n**3.1 Scale up workers**\n```\n# Increase worker count\nexport ORCHESTRATOR_WORKERS=16\npkill -SIGTERM orchestrator\nsleep 2\n\n# Build and restart\ncargo build --release -p orchestrator\ncargo run --release -p orchestrator &\n\n# Monitor queue depth\nwatch -n 5 'curl -s http://localhost:9090/api/v1/queue/status | jq .depth'\n```\n\n**3.2 Skip failing tasks (if safe)**\n```\n# Get failed task\ncurl -s http://localhost:9090/api/v1/tasks?status=failed&limit=1 | jq '.[] | .id'\n\n# Manually fail task to unblock queue\ncurl -X POST http://localhost:9090/api/v1/tasks/TASK_ID/skip \\n  -H "Content-Type: application/json" \\n  -d '{"reason": "operator override - unblock queue"}'\n\n# Monitor queue depth\ncurl -s http://localhost:9090/api/v1/queue/status | jq .depth\n```\n\n**3.3 Drain queue safely**\n```\n# Get queue depth before\ncurl -s http://localhost:9090/api/v1/queue/status | jq .depth\n\n# Process queue aggressively\nexport ORCHESTRATOR_BATCH_SIZE=100\npkill -SIGTERM orchestrator\nsleep 2\ncargo run --release -p orchestrator &\n\n# Monitor progress\nfor i in {1..30}; do\n  echo "$(date): $(curl -s http://localhost:9090/api/v1/queue/status | jq .depth) tasks"\n  sleep 10\ndone\n```\n\n**3.4 Clear stuck tasks (last resort)**\n```\n# Get list of stuck tasks\nSTUCK_TASKS=$(curl -s http://localhost:9090/api/v1/tasks?status=pending | \\n  jq -r '.[] | select(.duration_seconds > 900) | .id')\n\n# Fail all stuck tasks\nfor task_id in $STUCK_TASKS; do\n  curl -X POST http://localhost:9090/api/v1/tasks/$task_id/fail \\n    -H "Content-Type: application/json" \\n    -d '{"reason": "operator override - task stuck > 15 min"}'\ndone\n\n# Verify queue depth decreased\ncurl -s http://localhost:9090/api/v1/queue/status | jq .depth\n```\n\n### Phase 4: Verification (10-15 minutes)\n\n**4.1 Confirm queue depth decreasing**\n```\n# Check trend\ncurl -s 'http://localhost:9090/api/v1/query_range?query=orchestrator_queue_depth&start=10m&step=1m' | jq .\n\n# Should show downward trend\n\n# Monitor real-time\nwatch -n 5 'curl -s http://localhost:9090/api/v1/queue/status | jq ".depth"'\n\n# Should reach 0 or acceptable level\n```\n\n**4.2 Check task completion**\n```\n# Success rate\ncurl -s 'http://localhost:9090/api/v1/query?query=rate(orchestrator_tasks_successful_total[5m])' | jq .\n\n# Failure rate\ncurl -s 'http://localhost:9090/api/v1/query?query=rate(orchestrator_tasks_failed_total[5m])' | jq .\n\n# Processing time\ncurl -s 'http://localhost:9090/api/v1/query?query=histogram_quantile(0.95, rate(orchestrator_task_duration_seconds_bucket[5m]))' | jq .\n```\n\n---\n\n## Authentication Failure\n\n**Alert**: `RegistryAuthError` (severity: warning)\n\n**Impact**: Extension registry unavailable, deployment failures\n\n**Detection**: Auth failures > 5 in 5 minutes\n\n### Phase 1: Immediate Response (0-2 minutes)\n\n**1.1 Check registry status**\n```\n# Registry health\ncurl -s http://localhost:8081/health | jq .\n\n# Check auth endpoint\ncurl -X POST http://localhost:8081/api/auth/login \\n  -H "Content-Type: application/json" \\n  -d '{"username": "test", "password": "test"}'\n\n# Check logs\ntail -30 /var/log/provisioning/extension-registry.log | grep -i auth\n```\n\n**1.2 Check auth backend**\n```\n# If using Gitea\ncurl -s http://gitea:3000/health\n\n# If using local auth\ncurl -s http://localhost:8081/api/auth/status\n\n# Get auth errors\ngrep "auth.*error\|authentication.*failed" /var/log/provisioning/extension-registry.log\n```\n\n### Phase 2: Diagnosis (2-5 minutes)\n\n**2.1 Check auth configuration**\n```\n# View registry schema\nnickel typecheck provisioning/schemas/platform/schemas/extension-registry.ncl\n\n# Check defaults contain auth config\ncat provisioning/schemas/platform/defaults/extension-registry-defaults.ncl | grep -A 5 "auth\|gitea"\n\n# Verify mode-specific settings\ncat provisioning/schemas/platform/defaults/deployment/*-defaults.ncl | grep -A 5 "extension.registry"\n```\n\n**2.2 Check credential validity**\n```\n# Test Gitea login if applicable\ncurl -X POST http://gitea:3000/api/v1/user/login \\n  -H "Content-Type: application/json" \\n  -d '{"login_name": "admin", "password": "PASSWORD"}'\n\n# Test OCI registry auth if applicable\ndocker login registry.internal\n```\n\n**2.3 Check network connectivity**\n```\n# Test connection to auth backend\ncurl -v http://gitea:3000/health\ncurl -v http://ldap:389/health\n\n# Check DNS\nnslookup gitea\ndig gitea\n```\n\n### Phase 3: Remediation (5-10 minutes)\n\n**3.1 If credential expired**\n```\n# Generate new token\ncurl -X POST http://gitea:3000/api/v1/users/admin/tokens \\n  -H "Authorization: token OLD_TOKEN"\n\n# Update registry defaults\nvim provisioning/schemas/platform/defaults/extension-registry-defaults.ncl\n# Update: auth_token = "NEW_TOKEN", then re-generate\n\n# Re-generate runtime config\n./provisioning/.typedialog/platform/scripts/generate-configs.nu extension-registry solo\n\n# Restart registry\npkill -SIGTERM extension-registry\nsleep 2\ncargo run --release -p extension-registry &\n```\n\n**3.2 If auth backend down**\n```\n# Check if Gitea is running\npgrep gitea || echo "Gitea not running"\nsystemctl status gitea\n\n# Restart Gitea\nsystemctl restart gitea\nsleep 10\n\n# Verify it's up\ncurl http://gitea:3000/health\n\n# Restart registry to reconnect\npkill -SIGTERM extension-registry\nsleep 2\ncargo run --release -p extension-registry &\n```\n\n**3.3 If auth service misconfigured**\n```\n# Check auth configuration\ncat /etc/gitea/app.ini | grep -A 5 "\[auth"\n\n# Verify LDAP/auth settings if applicable\nldapwhoami -H ldap://ldap:389 -x\n\n# Restart auth service\nsystemctl restart gitea-auth\n\n# Restart registry\npkill extension-registry\ncargo run --release -p extension-registry &\n```\n\n### Phase 4: Verification (10-15 minutes)\n\n**4.1 Confirm auth working**\n```\n# Test login\ncurl -X POST http://localhost:8081/api/auth/login \\n  -H "Content-Type: application/json" \\n  -d '{"username": "admin", "password": "PASSWORD"}'\n\n# Should return 200 with token\n\n# Test token usage\nTOKEN="..."\ncurl -H "Authorization: Bearer $TOKEN" http://localhost:8081/api/extensions\n\n# Should return 200\n```\n\n**4.2 Check alert cleared**\n```\n# Verify no more auth errors\ncurl -s 'http://localhost:9090/api/v1/query?query=rate(registry_auth_failures_total[5m])' | jq .\n\n# Should be 0\n\n# Check alert status\ncurl -s http://localhost:9093/api/v1/alerts | jq '.data.alerts[] | select(.labels.alertname=="RegistryAuthError")'\n\n# Should be empty or resolved\n```\n\n---\n\n## Data Corruption\n\n**Alert**: None (detected through manual inspection or consistency checks)\n\n**Impact**: Severe, data loss, system unreliable\n\n**Detection**: Failed checksums, database validation errors\n\n### Phase 1: Immediate Response (0-2 minutes)\n\n**1.1 Isolate affected service**\n```\n# Stop affected service to prevent further corruption\npkill -SIGTERM affected-service\nsleep 5\npgrep affected-service || echo "Stopped"\n\n# Alert team immediately\n# Post to incident channel: "DATA CORRUPTION DETECTED - isolating affected-service"\n```\n\n**1.2 Assess scope**\n```\n# What data is affected\ncurl -s http://localhost:9090/api/v1/health | jq '.data_integrity'\n\n# Check database\nsurreal sql --endpoint http://surrealdb:8000 --username root --password root \\n  --query "SELECT count(*) FROM CORRUPTED_TABLE"\n\n# Check for error patterns\ngrep -i "corrupt\|checksum\|validation.*failed" /var/log/provisioning/*.log\n```\n\n### Phase 2: Diagnosis (2-10 minutes)\n\n**2.1 Determine extent of corruption**\n```\n# Run integrity check\ncurl -X POST http://localhost:9090/api/v1/admin/integrity-check\n\n# Get results\ncurl -s http://localhost:9090/api/v1/admin/integrity-check/status | jq .\n\n# Check what failed\ncurl -s http://localhost:9090/api/v1/admin/integrity-check/errors | jq .\n```\n\n**2.2 Locate backup**\n```\n# Find most recent backup\nls -lrt /backup/surrealdb*.sql | tail -5\nls -lrt /backup/provisioning*.tar.gz | tail -5\n\n# Verify backup integrity\nfile /backup/surrealdb-latest.sql\ntar -tzf /backup/provisioning-latest.tar.gz | head -5\n```\n\n**2.3 Determine restore point**\n```\n# When was corruption detected\ngrep "corrupt\|checksum" /var/log/provisioning/*.log | head -1\n\n# Work back to find clean backup\n# Usually safe to restore from backup 1-2 hours before detection\n```\n\n### Phase 3: Remediation (10-30 minutes)\n\n**3.1 Restore from backup**\n```\n# Stop all services using the data\npkill -SIGTERM -f "cargo run --release"\nsleep 5\n\n# Create backup of corrupted data (for forensics)\ntar -czf /archive/corrupted-data-$(date +%s).tar.gz /var/lib/provisioning/\n\n# Restore clean data\ntar -xzf /backup/provisioning-2026-01-05-0800.tar.gz -C /var/lib/\n\n# Verify restore\nls -la /var/lib/provisioning/\n```\n\n**3.2 If database restore needed**\n```\n# Backup current corrupted database\nmv /var/lib/surrealdb /var/lib/surrealdb.corrupted\n\n# Create fresh database directory\nmkdir -p /var/lib/surrealdb\nchmod 755 /var/lib/surrealdb\n\n# Restore from backup\nsurreal import --endpoint http://surrealdb:8000 \\n  --username root --password root < /backup/surrealdb-2026-01-05-0800.sql\n\n# Verify restore\nsurreal sql --endpoint http://surrealdb:8000 --username root --password root \\n  --query "SELECT count(*) FROM services"\n```\n\n**3.3 Restart services and verify**\n```\n# Start services one by one\ncargo run --release -p vault-service &\nsleep 5\ncurl http://localhost:8200/health\n\ncargo run --release -p extension-registry &\nsleep 5\ncurl http://localhost:8081/health\n\n# And others...\n\n# Run integrity check again\ncurl -X POST http://localhost:9090/api/v1/admin/integrity-check\n```\n\n### Phase 4: Post-Incident (After Incident)\n\n**4.1 Root cause analysis**\n```\n# What caused corruption\n# Check logs for:\n# - Crashes during writes\n# - Disk full errors\n# - Database errors\n# - Network interruptions\n\ngrep -B 5 -A 5 "corrupt" /var/log/provisioning/*.log > /tmp/corruption-analysis.txt\n```\n\n**4.2 Prevent recurrence**\n```\n# Implement integrity checks\n# Add scheduled checksum validation\n# Add crash consistency checks\n# Implement write-ahead logging (WAL)\n# Add monitoring for anomalies\n```\n\n**4.3 Document incident**\n```\n# Create detailed incident report\ncat > /tmp/data-corruption-incident.md << EOF\n## Data Corruption Incident\n\n**Detection**: [WHEN AND HOW]\n**Root Cause**: [TBD]\n**Impact**: [DATA LOST/MODIFIED]\n**Duration**: [TIME]\n**Recovery**: [WHAT WAS RESTORED]\n**Prevention**: [PLANNED FIXES]\nEOF\n```\n\n---\n\n## Full Cluster Failure\n\n**Alert**: Multiple critical alerts firing simultaneously\n\n**Impact**: Complete platform unavailable, business-critical\n\n**Detection**: Multiple services down, cascading failures\n\n### Phase 1: Immediate Response (0-5 minutes)\n\n**1.1 Assess situation**\n```\n# Check all services\nfor port in 8200 8081 8083 8082 9090 8080; do\n  curl -s http://localhost:$port/health || echo "Port $port DOWN"\ndone\n\n# Check infrastructure\ncurl http://surrealdb:8000/health\netcdctl --endpoints=http://etcd:2379 endpoint health\n\n# Check system\nfree -h\ndf -h\nps aux | head -20\n```\n\n**1.2 Declare incident**\n```\n# Create war room\n# Notify all stakeholders\n# Post: "INCIDENT: Platform unavailable - full cluster failure - details TBD"\n# Start recording\n# Get additional on-call people\n```\n\n**1.3 Check recent changes**\n```\n# What changed?\ngit log --oneline | head -5\n\n# Recent deploys\ngit log --oneline | grep "deploy\|release" | head -3\n\n# System updates\njournalctl --since "30 min ago" | grep -i "update\|upgrade\|restart"\n```\n\n### Phase 2: Diagnosis (5-15 minutes)\n\n**2.1 Identify common cause**\n```\n# All services crashed?\npgrep -a cargo | grep "release -p"\n\n# Database down?\ncurl http://surrealdb:8000/health\nps aux | grep surrealdb\n\n# Network issue?\nping -c 5 8.8.8.8\nss -s\nnetstat -an | grep ESTABLISHED | wc -l\n\n# Disk full?\ndf -h\n\n# OOM?\ndmesg | tail -20 | grep -i "killed\|oom"\n```\n\n**2.2 Check shared infrastructure**\n```\n# Load balancer status\ncurl http://loadbalancer:8080/health\n\n# DNS\nnslookup surrealdb\ndig services.internal\n\n# Network\ntraceroute surrealdb\niptables -L | head -20\n\n# Firewall\nsystemctl status firewalld\n```\n\n**2.3 Determine if issue is infrastructure or application**\n```\n# Infrastructure signs:\n# - No network connectivity\n# - DNS failures\n# - All services unable to start\n# - System running out of resources\n\n# Application signs:\n# - Services start but crash\n# - Database errors\n# - Dependency failures\n# - Configuration issues\n```\n\n### Phase 3: Recovery (15-60 minutes)\n\n**3.1 Infrastructure restart (if infrastructure issue)**\n```\n# Restart network\nsystemctl restart networking\n\n# Restart DNS\nsystemctl restart systemd-resolved\nsystemctl restart coredns\n\n# Restart load balancer\nsystemctl restart haproxy\n\n# Test connectivity\ncurl http://8.8.8.8\nnslookup google.com\n```\n\n**3.2 Restart all services (methodical order)**\n```\n# Stop everything\npkill -9 -f "cargo run"\n\n# Clean up resources\nrm -f /var/run/provisioning/*.lock\nrm -f /tmp/provisioning-*\n\n# Restart infrastructure first\nsystemctl restart surrealdb\nsleep 10\ncurl http://surrealdb:8000/health\n\nsystemctl restart etcd\nsleep 10\netcdctl --endpoints=http://etcd:2379 endpoint health\n\n# Then restart services in dependency order\ncargo run --release -p vault-service &\nsleep 5\ncurl http://localhost:8200/health\n\ncargo run --release -p extension-registry &\nsleep 5\ncurl http://localhost:8081/health\n\n# Continue with others...\n```\n\n**3.3 Monitor recovery**\n```\n# Watch startup\nwatch -n 5 'for port in 8200 8081 8083 8082 9090 8080; do\n  curl -s http://localhost:$port/health | jq -r .status && echo "Port $port: OK" || echo "Port $port: DOWN"\ndone'\n\n# Watch logs\ntail -f /var/log/provisioning/*.log | grep -v "debug"\n\n# Watch resources\nwatch -n 5 'free -h && echo "---" && df -h'\n```\n\n### Phase 4: Verification (After Recovery)\n\n**4.1 System fully recovered**\n```\n# All services responding\nfor port in 8200 8081 8083 8082 9090 8080; do\n  status=$(curl -s http://localhost:$port/health | jq -r .status 2>/dev/null)\n  if [ "$status" = "ok" ]; then\n    echo "✓ Port $port: OK"\n  else\n    echo "✗ Port $port: FAILED"\n  fi\ndone\n\n# No errors in logs\ntail -100 /var/log/provisioning/*.log | grep -i "error\|warn" | wc -l\n\n# System resources normal\nfree -h\ndf -h\n```\n\n**4.2 Data integrity verified**\n```\n# Run integrity check\ncurl -X POST http://localhost:9090/api/v1/admin/integrity-check\n\n# Verify no corruption\ncurl -s http://localhost:9090/api/v1/admin/integrity-check/status | jq '.corruption_detected'\n\n# Should be false\n```\n\n**4.3 Close incident**\n```\n# Resolve in incident tracker\n# Post: "✓ RESOLVED at HH:MM UTC - full cluster recovered"\n\n# Schedule post-incident review\n# Collect logs and metrics\n# Document timeline\n```\n\n---\n\n## Post-Incident Actions (All Incidents)\n\n### Immediate (Within 1 hour)\n\n1. Document incident start/end times\n2. Collect logs and metrics\n3. Note impact and scope\n4. List immediate actions taken\n5. Identify if data was lost\n\n### Short-term (Within 24 hours)\n\n1. Send incident notification to stakeholders\n2. Conduct war room debriefing\n3. Identify root cause\n4. Create action items to prevent recurrence\n5. Update runbooks based on learnings\n\n### Long-term (Within 1 week)\n\n1. Complete RCA (Root Cause Analysis)\n2. Implement fixes to prevent recurrence\n3. Update monitoring and alerting\n4. Schedule postmortem review\n5. Update documentation\n\n---\n\n## Quick Reference: Critical Numbers\n\n```\nService Availability: Should be 99.9% or better\nError Rate: Should be < 0.1% (< 1 error per 1000 requests)\nP95 Latency: Should be < 500 ms\nP99 Latency: Should be < 1000 ms\nQueue Depth: Should stay < 100 tasks\nMemory Usage: Should stay < 80% of limit\nDisk Usage: Should stay < 80% full\nDatabase Connection Time: Should be < 100 ms\n```\n\n---\n\n## Resources & Escalation\n\n**On-Call**: Check PagerDuty for current on-call rotation\n**Slack**: #platform-incidents for incident discussion\n**Runbook Updates**: After each incident, update relevant runbook\n**Documentation**: Keep these runbooks updated as platform evolves\n\n---\n\n**Last Updated**: 2026-01-05\n**Version**: 1.0.0\n**Status**: Production Ready ✅
