Vapora/docs/operations/incident-response-runbook.md

Incident Response Runbook

Procedures for responding to and resolving VAPORA production incidents.

---

Incident Severity Levels

Severity 1: Critical 🔴

Definition: Service completely down or severely degraded affecting all users

Examples:

• All backend pods crashed
• Database completely unreachable
• API returning 100% errors
• Frontend completely inaccessible

Response Time: Immediate (< 2 minutes) On-Call: Page immediately (not optional) Communication: Update status page every 2 minutes

Severity 2: Major 🟠

Definition: Service partially down or significantly degraded

Examples:

• 50% of requests returning errors
• Latency 10x normal
• Some services down but others working
• Intermittent connectivity issues

Response Time: 5 minutes On-Call: Alert on-call engineer Communication: Internal updates every 5 minutes

Severity 3: Minor 🟡

Definition: Service slow or minor issues affecting some users

Examples:

• 5-10% error rate
• Elevated latency (2x normal)
• One pod having issues, others recovering
• Non-critical features unavailable

Response Time: 15 minutes On-Call: Alert team, not necessarily emergency page Communication: Post-incident update

Severity 4: Informational 🟢

Definition: No user impact, system anomalies or preventive issues

Examples:

• Disk usage trending high
• SSL cert expiring in 30 days
• Deployment taking longer than normal
• Non-critical service warnings

Response Time: During business hours On-Call: No alert needed Communication: Team Slack message

---

Incident Response Process

Step 1: Report & Assess (Immediately)

When incident reported (via alert, user report, or discovery):

# 1. Create incident ticket
# Title: "INCIDENT: [Service] - [Brief description]"
# Example: "INCIDENT: API - 50% error rate since 14:30 UTC"
# Severity: [1-4]
# Reporter: [Your name]
# Time Detected: [UTC time]

# 2. Open dedicated Slack channel
#slack /create #incident-20260112-backend
# Then: /invite @on-call-engineer

# 3. Post initial message
# "🔴 INCIDENT DECLARED
#  Service: VAPORA Backend
#  Severity: 1 (Critical)
#  Time Detected: 14:32 UTC
#  Current Status: Unknown
#  Next Update: 14:34 UTC"

Step 2: Quick Diagnosis (First 2 minutes)

# Establish facts quickly
export NAMESPACE=vapora

# Q1: Is the service actually down?
curl -v http://api.vapora.com/health
# If: Connection refused → Service down
# If: 500 errors → Service crashed
# If: Timeout → Service hung

# Q2: What's the scope?
kubectl get pods -n $NAMESPACE
# Count Running vs non-Running pods
# All down → Complete outage
# Some down → Partial outage

# Q3: What's happening right now?
for deployment in vapora-backend vapora-agents vapora-llm-router; do
  echo "=== $deployment ==="
  kubectl get deployment $deployment -n $NAMESPACE
done
# Shows: DESIRED vs CURRENT vs AVAILABLE
# Example: 3 DESIRED, 0 CURRENT, 0 AVAILABLE → Pod startup failure

# Q4: Any obvious errors?
kubectl logs deployment/vapora-backend -n $NAMESPACE --tail=20 | grep -i "error\|fatal"
# Shows: What's in the logs right now

Step 3: Escalate Decision

Based on quick diagnosis, decide next action:

IF pods not starting (CrashLoopBackOff):
  → Likely config issue
  → Check ConfigMap values
  → Likely recent deployment
  → DECISION: Possible rollback

IF pods pending (not scheduled):
  → Likely resource issue
  → Check node capacity
  → DECISION: Scale down workloads or investigate nodes

IF pods running but unresponsive:
  → Likely application issue
  → Check application logs
  → DECISION: Investigate app logic

IF network/database issues:
  → Check connectivity
  → Check credentials
  → DECISION: Infrastructure escalation

IF unknown:
  → Ask: "What changed recently?"
  → Check deployment history
  → Check infrastructure changes

Step 4: Initial Response Actions

For Severity 1 (Critical):

# A. Escalate immediately
- Page senior engineer if not already responding
- Contact infrastructure team
- Notify product/support managers

# B. Buy time with failover if available
- Switch to backup environment if configured
- Scale to different region if multi-region

# C. Gather data for debugging
- Save current logs
- Save pod events
- Record current metrics
- Take screenshot of dashboards

# D. Keep team updated
# Update #incident-* channel every 2 minutes

For Severity 2 (Major):

# A. Alert on-call team
# B. Gather same diagnostics
# C. Start investigation
# D. Update every 5 minutes

For Severity 3 (Minor):

# A. Create ticket for later investigation
# B. Monitor closely
# C. Gather diagnostics
# D. Plan fix during normal hours if not urgent

Step 5: Detailed Diagnosis

Once immediate actions taken:

# Get comprehensive view of system state
kubectl describe node <nodename>      # Hardware/capacity issues
kubectl describe pod <podname> -n $NAMESPACE  # Pod-specific issues
kubectl events -n $NAMESPACE          # What happened recently
kubectl top nodes                     # CPU/memory usage
kubectl top pods -n $NAMESPACE        # Per-pod resource usage

# Check recent changes
git log -5 --oneline
git diff HEAD~1 HEAD provisioning/

# Check deployment history
kubectl rollout history deployment/vapora-backend -n $NAMESPACE | tail -5

# Timeline analysis
# What happened at 14:30 UTC? (incident time)
# Was there a deployment?
# Did metrics change suddenly?
# Any alerts triggered?

Step 6: Implement Fix

Depending on root cause:

Root Cause: Recent Bad Deployment

# Solution: Rollback
# See: Rollback Runbook
kubectl rollout undo deployment/vapora-backend -n $NAMESPACE
kubectl rollout status deployment/vapora-backend --timeout=5m

# Verify
curl http://localhost:8001/health

Root Cause: Insufficient Resources

# Solution: Either scale out or reduce load

# Option A: Add more nodes
kubectl scale nodes --increment=1
# (Requires infrastructure access)

# Option B: Scale down non-critical services
kubectl scale deployment/vapora-agents --replicas=1 -n $NAMESPACE
# Then scale back up when resolved

# Option C: Temporarily scale down pod replicas
kubectl scale deployment/vapora-backend --replicas=2 -n $NAMESPACE
# (Trade: Reduced capacity but faster recovery)

Root Cause: Configuration Error

# Solution: Fix ConfigMap

# 1. Identify wrong value
kubectl get configmap -n $NAMESPACE vapora-config -o yaml | grep -A 2 <suspicious-key>

# 2. Fix value
# Edit configmap in external editor or via kubectl patch:
kubectl patch configmap vapora-config -n $NAMESPACE \
  --type merge \
  -p '{"data":{"vapora.toml":"[corrected content]"}}'

# 3. Restart pods to pick up new config
kubectl rollout restart deployment/vapora-backend -n $NAMESPACE
kubectl rollout status deployment/vapora-backend --timeout=5m

Root Cause: Database Issues

# Solution: Depends on specific issue

# If database down:
- Contact DBA or database team
- Check database status: kubectl exec <pod> -- curl localhost:8000

# If credentials wrong:
kubectl patch configmap vapora-config -n $NAMESPACE \
  --type merge \
  -p '{"data":{"DB_PASSWORD":"[correct-password]"}}'

# If database full:
- Contact DBA for cleanup
- Free up space on database volume

# If connection pool exhausted:
- Scale down services to reduce connections
- Increase connection pool size if possible

Root Cause: External Service Down

# Examples: Third-party API, external database

# Solution: Depends on severity

# If critical: Failover
- Switch to backup provider if available
- Route traffic differently

# If non-critical: Degrade gracefully
- Disable feature temporarily
- Use cache if available
- Return cached data

# Communicate
- Notify users of reduced functionality
- Provide ETA for restoration

Step 7: Verify Recovery

# Once fix applied, verify systematically

# 1. Pod health
kubectl get pods -n $NAMESPACE
# All should show: Running, 1/1 Ready

# 2. Service endpoints
kubectl get endpoints -n $NAMESPACE
# All should have IP addresses

# 3. Health endpoints
curl http://localhost:8001/health
# Should return: 200 OK

# 4. Check errors
kubectl logs deployment/vapora-backend -n $NAMESPACE --since=2m | grep -i error
# Should return: few or no errors

# 5. Monitor metrics
kubectl top pods -n $NAMESPACE
# CPU/Memory should be normal (not spiking)

# 6. Check for new issues
kubectl get events -n $NAMESPACE
# Should show normal state, no warnings

Step 8: Incident Closure

# When everything verified healthy:

# 1. Document resolution
# Update incident ticket with:
# - Root cause
# - Fix applied
# - Verification steps
# - Resolution time
# - Impact (how many users, how long)

# 2. Post final update
# "#incident channel:
#  ✅ INCIDENT RESOLVED
#
#  Duration: [start] to [end] = [X minutes]
#  Root Cause: [brief description]
#  Fix Applied: [brief description]
#  Impact: ~X users affected for X minutes
#
#  Status: All services healthy
#  Monitoring: Continuing for 1 hour
#  Post-mortem: Scheduled for [date]"

# 3. Schedule post-mortem
# Within 24 hours: review what happened and why
# Document lessons learned

# 4. Update dashboards
# Document incident on status page history
# If public incident: close status page incident

# 5. Send all-clear message
# Notify: support team, product team, key stakeholders

---

Incident Response Roles & Responsibilities

Incident Commander

• Overall control of incident response
• Makes critical decisions
• Drives decision-making speed
• Communicates status updates
• Calls when to escalate
• You if you discovered the incident and best understands it

Technical Responders

• Investigate specific systems
• Implement fixes
• Report findings to commander
• Execute verified solutions

Communication Lead (if Severity 1)

• Updates #incident channel every 2 minutes
• Updates status page every 5 minutes
• Fields questions from support/product
• Notifies key stakeholders

On-Call Manager (if Severity 1)

• Pages additional resources if needed
• Escalates to senior engineers
• Engages infrastructure/DBA teams
• Tracks response timeline

---

Common Incidents & Responses

Incident Type: Service Unresponsive

Detection: curl returns "Connection refused"
Diagnosis Time: 1 minute
Response:
1. Check if pods are running: kubectl get pods
2. If not running: likely crash → check logs
3. If running but unresponsive: likely port/network issue
4. Verify service exists: kubectl get service vapora-backend

Solution:
- If pods crashed: check logs, likely config or deployment issue
- If pods hanging: restart pods: kubectl delete pods -l app=vapora-backend
- If service/endpoints missing: apply service manifest

Incident Type: High Error Rate

Detection: Dashboard shows >10% 5xx errors
Diagnosis Time: 2 minutes
Response:
1. Check which endpoint is failing
2. Check logs for error pattern
3. Identify affected service (backend, agents, router)
4. Compare with baseline (worked X minutes ago)

Solution:
- If recent deployment: rollback
- If config change: revert config
- If database issue: contact DBA
- If third-party down: implement fallback

Incident Type: High Latency

Detection: Dashboard shows p99 latency >2 seconds
Diagnosis Time: 2 minutes
Response:
1. Check if requests still succeeding (is it slow or failing?)
2. Check CPU/memory usage: kubectl top pods
3. Check if database slow: run query diagnostics
4. Check network: are there packet losses?

Solution:
- If resource exhausted: scale up or reduce load
- If database slow: DBA investigation
- If network issue: infrastructure team
- If legitimate increased load: no action needed (expected)

Incident Type: Pod Restarting Repeatedly

Detection: kubectl get pods shows high RESTARTS count
Diagnosis Time: 1 minute
Response:
1. Check restart count: kubectl get pods -n vapora
2. Get pod logs: kubectl logs <pod-name> -n vapora --previous
3. Get pod events: kubectl describe pod <pod-name> -n vapora

Solution:
- Application error: check logs, fix issue, redeploy
- Config issue: fix ConfigMap, restart pods
- Resource issue: increase limits or scale out
- Liveness probe failing: adjust probe timing or fix health check

Incident Type: Database Connectivity

Detection: Logs show "database connection refused"
Diagnosis Time: 2 minutes
Response:
1. Check database service running: kubectl get pod -n <db-namespace>
2. Check database credentials in ConfigMap
3. Test connectivity: kubectl exec <pod> -- psql $DB_URL
4. Check firewall/network policy

Solution:
- If DB down: escalate to DBA, possibly restore from backup
- If credentials wrong: fix ConfigMap, restart app pods
- If network issue: network team investigation
- If no space: DBA cleanup

---

Communication During Incident

Every 2 Minutes (Severity 1) or 5 Minutes (Severity 2)

Post update to #incident channel:

⏱️ 14:35 UTC UPDATE

Status: Investigating
Current Action: Checking pod logs
Findings: Backend pods in CrashLoopBackOff
Next Step: Review recent deployment
ETA for Update: 14:37 UTC

/cc @on-call-engineer

Status Page Updates (If Public)

INCIDENT: VAPORA API Partially Degraded

Investigating: Our team is investigating elevated error rates
Duration: 5 minutes
Impact: ~30% of API requests failing

Last Updated: 14:35 UTC
Next Update: 14:37 UTC

Escalation Communication

If Severity 1 and unable to identify cause in 5 minutes:

"Escalating to senior engineering team.
Page @senior-engineer-on-call immediately.
Activating Incident War Room."

Include:
- Service name
- Duration so far
- What's been tried
- Current symptoms
- Why stuck

---

Incident Severity Decision Tree

Question 1: Can any users access the service?
  NO → Severity 1 (Critical - complete outage)
  YES → Question 2

Question 2: What percentage of requests are failing?
  >50% → Severity 1 (Critical)
  10-50% → Severity 2 (Major)
  5-10% → Severity 3 (Minor)
  <5% → Question 3

Question 3: Is the service recovering on its own?
  NO (staying broken) → Severity 2
  YES (automatically recovering) → Question 4

Question 4: Does it require any user action/data loss?
  YES → Severity 2
  NO → Severity 3

---

Post-Incident Procedures

Immediate (Within 30 minutes)

• Close incident ticket
• Post final update to #incident channel
• Save all logs and diagnostics
• Create post-mortem ticket
• Notify team: "incident resolved"

Follow-Up (Within 24 hours)

• Schedule post-mortem meeting
• Identify root cause
• Document preventive measures
• Identify owner for each action item
• Create tickets for improvements

Prevention (Within 1 week)

• Implement identified fixes
• Update monitoring/alerting
• Update runbooks with findings
• Conduct team training if needed
• Close post-mortem ticket

---

Incident Checklist

☐ Incident severity determined
☐ Ticket created and updated
☐ #incident channel created
☐ On-call team alerted
☐ Initial diagnosis completed
☐ Fix identified and implemented
☐ Fix verified working
☐ Incident closed and communicated
☐ Post-mortem scheduled
☐ Team debriefed
☐ Root cause documented
☐ Prevention measures identified
☐ Tickets created for follow-up