Vapora/docs/disaster-recovery/database-recovery-procedures.md

# Database Recovery Procedures

Detailed procedures for recovering SurrealDB in various failure scenarios.

---

## Quick Reference: Recovery Methods

| Scenario | Method | Time | Data Loss |
|----------|--------|------|-----------|
| **Pod restart** | Automatic pod recovery | 2 min | 0 |
| **Pod crash** | Persistent volume intact | 3 min | 0 |
| **Corrupted pod** | Restart from snapshot | 5 min | 0 |
| **Corrupted database** | Restore from backup | 15 min | 0-60 min |
| **Complete loss** | Restore from backup | 30 min | 0-60 min |

---

## SurrealDB Architecture

```
VAPORA Database Layer

SurrealDB Pod (Kubernetes)
├── PersistentVolume: /var/lib/surrealdb/
├── Data file: data.db (RocksDB)
├── Index files: *.idx
└── Wal (Write-ahead log): *.wal

Backed up to:
├── Hourly exports: S3 backups/database/
├── CloudSQL snapshots: AWS/GCP snapshots
└── Archive backups: Glacier (monthly)
```

---

## Scenario 1: Pod Restart (Most Common)

**Cause**: Node maintenance, resource limits, health check failure

**Duration**: 2-3 minutes
**Data Loss**: None

### Recovery Procedure

```bash
# Most of the time, just restart the pod

# 1. Delete the pod
kubectl delete pod -n vapora surrealdb-0

# 2. Pod automatically restarts (via StatefulSet)
kubectl get pods -n vapora -w

# 3. Verify it's Ready
kubectl get pod surrealdb-0 -n vapora
# Should show: 1/1 Running

# 4. Verify database is accessible
kubectl exec -n vapora surrealdb-0 -- \
  surreal sql "SELECT 1"

# 5. Check data integrity
kubectl exec -n vapora surrealdb-0 -- \
  surreal sql "SELECT COUNT(*) FROM projects"
# Should return non-zero count
```

---

## Scenario 2: Pod CrashLoop (Container Issue)

**Cause**: Application crash, memory issues, corrupt index

**Duration**: 5-10 minutes
**Data Loss**: None (usually)

### Recovery Procedure

```bash
# 1. Examine pod logs to identify issue
kubectl logs surrealdb-0 -n vapora --previous
# Look for: "panic", "fatal", "out of memory"

# 2. Increase resource limits if memory issue
kubectl patch statefulset surrealdb -n vapora --type='json' \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value":"2Gi"}]'

# 3. If corrupt index, rebuild
kubectl exec -n vapora surrealdb-0 -- \
  surreal query "REBUILD INDEX"

# 4. If persistent issue, try volume snapshot
kubectl delete pod -n vapora surrealdb-0
# Use previous snapshot (if available)

# 5. Monitor restart
kubectl get pods -n vapora -w
```

---

## Scenario 3: Corrupted Database (Detected via Queries)

**Cause**: Unclean shutdown, disk issue, data corruption

**Duration**: 15-30 minutes
**Data Loss**: Minimal (last hour of transactions)

### Detection

```bash
# Symptoms to watch for
✗ Queries return error: "corrupted database"
✗ Disk check shows corruption
✗ Checksums fail
✗ Integrity check fails

# Verify corruption
kubectl exec -n vapora surrealdb-0 -- \
  surreal query "INFO FOR DB"
# Look for any error messages

# Try repair
kubectl exec -n vapora surrealdb-0 -- \
  surreal query "REBUILD INDEX"
```

### Recovery: Option A - Restart and Repair (Try First)

```bash
# 1. Delete pod to force restart
kubectl delete pod -n vapora surrealdb-0

# 2. Watch restart
kubectl get pods -n vapora -w
# Should restart within 30 seconds

# 3. Verify database accessible
kubectl exec -n vapora surrealdb-0 -- \
  surreal sql "SELECT COUNT(*) FROM projects"

# 4. If successful, done
# If still errors, proceed to Option B
```

### Recovery: Option B - Restore from Recent Backup

```bash
# 1. Stop database pod
kubectl scale statefulset surrealdb --replicas=0 -n vapora

# 2. Download latest backup
aws s3 cp s3://vapora-backups/database/ ./ --recursive
# Get most recent .sql.gz file

# 3. Clear corrupted data
kubectl delete pvc -n vapora surrealdb-data-surrealdb-0

# 4. Recreate pod (will create new PVC)
kubectl scale statefulset surrealdb --replicas=1 -n vapora

# 5. Wait for pod to be ready
kubectl wait --for=condition=Ready pod/surrealdb-0 \
  -n vapora --timeout=300s

# 6. Restore backup
# Extract and import
gunzip vapora-db-*.sql.gz
kubectl cp vapora-db-*.sql vapora/surrealdb-0:/tmp/

kubectl exec -n vapora surrealdb-0 -- \
  surreal import \
    --conn ws://localhost:8000 \
    --user root \
    --pass $DB_PASSWORD \
    --input /tmp/vapora-db-*.sql

# 7. Verify restored data
kubectl exec -n vapora surrealdb-0 -- \
  surreal sql "SELECT COUNT(*) FROM projects"
# Should match pre-corruption count
```

---

## Scenario 4: Storage Failure (PVC Issue)

**Cause**: Storage volume corruption, node storage failure

**Duration**: 20-30 minutes
**Data Loss**: None with backup

### Recovery Procedure

```bash
# 1. Detect storage issue
kubectl describe pvc -n vapora surrealdb-data-surrealdb-0
# Look for: "Pod pending", "volume binding failure"

# 2. Check if snapshot available (cloud)
aws ec2 describe-snapshots \
  --filters "Name=tag:database,Values=vapora" \
  --query 'Snapshots[].{SnapshotId:SnapshotId,StartTime:StartTime}' \
  --sort-by StartTime | tail -10

# 3. Create new PVC from snapshot
kubectl apply -f - << EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: surrealdb-data-surrealdb-0-restore
  namespace: vapora
spec:
  accessModes:
    - ReadWriteOnce
  dataSource:
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
    name: surrealdb-snapshot-latest
  resources:
    requests:
      storage: 100Gi
EOF

# 4. Update StatefulSet to use new PVC
kubectl patch statefulset surrealdb -n vapora --type='json' \
  -p='[{"op": "replace", "path": "/spec/volumeClaimTemplates/0/metadata/name", "value":"surrealdb-data-surrealdb-0-restore"}]'

# 5. Delete old pod to force remount
kubectl delete pod -n vapora surrealdb-0

# 6. Verify new pod runs
kubectl get pods -n vapora -w

# 7. Test database
kubectl exec -n vapora surrealdb-0 -- \
  surreal sql "SELECT COUNT(*) FROM projects"
```

---

## Scenario 5: Complete Data Loss (Restore from Backup)

**Cause**: User delete, accidental truncate, security incident

**Duration**: 30-60 minutes
**Data Loss**: Up to 1 hour

### Pre-Recovery Checklist

```
Before restoring, verify:
□ What data was lost? (specific tables or entire DB?)
□ When was it lost? (exact time if possible)
□ Is it just one table or entire database?
□ Do we have valid backups from before loss?
□ Has the backup been tested before?
```

### Recovery Procedure

```bash
# 1. Stop the database
kubectl scale statefulset surrealdb --replicas=0 -n vapora
sleep 10

# 2. Identify backup to restore
# Look for backup from time BEFORE data loss
aws s3 ls s3://vapora-backups/database/ --recursive | sort
# Example: surrealdb-2026-01-12-230000.sql.gz
# (from 11 PM, before 12 AM loss)

# 3. Download backup
aws s3 cp s3://vapora-backups/database/2026-01-12-surrealdb-230000.sql.gz ./

gunzip surrealdb-230000.sql

# 4. Verify backup integrity before restoring
# Extract first 100 lines to check format
head -100 surrealdb-230000.sql

# 5. Delete corrupted PVC
kubectl delete pvc -n vapora surrealdb-data-surrealdb-0

# 6. Restart database pod (will create new PVC)
kubectl scale statefulset surrealdb --replicas=1 -n vapora

# 7. Wait for pod to be ready and listening
kubectl wait --for=condition=Ready pod/surrealdb-0 \
  -n vapora --timeout=300s
sleep 10

# 8. Copy backup to pod
kubectl cp surrealdb-230000.sql vapora/surrealdb-0:/tmp/

# 9. Restore backup
kubectl exec -n vapora surrealdb-0 -- \
  surreal import \
    --conn ws://localhost:8000 \
    --user root \
    --pass $DB_PASSWORD \
    --input /tmp/surrealdb-230000.sql

# Expected output:
# Imported 1500+ records...
# This should take 5-15 minutes depending on backup size

# 10. Verify data restored
kubectl exec -n vapora surrealdb-0 -- \
  surreal sql \
    --conn ws://localhost:8000 \
    --user root \
    --pass $DB_PASSWORD \
    "SELECT COUNT(*) as project_count FROM projects"

# Should match pre-loss count
```

### Data Loss Assessment

```bash
# After restore, compare with lost version

# 1. Get current record count
RESTORED_COUNT=$(kubectl exec -n vapora surrealdb-0 -- \
  surreal sql "SELECT COUNT(*) FROM projects")

# 2. Get pre-loss count (from logs or ticket)
PRE_LOSS_COUNT=1500

# 3. Calculate data loss
if [ "$RESTORED_COUNT" -lt "$PRE_LOSS_COUNT" ]; then
  LOSS=$(( PRE_LOSS_COUNT - RESTORED_COUNT ))
  echo "Data loss: $LOSS records"
  echo "Data loss duration: ~1 hour"
  echo "Restore successful but incomplete"
else
  echo "Data loss: 0 records"
  echo "Full recovery complete"
fi
```

---

## Scenario 6: Backup Verification Failed

**Cause**: Corrupt backup file, incompatible format

**Duration**: 30-120 minutes (fallback to older backup)
**Data Loss**: 2+ hours possible

### Recovery Procedure

```bash
# 1. Identify backup corruption
# During restore, if backup fails import:

kubectl exec -n vapora surrealdb-0 -- \
  surreal import \
    --conn ws://localhost:8000 \
    --user root \
    --pass $DB_PASSWORD \
    --input /tmp/backup.sql

# Error: "invalid SQL format" or similar

# 2. Check backup file integrity
file vapora-db-backup.sql
# Should show: ASCII text

head -5 vapora-db-backup.sql
# Should show: SQL statements or surreal export format

# 3. If corrupt, try next-oldest backup
aws s3 ls s3://vapora-backups/database/ --recursive | sort | tail -5
# Get second-newest backup

# 4. Retry restore with older backup
aws s3 cp s3://vapora-backups/database/2026-01-12-210000/ ./
gunzip backup.sql.gz

# 5. Repeat restore procedure with older backup
# (As in Scenario 5, steps 8-10)
```

---

## Scenario 7: Database Size Growing Unexpectedly

**Cause**: Accumulation of data, logs not rotated, storage leak

**Duration**: Varies (prevention focus)
**Data Loss**: None

### Detection

```bash
# Monitor database size
kubectl exec -n vapora surrealdb-0 -- du -sh /var/lib/surrealdb/

# Check disk usage trend
# (Should be ~1-2% growth per week)

# If sudden spike:
kubectl exec -n vapora surrealdb-0 -- \
  find /var/lib/surrealdb/ -type f -exec ls -lh {} + | sort -k5 -h | tail -20
```

### Cleanup Procedure

```bash
# 1. Identify large tables
kubectl exec -n vapora surrealdb-0 -- \
  surreal sql "SELECT table, count(*) FROM meta::tb GROUP BY table ORDER BY count DESC"

# 2. If logs table too large
kubectl exec -n vapora surrealdb-0 -- \
  surreal sql "DELETE FROM audit_logs WHERE created_at < now() - 90d"

# 3. Rebuild indexes to reclaim space
kubectl exec -n vapora surrealdb-0 -- \
  surreal query "REBUILD INDEX"

# 4. If still large, delete old records from other tables
kubectl exec -n vapora surrealdb-0 -- \
  surreal sql "DELETE FROM tasks WHERE status = 'archived' AND updated_at < now() - 1y"

# 5. Monitor size after cleanup
kubectl exec -n vapora surrealdb-0 -- du -sh /var/lib/surrealdb/
```

---

## Scenario 8: Replication Lag (If Using Replicas)

**Cause**: Replica behind primary, network latency

**Duration**: Usually self-healing (seconds to minutes)
**Data Loss**: None

### Detection

```bash
# Check replica lag
kubectl exec -n vapora surrealdb-replica -- \
  surreal sql "SHOW REPLICATION STATUS"

# Look for: "Seconds_Behind_Master" > 5 seconds
```

### Recovery

```bash
# Usually self-healing, but if stuck:

# 1. Check network connectivity
kubectl exec -n vapora surrealdb-replica -- ping surrealdb-primary -c 5

# 2. Restart replica
kubectl delete pod -n vapora surrealdb-replica

# 3. Monitor replica catching up
kubectl logs -n vapora surrealdb-replica -f

# 4. Verify replica status
kubectl exec -n vapora surrealdb-replica -- \
  surreal sql "SHOW REPLICATION STATUS"
```

---

## Database Health Checks

### Pre-Recovery Verification

```bash
def verify_database_health [] {
  print "=== Database Health Check ==="

  # 1. Connection test
  let conn = (try (
    exec "surreal sql --conn ws://localhost:8000 \"SELECT 1\""
  ) catch {error make {msg: "Cannot connect to database"}})

  # 2. Data integrity test
  let integrity = (exec "surreal sql \"REBUILD INDEX\"")
  print "✓ Integrity check passed"

  # 3. Performance test
  let perf = (exec "surreal sql \"SELECT COUNT(*) FROM projects\"")
  print "✓ Performance acceptable"

  # 4. Replication lag (if applicable)
  # let lag = (exec "surreal sql \"SHOW REPLICATION STATUS\"")
  # print "✓ No replication lag"

  print "✓ All health checks passed"
}
```

### Post-Recovery Verification

```bash
def verify_recovery_success [] {
  print "=== Post-Recovery Verification ==="

  # 1. Database accessible
  kubectl exec -n vapora surrealdb-0 -- \
    surreal sql "SELECT 1"
  print "✓ Database accessible"

  # 2. All tables present
  kubectl exec -n vapora surrealdb-0 -- \
    surreal sql "SELECT table FROM meta::tb"
  print "✓ All tables present"

  # 3. Record counts reasonable
  kubectl exec -n vapora surrealdb-0 -- \
    surreal sql "SELECT table, count(*) FROM meta::tb"
  print "✓ Record counts verified"

  # 4. Application can connect
  kubectl logs -n vapora deployment/vapora-backend --tail=5 | grep -i connected
  print "✓ Application connected"

  # 5. API operational
  curl http://localhost:8001/api/projects
  print "✓ API operational"
}
```

---

## Database Recovery Checklist

### Before Recovery

```
□ Documented failure symptoms
□ Determined root cause
□ Selected appropriate recovery method
□ Located backup to restore
□ Verified backup integrity
□ Notified relevant teams
□ Have runbook available
□ Test environment ready (for testing)
```

### During Recovery

```
□ Followed procedure step-by-step
□ Monitored each step completion
□ Captured any error messages
□ Took notes of timings
□ Did NOT skip verification steps
□ Had backup plans ready
```

### After Recovery

```
□ Verified database accessible
□ Verified data integrity
□ Verified application can connect
□ Checked API endpoints working
□ Monitored error rates
□ Waited for 30 min stability check
□ Documented recovery procedure
□ Identified improvements needed
□ Updated runbooks if needed
```

---

## Recovery Troubleshooting

### Issue: "Cannot connect to database after restore"

**Cause**: Database not fully recovered, network issue

**Solution**:
```bash
# 1. Wait longer (import can take 15+ minutes)
sleep 60 && kubectl exec -n vapora surrealdb-0 -- surreal sql "SELECT 1"

# 2. Check pod logs
kubectl logs -n vapora surrealdb-0 | tail -50

# 3. Restart pod
kubectl delete pod -n vapora surrealdb-0

# 4. Check network connectivity
kubectl exec -n vapora surrealdb-0 -- ping localhost
```

### Issue: "Import corrupted data" error

**Cause**: Backup file corrupted or wrong format

**Solution**:
```bash
# 1. Try different backup
aws s3 ls s3://vapora-backups/database/ | sort | tail -5

# 2. Verify backup format
file vapora-db-backup.sql
# Should show: text

# 3. Manual inspection
head -20 vapora-db-backup.sql
# Should show SQL format

# 4. Try with older backup
```

### Issue: "Database running but data seems wrong"

**Cause**: Restored wrong backup or partial restore

**Solution**:
```bash
# 1. Verify record counts
kubectl exec -n vapora surrealdb-0 -- \
  surreal sql "SELECT table, count(*) FROM meta::tb"

# 2. Compare to pre-loss baseline
# (from documentation or logs)

# If counts don't match:
# - Used wrong backup
# - Restore incomplete
# - Try again with correct backup
```

---

## Database Recovery Reference

**Recovery Procedure Flowchart**:

```
Database Issue Detected
    ↓
Is it just a pod restart?
  YES → kubectl delete pod surrealdb-0
  NO → Continue
    ↓
Can queries connect and run?
  YES → Continue with application recovery
  NO → Continue
    ↓
Is data corrupted (errors in queries)?
  YES → Try REBUILD INDEX
  NO → Continue
    ↓
Still errors?
  YES → Scale replicas=0, clear PVC, restore from backup
  NO → Success, monitor for 30 min
```