Vapora/docs/operations/backup-recovery-automation.md

# VAPORA Automated Backup & Recovery Automation

Automated backup and recovery procedures using Nushell scripts and Kubernetes CronJobs. Supports both direct S3 backups and Restic-based incremental backups.

---

## Overview

**Backup Strategy**:
- Hourly: Database export + Restic backup (1-hour RPO)
- Daily: Kubernetes config backup + Restic backup
- Monthly: Cleanup old snapshots and archive

**Dual Backup Approach**:
- **S3 Direct**: Simple file upload for quick recovery
- **Restic**: Incremental, deduplicated backups with integrated encryption

**Recovery Procedures**:
- One-command restore from S3 or Restic
- Verification before committing to production
- Automated database readiness checks

---

## Files and Components

### Backup Scripts

All scripts follow NUSHELL_GUIDELINES.md (0.109.0+) strictly.

#### `scripts/backup/database-backup.nu`

Direct S3 backup of SurrealDB with encryption.

```bash
nu scripts/backup/database-backup.nu \
  --surreal-url "ws://localhost:8000" \
  --surreal-user "root" \
  --surreal-pass "$SURREAL_PASS" \
  --s3-bucket "vapora-backups" \
  --s3-prefix "backups/database" \
  --encryption-key "$ENCRYPTION_KEY_FILE"
```

**Process**:
1. Export SurrealDB to SQL
2. Compress with gzip
3. Encrypt with AES-256
4. Upload to S3 with metadata
5. Verify upload completed

**Output**: `s3://vapora-backups/backups/database/database-YYYYMMDD-HHMMSS.sql.gz.enc`

#### `scripts/backup/config-backup.nu`

Backup Kubernetes resources (ConfigMaps, Secrets, Deployments).

```bash
nu scripts/backup/config-backup.nu \
  --namespace "vapora" \
  --s3-bucket "vapora-backups" \
  --s3-prefix "backups/config"
```

**Process**:
1. Export ConfigMaps from namespace
2. Export Secrets
3. Export Deployments, Services, Ingress
4. Compress all to tar.gz
5. Upload to S3

**Output**: `s3://vapora-backups/backups/config/configs-YYYYMMDD-HHMMSS.tar.gz`

#### `scripts/backup/restic-backup.nu`

Incremental, deduplicated backup using Restic.

```bash
nu scripts/backup/restic-backup.nu \
  --repo "s3:s3.amazonaws.com/vapora-backups/restic" \
  --password "$RESTIC_PASSWORD" \
  --database-dir "/tmp/vapora-db-backup" \
  --k8s-dir "/tmp/vapora-k8s-backup" \
  --iac-dir "provisioning" \
  --backup-db \
  --backup-k8s \
  --backup-iac \
  --verify \
  --cleanup \
  --keep-daily 7 \
  --keep-weekly 4 \
  --keep-monthly 12
```

**Features**:
- Incremental backups (only changed data stored)
- Deduplication across snapshots
- Built-in compression and encryption
- Automatic retention policies
- Repository health verification

**Output**: Tagged snapshots in Restic repository with metadata

#### `scripts/orchestrate-backup-recovery.nu`

Coordinates all backup types (S3 + Restic).

```bash
# Full backup cycle
nu scripts/orchestrate-backup-recovery.nu \
  --operation backup \
  --mode full \
  --surreal-url "ws://localhost:8000" \
  --surreal-user "root" \
  --surreal-pass "$SURREAL_PASS" \
  --namespace "vapora" \
  --s3-bucket "vapora-backups" \
  --s3-prefix "backups/database" \
  --encryption-key "$ENCRYPTION_KEY_FILE" \
  --restic-repo "s3:s3.amazonaws.com/vapora-backups/restic" \
  --restic-password "$RESTIC_PASSWORD" \
  --iac-dir "provisioning"
```

**Modes**:
- `full`: Database export → S3 + Restic
- `database-only`: Database export only
- `config-only`: Kubernetes config only

### Recovery Scripts

#### `scripts/recovery/database-recovery.nu`

Restore SurrealDB from S3 backup (with decryption).

```bash
nu scripts/recovery/database-recovery.nu \
  --s3-location "s3://vapora-backups/backups/database/database-20260112-010000.sql.gz.enc" \
  --encryption-key "$ENCRYPTION_KEY_FILE" \
  --surreal-url "ws://localhost:8000" \
  --surreal-user "root" \
  --surreal-pass "$SURREAL_PASS" \
  --namespace "vapora" \
  --statefulset "surrealdb" \
  --pvc "surrealdb-data-surrealdb-0" \
  --verify
```

**Process**:
1. Download encrypted backup from S3
2. Decrypt backup file
3. Decompress backup
4. Scale down StatefulSet (for PVC replacement)
5. Delete current PVC
6. Scale up StatefulSet (creates new PVC)
7. Wait for pod readiness
8. Import backup to database
9. Verify data integrity

**Output**: Restored database at specified SurrealDB URL

#### `scripts/orchestrate-backup-recovery.nu` (Recovery Mode)

One-command recovery from backup.

```bash
nu scripts/orchestrate-backup-recovery.nu \
  --operation recovery \
  --s3-location "s3://vapora-backups/backups/database/database-20260112-010000.sql.gz.enc" \
  --encryption-key "$ENCRYPTION_KEY_FILE" \
  --surreal-url "ws://localhost:8000" \
  --surreal-user "root" \
  --surreal-pass "$SURREAL_PASS"
```

### Verification Scripts

#### `scripts/verify-backup-health.nu`

Health check for backup infrastructure.

```bash
# Basic health check
nu scripts/verify-backup-health.nu \
  --s3-bucket "vapora-backups" \
  --s3-prefix "backups/database" \
  --restic-repo "s3:s3.amazonaws.com/vapora-backups/restic" \
  --restic-password "$RESTIC_PASSWORD" \
  --surreal-url "ws://localhost:8000" \
  --surreal-user "root" \
  --surreal-pass "$SURREAL_PASS" \
  --max-age-hours 25
```

**Checks Performed**:
- ✓ S3 backups exist and have content
- ✓ Restic repository accessible and has snapshots
- ✓ Database connectivity verified
- ✓ Backup freshness (< 25 hours old)
- ✓ Backup rotation policy (daily, weekly, monthly)
- ✓ Restore test (if `--full-test` specified)

**Output**: Pass/fail for each check with detailed status

---

## Kubernetes Automation

### CronJob Configuration

File: `kubernetes/09-backup-cronjobs.yaml`

Defines four automated CronJobs:

#### 1. Hourly Database Backup

```yaml
schedule: "0 * * * *"  # Every hour
timeout: 1800 seconds  # 30 minutes
```

Runs `orchestrate-backup-recovery.nu --operation backup --mode full`

**Backups**:
- SurrealDB to S3 (encrypted)
- SurrealDB to Restic (incremental)
- IaC to Restic

#### 2. Daily Configuration Backup

```yaml
schedule: "0 2 * * *"  # 02:00 UTC daily
timeout: 3600 seconds  # 60 minutes
```

Runs `config-backup.nu` for Kubernetes resources.

#### 3. Daily Health Verification

```yaml
schedule: "0 3 * * *"  # 03:00 UTC daily
timeout: 900 seconds   # 15 minutes
```

Runs `verify-backup-health.nu` to verify backup infrastructure.

**Alerts if**:
- No S3 backups found
- Restic repository inaccessible
- Database unreachable
- Backups older than 25 hours
- Rotation policy violated

#### 4. Monthly Backup Rotation

```yaml
schedule: "0 4 1 * *"  # First day of month, 04:00 UTC
timeout: 3600 seconds
```

Cleans up old Restic snapshots per retention policy:
- Keep: 7 daily, 4 weekly, 12 monthly
- Prune: Remove unreferenced data

### Environment Configuration

CronJobs require these secrets and ConfigMaps:

**ConfigMap: `vapora-config`**

```yaml
backup_s3_bucket: "vapora-backups"
restic_repo: "s3:s3.amazonaws.com/vapora-backups/restic"
aws_region: "us-east-1"
```

**Secret: `vapora-secrets`**

```yaml
surreal_password: "<database-password>"
restic_password: "<restic-encryption-password>"
```

**Secret: `vapora-aws-credentials`**

```yaml
access_key_id: "<aws-access-key>"
secret_access_key: "<aws-secret-key>"
```

**Secret: `vapora-encryption-key`**

```yaml
# File containing AES-256 encryption key
encryption.key: "<binary-key-data>"
```

### Deployment

1. **Create secrets** (if not existing):

```bash
kubectl create secret generic vapora-secrets \
  --from-literal=surreal_password="$SURREAL_PASS" \
  --from-literal=restic_password="$RESTIC_PASSWORD" \
  -n vapora

kubectl create secret generic vapora-aws-credentials \
  --from-literal=access_key_id="$AWS_ACCESS_KEY_ID" \
  --from-literal=secret_access_key="$AWS_SECRET_ACCESS_KEY" \
  -n vapora

kubectl create secret generic vapora-encryption-key \
  --from-file=encryption.key=/path/to/encryption.key \
  -n vapora
```

2. **Deploy CronJobs**:

```bash
kubectl apply -f kubernetes/09-backup-cronjobs.yaml
```

3. **Verify CronJobs**:

```bash
kubectl get cronjobs -n vapora
kubectl describe cronjob vapora-backup-database-hourly -n vapora
```

4. **Monitor scheduled runs**:

```bash
# Watch CronJob executions
kubectl get jobs -n vapora -l job-type=backup --watch

# View logs from backup job
kubectl logs -n vapora -l backup-type=database --tail=100 -f
```

---

## Setup Instructions

### Prerequisites

- Kubernetes 1.18+ with CronJob support
- Nushell 0.109.0+
- AWS CLI v2+
- Restic installed (or container image with restic)
- SurrealDB CLI (`surreal` command)
- `kubectl` with cluster access

### Local Testing

1. **Setup environment variables**:

```bash
export SURREAL_URL="ws://localhost:8000"
export SURREAL_USER="root"
export SURREAL_PASS="password"
export S3_BUCKET="vapora-backups"
export ENCRYPTION_KEY_FILE="/path/to/encryption.key"
export RESTIC_REPO="s3:s3.amazonaws.com/vapora-backups/restic"
export RESTIC_PASSWORD="restic-password"
export AWS_REGION="us-east-1"
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"
```

2. **Run backup**:

```bash
nu scripts/orchestrate-backup-recovery.nu \
  --operation backup \
  --mode full \
  --surreal-url "$SURREAL_URL" \
  --surreal-user "$SURREAL_USER" \
  --surreal-pass "$SURREAL_PASS" \
  --s3-bucket "$S3_BUCKET" \
  --s3-prefix "backups/database" \
  --encryption-key "$ENCRYPTION_KEY_FILE" \
  --restic-repo "$RESTIC_REPO" \
  --restic-password "$RESTIC_PASSWORD" \
  --iac-dir "provisioning"
```

3. **Verify backup**:

```bash
nu scripts/verify-backup-health.nu \
  --s3-bucket "$S3_BUCKET" \
  --s3-prefix "backups/database" \
  --restic-repo "$RESTIC_REPO" \
  --restic-password "$RESTIC_PASSWORD" \
  --surreal-url "$SURREAL_URL" \
  --surreal-user "$SURREAL_USER" \
  --surreal-pass "$SURREAL_PASS"
```

4. **Test recovery**:

```bash
# First, list available backups
aws s3 ls s3://$S3_BUCKET/backups/database/

# Then recover from latest backup
nu scripts/orchestrate-backup-recovery.nu \
  --operation recovery \
  --s3-location "s3://$S3_BUCKET/backups/database/database-20260112-010000.sql.gz.enc" \
  --encryption-key "$ENCRYPTION_KEY_FILE" \
  --surreal-url "$SURREAL_URL" \
  --surreal-user "$SURREAL_USER" \
  --surreal-pass "$SURREAL_PASS"
```

### Production Deployment

1. **Create S3 bucket** for backups:

```bash
aws s3 mb s3://vapora-backups --region us-east-1
```

2. **Enable bucket versioning** for protection:

```bash
aws s3api put-bucket-versioning \
  --bucket vapora-backups \
  --versioning-configuration Status=Enabled
```

3. **Set lifecycle policy** for Glacier archival (optional):

```bash
# 30 days to standard-IA, 90 days to Glacier
aws s3api put-bucket-lifecycle-configuration \
  --bucket vapora-backups \
  --lifecycle-configuration file://s3-lifecycle-policy.json
```

4. **Create Restic repository**:

```bash
export RESTIC_REPO="s3:s3.amazonaws.com/vapora-backups/restic"
export RESTIC_PASSWORD="your-restic-password"

restic init
```

5. **Deploy to Kubernetes**:

```bash
# 1. Create namespace
kubectl create namespace vapora

# 2. Create secrets
kubectl create secret generic vapora-secrets \
  --from-literal=surreal_password="$SURREAL_PASS" \
  --from-literal=restic_password="$RESTIC_PASSWORD" \
  -n vapora

# 3. Create ConfigMap
kubectl create configmap vapora-config \
  --from-literal=backup_s3_bucket="vapora-backups" \
  --from-literal=restic_repo="s3:s3.amazonaws.com/vapora-backups/restic" \
  --from-literal=aws_region="us-east-1" \
  -n vapora

# 4. Deploy CronJobs
kubectl apply -f kubernetes/09-backup-cronjobs.yaml
```

6. **Monitor**:

```bash
# Watch CronJobs
kubectl get cronjobs -n vapora --watch

# View backup logs
kubectl logs -n vapora -l backup-type=database -f

# Check health status
kubectl get jobs -n vapora -l job-type=health-check -o wide
```

---

## Emergency Recovery

### Complete Database Loss

If production database is lost, restore from backup:

```bash
# 1. Scale down StatefulSet
kubectl scale statefulset surrealdb --replicas=0 -n vapora

# 2. Delete current PVC
kubectl delete pvc surrealdb-data-surrealdb-0 -n vapora

# 3. Run recovery
nu scripts/orchestrate-backup-recovery.nu \
  --operation recovery \
  --s3-location "s3://vapora-backups/backups/database/database-LATEST.sql.gz.enc" \
  --encryption-key "/path/to/encryption.key" \
  --surreal-url "ws://surrealdb:8000" \
  --surreal-user "root" \
  --surreal-pass "$SURREAL_PASS"

# 4. Verify database restored
kubectl exec -n vapora surrealdb-0 -- \
  surreal query \
    --conn ws://localhost:8000 \
    --user root \
    --pass "$SURREAL_PASS" \
    "SELECT COUNT() FROM projects"
```

### Backup Verification Failed

If health check fails:

1. **Check Restic repository**:

```bash
export RESTIC_PASSWORD="$RESTIC_PASSWORD"
restic -r "s3:s3.amazonaws.com/vapora-backups/restic" check
```

2. **Force full verification** (slow):

```bash
restic -r "s3:s3.amazonaws.com/vapora-backups/restic" check --read-data
```

3. **List recent snapshots**:

```bash
restic -r "s3:s3.amazonaws.com/vapora-backups/restic" snapshots --max 10
```

---

## Troubleshooting

| Issue | Cause | Solution |
|-------|-------|----------|
| **CronJob not running** | Schedule incorrect | Check `kubectl get cronjobs` and verify schedule format |
| **Backup file too large** | Database growing | Check for old data that can be cleaned up |
| **S3 upload fails** | Credentials invalid | Verify `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` |
| **Restic backup slow** | First backup or network latency | Expected on first run; use `--keep-*` flags to limit retention |
| **Recovery fails** | Database already running | Scale down StatefulSet before recovery |
| **Encryption key missing** | Secret not created | Create `vapora-encryption-key` secret in namespace |

---

## Related Documentation

- **Disaster Recovery Procedures**: `docs/disaster-recovery/README.md`
- **Backup Strategy**: `docs/disaster-recovery/backup-strategy.md`
- **Database Recovery**: `docs/disaster-recovery/database-recovery-procedures.md`
- **Operations Guide**: `docs/operations/README.md`

---

**Last Updated**: January 12, 2026
**Status**: Production-Ready
**Automation**: Full CronJob automation with health checks