Vapora/scripts/backup/README.md

# VAPORA Backup & Recovery Scripts

Automated backup and recovery procedures for VAPORA using Nushell 0.109.0+.

**Dual Backup Strategy**:
- **S3**: Direct file uploads with AES-256 encryption
- **Restic**: Incremental, deduplicated backups with compression

---

## Scripts Overview

### Backup Scripts

| Script | Purpose | Schedule | Target |
|--------|---------|----------|--------|
| `database-backup.nu` | Export SurrealDB to S3 (compressed + encrypted) | Manual or Hourly | S3 |
| `config-backup.nu` | Backup Kubernetes ConfigMaps/Secrets | Manual or Daily | S3 |
| `restic-backup.nu` | Incremental backup to Restic repository | Manual | Restic |
| `orchestrate-backup-recovery.nu` | Coordinate all backup types | CronJob | S3 + Restic |

### Recovery Scripts

| Script | Purpose | Input |
|--------|---------|-------|
| `database-recovery.nu` | Restore SurrealDB from S3 backup | Encrypted S3 file |
| `orchestrate-backup-recovery.nu` | One-command recovery | S3 or Restic location |

### Verification

| Script | Purpose | Checks |
|--------|---------|--------|
| `verify-backup-health.nu` | Health check for backup infrastructure | S3, Restic, DB, freshness, rotation |

---

## Quick Start

### Local Backup

```bash
# Set environment
export SURREAL_URL="ws://localhost:8000"
export SURREAL_PASS="your-password"
export S3_BUCKET="vapora-backups"
export ENCRYPTION_KEY_FILE="/path/to/key"

# Run full backup
nu scripts/orchestrate-backup-recovery.nu \
  --operation backup \
  --mode full \
  --surreal-url "$SURREAL_URL" \
  --surreal-pass "$SURREAL_PASS" \
  --s3-bucket "$S3_BUCKET" \
  --encryption-key "$ENCRYPTION_KEY_FILE" \
  --iac-dir "provisioning"
```

### Local Recovery

```bash
# Restore from S3 backup
nu scripts/orchestrate-backup-recovery.nu \
  --operation recovery \
  --s3-location "s3://vapora-backups/backups/database/database-20260112-010000.sql.gz.enc" \
  --encryption-key "$ENCRYPTION_KEY_FILE" \
  --surreal-url "$SURREAL_URL" \
  --surreal-pass "$SURREAL_PASS"
```

### Health Check

```bash
nu scripts/verify-backup-health.nu \
  --s3-bucket "$S3_BUCKET" \
  --surreal-url "$SURREAL_URL" \
  --surreal-pass "$SURREAL_PASS"
```

---

## Kubernetes Automation

CronJobs defined in `kubernetes/09-backup-cronjobs.yaml`:

- **Hourly** (00:00 UTC): Database backup (S3 + Restic)
- **Daily** (02:00 UTC): Configuration backup
- **Daily** (03:00 UTC): Health verification
- **Monthly** (04:00 first day): Snapshot rotation/cleanup

**Deploy**:

```bash
kubectl apply -f kubernetes/09-backup-cronjobs.yaml
```

**Monitor**:

```bash
kubectl get cronjobs -n vapora
kubectl logs -n vapora -l backup-type=database -f
```

---

## Features

✅ **Dual backup approach** (S3 + Restic)
✅ **Encryption** (AES-256 at rest, encrypted transfer)
✅ **Compression** (gzip for S3, built-in for Restic)
✅ **Incremental** (Restic only - no duplicate data)
✅ **Verification** (post-backup integrity checks)
✅ **Retention** (daily/weekly/monthly policies)
✅ **Health checks** (automated daily verification)
✅ **Recovery** (one-command restore)
✅ **Kubernetes native** (CronJobs, RBAC, secrets)

---

## Implementation Details

All scripts follow **NUSHELL_GUIDELINES.md (0.109.0+)** strictly:

✓ Function signatures with BOTH `:` and `->`
✓ NO mutable variables (use `reduce --fold`)
✓ External commands with `^` prefix
✓ Error handling with `do { } | complete`
✓ Variable interpolation with `[$var]` for variables, `($expr)` for expressions
✓ NO try-catch blocks
✓ NO type annotations on boolean flags
✓ Pipelines in conditionals are parenthesized

---

## Configuration

### Environment Variables

**SurrealDB**:
```bash
SURREAL_URL=ws://localhost:8000
SURREAL_USER=root
SURREAL_PASS=<password>
```

**AWS S3**:
```bash
S3_BUCKET=vapora-backups
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=<key>
AWS_SECRET_ACCESS_KEY=<secret>
```

**Restic**:
```bash
RESTIC_REPO=s3:s3.amazonaws.com/vapora-backups/restic
RESTIC_PASSWORD=<password>
```

**Encryption**:
```bash
ENCRYPTION_KEY_FILE=/path/to/encryption.key
```

### Kubernetes Secrets

```bash
# Database credentials
kubectl create secret generic vapora-secrets \
  --from-literal=surreal_password="$SURREAL_PASS" \
  --from-literal=restic_password="$RESTIC_PASSWORD" \
  -n vapora

# AWS credentials
kubectl create secret generic vapora-aws-credentials \
  --from-literal=access_key_id="$AWS_ACCESS_KEY_ID" \
  --from-literal=secret_access_key="$AWS_SECRET_ACCESS_KEY" \
  -n vapora

# Encryption key
kubectl create secret generic vapora-encryption-key \
  --from-file=encryption.key=/path/to/encryption.key \
  -n vapora
```

---

## Backup Locations

### S3 Paths

```
s3://vapora-backups/
├── backups/
│   ├── database/
│   │   └── database-20260112-010000.sql.gz.enc
│   └── config/
│       └── configs-20260112-020000.tar.gz
└── restic/
    ├── data/
    ├── index/
    ├── snapshots/
    └── config
```

### Restic Repository

```
s3://vapora-backups/restic/
├── data/          # Backup data files
├── index/         # Index files
├── snapshots/     # Snapshot metadata
└── config         # Repository config
```

---

## Recovery Procedures

### Database Recovery (S3)

1. Download encrypted backup from S3
2. Decrypt with AES-256 key
3. Decompress backup
4. Scale down StatefulSet
5. Delete current PVC
6. Scale up StatefulSet (creates new PVC)
7. Import backup to database
8. Verify data integrity

**Time**: 30-60 seconds (depends on backup size)

### Restic Recovery

```bash
# List available snapshots
restic -r "$RESTIC_REPO" snapshots

# Restore specific snapshot to directory
restic -r "$RESTIC_REPO" restore <snapshot-id> --target /recovery
```

---

## Troubleshooting

### Backup Fails

**Check logs**:
```bash
kubectl logs -n vapora -l backup-type=database --tail=100
```

**Verify credentials**:
```bash
# S3
aws s3 ls s3://vapora-backups/

# Restic
RESTIC_PASSWORD="$RESTIC_PASSWORD" restic -r "$RESTIC_REPO" list snapshots
```

### Recovery Fails

**Ensure database is stopped**:
```bash
kubectl scale statefulset surrealdb --replicas=0 -n vapora
```

**Verify PVC deleted**:
```bash
kubectl get pvc -n vapora
```

**Check encryption key exists**:
```bash
kubectl get secrets -n vapora vapora-encryption-key
```

### Health Check Fails

**Run detailed check**:
```bash
nu scripts/verify-backup-health.nu \
  --s3-bucket "$S3_BUCKET" \
  --surreal-url "$SURREAL_URL" \
  --surreal-pass "$SURREAL_PASS" \
  --max-age-hours 25
```

---

## Integration with Disaster Recovery

These scripts implement the backup strategy defined in:
- `docs/disaster-recovery/backup-strategy.md`
- `docs/disaster-recovery/database-recovery-procedures.md`

See `docs/operations/backup-recovery-automation.md` for comprehensive integration guide.

---

## Support

**Documentation**:
- Backup Strategy: `docs/disaster-recovery/backup-strategy.md`
- Disaster Recovery: `docs/disaster-recovery/README.md`
- Operations Guide: `docs/operations/README.md`

**Issues**:
- Check logs: `kubectl logs -n vapora -l backup-type=database`
- Verify configuration: Check all environment variables are set
- Test connectivity: `aws s3 ls`, `surreal list namespaces`

---

**Last Updated**: January 12, 2026
**Nushell Version**: 0.109.0+
**Status**: Production-Ready