# Backup & Recovery

Comprehensive backup strategies and disaster recovery procedures for the Provisioning platform.

## Overview

The platform backup strategy covers:

- Platform service data and state
- Database backups (SurrealDB)
- Configuration files and secrets
- Infrastructure definitions
- Workflow checkpoints and history
- Audit logs and compliance data

## Backup Components

### Critical Data

| Component | Location | Backup Priority | Recovery Time |
| --------- | -------- | --------------- | ------------- |
| Database | `/var/lib/provisioning/database` | Critical | < 15 min |
| Orchestrator State | `/var/lib/provisioning/orchestrator` | Critical | < 5 min |
| Configuration | `/etc/provisioning` | High | < 5 min |
| Secrets | SOPS-encrypted files | Critical | < 5 min |
| Audit Logs | `/var/log/provisioning/audit` | Compliance | < 30 min |
| Workspace Data | `workspace/` | High | < 15 min |
| Infrastructure Schemas | `provisioning/schemas` | High | < 10 min |

## Backup Strategies

### Full Backup

Complete system backup including all components:

```bash
# Create full backup
provisioning backup create --type full --output /backups/full-$(date +%Y%m%d).tar.gz

# Full backup includes:
# - Database dump
# - Service configuration
# - Workflow state
# - Audit logs
# - User data
```

Contents of full backup:

```text
full-20260116.tar.gz
├── database/
│   └── surrealdb-dump.sql
├── config/
│   ├── provisioning.toml
│   ├── orchestrator.toml
│   └── control-center.toml
├── state/
│   ├── workflows/
│   └── checkpoints/
├── logs/
│   └── audit/
├── workspace/
│   ├── infra/
│   └── config/
└── metadata.json
```

### Incremental Backup

Backup only changed data since last backup:

```bash
# Incremental backup (faster, smaller)
provisioning backup create --type incremental --since-backup full-20260116

# Incremental backup includes:
# - New workflows since last backup
# - Configuration changes
# - New audit log entries
# - Modified workspace files
```

### Continuous Backup

Real-time backup of critical data:

```bash
# Enable continuous backup
provisioning backup enable-continuous --destination s3://backups/continuous

# WAL archiving for database
# Real-time checkpoint backup
# Audit log streaming
```

## Backup Commands

### Create Backup

```bash
# Full backup to local directory
provisioning backup create --type full --output /backups

# Incremental backup
provisioning backup create --type incremental

# Backup specific components
provisioning backup create --components database,config

# Compressed backup
provisioning backup create --compress gzip

# Encrypted backup
provisioning backup create --encrypt --key-file /etc/provisioning/backup.key
```

### List Backups

```bash
# List all backups
provisioning backup list

# Output:
# NAME                  TYPE         SIZE    DATE                STATUS
# full-20260116        Full         2.5GB   2026-01-16 10:00   Complete
# incr-20260116-1200   Incremental  150MB   2026-01-16 12:00   Complete
# full-20260115        Full         2.4GB   2026-01-15 10:00   Complete
```

### Restore Backup

```bash
# Restore full backup
provisioning backup restore --backup full-20260116 --confirm

# Restore specific components
provisioning backup restore --backup full-20260116 --components database

# Point-in-time restore
provisioning backup restore --timestamp "2026-01-16 09:30:00"

# Dry-run restore
provisioning backup restore --backup full-20260116 --dry-run
```

### Verify Backup

```bash
# Verify backup integrity
provisioning backup verify --backup full-20260116

# Test restore in isolated environment
provisioning backup test-restore --backup full-20260116
```

## Automated Backup Scheduling

### Cron-based Backups

```bash
# Install backup cron jobs
provisioning backup schedule install

# Default schedule:
# Full backup: Daily at 2 AM
# Incremental: Every 6 hours
# Cleanup old backups: Weekly
```

Crontab entries:

```cron
# Full daily backup
0 2 * * * /usr/local/bin/provisioning backup create --type full --output /backups

# Incremental every 6 hours
0 */6 * * * /usr/local/bin/provisioning backup create --type incremental

# Cleanup backups older than 30 days
0 3 * * 0 /usr/local/bin/provisioning backup cleanup --older-than 30d
```

### Systemd Timer-based Backups

```ini
# /etc/systemd/system/provisioning-backup.timer
[Unit]
Description=Provisioning Platform Backup Timer

[Timer]
OnCalendar=daily
OnCalendar=02:00
Persistent=true

[Install]
WantedBy=timers.target
```

```ini
# /etc/systemd/system/provisioning-backup.service
[Unit]
Description=Provisioning Platform Backup

[Service]
Type=oneshot
ExecStart=/usr/local/bin/provisioning backup create --type full
User=provisioning
```

Enable timer:

```bash
sudo systemctl enable provisioning-backup.timer
sudo systemctl start provisioning-backup.timer
```

## Backup Destinations

### Local Filesystem

```bash
# Backup to local directory
provisioning backup create --output /mnt/backups
```

### Remote Storage

S3-compatible storage:

```bash
# Backup to S3
provisioning backup create --destination s3://my-bucket/backups \
  --s3-region us-east-1

# Backup to MinIO
provisioning backup create --destination s3://backups \
  --s3-endpoint  [http://minio.local:9000](http://minio.local:9000)
```

Network filesystem:

```bash
# Backup to NFS mount
provisioning backup create --output /mnt/nfs/backups

# Backup to SMB share
provisioning backup create --output /mnt/smb/backups
```

### Off-site Backup

Rsync to remote server:

```bash
# Backup and sync to remote
provisioning backup create --output /backups
rsync -avz /backups/ backup-server:/backups/provisioning/
```

## Database Backup

### SurrealDB Backup

```bash
# Export database
surreal export --conn  [http://localhost:8000](http://localhost:8000) \
  --user root --pass root \
  --ns provisioning --db main \
  /backups/database-$(date +%Y%m%d).surql

# Import database
surreal import --conn  [http://localhost:8000](http://localhost:8000) \
  --user root --pass root \
  --ns provisioning --db main \
  /backups/database-20260116.surql
```

### Automated Database Backups

```bash
# Enable automatic database backups
provisioning backup database enable --interval daily

# Backup with point-in-time recovery
provisioning backup database create --enable-pitr
```

## Disaster Recovery

### Recovery Procedures

Complete platform recovery from backup:

```bash
# 1. Stop all services
sudo systemctl stop provisioning-*

# 2. Restore database
provisioning backup restore --backup full-20260116 --components database

# 3. Restore configuration
provisioning backup restore --backup full-20260116 --components config

# 4. Restore service state
provisioning backup restore --backup full-20260116 --components state

# 5. Verify data integrity
provisioning validate-installation

# 6. Start services
sudo systemctl start provisioning-*

# 7. Verify services
provisioning platform status
```

### Recovery Time Objectives

| Scenario | RTO | RPO | Procedure |
| -------- | --- | --- | --------- |
| Service failure | 5 min | 0 | Restart service from checkpoint |
| Database corruption | 15 min | 6 hours | Restore from incremental backup |
| Complete data loss | 30 min | 24 hours | Restore from full backup |
| Site disaster | 2 hours | 24 hours | Restore from off-site backup |

### Point-in-Time Recovery

Restore to specific timestamp:

```bash
# List available recovery points
provisioning backup list-recovery-points

# Restore to specific time
provisioning backup restore --timestamp "2026-01-16 09:30:00"

# Recovery with workflow replay
provisioning backup restore --timestamp "2026-01-16 09:30:00" --replay-workflows
```

## Backup Encryption

### SOPS Encryption

Encrypt backups with SOPS:

```bash
# Create encrypted backup
provisioning backup create --encrypt sops --key-file /etc/provisioning/age.key

# Restore encrypted backup
provisioning backup restore --backup encrypted-20260116.tar.gz.enc \
  --decrypt sops --key-file /etc/provisioning/age.key
```

### Age Encryption

```bash
# Generate age key pair
age-keygen -o /etc/provisioning/backup-key.txt

# Create encrypted backup with age
provisioning backup create --encrypt age --recipient "age1..."

# Decrypt and restore
age -d -i /etc/provisioning/backup-key.txt backup.tar.gz.age | \
  provisioning backup restore --stdin
```

## Backup Retention

### Retention Policies

```toml
# /etc/provisioning/backup-retention.toml
[retention]
# Keep daily backups for 7 days
daily = 7

# Keep weekly backups for 4 weeks
weekly = 4

# Keep monthly backups for 12 months
monthly = 12

# Keep yearly backups for 7 years (compliance)
yearly = 7
```

Apply retention policy:

```bash
# Cleanup old backups according to policy
provisioning backup cleanup --policy /etc/provisioning/backup-retention.toml
```

## Backup Monitoring

### Backup Alerts

Configure alerts for backup failures:

```yaml
# Prometheus alert for failed backups
- alert: BackupFailed
  expr: provisioning_backup_status{status="failed"} > 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Backup failed"
    description: "Backup has failed, investigate immediately"
```

### Backup Metrics

Monitor backup health:

```bash
# Backup success rate
provisioning_backup_success_rate{type="full"} 1.0

# Time since last backup
time() - provisioning_backup_last_success_timestamp > 86400

# Backup size trend
increase(provisioning_backup_size_bytes[7d])
```

## Testing Recovery Procedures

### Regular DR Drills

```bash
# Automated disaster recovery test
provisioning backup test-recovery --backup full-20260116 \
  --test-environment isolated

# Steps performed:
# 1. Spin up isolated test environment
# 2. Restore backup
# 3. Verify data integrity
# 4. Run smoke tests
# 5. Generate test report
# 6. Teardown test environment
```

Schedule monthly DR tests:

```cron
# Monthly disaster recovery drill
0 4 1 * * /usr/local/bin/provisioning backup test-recovery --latest
```

## Best Practices

- Implement 3-2-1 backup rule: 3 copies, 2 different media, 1 off-site
- Encrypt all backups containing sensitive data
- Test restore procedures regularly (monthly minimum)
- Monitor backup success/failure metrics
- Automate backup verification
- Document recovery procedures and RTO/RPO
- Maintain off-site backups for disaster recovery
- Use incremental backups to reduce storage costs
- Version control infrastructure schemas separately
- Retain audit logs per compliance requirements (7 years)

## Related Documentation

- [Service Management](service-management.md) - Service lifecycle
- [Platform Health](platform-health.md) - Health monitoring
- [Troubleshooting](troubleshooting.md) - Recovery procedures
