# Disaster Recovery

Comprehensive disaster recovery procedures for the Provisioning platform and managed infrastructure.

## Overview

Disaster recovery (DR) ensures business continuity through:
- Automated backups
- Point-in-time recovery
- Multi-region failover
- Data replication
- DR testing procedures

## Recovery Objectives

### RTO (Recovery Time Objective)

Target time to restore service:

- **Critical Services**: < 1 hour
- **Production Infrastructure**: < 4 hours
- **Development Environment**: < 24 hours

### RPO (Recovery Point Objective)

Maximum acceptable data loss:

- **Production Databases**: < 5 minutes (continuous replication)
- **Configuration**: < 1 hour (hourly backups)
- **Workspace State**: < 15 minutes (incremental backups)

## Backup Strategy

### Automated Backups

Configure automatic backups:

```nickel
{
  backup = {
    enabled = true,
    schedule = "0 */6 * * *",  # Every 6 hours
    retention_days = 30,

    targets = [
      {type = 'workspace_state, enabled = true},
      {type = 'infrastructure_config, enabled = true},
      {type = 'platform_data, enabled = true},
    ],

    storage = {
      backend = 's3,
      bucket = "provisioning-backups",
      encryption = true,
    }
  }
}
```

### Backup Types

**Full Backups**:

```bash
# Full platform backup
provisioning backup create --type full --name "pre-upgrade-$(date +%Y%m%d)"

# Full workspace backup
provisioning workspace backup production --full
```

**Incremental Backups**:

```bash
# Incremental backup (changed files only)
provisioning backup create --type incremental

# Automated incremental
provisioning config set backup.incremental_enabled true
```

**Snapshot Backups**:

```bash
# Infrastructure snapshot
provisioning infrastructure snapshot --name "stable-v2"

# Database snapshot
provisioning taskserv backup postgresql --snapshot
```

## Data Replication

### Cross-Region Replication

Replicate to secondary region:

```nickel
{
  replication = {
    enabled = true,
    mode = 'async,

    primary = {region = "eu-west-1", provider = 'aws},
    secondary = {region = "us-east-1", provider = 'aws},

    replication_lag_max_seconds = 300,
  }
}
```

### Database Replication

```bash
# Configure database replication
provisioning taskserv configure postgresql --replication \
  --primary db-eu-west-1 \
  --standby db-us-east-1 \
  --sync-mode async
```

## Disaster Scenarios

### Complete Region Failure

**Procedure**:

1. **Detect Failure**:

```bash
# Check region health
provisioning health check --region eu-west-1
```

2. **Initiate Failover**:

```bash
# Promote secondary region
provisioning disaster-recovery failover --to us-east-1 --confirm

# Verify services
provisioning health check --all
```

3. **Update DNS**:

```bash
# Point traffic to secondary region
provisioning dns update --region us-east-1
```

4. **Monitor**:

```bash
# Watch recovery progress
provisioning disaster-recovery status --follow
```

### Data Corruption

**Procedure**:

1. **Identify Corruption**:

```bash
# Validate data integrity
provisioning validate data --workspace production
```

2. **Find Clean Backup**:

```bash
# List available backups
provisioning backup list --before "2024-01-15 10:00"

# Verify backup integrity
provisioning backup verify backup-20240115-0900
```

3. **Restore from Backup**:

```bash
# Restore to point in time
provisioning restore --backup backup-20240115-0900 \
  --workspace production --confirm
```

### Platform Service Failure

**Procedure**:

1. **Identify Failed Service**:

```bash
# Check platform health
provisioning platform health

# Service logs
provisioning platform logs orchestrator --tail 100
```

2. **Restart Service**:

```bash
# Restart failed service
provisioning platform restart orchestrator

# Verify health
provisioning platform health orchestrator
```

3. **Restore from Backup** (if needed):

```bash
# Restore service data
provisioning platform restore orchestrator \
  --from-backup latest
```

## Failover Procedures

### Automated Failover

Configure automatic failover:

```nickel
{
  failover = {
    enabled = true,
    health_check_interval_seconds = 30,
    failure_threshold = 3,

    primary = {region = "eu-west-1"},
    secondary = {region = "us-east-1"},

    auto_failback = false,  # Manual failback
  }
}
```

### Manual Failover

```bash
# Initiate manual failover
provisioning disaster-recovery failover \
  --from eu-west-1 \
  --to us-east-1 \
  --verify-replication \
  --confirm

# Verify failover
provisioning disaster-recovery verify

# Update routing
provisioning disaster-recovery update-routing
```

## Recovery Procedures

### Workspace Recovery

```bash
# List workspace backups
provisioning workspace backups production

# Restore workspace
provisioning workspace restore production \
  --backup backup-20240115-1200 \
  --target-region us-east-1

# Verify recovery
provisioning workspace validate production
```

### Infrastructure Recovery

```bash
# Restore infrastructure from Nickel config
provisioning infrastructure restore \
  --config workspace/infra/production.ncl \
  --region us-east-1

# Restore from snapshot
provisioning infrastructure restore \
  --snapshot infra-snapshot-20240115

# Verify deployment
provisioning infrastructure validate
```

### Platform Recovery

```bash
# Reinstall platform services
provisioning platform install --region us-east-1

# Restore platform data
provisioning platform restore --from-backup latest

# Verify platform health
provisioning platform health --all
```

## DR Testing

### Test Schedule

- **Monthly**: Backup restore test
- **Quarterly**: Regional failover drill
- **Annually**: Full DR simulation

### Backup Restore Test

```bash
# Create test workspace
provisioning workspace create dr-test-$(date +%Y%m%d)

# Restore latest backup
provisioning workspace restore dr-test --backup latest

# Validate restore
provisioning workspace validate dr-test

# Cleanup
provisioning workspace delete dr-test --yes
```

### Failover Drill

```bash
# Simulate regional failure
provisioning disaster-recovery simulate-failure \
  --region eu-west-1 \
  --duration 30m

# Monitor automated failover
provisioning disaster-recovery status --follow

# Validate services in secondary region
provisioning health check --region us-east-1 --all

# Manual failback after drill
provisioning disaster-recovery failback --to eu-west-1
```

## Monitoring and Alerts

### Backup Monitoring

```bash
# Check backup status
provisioning backup status

# Verify backup integrity
provisioning backup verify --all --schedule daily

# Alert on backup failures
provisioning alert create backup-failure \
  --condition "backup.status == 'failed'" \
  --notify [ops@example.com](mailto:ops@example.com)
```

### Replication Monitoring

```bash
# Check replication lag
provisioning replication status

# Alert on lag exceeding threshold
provisioning alert create replication-lag \
  --condition "replication.lag_seconds > 300" \
  --notify [ops@example.com](mailto:ops@example.com)
```

## Best Practices

1. **Regular testing** - Test DR procedures quarterly
2. **Automated backups** - Never rely on manual backups
3. **Multiple regions** - Geographic redundancy
4. **Monitor replication** - Track replication lag
5. **Document procedures** - Keep runbooks updated
6. **Encrypt backups** - Protect backup data
7. **Verify restores** - Test backup integrity
8. **Automate failover** - Reduce recovery time

## References

- [Backup & Recovery](../operations/backup-recovery.md) - Backup operations
- [Monitoring](../operations/monitoring.md) - Health monitoring
- [Platform Health](../operations/platform-health.md) - Service health
