provisioning-platform/prov-ecosystem/crates/backup/INTEGRATION.md

# Backup Module Integration Guide

This guide covers integrating the backup module with Provctl and the Provisioning system.

## Table of Contents

1. [Standalone Usage](#standalone-usage)
2. [Provctl Integration](#provctl-integration)
3. [Provisioning System Integration](#provisioning-system-integration)
4. [KCL Configuration Examples](#kcl-configuration-examples)
5. [Automation Workflows](#automation-workflows)

## Standalone Usage

### As a CLI Tool

Create a backup script that uses the backup library:

```rust
// bin/backup-tool/main.rs
use backup::config::BackupConfig;
use backup::backends::{ResticBackend, BorgBackend, TarBackend, CpioBackend, RsyncBackend};
use backup::backend::BackupBackend;
use backup::backend_type::BackendType;
use std::path::Path;
use clap::{Parser, Subcommand};

#[derive(Parser)]
struct Args {
    #[command(subcommand)]
    command: Commands,
}

#[derive(Subcommand)]
enum Commands {
    Backup {
        #[arg(short, long)]
        config: String,
    },
    Restore {
        #[arg(short, long)]
        config: String,
        #[arg(short, long)]
        snapshot: String,
        #[arg(short, long)]
        target: String,
    },
    List {
        #[arg(short, long)]
        config: String,
    },
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let args = Args::parse();

    match args.command {
        Commands::Backup { config } => {
            let config = BackupConfig::from_toml(Path::new(&config))?;
            config.validate()?;

            let backend = create_backend(&config.backend)?;

            for target in config.targets {
                println!("Backing up: {}", target.name);
                let report = backend.backup(&target).await?;
                println!("Complete: {} files", report.files_processed);
            }
        }
        Commands::List { config } => {
            let config = BackupConfig::from_toml(Path::new(&config))?;
            let backend = create_backend(&config.backend)?;

            let snapshots = backend.list_snapshots().await?;
            for snapshot in snapshots {
                println!("{}: {} bytes", snapshot.id, snapshot.size);
            }
        }
        Commands::Restore { config, snapshot, target } => {
            let config = BackupConfig::from_toml(Path::new(&config))?;
            let backend = create_backend(&config.backend)?;

            backend.restore(&snapshot, Path::new(&target), &Default::default()).await?;
            println!("Restore complete");
        }
    }

    Ok(())
}

fn create_backend(backend_type: &BackendType) -> Result<Box<dyn BackupBackend>, Box<dyn std::error::Error>> {
    match backend_type {
        BackendType::Restic => Ok(Box::new(ResticBackend::new(None))),
        BackendType::Borg => Ok(Box::new(BorgBackend::new(None))),
        BackendType::Tar => Ok(Box::new(TarBackend::new(None))),
        BackendType::Cpio => Ok(Box::new(CpioBackend::new(None))),
        BackendType::Rsync => Ok(Box::new(RsyncBackend::new(None))),
    }
}
```

### As a Library in Your App

```rust
use backup::config::BackupConfig;
use backup::backends::ResticBackend;
use backup::backend::BackupBackend;

pub async fn create_daily_backup(config_path: &str) -> Result<(), Box<dyn std::error::Error>> {
    let config = BackupConfig::from_toml(std::path::Path::new(config_path))?;
    config.validate()?;

    let backend = ResticBackend::new(None);

    // Initialize repository if needed
    if !backend.is_available().await? {
        eprintln!("Restic not available");
        return Ok(());
    }

    backend.init_repository(&config.repository).await?;

    // Execute backups
    for target in &config.targets {
        backend.backup(target).await?;
    }

    // Apply retention policy
    backend.prune(&config.retention).await?;

    Ok(())
}
```

## Provctl Integration

### Adding Backup Commands to Provctl

Add to `crates/provctl/src/commands/backup.rs`:

```rust
use backup::config::BackupConfig;
use backup::backends::ResticBackend;
use backup::backend::BackupBackend;
use clap::Args;

#[derive(Args)]
pub struct BackupArgs {
    #[command(subcommand)]
    pub command: BackupCommand,
}

#[derive(clap::Subcommand)]
pub enum BackupCommand {
    Init {
        #[arg(short, long)]
        config: String,
        #[arg(short, long)]
        backend: Option<String>,
    },
    Run {
        #[arg(short, long)]
        config: String,
    },
    Status {
        #[arg(short, long)]
        config: String,
    },
    Check {
        #[arg(short, long)]
        config: String,
    },
}

pub async fn handle_backup(args: BackupArgs) -> Result<(), Box<dyn std::error::Error>> {
    match args.command {
        BackupCommand::Init { config, backend } => {
            let config = BackupConfig::from_toml(std::path::Path::new(&config))?;
            config.validate()?;

            let backend = ResticBackend::new(None);
            backend.init_repository(&config.repository).await?;
            println!("Repository initialized");
        }
        BackupCommand::Run { config } => {
            let config = BackupConfig::from_toml(std::path::Path::new(&config))?;
            let backend = ResticBackend::new(None);

            for target in config.targets {
                let report = backend.backup(&target).await?;
                println!("✓ {} backed up", target.name);
            }
        }
        BackupCommand::Status { config } => {
            let config = BackupConfig::from_toml(std::path::Path::new(&config))?;
            let backend = ResticBackend::new(None);

            let stats = backend.stats().await?;
            println!("Repository size: {} bytes", stats.total_size);
            println!("Snapshots: {}", stats.snapshot_count);
        }
        BackupCommand::Check { config } => {
            let config = BackupConfig::from_toml(std::path::Path::new(&config))?;
            let backend = ResticBackend::new(None);

            let check = backend.check().await?;
            if check.valid {
                println!("✓ Repository is valid");
            } else {
                println!("✗ Repository has issues:");
                for issue in check.issues {
                    println!("  - {}", issue);
                }
            }
        }
    }

    Ok(())
}
```

Add to Provctl's main command handler:

```rust
mod commands;
use commands::backup;

pub enum Command {
    Backup(backup::BackupArgs),
    // ... other commands
}

pub async fn handle_command(cmd: Command) -> Result<(), Box<dyn std::error::Error>> {
    match cmd {
        Command::Backup(args) => backup::handle_backup(args).await,
        // ... other commands
    }
}
```

### Usage

```bash
# Initialize backup repository
provctl backup init --config backup.toml

# Run a backup
provctl backup run --config backup.toml

# Check backup status
provctl backup status --config backup.toml

# Verify repository integrity
provctl backup check --config backup.toml
```

## Provisioning System Integration

### Using Backup in Provisioning Configs

Create `provisioning/backup-deployment.kcl`:

```kcl
import backup

# Backup configuration for production deployment
backup_config = backup.BackupConfig {
    version = "1.0"
    name = "prod-database"
    description = "Production database daily backups"
    backend = "restic"

    repository = {
        type = "s3"
        path = "s3://company-backups/prod-db"
        env_vars = {
            AWS_ACCESS_KEY_ID = env("AWS_ACCESS_KEY_ID")
            AWS_SECRET_ACCESS_KEY = env("AWS_SECRET_ACCESS_KEY")
        }
        password_file = "/etc/backup/db.pwd"
    }

    targets = [
        {
            name = "postgresql"
            paths = ["/var/lib/postgresql"]
            excludes = ["pg_wal/*", "*.tmp"]
            tags = ["production", "daily"]
            pre_hooks = [
                "systemctl stop postgresql",
                "pg_dumpall -U postgres > /tmp/pg_backup.sql"
            ]
            post_hooks = [
                "rm /tmp/pg_backup.sql",
                "systemctl start postgresql"
            ]
        }
    ]

    retention = {
        keep_daily = 7
        keep_weekly = 4
        keep_monthly = 12
        keep_yearly = 5
    }

    schedule = {
        enabled = true
        cron = "0 2 * * *"
        init_system = "systemd"
        on_failure = "notify"
    }

    notifications = {
        enabled = true
        on_success = false
        on_failure = true
        command = "mail -s 'Backup alert' ops@company.com"
    }

    hooks = {
        on_error = [
            "echo 'Backup failed at $(date)' | mail -s 'Critical: Backup failure' ops@company.com"
        ]
    }
}

# Output the configuration
backup_config
```

### Integration with Init Services

Create `provisioning/backup-service.kcl`:

```kcl
import init_system

# Generate systemd service for backups
backup_service = init_system.Service {
    name = "prod-backup"
    description = "Production backup service"
    type = "oneshot"

    exec_start = "/usr/local/bin/backup-tool --config /etc/backup/prod.toml"

    # Run as backup user
    user = "backup"
    group = "backup"

    # Restart policy
    restart = "on-failure"
    restart_sec = 300
}

# Generate the timer
backup_timer = init_system.Timer {
    name = "prod-backup"
    description = "Production backup timer"

    # Run at 2 AM daily
    on_calendar = "*-*-* 02:00:00"
    unit = "prod-backup.service"

    persistent = true
}
```

## KCL Configuration Examples

### Simple Local Backup

```kcl
import backup

config = backup.BackupConfig {
    version = "1.0"
    name = "local-backup"
    backend = "tar"

    repository = {
        type = "local"
        path = "/mnt/backups/local"
    }

    targets = [
        {
            name = "home"
            paths = ["/home/user/documents", "/home/user/projects"]
            excludes = ["node_modules/*", ".git/*"]
        }
    ]

    retention = {
        keep_daily = 7
    }

    schedule = {
        enabled = true
        cron = "0 3 * * *"
        init_system = "cron"
    }
}
```

### Cloud S3 Backup with Encryption

```kcl
import backup

config = backup.BackupConfig {
    version = "1.0"
    name = "cloud-backup"
    backend = "restic"

    repository = {
        type = "s3"
        path = "s3://my-bucket/backups"
        password_file = "/etc/backup/restic.pwd"
        env_vars = {
            AWS_ACCESS_KEY_ID = env("AWS_KEY")
            AWS_SECRET_ACCESS_KEY = env("AWS_SECRET")
        }
    }

    targets = [
        {
            name = "app-data"
            paths = ["/var/lib/myapp", "/etc/myapp"]
            excludes = ["*.tmp", "*.log"]
            tags = ["production"]
        }
    ]

    retention = {
        keep_daily = 7
        keep_weekly = 4
        keep_monthly = 12
    }

    schedule = {
        enabled = true
        cron = "0 */6 * * *"  # Every 6 hours
        init_system = "systemd"
        on_failure = "notify"
    }
}
```

### Multi-Target SFTP Backup

```kcl
import backup

config = backup.BackupConfig {
    version = "1.0"
    name = "sftp-backup"
    backend = "rsync"

    repository = {
        type = "sftp"
        path = "sftp://backup.example.com/backups"
        env_vars = {
            RSYNC_RSH = "ssh -i /etc/backup/id_rsa"
        }
    }

    targets = [
        {
            name = "web-content"
            paths = ["/var/www/html", "/var/www/uploads"]
            excludes = ["*.temp", "cache/*"]
            tags = ["web"]
        }
        {
            name = "configs"
            paths = ["/etc", "/opt/app"]
            excludes = ["/etc/ssl/private"]
            tags = ["system"]
        }
    ]

    retention = {
        keep_daily = 7
        keep_weekly = 4
    }

    schedule = {
        enabled = true
        cron = "30 1 * * *"
        init_system = "systemd"
    }
}
```

## Automation Workflows

### Database Backup Workflow

```rust
use backup::config::BackupConfig;
use backup::backends::ResticBackend;
use backup::backend::BackupBackend;

#[tokio::main]
async fn backup_database() -> Result<(), Box<dyn std::error::Error>> {
    // Pre-backup: dump database
    println!("Dumping database...");
    std::process::Command::new("pg_dump")
        .arg("mydb")
        .arg("-f")
        .arg("/tmp/db_backup.sql")
        .output()?;

    // Run backup
    let config = BackupConfig::from_toml("backup.toml")?;
    let backend = ResticBackend::new(None);

    for target in config.targets {
        println!("Backing up: {}", target.name);
        let report = backend.backup(&target).await?;

        if !report.success {
            eprintln!("Backup failed!");
            // Trigger error handlers
            for cmd in config.hooks.on_error {
                std::process::Command::new("sh")
                    .arg("-c")
                    .arg(&cmd)
                    .output()?;
            }
            return Err("Backup failed".into());
        }
    }

    // Post-backup: cleanup and prune
    println!("Cleaning up...");
    std::fs::remove_file("/tmp/db_backup.sql")?;

    println!("Pruning old snapshots...");
    backend.prune(&config.retention).await?;

    // Verify
    let check = backend.check().await?;
    if !check.valid {
        eprintln!("Repository check failed!");
    }

    println!("✓ Backup complete");
    Ok(())
}
```

### Incremental Backup with Verification

```rust
use backup::config::BackupConfig;
use backup::backends::ResticBackend;
use backup::backend::BackupBackend;

#[tokio::main]
async fn verified_backup() -> Result<(), Box<dyn std::error::Error>> {
    let config = BackupConfig::from_toml("backup.toml")?;
    let backend = ResticBackend::new(None)?;

    // Step 1: Check repository before backup
    println!("Step 1: Checking repository...");
    let pre_check = backend.check().await?;
    if !pre_check.valid {
        eprintln!("Repository has issues before backup!");
        return Err("Pre-backup check failed".into());
    }

    // Step 2: Run backups
    println!("Step 2: Running backups...");
    for target in config.targets {
        let report = backend.backup(&target).await?;
        println!("  {} - {} files processed", target.name, report.files_processed);
    }

    // Step 3: Check repository after backup
    println!("Step 3: Post-backup verification...");
    let post_check = backend.check().await?;
    if !post_check.valid {
        eprintln!("Repository has issues after backup!");
        return Err("Post-backup check failed".into());
    }

    // Step 4: Get and display stats
    println!("Step 4: Repository statistics");
    let stats = backend.stats().await?;
    println!("  Total size: {} MB", stats.total_size / (1024 * 1024));
    println!("  Snapshots: {}", stats.snapshot_count);

    println!("✓ Backup verified and complete");
    Ok(())
}
```

### Selective Restore

```rust
use backup::config::BackupConfig;
use backup::backends::ResticBackend;
use backup::backend::BackupBackend;
use backup::types::RestoreOptions;
use std::path::Path;

#[tokio::main]
async fn selective_restore() -> Result<(), Box<dyn std::error::Error>> {
    let config = BackupConfig::from_toml("backup.toml")?;
    let backend = ResticBackend::new(None)?;

    // List available snapshots
    println!("Available snapshots:");
    let snapshots = backend.list_snapshots().await?;
    for (idx, snapshot) in snapshots.iter().enumerate() {
        println!("  [{}] {} - {} bytes", idx, snapshot.id, snapshot.size);
    }

    // Restore specific snapshot
    let restore_opts = RestoreOptions {
        overwrite: false,
        verify: true,
        restore_permissions: true,
    };

    println!("\nRestoring latest snapshot...");
    if let Some(latest) = snapshots.first() {
        backend.restore(&latest.id, Path::new("/restore"), &restore_opts).await?;
        println!("✓ Restore complete");
    }

    Ok(())
}
```

## Testing Integration

Create integration tests in `crates/backup/tests/integration.rs`:

```rust
use backup::config::BackupConfig;
use backup::backends::TarBackend;
use backup::backend::BackupBackend;
use tempfile::TempDir;

#[tokio::test]
async fn test_tar_backup_and_restore() -> Result<(), Box<dyn std::error::Error>> {
    let temp_repo = TempDir::new()?;
    let temp_restore = TempDir::new()?;

    let backend = TarBackend::new(None);

    // Create test configuration
    let config = create_test_config(temp_repo.path().to_str().unwrap())?;

    // Initialize
    backend.init_repository(&config.repository).await?;

    // Backup
    let target = &config.targets[0];
    let report = backend.backup(target).await?;
    assert!(report.success);

    // List snapshots
    let snapshots = backend.list_snapshots().await?;
    assert!(!snapshots.is_empty());

    // Restore
    backend.restore(&snapshots[0].id, temp_restore.path(), &Default::default()).await?;

    Ok(())
}
```

## Monitoring and Alerting

Integrate with monitoring systems:

```rust
use backup::config::BackupConfig;
use backup::backends::ResticBackend;
use backup::backend::BackupBackend;

#[tokio::main]
async fn monitored_backup() -> Result<(), Box<dyn std::error::Error>> {
    let config = BackupConfig::from_toml("backup.toml")?;
    let backend = ResticBackend::new(None)?;

    let start = std::time::Instant::now();

    match perform_backups(&backend, &config).await {
        Ok(stats) => {
            let duration = start.elapsed();
            println!("gauge backup_duration_seconds {}", duration.as_secs());
            println!("gauge backup_files_processed {}", stats.files_processed);
            println!("gauge backup_size_bytes {}", stats.size_processed);
            println!("gauge backup_success 1");
        }
        Err(e) => {
            println!("gauge backup_success 0");
            eprintln!("Backup failed: {}", e);
        }
    }

    Ok(())
}

async fn perform_backups(
    backend: &(impl BackupBackend + ?Sized),
    config: &BackupConfig,
) -> Result<BackupStats, Box<dyn std::error::Error>> {
    // ... backup implementation
    Ok(Default::default())
}
```

## Next Steps

1. Install required backend tools (restic, borg, etc.)
2. Create backup configuration files
3. Test configurations with dry runs
4. Set up scheduling with init system
5. Configure monitoring and alerting
6. Document recovery procedures
7. Test restore procedures regularly