# Multi-Provider Deployment Guide

This guide covers strategies and patterns for deploying infrastructure across multiple cloud providers using the provisioning system. Multi-provider deployments enable high availability, disaster recovery, cost optimization, compliance with regional requirements, and vendor lock-in avoidance.

## Table of Contents

- [Overview](#overview)
- [Why Multiple Providers](#why-multiple-providers)
- [Provider Selection Strategy](#provider-selection-strategy)
- [Workspace Configuration](#workspace-configuration)
- [Architecture Patterns](#architecture-patterns)
  - [Pattern 1: Compute + Storage Split](#pattern-1-compute--storage-split)
  - [Pattern 2: Primary + Backup](#pattern-2-primary--backup)
  - [Pattern 3: Multi-Region High Availability](#pattern-3-multi-region-high-availability)
  - [Pattern 4: Hybrid Cloud](#pattern-4-hybrid-cloud)
- [Implementation Examples](#implementation-examples)
- [Best Practices](#best-practices)
- [Troubleshooting](#troubleshooting)

## Overview

The provisioning system provides a provider-agnostic abstraction layer that enables seamless deployment across Hetzner, UpCloud, AWS, and DigitalOcean. Each provider implements a standard interface with compute, storage, networking, and management capabilities.

### Supported Providers

| Provider | Compute | Storage | Load Balancer | Managed Services | Network Isolation |
|----------|---------|---------|---------------|------------------|-------------------|
| Hetzner | Cloud Servers | Volumes | Load Balancer | No | vSwitch/Private Networks |
| UpCloud | Servers | Storage | Load Balancer | No | VLAN |
| AWS | EC2 | EBS/S3 | ALB/NLB | RDS, ElastiCache, etc | VPC/Security Groups |
| DigitalOcean | Droplets | Volumes | Load Balancer | Managed DB | VPC/Firewall |

### Key Concepts

- **Provider Abstraction**: Consistent interface across all providers hides provider-specific details
- **Workspace**: Defines infrastructure components, resource allocation, and provider configuration
- **Multi-Provider Workspace**: A single workspace that spans multiple providers with coordinated deployment
- **Batch Workflows**: Orchestrate deployment across providers with dependency tracking and rollback capability

## Why Multiple Providers

### Cost Optimization

Different providers excel at different workloads:

- **Compute-Heavy**: Hetzner offers best price/performance ratio for compute-intensive workloads
- **Managed Services**: AWS RDS or DigitalOcean Managed Databases often more cost-effective than self-managed
- **Storage-Intensive**: AWS S3 or Google Cloud Storage for large object storage requirements
- **Edge Locations**: DigitalOcean's CDN and global regions for geographically distributed serving

**Example**: Store application data in Hetzner compute nodes (cost-effective), analytics database in AWS RDS (managed), and backups in DigitalOcean Spaces (affordable object storage).

### High Availability and Disaster Recovery

- **Active-Active**: Run identical infrastructure in multiple providers for load balancing
- **Active-Standby**: Primary on Provider A, warm standby on Provider B with automated failover
- **Multi-Region**: Distribute across geographic regions within and between providers
- **Time-to-Recovery**: Multiple providers reduce dependency on single provider's infrastructure

### Compliance and Data Residency

- **GDPR**: European data must stay in EU providers (Hetzner DE, UpCloud FI/SE)
- **Regional Requirements**: Some compliance frameworks require data in specific countries
- **Provider Certifications**: Different providers have different compliance certifications (SOC2, ISO 27001, HIPAA)

**Example**: Production data in Hetzner (EU-based), analytics in AWS (GDPR-compliant regions), backups in DigitalOcean.

### Vendor Lock-in Avoidance

- **Portability**: Multi-provider setup enables migration without complete outage
- **Flexibility**: Switch providers for cost negotiation or service issues
- **Resilience**: Not dependent on single provider's reliability or pricing changes

### Performance and Latency

- **Geographic Distribution**: Serve users from nearest provider
- **Provider-Specific Performance**: Some providers have better infrastructure for specific regions
- **Regional Redundancy**: Maintain service availability during provider-wide outages

## Provider Selection Strategy

### Decision Framework

#### 1. Workload Characteristics

**Compute-Intensive (batch processing, ML, heavy calculations)**
- Recommended: Hetzner (best price), UpCloud (mid-range)
- Avoid: AWS on-demand (unless spot instances), DigitalOcean premium tier

**Web/Application (stateless serving, APIs)**
- Recommended: DigitalOcean (simple management), Hetzner (cost), AWS (multi-region)
- Consider: Geographic proximity to users

**Stateful/Database (databases, caches, queues)**
- Recommended: AWS RDS/ElastiCache, DigitalOcean Managed DB
- Alternative: Self-managed on any provider with replication

**Storage/File Serving (object storage, backups)**
- Recommended: AWS S3, DigitalOcean Spaces, Hetzner Object Storage
- Consider: Cost per GB, access patterns, bandwidth

### Regional Availability

**North America**
- AWS: Multiple regions (us-east-1, us-west-2, etc)
- DigitalOcean: NYC, SFO
- Hetzner: Ashburn, Virginia
- UpCloud: Multiple US locations

**Europe**
- Hetzner: Falkenstein (DE), Nuremberg (DE), Helsinki (FI)
- UpCloud: Multiple EU locations
- AWS: eu-west-1 (IE), eu-central-1 (DE), etc
- DigitalOcean: London, Frankfurt, Amsterdam

**Asia**
- AWS: ap-southeast-1 (SG), ap-northeast-1 (Tokyo)
- DigitalOcean: Singapore, Bangalore
- Hetzner: Limited
- UpCloud: Singapore

**Recommendation for Multi-Region**: Combine Hetzner (EU backbone), DigitalOcean (global presence), AWS (comprehensive regions).

### Cost Analysis

#### Monthly Compute Comparison (2 vCPU, 4 GB RAM)

| Provider | Price | Notes |
|----------|-------|-------|
| Hetzner | €6.90 (~$7.50) | Cheapest, good performance |
| DigitalOcean | $24 | Premium pricing, simplicity |
| UpCloud | $30 | Mid-range, good support |
| AWS t3.medium | $60+ | On-demand pricing (spot: $18-25) |

#### Recommendations by Budget

**Minimal Budget (<$50/month)**
- Single Hetzner server: €6.90
- Alternative: DigitalOcean $24 + DigitalOcean Spaces for backup

**Small Team ($100-500/month)**
- Hetzner primary (€50-150), DigitalOcean backup (60-80)
- Good HA coverage with cost control

**Enterprise ($1000+/month)**
- AWS primary (managed services, compliance)
- Hetzner backup (cost-effective)
- DigitalOcean edge locations (CDN)

### Compliance and Certifications

| Provider | GDPR | SOC 2 | ISO 27001 | HIPAA | FIPS | PCI-DSS |
|----------|------|-------|-----------|-------|------|---------|
| Hetzner | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |
| UpCloud | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |
| AWS | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| DigitalOcean | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

**Compliance Selection Matrix**

- **GDPR Only**: Hetzner, UpCloud (EU-based), all AWS/DO EU regions
- **HIPAA Required**: AWS, DigitalOcean (DigitalOcean requires BAA)
- **FIPS Required**: AWS (all regions)
- **PCI-DSS**: All providers support, AWS most comprehensive

## Workspace Configuration

### Multi-Provider Workspace Structure

```bash
provisioning/examples/workspaces/my-multi-provider-app/
├── workspace.ncl                # Infrastructure definition
├── config.toml                  # Provider credentials, regions, defaults
├── README.md                    # Setup and deployment instructions
└── deploy.nu                    # Deployment orchestration script
```

### Provider Credential Management

#### Environment Variables

Each provider requires authentication via environment variables:

```bash
# Hetzner
export HCLOUD_TOKEN="your-hetzner-api-token"

# UpCloud
export UPCLOUD_USERNAME="your-upcloud-username"
export UPCLOUD_PASSWORD="your-upcloud-password"

# AWS
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_DEFAULT_REGION="us-east-1"

# DigitalOcean
export DIGITALOCEAN_TOKEN="your-do-api-token"
```

#### Configuration File Structure (config.toml)

```toml
[providers]

[providers.hetzner]
enabled = true
api_token_env = "HCLOUD_TOKEN"
default_region = "nbg1"
default_datacenter = "nbg1-dc8"

[providers.upcloud]
enabled = true
username_env = "UPCLOUD_USERNAME"
password_env = "UPCLOUD_PASSWORD"
default_region = "fi-hel1"

[providers.aws]
enabled = true
region = "us-east-1"
access_key_env = "AWS_ACCESS_KEY_ID"
secret_key_env = "AWS_SECRET_ACCESS_KEY"

[providers.digitalocean]
enabled = true
token_env = "DIGITALOCEAN_TOKEN"
default_region = "nyc3"

[workspace]
name = "my-multi-provider-app"
environment = "production"
owner = "platform-team"
```

### Multi-Provider Workspace Definition

Nickel workspace with multiple providers:

```nickel
# workspace.ncl - Multi-provider infrastructure definition

let hetzner = import "../../extensions/providers/hetzner/nickel/main.ncl" in
let upcloud = import "../../extensions/providers/upcloud/nickel/main.ncl" in
let aws = import "../../extensions/providers/aws/nickel/main.ncl" in
let digitalocean = import "../../extensions/providers/digitalocean/nickel/main.ncl" in

{
  workspace_name = "multi-provider-app",
  description = "Multi-provider infrastructure example",

  # Provider routing configuration
  providers = {
    primary_compute = "hetzner",
    secondary_compute = "digitalocean",
    database = "aws",
    backup = "upcloud"
  },

  # Infrastructure defined per provider
  infrastructure = {
    # Hetzner: Primary compute tier
    primary_servers = hetzner.Server & {
      name = "primary-server",
      server_type = "cx31",
      image = "ubuntu-22.04",
      location = "nbg1",
      count = 3,
      ssh_keys = ["your-ssh-key"],
      firewalls = ["primary-fw"]
    },

    # DigitalOcean: Secondary compute tier
    secondary_servers = digitalocean.Droplet & {
      name = "secondary-droplet",
      size = "s-2vcpu-4gb",
      image = "ubuntu-22-04-x64",
      region = "nyc3",
      count = 2
    },

    # AWS: Managed database
    database = aws.RDS & {
      identifier = "prod-db",
      engine = "postgresql",
      engine_version = "14.6",
      instance_class = "db.t3.medium",
      allocated_storage = 100
    },

    # UpCloud: Backup storage
    backup_storage = upcloud.Storage & {
      name = "backup-volume",
      size = 500,
      location = "fi-hel1"
    }
  }
}
```

## Architecture Patterns

### Pattern 1: Compute + Storage Split

**Scenario**: Cost-effective compute with specialized managed storage.

**Example**: Use Hetzner for compute (cheap), AWS S3 for object storage (reliable), managed database on AWS RDS.

#### Benefits

- Compute optimization (Hetzner's low cost)
- Storage specialization (AWS S3 reliability and features)
- Separation of concerns (different performance tuning)

#### Architecture

```bash
                    ┌─────────────────────┐
                    │   Client Requests   │
                    └──────────┬──────────┘
                               │
                ┌──────────────┼──────────────┐
                │              │              │
         ┌──────▼─────┐  ┌────▼─────┐  ┌───▼──────┐
         │  Hetzner   │  │    AWS   │  │ AWS S3   │
         │  Servers   │  │    RDS   │  │ Storage  │
         │ (Compute)  │  │(Database)│  │(Backups) │
         └────────────┘  └──────────┘  └──────────┘
```

#### Nickel Configuration

```nickel
let hetzner = import "../../extensions/providers/hetzner/nickel/main.ncl" in
let aws = import "../../extensions/providers/aws/nickel/main.ncl" in

{
  compute = hetzner.Server & {
    name = "app-server",
    server_type = "cpx21",  # 4 vCPU, 8 GB RAM
    image = "ubuntu-22.04",
    location = "nbg1",
    count = 2,
    volumes = [
      {
        size = 100,
        format = "ext4",
        mount = "/app"
      }
    ]
  },

  database = aws.RDS & {
    identifier = "app-database",
    engine = "postgresql",
    instance_class = "db.t3.medium",
    allocated_storage = 100
  },

  backup_bucket = aws.S3 & {
    bucket = "app-backups",
    region = "us-east-1",
    versioning = true,
    lifecycle_rules = [
      {
        id = "delete-old-backups",
        days = 90,
        action = "delete"
      }
    ]
  }
}
```

#### Network Configuration

Hetzner servers connect to AWS RDS via VPN or public endpoint:

```nushell
# Network setup script
def setup_database_connection [] {
  let hetzner_servers = (hetzner_list_servers)
  let db_endpoint = (aws_get_rds_endpoint "app-database")

  # Install PostgreSQL client
  $hetzner_servers | each {|server|
    ssh $server.ip "apt-get install -y postgresql-client"
    ssh $server.ip $"echo 'DB_HOST=($db_endpoint)' >> /app/.env"
  }
}
```

#### Cost Analysis

Monthly estimate:
- Hetzner cx31 × 2: €13.80 (~$15)
- AWS RDS t3.medium: $60
- AWS S3 (100 GB): $2.30
- **Total: ~$77/month** (vs $120+ for all-AWS)

### Pattern 2: Primary + Backup

**Scenario**: Active-standby deployment for disaster recovery.

**Example**: DigitalOcean primary datacenter, Hetzner warm standby with automated failover.

#### Benefits

- Disaster recovery capability
- Zero data loss (with replication)
- Tested failover procedure
- Cost-effective backup (warm standby vs hot standby)

#### Architecture

```bash
         Primary (DigitalOcean NYC)        Backup (Hetzner DE)
         ┌──────────────────────┐          ┌─────────────────┐
         │   DigitalOcean LB    │◄────────►│ HAProxy Monitor │
         └──────────┬───────────┘          └────────┬────────┘
                    │                               │
         ┌──────────┴──────────┐                    │
         │                     │                    │
     ┌───▼───┐ ┌───▼───┐   ┌──▼──┐ ┌──────┐    ┌──▼───┐
     │ APP 1 │ │ APP 2 │   │ DB  │ │ ELK  │    │ WARM │
     │PRIMARY│ │PRIMARY│   │REPL │ │MON   │    │STANDBY
     └───────┘ └───────┘   └─────┘ └──────┘    └──────┘
         │                     │                    ▲
         └─────────────────────┼────────────────────┘
                        Async Replication
```

#### Failover Trigger

```nushell
def monitor_primary_health [do_region, hetzner_region] {
  loop {
    let health = (do_health_check $do_region)

    if $health.status == "degraded" or $health.status == "down" {
      print "Primary degraded, triggering failover"
      trigger_failover $hetzner_region
      break
    }

    sleep 30sec
  }
}

def trigger_failover [backup_region] {
  # 1. Promote backup database
  promote_replica_to_primary $backup_region

  # 2. Update DNS to point to backup
  update_dns_to_backup $backup_region

  # 3. Scale up backup servers
  scale_servers $backup_region 3

  # 4. Verify traffic flowing
  wait_for_traffic_migration $backup_region 120sec
}
```

#### Nickel Configuration

```nickel
let digitalocean = import "../../extensions/providers/digitalocean/nickel/main.ncl" in
let hetzner = import "../../extensions/providers/hetzner/nickel/main.ncl" in

{
  # Primary: DigitalOcean
  primary = {
    region = "nyc3",
    provider = "digitalocean",

    servers = digitalocean.Droplet & {
      name = "primary-app",
      size = "s-2vcpu-4gb",
      count = 3,
      region = "nyc3",
      firewall = {
        inbound = [
          { protocol = "tcp", ports = "80", sources = ["0.0.0.0/0"] },
          { protocol = "tcp", ports = "443", sources = ["0.0.0.0/0"] },
          { protocol = "tcp", ports = "5432", sources = ["10.0.0.0/8"] }
        ]
      }
    },

    database = digitalocean.Database & {
      name = "primary-db",
      engine = "pg",
      version = "14",
      size = "db-s-2vcpu-4gb",
      region = "nyc3"
    }
  },

  # Backup: Hetzner (warm standby)
  backup = {
    region = "nbg1",
    provider = "hetzner",

    servers = hetzner.Server & {
      name = "backup-app",
      server_type = "cx31",
      count = 1,  # Minimal for cost
      location = "nbg1",
      automount = true
    },

    # Replica database (read-only until promoted)
    database_replica = hetzner.Volume & {
      name = "db-replica",
      size = 100,
      location = "nbg1"
    }
  },

  replication = {
    type = "async",
    primary_to_backup = true,
    recovery_point_objective = 300  # 5 minutes
  }
}
```

#### Failover Testing

```nushell
# Test failover without affecting production
def test_failover_dry_run [config] {
  print "Starting failover dry-run test..."

  # 1. Snapshot primary database
  let snapshot = (do_create_db_snapshot "primary-db")

  # 2. Create temporary replica from snapshot
  let temp_replica = (hetzner_create_from_snapshot $snapshot)

  # 3. Run traffic tests against temp replica
  let test_results = (run_integration_tests $temp_replica.ip)

  # 4. Verify database consistency
  let consistency = (verify_db_consistency $temp_replica.ip)

  # 5. Cleanup temp resources
  hetzner_destroy $temp_replica.id
  do_delete_snapshot $snapshot.id

  {
    status: "passed",
    results: $test_results,
    consistency_check: $consistency
  }
}
```

### Pattern 3: Multi-Region High Availability

**Scenario**: Distributed deployment across 3+ geographic regions with global load balancing.

**Example**: DigitalOcean US (NYC), Hetzner EU (Germany), AWS Asia (Singapore) with DNS-based failover.

#### Benefits

- Geographic distribution for low latency
- Protection against regional outages
- Compliance with data residency (data stays in region)
- Load distribution across regions

#### Architecture

```bash
                    ┌─────────────────┐
                    │  Global DNS     │
                    │  (Geofencing)   │
                    └────────┬────────┘
                    ┌────────┴────────┐
                    │                 │
         ┌──────────▼──────┐  ┌──────▼─────────┐  ┌─────────────┐
         │  DigitalOcean   │  │  Hetzner       │  │    AWS      │
         │  US/NYC Region  │  │  EU/Germany    │  │  Asia/SG    │
         ├─────────────────┤  ├────────────────┤  ├─────────────┤
         │ Droplets (3)    │  │ Servers (3)    │  │ EC2 (3)     │
         │ LB              │  │ HAProxy        │  │ ALB         │
         │ DB (Primary)    │  │ DB (Replica)   │  │ DB (Replica)│
         └─────────────────┘  └────────────────┘  └─────────────┘
                    │                 │                    │
                    └─────────────────┴────────────────────┘
                           Cross-Region Sync
```

#### Global Load Balancing

```nushell
def setup_global_dns [] {
  # Using Route53 or Cloudflare for DNS failover
  let regions = [
    { name: "us-nyc", provider: "digitalocean", endpoint: "us.app.example.com" },
    { name: "eu-de", provider: "hetzner", endpoint: "eu.app.example.com" },
    { name: "asia-sg", provider: "aws", endpoint: "asia.app.example.com" }
  ]

  # Create health checks
  $regions | each {|region|
    configure_health_check $region.name $region.endpoint
  }

  # Setup failover policy
  # Primary: US, Secondary: EU, Tertiary: Asia
  configure_dns_failover {
    primary: "us-nyc",
    secondary: "eu-de",
    tertiary: "asia-sg"
  }
}
```

#### Nickel Configuration

```nickel
{
  regions = {
    us_east = {
      provider = "digitalocean",
      region = "nyc3",

      servers = digitalocean.Droplet & {
        name = "us-app",
        size = "s-2vcpu-4gb",
        count = 3,
        region = "nyc3"
      },

      database = digitalocean.Database & {
        name = "us-db",
        engine = "pg",
        size = "db-s-2vcpu-4gb",
        region = "nyc3",
        replica_regions = ["eu-de", "asia-sg"]
      }
    },

    eu_central = {
      provider = "hetzner",
      region = "nbg1",

      servers = hetzner.Server & {
        name = "eu-app",
        server_type = "cx31",
        count = 3,
        location = "nbg1"
      }
    },

    asia_southeast = {
      provider = "aws",
      region = "ap-southeast-1",

      servers = aws.EC2 & {
        name = "asia-app",
        instance_type = "t3.medium",
        count = 3,
        region = "ap-southeast-1"
      }
    }
  },

  global_config = {
    dns_provider = "route53",
    ttl = 60,
    health_check_interval = 30
  }
}
```

#### Data Synchronization

```nushell
# Multi-region data sync strategy
def sync_data_across_regions [primary_region, secondary_regions] {
  let sync_config = {
    strategy: "async",
    consistency: "eventual",
    conflict_resolution: "last-write-wins",
    replication_lag: "300s"  # 5 minute max lag
  }

  # Setup replication from primary to all secondaries
  $secondary_regions | each {|region|
    setup_async_replication $primary_region $region $sync_config
  }

  # Monitor replication lag
  loop {
    let lag = (check_replication_lag)
    if $lag > 300 {
      print "Warning: replication lag exceeds threshold"
      trigger_alert "replication-lag-warning"
    }
    sleep 60sec
  }
}
```

### Pattern 4: Hybrid Cloud

**Scenario**: On-premises infrastructure with public cloud providers for burst capacity and backup.

**Example**: On-premise data center + AWS for burst capacity + DigitalOcean for disaster recovery.

#### Benefits

- Existing infrastructure utilization
- Burst capacity in public cloud
- Disaster recovery site
- Compliance with on-premise requirements
- Cost control (scale only when needed)

#### Architecture

```bash
    On-Premises Data Center           Public Cloud (Burst)
    ┌─────────────────────────┐      ┌────────────────────┐
    │  Physical Servers       │◄────►│  AWS Auto-Scaling  │
    │  - App Tier (24 cores)  │      │  - Elasticity      │
    │  - DB Tier (48 cores)   │      │  - Pay-as-you-go   │
    │  - Storage (50 TB)       │      │  - CloudFront CDN  │
    └─────────────────────────┘      └────────────────────┘
               │                               ▲
               │ VPN Tunnel                    │
               └───────────────────────────────┘

    On-Premises                        DR Site (DigitalOcean)
    │ Production                        │ Warm Standby
    ├─ 95% Utilization                  ├─ Cold VM Snapshots
    ├─ Full Data                        ├─ Async Replication
    ├─ Peak Load Handling               ├─ Ready for 15 min RTO
    │                                   │
```

#### VPN Configuration

```nushell
def setup_hybrid_vpn [] {
  # AWS VPN to on-premise datacenter
  let vpn_config = {
    type: "site-to-site",
    protocol: "ipsec",
    encryption: "aes-256",
    authentication: "sha256",
    on_prem_cidr: "192.168.0.0/16",
    aws_cidr: "10.0.0.0/16",
    do_cidr: "172.16.0.0/16"
  }

  # Create AWS Site-to-Site VPN
  let vpn = (aws_create_vpn_connection $vpn_config)

  # Configure on-prem gateway
  configure_on_prem_vpn_gateway $vpn

  # Verify tunnel status
  wait_for_vpn_ready 300
}
```

#### Nickel Configuration

```nickel
{
  on_premises = {
    provider = "manual",
    gateway = "192.168.1.1",
    cidr = "192.168.0.0/16",
    bandwidth = "1gbps",

    # Resources remain on-prem (managed manually)
    servers = {
      app_tier = { cores = 24, memory = 128 },
      db_tier = { cores = 48, memory = 256 },
      storage = { capacity = "50 TB" }
    }
  },

  aws_burst_capacity = {
    provider = "aws",
    region = "us-east-1",

    auto_scaling_group = aws.ASG & {
      name = "burst-asg",
      min_size = 0,
      desired_capacity = 0,
      max_size = 20,
      instance_type = "c5.2xlarge",
      scale_up_trigger = "on_prem_cpu > 80%",
      scale_down_trigger = "on_prem_cpu < 40%"
    },

    cdn = aws.CloudFront & {
      origin = "on-prem-origin",
      regional_origins = ["us-east-1", "eu-west-1", "ap-southeast-1"]
    }
  },

  dr_site = {
    provider = "digitalocean",
    region = "nyc3",

    snapshot_storage = digitalocean.Droplet & {
      name = "dr-snapshot",
      size = "s-24vcpu-48gb",
      count = 0,  # Powered off until needed
      image = "on-prem-snapshot"
    }
  },

  replication = {
    on_prem_to_aws: {
      strategy = "continuous",
      target = "aws-s3-bucket",
      retention = "7days"
    },

    on_prem_to_do: {
      strategy = "nightly",
      target = "do-spaces-bucket",
      retention = "30days"
    }
  }
}
```

#### Burst Capacity Orchestration

```nushell
# Monitor on-prem and trigger AWS burst
def monitor_and_burst [] {
  loop {
    let on_prem_metrics = (collect_on_prem_metrics)

    if $on_prem_metrics.cpu_avg > 80 {
      # Trigger AWS burst scaling
      let scale_size = ((100 - $on_prem_metrics.cpu_avg) / 10)
      scale_aws_burst $scale_size
    } else if $on_prem_metrics.cpu_avg < 40 {
      # Scale down AWS
      scale_aws_burst 0
    }

    sleep 60sec
  }
}
```

## Implementation Examples

### Example 1: Three-Provider Web Application

**Scenario**: Production web application with DigitalOcean web servers, AWS managed database, and Hetzner backup storage.

**Architecture**:
- DigitalOcean: 3 web servers with load balancer (cost-effective compute)
- AWS: RDS PostgreSQL database (managed, high availability)
- Hetzner: Backup volumes (low-cost storage)

**Files to Create**:

**workspace.ncl**:

```nickel
let digitalocean = import "../../extensions/providers/digitalocean/nickel/main.ncl" in
let aws = import "../../extensions/providers/aws/nickel/main.ncl" in
let hetzner = import "../../extensions/providers/hetzner/nickel/main.ncl" in

{
  workspace_name = "three-provider-webapp",
  description = "Web application across three providers",

  infrastructure = {
    web_tier = digitalocean.Droplet & {
      name = "web-server",
      region = "nyc3",
      size = "s-2vcpu-4gb",
      image = "ubuntu-22-04-x64",
      count = 3,
      firewall = {
        inbound_rules = [
          { protocol = "tcp", ports = "22", sources = { addresses = ["your-ip/32"] } },
          { protocol = "tcp", ports = "80", sources = { addresses = ["0.0.0.0/0"] } },
          { protocol = "tcp", ports = "443", sources = { addresses = ["0.0.0.0/0"] } }
        ],
        outbound_rules = [
          { protocol = "tcp", destinations = { addresses = ["0.0.0.0/0"] } }
        ]
      }
    },

    load_balancer = digitalocean.LoadBalancer & {
      name = "web-lb",
      algorithm = "round_robin",
      region = "nyc3",
      forwarding_rules = [
        {
          entry_protocol = "http",
          entry_port = 80,
          target_protocol = "http",
          target_port = 80,
          certificate_id = null
        },
        {
          entry_protocol = "https",
          entry_port = 443,
          target_protocol = "http",
          target_port = 80,
          certificate_id = "your-cert-id"
        }
      ],
      sticky_sessions = {
        type = "cookies",
        cookie_name = "lb",
        cookie_ttl_seconds = 300
      }
    },

    database = aws.RDS & {
      identifier = "webapp-db",
      engine = "postgres",
      engine_version = "14.6",
      instance_class = "db.t3.medium",
      allocated_storage = 100,
      storage_type = "gp3",
      multi_az = true,
      backup_retention_days = 30,
      subnet_group = "default",
      parameter_group = "default.postgres14",
      tags = [
        { key = "Environment", value = "production" },
        { key = "Application", value = "web-app" }
      ]
    },

    backup_volume = hetzner.Volume & {
      name = "webapp-backups",
      size = 500,
      location = "nbg1",
      automount = false,
      format = "ext4"
    }
  }
}
```

**config.toml**:

```toml
[workspace]
name = "three-provider-webapp"
environment = "production"
owner = "platform-team"

[providers.digitalocean]
enabled = true
token_env = "DIGITALOCEAN_TOKEN"
default_region = "nyc3"

[providers.aws]
enabled = true
region = "us-east-1"
access_key_env = "AWS_ACCESS_KEY_ID"
secret_key_env = "AWS_SECRET_ACCESS_KEY"

[providers.hetzner]
enabled = true
token_env = "HCLOUD_TOKEN"
default_location = "nbg1"

[deployment]
strategy = "rolling"
batch_size = 1
health_check_wait = 60
rollback_on_failure = true
```

**deploy.nu**:

```nushell
#!/usr/bin/env nu

# Deploy three-provider web application
def main [environment = "staging"] {
  print "Deploying three-provider web application to ($environment)..."

  # 1. Validate configuration
  print "Step 1: Validating configuration..."
  validate_config "workspace.ncl"

  # 2. Create infrastructure
  print "Step 2: Creating infrastructure..."
  create_digitalocean_resources
  create_aws_resources
  create_hetzner_resources

  # 3. Configure networking
  print "Step 3: Configuring networking..."
  setup_vpc_peering
  configure_security_groups

  # 4. Deploy application
  print "Step 4: Deploying application..."
  deploy_app_to_web_servers

  # 5. Verify deployment
  print "Step 5: Verifying deployment..."
  verify_health_checks
  verify_database_connectivity
  verify_backups

  print "Deployment complete!"
}

def validate_config [config_file] {
  print $"Validating ($config_file)..."
  nickel export $config_file | from json
}

def create_digitalocean_resources [] {
  print "Creating DigitalOcean resources (3 droplets + load balancer)..."
  # Implementation
}

def create_aws_resources [] {
  print "Creating AWS resources (RDS database)..."
  # Implementation
}

def create_hetzner_resources [] {
  print "Creating Hetzner resources (backup volume)..."
  # Implementation
}

def setup_vpc_peering [] {
  print "Setting up cross-provider networking..."
  # Implementation
}

def configure_security_groups [] {
  print "Configuring security groups..."
  # Implementation
}

def deploy_app_to_web_servers [] {
  print "Deploying application..."
  # Implementation
}

def verify_health_checks [] {
  print "Verifying health checks..."
  # Implementation
}

def verify_database_connectivity [] {
  print "Verifying database connectivity..."
  # Implementation
}

def verify_backups [] {
  print "Verifying backup configuration..."
  # Implementation
}

main $env.ENVIRONMENT?
```

### Example 2: Multi-Region Disaster Recovery

**Scenario**: Active-standby DR setup with DigitalOcean primary and Hetzner backup.

**Architecture**:
- DigitalOcean NYC: Production environment (active)
- Hetzner Germany: Warm standby (scales down until needed)
- Async database replication
- DNS-based failover
- RPO: 5 minutes, RTO: 15 minutes

### Example 3: Cost-Optimized Deployment

**Scenario**: Optimize across provider strengths: Hetzner compute, AWS managed services, DigitalOcean CDN.

**Architecture**:
- Hetzner: 5 application servers (best compute price)
- AWS: RDS database, ElastiCache (managed services)
- DigitalOcean: Spaces for backups, CDN endpoints

## Best Practices

### 1. Provider Selection

- **Document provider choices**: Keep record of which workloads run where and why
- **Audit provider capabilities**: Ensure chosen provider supports required features
- **Monitor provider health**: Track outages and issues per provider
- **Cost tracking per provider**: Understand where money is spent

### 2. Network Security

- **Encrypt inter-provider traffic**: Use VPN, mTLS, or encrypted tunnels
- **Implement firewall rules**: Limit traffic between providers to necessary ports
- **Use security groups**: AWS-style security groups where available
- **Monitor network traffic**: Detect unusual patterns across providers

### 3. Data Consistency

- **Choose replication strategy**: Synchronous (consistency), asynchronous (performance)
- **Implement conflict resolution**: Define how conflicts are resolved
- **Monitor replication lag**: Alert on excessive lag
- **Test failover regularly**: Verify data integrity during failover

### 4. Disaster Recovery

- **Define RPO/RTO targets**: Recovery Point Objective and Recovery Time Objective
- **Document failover procedures**: Step-by-step instructions
- **Test failover regularly**: At least quarterly, ideally monthly
- **Maintain DR site readiness**: Cold, warm, or hot standby based on RTO

### 5. Compliance and Governance

- **Data residency**: Ensure data stays in required regions
- **Encryption at rest**: Use provider-native encryption
- **Encryption in transit**: TLS/mTLS for all inter-provider communication
- **Audit logging**: Enable audit logs in all providers
- **Access control**: Implement least privilege across all providers

### 6. Monitoring and Alerting

- **Unified monitoring**: Aggregate metrics from all providers
- **Cross-provider dashboards**: Visualize health across providers
- **Provider-specific alerts**: Configure alerts per provider
- **Escalation procedures**: Clear escalation for failures

### 7. Cost Management

- **Set budget alerts**: Per provider and total
- **Reserved instances**: Use provider discounts
- **Spot instances**: AWS spot for non-critical workloads
- **Auto-scaling policies**: Scale based on demand
- **Regular cost reviews**: Monthly cost analysis and optimization

## Troubleshooting

### Issue: Network Connectivity Between Providers

**Symptoms**: Droplets can't reach AWS database, high latency between regions

**Diagnosis**:

```nushell
# Check network connectivity
def diagnose_network_issue [source_ip, dest_ip] {
  print "Diagnosing network connectivity..."

  # 1. Check routing
  ssh $source_ip "ip route show"

  # 2. Check firewall rules
  check_security_groups $source_ip $dest_ip

  # 3. Test connectivity
  ssh $source_ip "ping -c 3 $dest_ip"
  ssh $source_ip "traceroute $dest_ip"

  # 4. Check DNS resolution
  ssh $source_ip "nslookup $dest_ip"
}
```

**Solutions**:
- Verify firewall rules allow traffic on required ports
- Check VPN tunnel status if using site-to-site VPN
- Verify DNS resolution in both providers
- Check MTU size for jumbo frames (1500 bytes)
- Enable debug logging on network components

### Issue: Database Replication Lag

**Symptoms**: Secondary database lagging behind primary

**Diagnosis**:

```nushell
def check_replication_lag [] {
  # AWS RDS
  aws rds describe-db-instances --query 'DBInstances[].{ID:DBInstanceIdentifier,Lag:ReplicationLag}'

  # DigitalOcean
  doctl databases backups list --format Name,Created
}
```

**Solutions**:
- Check network bandwidth between providers
- Review write throughput on primary
- Monitor CPU/IO on secondary
- Adjust replication thread pool size
- Check for long-running queries blocking replication

### Issue: Failover Not Working

**Symptoms**: Failover script fails, DNS not updating

**Diagnosis**:

```nushell
def test_failover_chain [] {
  # 1. Verify backup infrastructure is ready
  verify_backup_infrastructure

  # 2. Test DNS failover
  test_dns_failover

  # 3. Verify database promotion
  test_db_promotion

  # 4. Check application configuration
  verify_app_failover_config
}
```

**Solutions**:
- Ensure backup infrastructure is powered on and running
- Verify DNS TTL is appropriate (typically 60 seconds)
- Test failover in staging environment first
- Check VPN connectivity to backup provider
- Verify database promotion scripts
- Ensure application connection strings support both endpoints

### Issue: Cost Spike Across Providers

**Symptoms**: Monthly bill unexpectedly high

**Diagnosis**:

```nushell
def analyze_cost_spike [] {
  print "Analyzing cost spike..."

  # Compare current vs previous month
  let current = (get_current_month_costs)
  let previous = (get_previous_month_costs)
  let delta = ($current - $previous)

  # Break down by provider
  $current | group-by provider | each {|group|
    let provider = ($group.0.provider)
    let cost = ($group | map {|x| $x.cost} | math sum)
    print $"($provider): $($cost)"
  }

  # Identify largest increases
  ($delta | sort-by cost_change | reverse | first 5)
}
```

**Solutions**:
- Review auto-scaling activities
- Check for unintended resource creation
- Verify reserved instances are being used
- Review data transfer costs (cross-region expensive)
- Cancel idle resources
- Contact provider support if billing seems incorrect

## Conclusion

Multi-provider deployments provide significant benefits in cost optimization, reliability, and compliance. Start with a simple pattern (Compute + Storage Split) and evolve to more complex patterns as needs grow. Always test failover procedures and maintain clear documentation of provider responsibilities and network configurations.

For more information, see:
- Provider-agnostic architecture guide
- Batch workflow orchestration guide
- Individual provider implementation guides
