# Multi-Provider Deployment Guide\n\nThis guide covers strategies and patterns for deploying infrastructure across multiple cloud providers using the provisioning system. Multi-provider\ndeployments enable high availability, disaster recovery, cost optimization, compliance with regional requirements, and vendor lock-in avoidance.\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Why Multiple Providers](#why-multiple-providers)\n- [Provider Selection Strategy](#provider-selection-strategy)\n- [Workspace Configuration](#workspace-configuration)\n- [Architecture Patterns](#architecture-patterns)\n  - [Pattern 1: Compute + Storage Split](#pattern-1-compute--storage-split)\n  - [Pattern 2: Primary + Backup](#pattern-2-primary--backup)\n  - [Pattern 3: Multi-Region High Availability](#pattern-3-multi-region-high-availability)\n  - [Pattern 4: Hybrid Cloud](#pattern-4-hybrid-cloud)\n- [Implementation Examples](#implementation-examples)\n- [Best Practices](#best-practices)\n- [Troubleshooting](#troubleshooting)\n\n## Overview\n\nThe provisioning system provides a provider-agnostic abstraction layer that enables seamless deployment across Hetzner, UpCloud, AWS, and\nDigitalOcean. Each provider implements a standard interface with compute, storage, networking, and management capabilities.\n\n### Supported Providers\n\n| Provider | Compute | Storage | Load Balancer | Managed Services | Network Isolation |\n| ---------- | --------- | --------- | --------------- | ------------------ | ------------------- |\n| Hetzner | Cloud Servers | Volumes | Load Balancer | No | vSwitch/Private Networks |\n| UpCloud | Servers | Storage | Load Balancer | No | VLAN |\n| AWS | EC2 | EBS/S3 | ALB/NLB | RDS, ElastiCache, etc | VPC/Security Groups |\n| DigitalOcean | Droplets | Volumes | Load Balancer | Managed DB | VPC/Firewall |\n\n### Key Concepts\n\n- **Provider Abstraction**: Consistent interface across all providers hides provider-specific details\n- **Workspace**: Defines infrastructure components, resource allocation, and provider configuration\n- **Multi-Provider Workspace**: A single workspace that spans multiple providers with coordinated deployment\n- **Batch Workflows**: Orchestrate deployment across providers with dependency tracking and rollback capability\n\n## Why Multiple Providers\n\n### Cost Optimization\n\nDifferent providers excel at different workloads:\n\n- **Compute-Heavy**: Hetzner offers best price/performance ratio for compute-intensive workloads\n- **Managed Services**: AWS RDS or DigitalOcean Managed Databases often more cost-effective than self-managed\n- **Storage-Intensive**: AWS S3 or Google Cloud Storage for large object storage requirements\n- **Edge Locations**: DigitalOcean's CDN and global regions for geographically distributed serving\n\n**Example**: Store application data in Hetzner compute nodes (cost-effective), analytics database in AWS RDS (managed), and backups in DigitalOcean\nSpaces (affordable object storage).\n\n### High Availability and Disaster Recovery\n\n- **Active-Active**: Run identical infrastructure in multiple providers for load balancing\n- **Active-Standby**: Primary on Provider A, warm standby on Provider B with automated failover\n- **Multi-Region**: Distribute across geographic regions within and between providers\n- **Time-to-Recovery**: Multiple providers reduce dependency on single provider's infrastructure\n\n### Compliance and Data Residency\n\n- **GDPR**: European data must stay in EU providers (Hetzner DE, UpCloud FI/SE)\n- **Regional Requirements**: Some compliance frameworks require data in specific countries\n- **Provider Certifications**: Different providers have different compliance certifications (SOC2, ISO 27001, HIPAA)\n\n**Example**: Production data in Hetzner (EU-based), analytics in AWS (GDPR-compliant regions), backups in DigitalOcean.\n\n### Vendor Lock-in Avoidance\n\n- **Portability**: Multi-provider setup enables migration without complete outage\n- **Flexibility**: Switch providers for cost negotiation or service issues\n- **Resilience**: Not dependent on single provider's reliability or pricing changes\n\n### Performance and Latency\n\n- **Geographic Distribution**: Serve users from nearest provider\n- **Provider-Specific Performance**: Some providers have better infrastructure for specific regions\n- **Regional Redundancy**: Maintain service availability during provider-wide outages\n\n## Provider Selection Strategy\n\n### Decision Framework\n\n#### 1. Workload Characteristics\n\n**Compute-Intensive (batch processing, ML, heavy calculations)**\n- Recommended: Hetzner (best price), UpCloud (mid-range)\n- Avoid: AWS on-demand (unless spot instances), DigitalOcean premium tier\n\n**Web/Application (stateless serving, APIs)**\n- Recommended: DigitalOcean (simple management), Hetzner (cost), AWS (multi-region)\n- Consider: Geographic proximity to users\n\n**Stateful/Database (databases, caches, queues)**\n- Recommended: AWS RDS/ElastiCache, DigitalOcean Managed DB\n- Alternative: Self-managed on any provider with replication\n\n**Storage/File Serving (object storage, backups)**\n- Recommended: AWS S3, DigitalOcean Spaces, Hetzner Object Storage\n- Consider: Cost per GB, access patterns, bandwidth\n\n### Regional Availability\n\n**North America**\n- AWS: Multiple regions (us-east-1, us-west-2, etc)\n- DigitalOcean: NYC, SFO\n- Hetzner: Ashburn, Virginia\n- UpCloud: Multiple US locations\n\n**Europe**\n- Hetzner: Falkenstein (DE), Nuremberg (DE), Helsinki (FI)\n- UpCloud: Multiple EU locations\n- AWS: eu-west-1 (IE), eu-central-1 (DE), etc\n- DigitalOcean: London, Frankfurt, Amsterdam\n\n**Asia**\n- AWS: ap-southeast-1 (SG), ap-northeast-1 (Tokyo)\n- DigitalOcean: Singapore, Bangalore\n- Hetzner: Limited\n- UpCloud: Singapore\n\n**Recommendation for Multi-Region**: Combine Hetzner (EU backbone), DigitalOcean (global presence), AWS (comprehensive regions).\n\n### Cost Analysis\n\n#### Monthly Compute Comparison (2 vCPU, 4 GB RAM)\n\n| Provider | Price | Notes |\n| ---------- | ------- | ------- |\n| Hetzner | €6.90 (~$7.50) | Cheapest, good performance |\n| DigitalOcean | $24 | Premium pricing, simplicity |\n| UpCloud | $30 | Mid-range, good support |\n| AWS t3.medium | $60+ | On-demand pricing (spot: $18-25) |\n\n#### Recommendations by Budget\n\n**Minimal Budget (<$50/month)**\n- Single Hetzner server: €6.90\n- Alternative: DigitalOcean $24 + DigitalOcean Spaces for backup\n\n**Small Team ($100-500/month)**\n- Hetzner primary (€50-150), DigitalOcean backup (60-80)\n- Good HA coverage with cost control\n\n**Enterprise ($1000+/month)**\n- AWS primary (managed services, compliance)\n- Hetzner backup (cost-effective)\n- DigitalOcean edge locations (CDN)\n\n### Compliance and Certifications\n\n| Provider | GDPR | SOC 2 | ISO 27001 | HIPAA | FIPS | PCI-DSS |\n| ---------- | ------ | ------- | ----------- | ------- | ------ | --------- |\n| Hetzner | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |\n| UpCloud | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |\n| AWS | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |\n| DigitalOcean | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |\n\n**Compliance Selection Matrix**\n\n- **GDPR Only**: Hetzner, UpCloud (EU-based), all AWS/DO EU regions\n- **HIPAA Required**: AWS, DigitalOcean (DigitalOcean requires BAA)\n- **FIPS Required**: AWS (all regions)\n- **PCI-DSS**: All providers support, AWS most comprehensive\n\n## Workspace Configuration\n\n### Multi-Provider Workspace Structure\n\n```\nprovisioning/examples/workspaces/my-multi-provider-app/\n├── workspace.ncl                # Infrastructure definition\n├── config.toml                  # Provider credentials, regions, defaults\n├── README.md                    # Setup and deployment instructions\n└── deploy.nu                    # Deployment orchestration script\n```\n\n### Provider Credential Management\n\n#### Environment Variables\n\nEach provider requires authentication via environment variables:\n\n```\n# Hetzner\nexport HCLOUD_TOKEN="your-hetzner-api-token"\n\n# UpCloud\nexport UPCLOUD_USERNAME="your-upcloud-username"\nexport UPCLOUD_PASSWORD="your-upcloud-password"\n\n# AWS\nexport AWS_ACCESS_KEY_ID="your-access-key"\nexport AWS_SECRET_ACCESS_KEY="your-secret-key"\nexport AWS_DEFAULT_REGION="us-east-1"\n\n# DigitalOcean\nexport DIGITALOCEAN_TOKEN="your-do-api-token"\n```\n\n#### Configuration File Structure (config.toml)\n\n```\n[providers]\n\n[providers.hetzner]\nenabled = true\napi_token_env = "HCLOUD_TOKEN"\ndefault_region = "nbg1"\ndefault_datacenter = "nbg1-dc8"\n\n[providers.upcloud]\nenabled = true\nusername_env = "UPCLOUD_USERNAME"\npassword_env = "UPCLOUD_PASSWORD"\ndefault_region = "fi-hel1"\n\n[providers.aws]\nenabled = true\nregion = "us-east-1"\naccess_key_env = "AWS_ACCESS_KEY_ID"\nsecret_key_env = "AWS_SECRET_ACCESS_KEY"\n\n[providers.digitalocean]\nenabled = true\ntoken_env = "DIGITALOCEAN_TOKEN"\ndefault_region = "nyc3"\n\n[workspace]\nname = "my-multi-provider-app"\nenvironment = "production"\nowner = "platform-team"\n```\n\n### Multi-Provider Workspace Definition\n\nNickel workspace with multiple providers:\n\n```\n# workspace.ncl - Multi-provider infrastructure definition\n\nlet hetzner = import "../../extensions/providers/hetzner/nickel/main.ncl" in\nlet upcloud = import "../../extensions/providers/upcloud/nickel/main.ncl" in\nlet aws = import "../../extensions/providers/aws/nickel/main.ncl" in\nlet digitalocean = import "../../extensions/providers/digitalocean/nickel/main.ncl" in\n\n{\n  workspace_name = "multi-provider-app",\n  description = "Multi-provider infrastructure example",\n\n  # Provider routing configuration\n  providers = {\n    primary_compute = "hetzner",\n    secondary_compute = "digitalocean",\n    database = "aws",\n    backup = "upcloud"\n  },\n\n  # Infrastructure defined per provider\n  infrastructure = {\n    # Hetzner: Primary compute tier\n    primary_servers = hetzner.Server & {\n      name = "primary-server",\n      server_type = "cx31",\n      image = "ubuntu-22.04",\n      location = "nbg1",\n      count = 3,\n      ssh_keys = ["your-ssh-key"],\n      firewalls = ["primary-fw"]\n    },\n\n    # DigitalOcean: Secondary compute tier\n    secondary_servers = digitalocean.Droplet & {\n      name = "secondary-droplet",\n      size = "s-2vcpu-4gb",\n      image = "ubuntu-22-04-x64",\n      region = "nyc3",\n      count = 2\n    },\n\n    # AWS: Managed database\n    database = aws.RDS & {\n      identifier = "prod-db",\n      engine = "postgresql",\n      engine_version = "14.6",\n      instance_class = "db.t3.medium",\n      allocated_storage = 100\n    },\n\n    # UpCloud: Backup storage\n    backup_storage = upcloud.Storage & {\n      name = "backup-volume",\n      size = 500,\n      location = "fi-hel1"\n    }\n  }\n}\n```\n\n## Architecture Patterns\n\n### Pattern 1: Compute + Storage Split\n\n**Scenario**: Cost-effective compute with specialized managed storage.\n\n**Example**: Use Hetzner for compute (cheap), AWS S3 for object storage (reliable), managed database on AWS RDS.\n\n#### Benefits\n\n- Compute optimization (Hetzner's low cost)\n- Storage specialization (AWS S3 reliability and features)\n- Separation of concerns (different performance tuning)\n\n#### Architecture\n\n```\n                    ┌─────────────────────┐\n                    │   Client Requests   │\n                    └──────────┬──────────┘\n                               │\n                ┌──────────────┼──────────────┐\n                │              │              │\n         ┌──────▼─────┐  ┌────▼─────┐  ┌───▼──────┐\n         │  Hetzner   │  │    AWS   │  │ AWS S3   │\n         │  Servers   │  │    RDS   │  │ Storage  │\n         │ (Compute)  │  │(Database)│  │(Backups) │\n         └────────────┘  └──────────┘  └──────────┘\n```\n\n#### Nickel Configuration\n\n```\nlet hetzner = import "../../extensions/providers/hetzner/nickel/main.ncl" in\nlet aws = import "../../extensions/providers/aws/nickel/main.ncl" in\n\n{\n  compute = hetzner.Server & {\n    name = "app-server",\n    server_type = "cpx21",  # 4 vCPU, 8 GB RAM\n    image = "ubuntu-22.04",\n    location = "nbg1",\n    count = 2,\n    volumes = [\n      {\n        size = 100,\n        format = "ext4",\n        mount = "/app"\n      }\n    ]\n  },\n\n  database = aws.RDS & {\n    identifier = "app-database",\n    engine = "postgresql",\n    instance_class = "db.t3.medium",\n    allocated_storage = 100\n  },\n\n  backup_bucket = aws.S3 & {\n    bucket = "app-backups",\n    region = "us-east-1",\n    versioning = true,\n    lifecycle_rules = [\n      {\n        id = "delete-old-backups",\n        days = 90,\n        action = "delete"\n      }\n    ]\n  }\n}\n```\n\n#### Network Configuration\n\nHetzner servers connect to AWS RDS via VPN or public endpoint:\n\n```\n# Network setup script\ndef setup_database_connection [] {\n  let hetzner_servers = (hetzner_list_servers)\n  let db_endpoint = (aws_get_rds_endpoint "app-database")\n\n  # Install PostgreSQL client\n  $hetzner_servers | each {|server|\n    ssh $server.ip "apt-get install -y postgresql-client"\n    ssh $server.ip $"echo 'DB_HOST=($db_endpoint)' >> /app/.env"\n  }\n}\n```\n\n#### Cost Analysis\n\nMonthly estimate:\n- Hetzner cx31 × 2: €13.80 (~$15)\n- AWS RDS t3.medium: $60\n- AWS S3 (100 GB): $2.30\n- **Total: ~$77/month** (vs $120+ for all-AWS)\n\n### Pattern 2: Primary + Backup\n\n**Scenario**: Active-standby deployment for disaster recovery.\n\n**Example**: DigitalOcean primary datacenter, Hetzner warm standby with automated failover.\n\n#### Benefits\n\n- Disaster recovery capability\n- Zero data loss (with replication)\n- Tested failover procedure\n- Cost-effective backup (warm standby vs hot standby)\n\n#### Architecture\n\n```\n         Primary (DigitalOcean NYC)        Backup (Hetzner DE)\n         ┌──────────────────────┐          ┌─────────────────┐\n         │   DigitalOcean LB    │◄────────►│ HAProxy Monitor │\n         └──────────┬───────────┘          └────────┬────────┘\n                    │                               │\n         ┌──────────┴──────────┐                    │\n         │                     │                    │\n     ┌───▼───┐ ┌───▼───┐   ┌──▼──┐ ┌──────┐    ┌──▼───┐\n     │ APP 1 │ │ APP 2 │   │ DB  │ │ ELK  │    │ WARM │\n     │PRIMARY│ │PRIMARY│   │REPL │ │MON   │    │STANDBY\n     └───────┘ └───────┘   └─────┘ └──────┘    └──────┘\n         │                     │                    ▲\n         └─────────────────────┼────────────────────┘\n                        Async Replication\n```\n\n#### Failover Trigger\n\n```\ndef monitor_primary_health [do_region, hetzner_region] {\n  loop {\n    let health = (do_health_check $do_region)\n\n    if $health.status == "degraded" or $health.status == "down" {\n      print "Primary degraded, triggering failover"\n      trigger_failover $hetzner_region\n      break\n    }\n\n    sleep 30sec\n  }\n}\n\ndef trigger_failover [backup_region] {\n  # 1. Promote backup database\n  promote_replica_to_primary $backup_region\n\n  # 2. Update DNS to point to backup\n  update_dns_to_backup $backup_region\n\n  # 3. Scale up backup servers\n  scale_servers $backup_region 3\n\n  # 4. Verify traffic flowing\n  wait_for_traffic_migration $backup_region 120sec\n}\n```\n\n#### Nickel Configuration\n\n```\nlet digitalocean = import "../../extensions/providers/digitalocean/nickel/main.ncl" in\nlet hetzner = import "../../extensions/providers/hetzner/nickel/main.ncl" in\n\n{\n  # Primary: DigitalOcean\n  primary = {\n    region = "nyc3",\n    provider = "digitalocean",\n\n    servers = digitalocean.Droplet & {\n      name = "primary-app",\n      size = "s-2vcpu-4gb",\n      count = 3,\n      region = "nyc3",\n      firewall = {\n        inbound = [\n          { protocol = "tcp", ports = "80", sources = ["0.0.0.0/0"] },\n          { protocol = "tcp", ports = "443", sources = ["0.0.0.0/0"] },\n          { protocol = "tcp", ports = "5432", sources = ["10.0.0.0/8"] }\n        ]\n      }\n    },\n\n    database = digitalocean.Database & {\n      name = "primary-db",\n      engine = "pg",\n      version = "14",\n      size = "db-s-2vcpu-4gb",\n      region = "nyc3"\n    }\n  },\n\n  # Backup: Hetzner (warm standby)\n  backup = {\n    region = "nbg1",\n    provider = "hetzner",\n\n    servers = hetzner.Server & {\n      name = "backup-app",\n      server_type = "cx31",\n      count = 1,  # Minimal for cost\n      location = "nbg1",\n      automount = true\n    },\n\n    # Replica database (read-only until promoted)\n    database_replica = hetzner.Volume & {\n      name = "db-replica",\n      size = 100,\n      location = "nbg1"\n    }\n  },\n\n  replication = {\n    type = "async",\n    primary_to_backup = true,\n    recovery_point_objective = 300  # 5 minutes\n  }\n}\n```\n\n#### Failover Testing\n\n```\n# Test failover without affecting production\ndef test_failover_dry_run [config] {\n  print "Starting failover dry-run test..."\n\n  # 1. Snapshot primary database\n  let snapshot = (do_create_db_snapshot "primary-db")\n\n  # 2. Create temporary replica from snapshot\n  let temp_replica = (hetzner_create_from_snapshot $snapshot)\n\n  # 3. Run traffic tests against temp replica\n  let test_results = (run_integration_tests $temp_replica.ip)\n\n  # 4. Verify database consistency\n  let consistency = (verify_db_consistency $temp_replica.ip)\n\n  # 5. Cleanup temp resources\n  hetzner_destroy $temp_replica.id\n  do_delete_snapshot $snapshot.id\n\n  {\n    status: "passed",\n    results: $test_results,\n    consistency_check: $consistency\n  }\n}\n```\n\n### Pattern 3: Multi-Region High Availability\n\n**Scenario**: Distributed deployment across 3+ geographic regions with global load balancing.\n\n**Example**: DigitalOcean US (NYC), Hetzner EU (Germany), AWS Asia (Singapore) with DNS-based failover.\n\n#### Benefits\n\n- Geographic distribution for low latency\n- Protection against regional outages\n- Compliance with data residency (data stays in region)\n- Load distribution across regions\n\n#### Architecture\n\n```\n                    ┌─────────────────┐\n                    │  Global DNS     │\n                    │  (Geofencing)   │\n                    └────────┬────────┘\n                    ┌────────┴────────┐\n                    │                 │\n         ┌──────────▼──────┐  ┌──────▼─────────┐  ┌─────────────┐\n         │  DigitalOcean   │  │  Hetzner       │  │    AWS      │\n         │  US/NYC Region  │  │  EU/Germany    │  │  Asia/SG    │\n         ├─────────────────┤  ├────────────────┤  ├─────────────┤\n         │ Droplets (3)    │  │ Servers (3)    │  │ EC2 (3)     │\n         │ LB              │  │ HAProxy        │  │ ALB         │\n         │ DB (Primary)    │  │ DB (Replica)   │  │ DB (Replica)│\n         └─────────────────┘  └────────────────┘  └─────────────┘\n                    │                 │                    │\n                    └─────────────────┴────────────────────┘\n                           Cross-Region Sync\n```\n\n#### Global Load Balancing\n\n```\ndef setup_global_dns [] {\n  # Using Route53 or Cloudflare for DNS failover\n  let regions = [\n    { name: "us-nyc", provider: "digitalocean", endpoint: "us.app.example.com" },\n    { name: "eu-de", provider: "hetzner", endpoint: "eu.app.example.com" },\n    { name: "asia-sg", provider: "aws", endpoint: "asia.app.example.com" }\n  ]\n\n  # Create health checks\n  $regions | each {|region|\n    configure_health_check $region.name $region.endpoint\n  }\n\n  # Setup failover policy\n  # Primary: US, Secondary: EU, Tertiary: Asia\n  configure_dns_failover {\n    primary: "us-nyc",\n    secondary: "eu-de",\n    tertiary: "asia-sg"\n  }\n}\n```\n\n#### Nickel Configuration\n\n```\n{\n  regions = {\n    us_east = {\n      provider = "digitalocean",\n      region = "nyc3",\n\n      servers = digitalocean.Droplet & {\n        name = "us-app",\n        size = "s-2vcpu-4gb",\n        count = 3,\n        region = "nyc3"\n      },\n\n      database = digitalocean.Database & {\n        name = "us-db",\n        engine = "pg",\n        size = "db-s-2vcpu-4gb",\n        region = "nyc3",\n        replica_regions = ["eu-de", "asia-sg"]\n      }\n    },\n\n    eu_central = {\n      provider = "hetzner",\n      region = "nbg1",\n\n      servers = hetzner.Server & {\n        name = "eu-app",\n        server_type = "cx31",\n        count = 3,\n        location = "nbg1"\n      }\n    },\n\n    asia_southeast = {\n      provider = "aws",\n      region = "ap-southeast-1",\n\n      servers = aws.EC2 & {\n        name = "asia-app",\n        instance_type = "t3.medium",\n        count = 3,\n        region = "ap-southeast-1"\n      }\n    }\n  },\n\n  global_config = {\n    dns_provider = "route53",\n    ttl = 60,\n    health_check_interval = 30\n  }\n}\n```\n\n#### Data Synchronization\n\n```\n# Multi-region data sync strategy\ndef sync_data_across_regions [primary_region, secondary_regions] {\n  let sync_config = {\n    strategy: "async",\n    consistency: "eventual",\n    conflict_resolution: "last-write-wins",\n    replication_lag: "300s"  # 5 minute max lag\n  }\n\n  # Setup replication from primary to all secondaries\n  $secondary_regions | each {|region|\n    setup_async_replication $primary_region $region $sync_config\n  }\n\n  # Monitor replication lag\n  loop {\n    let lag = (check_replication_lag)\n    if $lag > 300 {\n      print "Warning: replication lag exceeds threshold"\n      trigger_alert "replication-lag-warning"\n    }\n    sleep 60sec\n  }\n}\n```\n\n### Pattern 4: Hybrid Cloud\n\n**Scenario**: On-premises infrastructure with public cloud providers for burst capacity and backup.\n\n**Example**: On-premise data center + AWS for burst capacity + DigitalOcean for disaster recovery.\n\n#### Benefits\n\n- Existing infrastructure utilization\n- Burst capacity in public cloud\n- Disaster recovery site\n- Compliance with on-premise requirements\n- Cost control (scale only when needed)\n\n#### Architecture\n\n```\n    On-Premises Data Center           Public Cloud (Burst)\n    ┌─────────────────────────┐      ┌────────────────────┐\n    │  Physical Servers       │◄────►│  AWS Auto-Scaling  │\n    │  - App Tier (24 cores)  │      │  - Elasticity      │\n    │  - DB Tier (48 cores)   │      │  - Pay-as-you-go   │\n    │  - Storage (50 TB)       │      │  - CloudFront CDN  │\n    └─────────────────────────┘      └────────────────────┘\n               │                               ▲\n               │ VPN Tunnel                    │\n               └───────────────────────────────┘\n\n    On-Premises                        DR Site (DigitalOcean)\n    │ Production                        │ Warm Standby\n    ├─ 95% Utilization                  ├─ Cold VM Snapshots\n    ├─ Full Data                        ├─ Async Replication\n    ├─ Peak Load Handling               ├─ Ready for 15 min RTO\n    │                                   │\n```\n\n#### VPN Configuration\n\n```\ndef setup_hybrid_vpn [] {\n  # AWS VPN to on-premise datacenter\n  let vpn_config = {\n    type: "site-to-site",\n    protocol: "ipsec",\n    encryption: "aes-256",\n    authentication: "sha256",\n    on_prem_cidr: "192.168.0.0/16",\n    aws_cidr: "10.0.0.0/16",\n    do_cidr: "172.16.0.0/16"\n  }\n\n  # Create AWS Site-to-Site VPN\n  let vpn = (aws_create_vpn_connection $vpn_config)\n\n  # Configure on-prem gateway\n  configure_on_prem_vpn_gateway $vpn\n\n  # Verify tunnel status\n  wait_for_vpn_ready 300\n}\n```\n\n#### Nickel Configuration\n\n```\n{\n  on_premises = {\n    provider = "manual",\n    gateway = "192.168.1.1",\n    cidr = "192.168.0.0/16",\n    bandwidth = "1gbps",\n\n    # Resources remain on-prem (managed manually)\n    servers = {\n      app_tier = { cores = 24, memory = 128 },\n      db_tier = { cores = 48, memory = 256 },\n      storage = { capacity = "50 TB" }\n    }\n  },\n\n  aws_burst_capacity = {\n    provider = "aws",\n    region = "us-east-1",\n\n    auto_scaling_group = aws.ASG & {\n      name = "burst-asg",\n      min_size = 0,\n      desired_capacity = 0,\n      max_size = 20,\n      instance_type = "c5.2xlarge",\n      scale_up_trigger = "on_prem_cpu > 80%",\n      scale_down_trigger = "on_prem_cpu < 40%"\n    },\n\n    cdn = aws.CloudFront & {\n      origin = "on-prem-origin",\n      regional_origins = ["us-east-1", "eu-west-1", "ap-southeast-1"]\n    }\n  },\n\n  dr_site = {\n    provider = "digitalocean",\n    region = "nyc3",\n\n    snapshot_storage = digitalocean.Droplet & {\n      name = "dr-snapshot",\n      size = "s-24vcpu-48gb",\n      count = 0,  # Powered off until needed\n      image = "on-prem-snapshot"\n    }\n  },\n\n  replication = {\n    on_prem_to_aws: {\n      strategy = "continuous",\n      target = "aws-s3-bucket",\n      retention = "7days"\n    },\n\n    on_prem_to_do: {\n      strategy = "nightly",\n      target = "do-spaces-bucket",\n      retention = "30days"\n    }\n  }\n}\n```\n\n#### Burst Capacity Orchestration\n\n```\n# Monitor on-prem and trigger AWS burst\ndef monitor_and_burst [] {\n  loop {\n    let on_prem_metrics = (collect_on_prem_metrics)\n\n    if $on_prem_metrics.cpu_avg > 80 {\n      # Trigger AWS burst scaling\n      let scale_size = ((100 - $on_prem_metrics.cpu_avg) / 10)\n      scale_aws_burst $scale_size\n    } else if $on_prem_metrics.cpu_avg < 40 {\n      # Scale down AWS\n      scale_aws_burst 0\n    }\n\n    sleep 60sec\n  }\n}\n```\n\n## Implementation Examples\n\n### Example 1: Three-Provider Web Application\n\n**Scenario**: Production web application with DigitalOcean web servers, AWS managed database, and Hetzner backup storage.\n\n**Architecture**:\n- DigitalOcean: 3 web servers with load balancer (cost-effective compute)\n- AWS: RDS PostgreSQL database (managed, high availability)\n- Hetzner: Backup volumes (low-cost storage)\n\n**Files to Create**:\n\n**workspace.ncl**:\n\n```\nlet digitalocean = import "../../extensions/providers/digitalocean/nickel/main.ncl" in\nlet aws = import "../../extensions/providers/aws/nickel/main.ncl" in\nlet hetzner = import "../../extensions/providers/hetzner/nickel/main.ncl" in\n\n{\n  workspace_name = "three-provider-webapp",\n  description = "Web application across three providers",\n\n  infrastructure = {\n    web_tier = digitalocean.Droplet & {\n      name = "web-server",\n      region = "nyc3",\n      size = "s-2vcpu-4gb",\n      image = "ubuntu-22-04-x64",\n      count = 3,\n      firewall = {\n        inbound_rules = [\n          { protocol = "tcp", ports = "22", sources = { addresses = ["your-ip/32"] } },\n          { protocol = "tcp", ports = "80", sources = { addresses = ["0.0.0.0/0"] } },\n          { protocol = "tcp", ports = "443", sources = { addresses = ["0.0.0.0/0"] } }\n        ],\n        outbound_rules = [\n          { protocol = "tcp", destinations = { addresses = ["0.0.0.0/0"] } }\n        ]\n      }\n    },\n\n    load_balancer = digitalocean.LoadBalancer & {\n      name = "web-lb",\n      algorithm = "round_robin",\n      region = "nyc3",\n      forwarding_rules = [\n        {\n          entry_protocol = "http",\n          entry_port = 80,\n          target_protocol = "http",\n          target_port = 80,\n          certificate_id = null\n        },\n        {\n          entry_protocol = "https",\n          entry_port = 443,\n          target_protocol = "http",\n          target_port = 80,\n          certificate_id = "your-cert-id"\n        }\n      ],\n      sticky_sessions = {\n        type = "cookies",\n        cookie_name = "lb",\n        cookie_ttl_seconds = 300\n      }\n    },\n\n    database = aws.RDS & {\n      identifier = "webapp-db",\n      engine = "postgres",\n      engine_version = "14.6",\n      instance_class = "db.t3.medium",\n      allocated_storage = 100,\n      storage_type = "gp3",\n      multi_az = true,\n      backup_retention_days = 30,\n      subnet_group = "default",\n      parameter_group = "default.postgres14",\n      tags = [\n        { key = "Environment", value = "production" },\n        { key = "Application", value = "web-app" }\n      ]\n    },\n\n    backup_volume = hetzner.Volume & {\n      name = "webapp-backups",\n      size = 500,\n      location = "nbg1",\n      automount = false,\n      format = "ext4"\n    }\n  }\n}\n```\n\n**config.toml**:\n\n```\n[workspace]\nname = "three-provider-webapp"\nenvironment = "production"\nowner = "platform-team"\n\n[providers.digitalocean]\nenabled = true\ntoken_env = "DIGITALOCEAN_TOKEN"\ndefault_region = "nyc3"\n\n[providers.aws]\nenabled = true\nregion = "us-east-1"\naccess_key_env = "AWS_ACCESS_KEY_ID"\nsecret_key_env = "AWS_SECRET_ACCESS_KEY"\n\n[providers.hetzner]\nenabled = true\ntoken_env = "HCLOUD_TOKEN"\ndefault_location = "nbg1"\n\n[deployment]\nstrategy = "rolling"\nbatch_size = 1\nhealth_check_wait = 60\nrollback_on_failure = true\n```\n\n**deploy.nu**:\n\n```\n#!/usr/bin/env nu\n\n# Deploy three-provider web application\ndef main [environment = "staging"] {\n  print "Deploying three-provider web application to ($environment)..."\n\n  # 1. Validate configuration\n  print "Step 1: Validating configuration..."\n  validate_config "workspace.ncl"\n\n  # 2. Create infrastructure\n  print "Step 2: Creating infrastructure..."\n  create_digitalocean_resources\n  create_aws_resources\n  create_hetzner_resources\n\n  # 3. Configure networking\n  print "Step 3: Configuring networking..."\n  setup_vpc_peering\n  configure_security_groups\n\n  # 4. Deploy application\n  print "Step 4: Deploying application..."\n  deploy_app_to_web_servers\n\n  # 5. Verify deployment\n  print "Step 5: Verifying deployment..."\n  verify_health_checks\n  verify_database_connectivity\n  verify_backups\n\n  print "Deployment complete!"\n}\n\ndef validate_config [config_file] {\n  print $"Validating ($config_file)..."\n  nickel export $config_file | from json\n}\n\ndef create_digitalocean_resources [] {\n  print "Creating DigitalOcean resources (3 droplets + load balancer)..."\n  # Implementation\n}\n\ndef create_aws_resources [] {\n  print "Creating AWS resources (RDS database)..."\n  # Implementation\n}\n\ndef create_hetzner_resources [] {\n  print "Creating Hetzner resources (backup volume)..."\n  # Implementation\n}\n\ndef setup_vpc_peering [] {\n  print "Setting up cross-provider networking..."\n  # Implementation\n}\n\ndef configure_security_groups [] {\n  print "Configuring security groups..."\n  # Implementation\n}\n\ndef deploy_app_to_web_servers [] {\n  print "Deploying application..."\n  # Implementation\n}\n\ndef verify_health_checks [] {\n  print "Verifying health checks..."\n  # Implementation\n}\n\ndef verify_database_connectivity [] {\n  print "Verifying database connectivity..."\n  # Implementation\n}\n\ndef verify_backups [] {\n  print "Verifying backup configuration..."\n  # Implementation\n}\n\nmain $env.ENVIRONMENT?\n```\n\n### Example 2: Multi-Region Disaster Recovery\n\n**Scenario**: Active-standby DR setup with DigitalOcean primary and Hetzner backup.\n\n**Architecture**:\n- DigitalOcean NYC: Production environment (active)\n- Hetzner Germany: Warm standby (scales down until needed)\n- Async database replication\n- DNS-based failover\n- RPO: 5 minutes, RTO: 15 minutes\n\n### Example 3: Cost-Optimized Deployment\n\n**Scenario**: Optimize across provider strengths: Hetzner compute, AWS managed services, DigitalOcean CDN.\n\n**Architecture**:\n- Hetzner: 5 application servers (best compute price)\n- AWS: RDS database, ElastiCache (managed services)\n- DigitalOcean: Spaces for backups, CDN endpoints\n\n## Best Practices\n\n### 1. Provider Selection\n\n- **Document provider choices**: Keep record of which workloads run where and why\n- **Audit provider capabilities**: Ensure chosen provider supports required features\n- **Monitor provider health**: Track outages and issues per provider\n- **Cost tracking per provider**: Understand where money is spent\n\n### 2. Network Security\n\n- **Encrypt inter-provider traffic**: Use VPN, mTLS, or encrypted tunnels\n- **Implement firewall rules**: Limit traffic between providers to necessary ports\n- **Use security groups**: AWS-style security groups where available\n- **Monitor network traffic**: Detect unusual patterns across providers\n\n### 3. Data Consistency\n\n- **Choose replication strategy**: Synchronous (consistency), asynchronous (performance)\n- **Implement conflict resolution**: Define how conflicts are resolved\n- **Monitor replication lag**: Alert on excessive lag\n- **Test failover regularly**: Verify data integrity during failover\n\n### 4. Disaster Recovery\n\n- **Define RPO/RTO targets**: Recovery Point Objective and Recovery Time Objective\n- **Document failover procedures**: Step-by-step instructions\n- **Test failover regularly**: At least quarterly, ideally monthly\n- **Maintain DR site readiness**: Cold, warm, or hot standby based on RTO\n\n### 5. Compliance and Governance\n\n- **Data residency**: Ensure data stays in required regions\n- **Encryption at rest**: Use provider-native encryption\n- **Encryption in transit**: TLS/mTLS for all inter-provider communication\n- **Audit logging**: Enable audit logs in all providers\n- **Access control**: Implement least privilege across all providers\n\n### 6. Monitoring and Alerting\n\n- **Unified monitoring**: Aggregate metrics from all providers\n- **Cross-provider dashboards**: Visualize health across providers\n- **Provider-specific alerts**: Configure alerts per provider\n- **Escalation procedures**: Clear escalation for failures\n\n### 7. Cost Management\n\n- **Set budget alerts**: Per provider and total\n- **Reserved instances**: Use provider discounts\n- **Spot instances**: AWS spot for non-critical workloads\n- **Auto-scaling policies**: Scale based on demand\n- **Regular cost reviews**: Monthly cost analysis and optimization\n\n## Troubleshooting\n\n### Issue: Network Connectivity Between Providers\n\n**Symptoms**: Droplets can't reach AWS database, high latency between regions\n\n**Diagnosis**:\n\n```\n# Check network connectivity\ndef diagnose_network_issue [source_ip, dest_ip] {\n  print "Diagnosing network connectivity..."\n\n  # 1. Check routing\n  ssh $source_ip "ip route show"\n\n  # 2. Check firewall rules\n  check_security_groups $source_ip $dest_ip\n\n  # 3. Test connectivity\n  ssh $source_ip "ping -c 3 $dest_ip"\n  ssh $source_ip "traceroute $dest_ip"\n\n  # 4. Check DNS resolution\n  ssh $source_ip "nslookup $dest_ip"\n}\n```\n\n**Solutions**:\n- Verify firewall rules allow traffic on required ports\n- Check VPN tunnel status if using site-to-site VPN\n- Verify DNS resolution in both providers\n- Check MTU size for jumbo frames (1500 bytes)\n- Enable debug logging on network components\n\n### Issue: Database Replication Lag\n\n**Symptoms**: Secondary database lagging behind primary\n\n**Diagnosis**:\n\n```\ndef check_replication_lag [] {\n  # AWS RDS\n  aws rds describe-db-instances --query 'DBInstances[].{ID:DBInstanceIdentifier,Lag:ReplicationLag}'\n\n  # DigitalOcean\n  doctl databases backups list --format Name,Created\n}\n```\n\n**Solutions**:\n- Check network bandwidth between providers\n- Review write throughput on primary\n- Monitor CPU/IO on secondary\n- Adjust replication thread pool size\n- Check for long-running queries blocking replication\n\n### Issue: Failover Not Working\n\n**Symptoms**: Failover script fails, DNS not updating\n\n**Diagnosis**:\n\n```\ndef test_failover_chain [] {\n  # 1. Verify backup infrastructure is ready\n  verify_backup_infrastructure\n\n  # 2. Test DNS failover\n  test_dns_failover\n\n  # 3. Verify database promotion\n  test_db_promotion\n\n  # 4. Check application configuration\n  verify_app_failover_config\n}\n```\n\n**Solutions**:\n- Ensure backup infrastructure is powered on and running\n- Verify DNS TTL is appropriate (typically 60 seconds)\n- Test failover in staging environment first\n- Check VPN connectivity to backup provider\n- Verify database promotion scripts\n- Ensure application connection strings support both endpoints\n\n### Issue: Cost Spike Across Providers\n\n**Symptoms**: Monthly bill unexpectedly high\n\n**Diagnosis**:\n\n```\ndef analyze_cost_spike [] {\n  print "Analyzing cost spike..."\n\n  # Compare current vs previous month\n  let current = (get_current_month_costs)\n  let previous = (get_previous_month_costs)\n  let delta = ($current - $previous)\n\n  # Break down by provider\n  $current | group-by provider | each {|group|\n    let provider = ($group.0.provider)\n    let cost = ($group | map {|x| $x.cost} | math sum)\n    print $"($provider): $($cost)"\n  }\n\n  # Identify largest increases\n  ($delta | sort-by cost_change | reverse | first 5)\n}\n```\n\n**Solutions**:\n- Review auto-scaling activities\n- Check for unintended resource creation\n- Verify reserved instances are being used\n- Review data transfer costs (cross-region expensive)\n- Cancel idle resources\n- Contact provider support if billing seems incorrect\n\n## Conclusion\n\nMulti-provider deployments provide significant benefits in cost optimization, reliability, and compliance. Start with a simple pattern (Compute +\nStorage Split) and evolve to more complex patterns as needs grow. Always test failover procedures and maintain clear documentation of provider\nresponsibilities and network configurations.\n\nFor more information, see:\n- Provider-agnostic architecture guide\n- Batch workflow orchestration guide\n- Individual provider implementation guides
