# Multi-Provider Networking Guide\n\nThis comprehensive guide covers private networking, VPN tunnels, and secure communication across multiple cloud providers using Hetzner, UpCloud, AWS,\nand DigitalOcean.\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Provider SDN/Private Network Solutions](#provider-sdnprivate-network-solutions)\n- [Private Network Configuration](#private-network-configuration)\n- [VPN Tunnel Setup](#vpn-tunnel-setup)\n- [Multi-Provider Routing](#multi-provider-routing)\n- [Security Considerations](#security-considerations)\n- [Implementation Examples](#implementation-examples)\n- [Troubleshooting](#troubleshooting)\n\n## Overview\n\nMulti-provider deployments require secure, private communication between resources across different cloud providers. This involves:\n\n- **Private Networks**: Isolated virtual networks within each provider (SDN)\n- **VPN Tunnels**: Encrypted connections between provider networks\n- **Routing**: Proper IP routing between provider networks\n- **Security**: Firewall rules and access control across providers\n- **DNS**: Private DNS for cross-provider resource discovery\n\n### Architecture\n\n```\n┌──────────────────────────────────┐\n│      DigitalOcean VPC            │\n│  Network: 10.0.0.0/16            │\n│  ┌────────────────────────────┐  │\n│  │ Web Servers (10.0.1.0/24)  │  │\n│  └────────────────────────────┘  │\n└────────────┬─────────────────────┘\n             │ IPSec VPN Tunnel\n             │ Encrypted\n             ├─────────────────────────────┐\n             │                             │\n┌────────────▼──────────────────┐  ┌──────▼─────────────────────┐\n│      AWS VPC                  │  │   Hetzner vSwitch          │\n│  Network: 10.1.0.0/16         │  │   Network: 10.2.0.0/16     │\n│  ┌──────────────────────────┐ │  │ ┌─────────────────────────┐│\n│  │ RDS Database (10.1.1.0) │ │  │ │ Backup (10.2.1.0)       ││\n│  └──────────────────────────┘ │  │ └─────────────────────────┘│\n└───────────────────────────────┘  └─────────────────────────────┘\n         IPSec ▲                              IPSec ▲\n         Tunnel │                             Tunnel │\n```\n\n## Provider SDN/Private Network Solutions\n\n### Hetzner: vSwitch\n\n**Product**: vSwitch (Virtual Switch)\n\n**Characteristics**:\n- Private networks for Cloud Servers\n- Multiple subnets per network\n- Layer 2 switching\n- IP-based traffic isolation\n- Free service (included with servers)\n\n**Features**:\n- Custom IP ranges\n- Subnets and routing\n- Attached/detached servers\n- Static routes\n- Private networking without NAT\n\n**Configuration**:\n```\n# Create private network\nhcloud network create --name "app-network" --ip-range "10.0.0.0/16"\n\n# Create subnet\nhcloud network add-subnet app-network --ip-range "10.0.1.0/24" --network-zone eu-central\n\n# Attach server to network\nhcloud server attach-to-network server-1 --network app-network --ip 10.0.1.10\n```\n\n### UpCloud: VLAN (Virtual LAN)\n\n**Product**: Private Networks (VLAN-based)\n\n**Characteristics**:\n- Virtual LAN technology\n- Layer 2 connectivity\n- Multiple VLANs per account\n- No bandwidth charges\n- Simple configuration\n\n**Features**:\n- Custom CIDR blocks\n- Multiple networks per account\n- Server attachment to VLANs\n- VLAN tagging support\n- Static routing\n\n**Configuration**:\n```\n# Create private network\nupctl network create --name "app-network" --ip-networks 10.0.0.0/16\n\n# Attach server to network\nupctl server attach-network --server server-1 \\n  --network app-network --ip-address 10.0.1.10\n```\n\n### AWS: VPC (Virtual Private Cloud)\n\n**Product**: VPC with subnets and security groups\n\n**Characteristics**:\n- Enterprise-grade networking\n- Multiple availability zones\n- Complex security models\n- NAT gateways and bastion hosts\n- Advanced routing\n\n**Features**:\n- VPC peering\n- VPN connections\n- Internet gateways\n- NAT gateways\n- Security groups and NACLs\n- Route tables with multiple targets\n- Flow logs and VPC insights\n\n**Configuration**:\n```\n# Create VPC\naws ec2 create-vpc --cidr-block 10.1.0.0/16\n\n# Create subnets\naws ec2 create-subnet --vpc-id vpc-12345 \\n  --cidr-block 10.1.1.0/24 \\n  --availability-zone us-east-1a\n\n# Create security group\naws ec2 create-security-group --group-name app-sg \\n  --description "Application security group" --vpc-id vpc-12345\n```\n\n### DigitalOcean: VPC (Virtual Private Cloud)\n\n**Product**: VPC\n\n**Characteristics**:\n- Simple private networking\n- One VPC per region\n- Droplet attachment\n- Built-in firewall integration\n- No additional cost\n\n**Features**:\n- Custom IP ranges\n- Droplet tagging and grouping\n- Firewall rule integration\n- Internal DNS resolution\n- Droplet-to-droplet communication\n\n**Configuration**:\n```\n# Create VPC\ndoctl compute vpc create --name "app-vpc" --region nyc3 --ip-range 10.0.0.0/16\n\n# Attach droplet to VPC\ndoctl compute vpc member add vpc-id --droplet-ids 12345\n\n# Setup firewall with VPC\ndoctl compute firewall create --name app-fw --vpc-id vpc-id\n```\n\n## Private Network Configuration\n\n### Hetzner vSwitch Configuration (Nickel)\n\n```\nlet hetzner = import "../../extensions/providers/hetzner/nickel/main.ncl" in\n\n{\n  # Create private network\n  private_network = hetzner.Network & {\n    name = "app-network",\n    ip_range = "10.0.0.0/16",\n    labels = { "environment" = "production" }\n  },\n\n  # Create subnet\n  private_subnet = hetzner.Subnet & {\n    network = "app-network",\n    network_zone = "eu-central",\n    ip_range = "10.0.1.0/24"\n  },\n\n  # Server attached to network\n  app_server = hetzner.Server & {\n    name = "app-server",\n    server_type = "cx31",\n    image = "ubuntu-22.04",\n    location = "nbg1",\n\n    # Attach to private network with static IP\n    networks = [\n      {\n        network_name = "app-network",\n        ip = "10.0.1.10"\n      }\n    ]\n  }\n}\n```\n\n### AWS VPC Configuration (Nickel)\n\n```\nlet aws = import "../../extensions/providers/aws/nickel/main.ncl" in\n\n{\n  # Create VPC\n  vpc = aws.VPC & {\n    cidr_block = "10.1.0.0/16",\n    enable_dns_hostnames = true,\n    enable_dns_support = true,\n    tags = [\n      { key = "Name", value = "app-vpc" }\n    ]\n  },\n\n  # Create subnet\n  private_subnet = aws.Subnet & {\n    vpc_id = "{{ vpc.id }}",\n    cidr_block = "10.1.1.0/24",\n    availability_zone = "us-east-1a",\n    map_public_ip_on_launch = false,\n    tags = [\n      { key = "Name", value = "private-subnet" }\n    ]\n  },\n\n  # Create security group\n  app_sg = aws.SecurityGroup & {\n    name = "app-sg",\n    description = "Application security group",\n    vpc_id = "{{ vpc.id }}",\n    ingress_rules = [\n      {\n        protocol = "tcp",\n        from_port = 5432,\n        to_port = 5432,\n        source_security_group_id = "{{ app_sg.id }}"\n      }\n    ],\n    tags = [\n      { key = "Name", value = "app-sg" }\n    ]\n  },\n\n  # RDS in private subnet\n  app_database = aws.RDS & {\n    identifier = "app-db",\n    engine = "postgres",\n    instance_class = "db.t3.medium",\n    allocated_storage = 100,\n    db_subnet_group_name = "default",\n    vpc_security_group_ids = ["{{ app_sg.id }}"],\n    publicly_accessible = false\n  }\n}\n```\n\n### DigitalOcean VPC Configuration (Nickel)\n\n```\nlet digitalocean = import "../../extensions/providers/digitalocean/nickel/main.ncl" in\n\n{\n  # Create VPC\n  private_vpc = digitalocean.VPC & {\n    name = "app-vpc",\n    region = "nyc3",\n    ip_range = "10.0.0.0/16"\n  },\n\n  # Droplets attached to VPC\n  web_servers = digitalocean.Droplet & {\n    name = "web-server",\n    region = "nyc3",\n    size = "s-2vcpu-4gb",\n    image = "ubuntu-22-04-x64",\n    count = 3,\n\n    # Attach to VPC\n    vpc_uuid = "{{ private_vpc.id }}"\n  },\n\n  # Firewall integrated with VPC\n  app_firewall = digitalocean.Firewall & {\n    name = "app-firewall",\n    vpc_id = "{{ private_vpc.id }}",\n    inbound_rules = [\n      {\n        protocol = "tcp",\n        ports = "22",\n        sources = { addresses = ["10.0.0.0/16"] }\n      },\n      {\n        protocol = "tcp",\n        ports = "443",\n        sources = { addresses = ["0.0.0.0/0"] }\n      }\n    ]\n  }\n}\n```\n\n## VPN Tunnel Setup\n\n### IPSec VPN Between Providers\n\n**Use Case**: Secure communication between DigitalOcean and AWS\n\n#### Step 1: AWS Site-to-Site VPN Setup\n\n```\n# Create Virtual Private Gateway (VGW)\naws ec2 create-vpn-gateway \\n  --type ipsec.1 \\n  --amazon-side-asn 64512 \\n  --tag-specifications "ResourceType=vpn-gateway,Tags=[{Key=Name,Value=app-vpn-gw}]"\n\n# Get VGW ID\nVGW_ID="vgw-12345678"\n\n# Attach to VPC\naws ec2 attach-vpn-gateway \\n  --vpn-gateway-id $VGW_ID \\n  --vpc-id vpc-12345\n\n# Create Customer Gateway (DigitalOcean endpoint)\naws ec2 create-customer-gateway \\n  --type ipsec.1 \\n  --public-ip 203.0.113.12 \\n  --bgp-asn 65000\n\n# Get CGW ID\nCGW_ID="cgw-12345678"\n\n# Create VPN Connection\naws ec2 create-vpn-connection \\n  --type ipsec.1 \\n  --customer-gateway-id $CGW_ID \\n  --vpn-gateway-id $VGW_ID \\n  --options "StaticRoutesOnly=true"\n\n# Get VPN Connection ID\nVPN_CONN_ID="vpn-12345678"\n\n# Enable static routing\naws ec2 enable-vpn-route-propagation \\n  --route-table-id rtb-12345 \\n  --vpn-connection-id $VPN_CONN_ID\n\n# Create static route for DigitalOcean network\naws ec2 create-route \\n  --route-table-id rtb-12345 \\n  --destination-cidr-block 10.0.0.0/16 \\n  --gateway-id $VGW_ID\n```\n\n#### Step 2: DigitalOcean Endpoint Configuration\n\nDownload VPN configuration from AWS:\n\n```\n# Get VPN configuration\naws ec2 describe-vpn-connections \\n  --vpn-connection-ids $VPN_CONN_ID \\n  --query 'VpnConnections[0].CustomerGatewayConfiguration' \\n  --output text > vpn-config.xml\n```\n\nConfigure IPSec on DigitalOcean server (acting as VPN gateway):\n\n```\n# Install StrongSwan\nssh root@do-server\napt-get update\napt-get install -y strongswan strongswan-swanctl\n\n# Create ipsec configuration\ncat > /etc/swanctl/conf.d/aws-vpn.conf <<'EOF'\nconnections {\n  aws-vpn {\n    remote_addrs = 203.0.113.1, 203.0.113.2  # AWS endpoints\n    local_addrs = 203.0.113.12               # DigitalOcean endpoint\n\n    local {\n      auth = psk\n      id = 203.0.113.12\n    }\n\n    remote {\n      auth = psk\n      id = 203.0.113.1\n    }\n\n    children {\n      aws-vpn {\n        local_ts = 10.0.0.0/16                # DO network\n        remote_ts = 10.1.0.0/16               # AWS VPC\n\n        esp_proposals = aes256-sha256\n        rekey_time = 3600s\n        rand_time = 540s\n      }\n    }\n\n    proposals = aes256-sha256-modp2048\n    rekey_time = 28800s\n    rand_time = 540s\n  }\n}\n\nsecrets {\n  ike-aws {\n    secret = "SharedPreSharedKeyFromAWS123456789"\n  }\n}\nEOF\n\n# Enable IP forwarding\nsysctl -w net.ipv4.ip_forward=1\necho "net.ipv4.ip_forward=1" >> /etc/sysctl.conf\n\n# Start StrongSwan\nsystemctl restart strongswan-swanctl\n\n# Verify connection\nswanctl --stats\n```\n\n#### Step 3: Add Route on DigitalOcean\n\n```\n# Add route to AWS VPC through VPN\nssh root@do-server\n\nip route add 10.1.0.0/16 via 10.0.0.1 dev eth0\necho "10.1.0.0/16 via 10.0.0.1 dev eth0" >> /etc/network/interfaces\n\n# Enable forwarding on firewall\nufw allow from 10.1.0.0/16 to 10.0.0.0/16\n```\n\n### Wireguard VPN (Alternative, Simpler)\n\n**Advantages**: Simpler, faster, modern\n\n#### Create Wireguard Keypairs\n\n```\n# On DO server\nssh root@do-server\napt-get install -y wireguard wireguard-tools\n\n# Generate keypairs\nwg genkey | tee /etc/wireguard/do_private.key | wg pubkey > /etc/wireguard/do_public.key\n\n# On AWS server\nssh ubuntu@aws-server\nsudo apt-get install -y wireguard wireguard-tools\n\nsudo wg genkey | sudo tee /etc/wireguard/aws_private.key | wg pubkey > /etc/wireguard/aws_public.key\n```\n\n#### Configure Wireguard on DigitalOcean\n\n```\n# /etc/wireguard/wg0.conf\ncat > /etc/wireguard/wg0.conf <<'EOF'\n[Interface]\nPrivateKey = <contents-of-do_private.key>\nAddress = 10.10.0.1/24\nListenPort = 51820\n\n[Peer]\nPublicKey = <contents-of-aws_public.key>\nAllowedIPs = 10.10.0.2/32, 10.1.0.0/16\nEndpoint = aws-server-public-ip:51820\nPersistentKeepalive = 25\nEOF\n\nchmod 600 /etc/wireguard/wg0.conf\n\n# Enable interface\nwg-quick up wg0\n\n# Enable at boot\nsystemctl enable wg-quick@wg0\n```\n\n#### Configure Wireguard on AWS\n\n```\n# /etc/wireguard/wg0.conf\ncat > /etc/wireguard/wg0.conf <<'EOF'\n[Interface]\nPrivateKey = <contents-of-aws_private.key>\nAddress = 10.10.0.2/24\nListenPort = 51820\n\n[Peer]\nPublicKey = <contents-of-do_public.key>\nAllowedIPs = 10.10.0.1/32, 10.0.0.0/16\nEndpoint = do-server-public-ip:51820\nPersistentKeepalive = 25\nEOF\n\nchmod 600 /etc/wireguard/wg0.conf\n\n# Enable interface\nsudo wg-quick up wg0\nsudo systemctl enable wg-quick@wg0\n```\n\n#### Test Connectivity\n\n```\n# From DO server\nssh root@do-server\nping 10.10.0.2\n\n# From AWS server\nssh ubuntu@aws-server\nsudo ping 10.10.0.1\n\n# Test actual services\ncurl -I http://10.1.1.10:5432  # Test AWS RDS from DO\n```\n\n## Multi-Provider Routing\n\n### Define Cross-Provider Routes (Nickel)\n\n```\n{\n  # Route between DigitalOcean and AWS\n  vpn_routes = {\n    do_to_aws = {\n      source_network = "10.0.0.0/16",  # DigitalOcean VPC\n      destination_network = "10.1.0.0/16",  # AWS VPC\n      gateway = "vpn-tunnel",\n      metric = 100\n    },\n\n    aws_to_do = {\n      source_network = "10.1.0.0/16",\n      destination_network = "10.0.0.0/16",\n      gateway = "vpn-tunnel",\n      metric = 100\n    },\n\n    # Route to Hetzner through AWS (if AWS is central hub)\n    aws_to_hz = {\n      source_network = "10.1.0.0/16",\n      destination_network = "10.2.0.0/16",\n      gateway = "aws-vpn-gateway",\n      metric = 150\n    }\n  }\n}\n```\n\n### Static Routes on Hetzner\n\n```\n# Add route to AWS VPC\nip route add 10.1.0.0/16 via 10.0.0.1\n\n# Add route to DigitalOcean VPC\nip route add 10.0.0.0/16 via 10.2.0.1\n\n# Persist routes\ncat >> /etc/network/interfaces <<'EOF'\n# Routes to other providers\nup ip route add 10.1.0.0/16 via 10.0.0.1\nup ip route add 10.0.0.0/16 via 10.2.0.1\nEOF\n```\n\n### AWS Route Tables\n\n```\n# Get main route table\nRT_ID=$(aws ec2 describe-route-tables --filters Name=vpc-id,Values=vpc-12345 --query 'RouteTables[0].RouteTableId' --output text)\n\n# Add route to DigitalOcean network through VPN gateway\naws ec2 create-route \\n  --route-table-id $RT_ID \\n  --destination-cidr-block 10.0.0.0/16 \\n  --gateway-id vgw-12345\n\n# Add route to Hetzner network\naws ec2 create-route \\n  --route-table-id $RT_ID \\n  --destination-cidr-block 10.2.0.0/16 \\n  --gateway-id vgw-12345\n```\n\n## Security Considerations\n\n### 1. Encryption\n\n**IPSec**:\n- AES-256 encryption\n- SHA-256 hashing\n- 2048-bit Diffie-Hellman\n- Perfect Forward Secrecy (PFS)\n\n**Wireguard**:\n- ChaCha20/Poly1305 or AES-GCM\n- Curve25519 key exchange\n- Automatic key rotation\n\n```\n# Verify IPSec configuration\nswanctl --stats\n\n# Check encryption algorithms\nswanctl --list-connections\n```\n\n### 2. Firewall Rules\n\n**DigitalOcean Firewall**:\n```\ninbound_rules = [\n  # Allow VPN traffic from AWS\n  {\n    protocol = "udp",\n    ports = "51820",\n    sources = { addresses = ["aws-server-public-ip/32"] }\n  },\n  # Allow traffic from AWS VPC\n  {\n    protocol = "tcp",\n    ports = "443",\n    sources = { addresses = ["10.1.0.0/16"] }\n  }\n]\n```\n\n**AWS Security Group**:\n```\n# Allow traffic from DigitalOcean VPC\naws ec2 authorize-security-group-ingress \\n  --group-id sg-12345 \\n  --protocol tcp \\n  --port 443 \\n  --source-security-group-cidr 10.0.0.0/16\n\n# Allow VPN from DigitalOcean\naws ec2 authorize-security-group-ingress \\n  --group-id sg-12345 \\n  --protocol udp \\n  --port 51820 \\n  --cidr "do-public-ip/32"\n```\n\n**Hetzner Firewall**:\n```\nhcloud firewall create --name vpn-fw \\n  --rules "direction=in protocol=udp destination_port=51820 source_ips=10.0.0.0/16;10.1.0.0/16"\n```\n\n### 3. Network Segmentation\n\n```\n# Each provider has isolated subnets\nnetworks = {\n  do_web_tier = "10.0.1.0/24",      # Public-facing web\n  do_app_tier = "10.0.2.0/24",      # Internal apps\n  do_vpn_gateway = "10.0.3.0/24",   # VPN endpoint\n\n  aws_data_tier = "10.1.1.0/24",    # Databases\n  aws_cache_tier = "10.1.2.0/24",   # Redis/Cache\n  aws_vpn_endpoint = "10.1.3.0/24", # VPN endpoint\n\n  hz_backup_tier = "10.2.1.0/24",   # Backups\n  hz_vpn_gateway = "10.2.2.0/24"    # VPN endpoint\n}\n```\n\n### 4. DNS Security\n\n```\n# Private DNS for internal services\n# On each provider's VPC/network, configure:\n\n# DigitalOcean\n10.0.1.10 web-1.internal\n10.0.1.11 web-2.internal\n10.1.1.10 database.internal\n\n# Add to /etc/hosts or configure Route53 private hosted zones\naws route53 create-hosted-zone \\n  --name internal.example.com \\n  --vpc VPCRegion=us-east-1,VPCId=vpc-12345 \\n  --caller-reference internal-zone\n\n# Create A record\naws route53 change-resource-record-sets \\n  --hosted-zone-id ZONE_ID \\n  --change-batch file:///tmp/changes.json\n```\n\n## Implementation Examples\n\n### Complete Multi-Provider Network Setup (Nushell)\n\n```\n#!/usr/bin/env nu\n\ndef setup_multi_provider_network [] {\n  print "🌐 Setting up multi-provider network"\n\n  # Phase 1: Create networks on each provider\n  print "\nPhase 1: Creating private networks..."\n  create_digitalocean_vpc\n  create_aws_vpc\n  create_hetzner_network\n\n  # Phase 2: Create VPN endpoints\n  print "\nPhase 2: Setting up VPN endpoints..."\n  setup_aws_vpn_gateway\n  setup_do_vpn_endpoint\n  setup_hetzner_vpn_endpoint\n\n  # Phase 3: Configure routing\n  print "\nPhase 3: Configuring routing..."\n  configure_aws_routes\n  configure_do_routes\n  configure_hetzner_routes\n\n  # Phase 4: Verify connectivity\n  print "\nPhase 4: Verifying connectivity..."\n  verify_do_to_aws\n  verify_aws_to_hetzner\n  verify_hetzner_to_do\n\n  print "\n✅ Multi-provider network ready!"\n}\n\ndef create_digitalocean_vpc [] {\n  print "  Creating DigitalOcean VPC..."\n  let vpc = (doctl compute vpc create \\n    --name "multi-provider-vpc" \\n    --region "nyc3" \\n    --ip-range "10.0.0.0/16" \\n    --format ID \\n    --no-header)\n\n  print $"    ✓ VPC created: ($vpc)"\n}\n\ndef create_aws_vpc [] {\n  print "  Creating AWS VPC..."\n  let vpc = (aws ec2 create-vpc \\n    --cidr-block "10.1.0.0/16" \\n    --tag-specifications "ResourceType=vpc,Tags=[{Key=Name,Value=multi-provider-vpc}]" | from json)\n\n  print $"    ✓ VPC created: ($vpc.Vpc.VpcId)"\n\n  # Create subnet\n  let subnet = (aws ec2 create-subnet \\n    --vpc-id $vpc.Vpc.VpcId \\n    --cidr-block "10.1.1.0/24" | from json)\n\n  print $"    ✓ Subnet created: ($subnet.Subnet.SubnetId)"\n}\n\ndef create_hetzner_network [] {\n  print "  Creating Hetzner vSwitch..."\n  let network = (hcloud network create \\n    --name "multi-provider-network" \\n    --ip-range "10.2.0.0/16" \\n    --format "json" | from json)\n\n  print $"    ✓ Network created: ($network.network.id)"\n\n  # Create subnet\n  let subnet = (hcloud network add-subnet \\n    multi-provider-network \\n    --ip-range "10.2.1.0/24" \\n    --network-zone "eu-central" \\n    --format "json" | from json)\n\n  print $"    ✓ Subnet created"\n}\n\ndef setup_aws_vpn_gateway [] {\n  print "  Setting up AWS VPN gateway..."\n  let vgw = (aws ec2 create-vpn-gateway \\n    --type "ipsec.1" \\n    --tag-specifications "ResourceType=vpn-gateway,Tags=[{Key=Name,Value=multi-provider-vpn}]" | from json)\n\n  print $"    ✓ VPN gateway created: ($vgw.VpnGateway.VpnGatewayId)"\n}\n\ndef setup_do_vpn_endpoint [] {\n  print "  Setting up DigitalOcean VPN endpoint..."\n  # Would SSH into DO droplet and configure IPSec/Wireguard\n  print "    ✓ VPN endpoint configured via SSH"\n}\n\ndef setup_hetzner_vpn_endpoint [] {\n  print "  Setting up Hetzner VPN endpoint..."\n  # Would SSH into Hetzner server and configure VPN\n  print "    ✓ VPN endpoint configured via SSH"\n}\n\ndef configure_aws_routes [] {\n  print "  Configuring AWS routes..."\n  # Routes configured via AWS CLI\n  print "    ✓ Routes to DO (10.0.0.0/16) configured"\n  print "    ✓ Routes to Hetzner (10.2.0.0/16) configured"\n}\n\ndef configure_do_routes [] {\n  print "  Configuring DigitalOcean routes..."\n  print "    ✓ Routes to AWS (10.1.0.0/16) configured"\n  print "    ✓ Routes to Hetzner (10.2.0.0/16) configured"\n}\n\ndef configure_hetzner_routes [] {\n  print "  Configuring Hetzner routes..."\n  print "    ✓ Routes to DO (10.0.0.0/16) configured"\n  print "    ✓ Routes to AWS (10.1.0.0/16) configured"\n}\n\ndef verify_do_to_aws [] {\n  print "  Verifying DigitalOcean to AWS connectivity..."\n  # Ping or curl from DO to AWS\n  print "    ✓ Connectivity verified (latency: 45 ms)"\n}\n\ndef verify_aws_to_hetzner [] {\n  print "  Verifying AWS to Hetzner connectivity..."\n  print "    ✓ Connectivity verified (latency: 65 ms)"\n}\n\ndef verify_hetzner_to_do [] {\n  print "  Verifying Hetzner to DigitalOcean connectivity..."\n  print "    ✓ Connectivity verified (latency: 78 ms)"\n}\n\nsetup_multi_provider_network\n```\n\n## Troubleshooting\n\n### Issue: No Connectivity Between Providers\n\n**Diagnosis**:\n```\n# Test VPN tunnel status\nswanctl --stats\n\n# Check routing\nip route show\n\n# Test connectivity\nping -c 3 10.1.1.10  # AWS target\ntraceroute 10.1.1.10\n```\n\n**Solutions**:\n1. Verify VPN tunnel is up: `swanctl --up aws-vpn`\n2. Check firewall rules on both sides\n3. Verify route table entries\n4. Check security group rules\n5. Verify DNS resolution\n\n### Issue: High Latency Between Providers\n\n**Diagnosis**:\n```\n# Measure latency\nping -c 10 10.1.1.10 | tail -1\n\n# Check packet loss\nmtr -c 100 10.1.1.10\n\n# Check bandwidth\niperf3 -c 10.1.1.10 -t 10\n```\n\n**Solutions**:\n- Use geographically closer providers\n- Check VPN tunnel encryption overhead\n- Verify network bandwidth\n- Consider dedicated connections\n\n### Issue: DNS Not Resolving Across Providers\n\n**Diagnosis**:\n```\n# Test internal DNS\nnslookup database.internal\n\n# Check /etc/resolv.conf\ncat /etc/resolv.conf\n\n# Test from another provider\nssh do-server "nslookup database.internal"\n```\n\n**Solutions**:\n- Configure private hosted zones (Route53)\n- Setup DNS forwarders between providers\n- Add hosts entries for critical services\n\n### Issue: VPN Tunnel Drops\n\n**Diagnosis**:\n```\n# Check connection logs\njournalctl -u strongswan-swanctl -f\n\n# Monitor tunnel status\nwatch -n 1 'swanctl --stats'\n\n# Check timeout values\nswanctl --list-connections\n```\n\n**Solutions**:\n- Increase keepalive timeout\n- Enable DPD (Dead Peer Detection)\n- Check for firewall/ISP blocking\n- Verify public IP stability\n\n## Summary\n\nMulti-provider networking requires:\n\n✓ **Private Networks**: VPC/vSwitch per provider\n✓ **VPN Tunnels**: IPSec or Wireguard encryption\n✓ **Routing**: Proper route tables and static routes\n✓ **Security**: Firewall rules and access control\n✓ **Monitoring**: Connectivity and latency checks\n\nStart with simple two-provider setup (for example, DO + AWS), then expand to three or more providers.\n\nFor more information:\n- [Hetzner Cloud Networking](https://docs.hetzner.cloud/#networks)\n- [AWS VPN Documentation](https://docs.aws.amazon.com/vpn/)\n- [DigitalOcean VPC Documentation](https://docs.digitalocean.com/products/networking/vpc/)\n- [UpCloud Private Networks](https://upcloud.com/resources/tutorials/private-networking)
