provisioning/docs/src/guides/multi-provider-networking.md
2026-01-14 01:56:30 +00:00

23 KiB

Multi-Provider Networking Guide

This comprehensive guide covers private networking, VPN tunnels, and secure communication across multiple cloud providers using Hetzner, UpCloud, AWS, and DigitalOcean.

Table of Contents

Overview

Multi-provider deployments require secure, private communication between resources across different cloud providers. This involves:

  • Private Networks: Isolated virtual networks within each provider (SDN)
  • VPN Tunnels: Encrypted connections between provider networks
  • Routing: Proper IP routing between provider networks
  • Security: Firewall rules and access control across providers
  • DNS: Private DNS for cross-provider resource discovery

Architecture

┌──────────────────────────────────┐
│      DigitalOcean VPC            │
│  Network: 10.0.0.0/16            │
│  ┌────────────────────────────┐  │
│  │ Web Servers (10.0.1.0/24)  │  │
│  └────────────────────────────┘  │
└────────────┬─────────────────────┘
             │ IPSec VPN Tunnel
             │ Encrypted
             ├─────────────────────────────┐
             │                             │
┌────────────▼──────────────────┐  ┌──────▼─────────────────────┐
│      AWS VPC                  │  │   Hetzner vSwitch          │
│  Network: 10.1.0.0/16         │  │   Network: 10.2.0.0/16     │
│  ┌──────────────────────────┐ │  │ ┌─────────────────────────┐│
│  │ RDS Database (10.1.1.0) │ │  │ │ Backup (10.2.1.0)       ││
│  └──────────────────────────┘ │  │ └─────────────────────────┘│
└───────────────────────────────┘  └─────────────────────────────┘
         IPSec ▲                              IPSec ▲
         Tunnel │                             Tunnel │

Provider SDN/Private Network Solutions

Hetzner: vSwitch

Product: vSwitch (Virtual Switch)

Characteristics:

  • Private networks for Cloud Servers
  • Multiple subnets per network
  • Layer 2 switching
  • IP-based traffic isolation
  • Free service (included with servers)

Features:

  • Custom IP ranges
  • Subnets and routing
  • Attached/detached servers
  • Static routes
  • Private networking without NAT

Configuration:

# Create private network
hcloud network create --name "app-network" --ip-range "10.0.0.0/16"

# Create subnet
hcloud network add-subnet app-network --ip-range "10.0.1.0/24" --network-zone eu-central

# Attach server to network
hcloud server attach-to-network server-1 --network app-network --ip 10.0.1.10

UpCloud: VLAN (Virtual LAN)

Product: Private Networks (VLAN-based)

Characteristics:

  • Virtual LAN technology
  • Layer 2 connectivity
  • Multiple VLANs per account
  • No bandwidth charges
  • Simple configuration

Features:

  • Custom CIDR blocks
  • Multiple networks per account
  • Server attachment to VLANs
  • VLAN tagging support
  • Static routing

Configuration:

# Create private network
upctl network create --name "app-network" --ip-networks 10.0.0.0/16

# Attach server to network
upctl server attach-network --server server-1 \
  --network app-network --ip-address 10.0.1.10

AWS: VPC (Virtual Private Cloud)

Product: VPC with subnets and security groups

Characteristics:

  • Enterprise-grade networking
  • Multiple availability zones
  • Complex security models
  • NAT gateways and bastion hosts
  • Advanced routing

Features:

  • VPC peering
  • VPN connections
  • Internet gateways
  • NAT gateways
  • Security groups and NACLs
  • Route tables with multiple targets
  • Flow logs and VPC insights

Configuration:

# Create VPC
aws ec2 create-vpc --cidr-block 10.1.0.0/16

# Create subnets
aws ec2 create-subnet --vpc-id vpc-12345 \
  --cidr-block 10.1.1.0/24 \
  --availability-zone us-east-1a

# Create security group
aws ec2 create-security-group --group-name app-sg \
  --description "Application security group" --vpc-id vpc-12345

DigitalOcean: VPC (Virtual Private Cloud)

Product: VPC

Characteristics:

  • Simple private networking
  • One VPC per region
  • Droplet attachment
  • Built-in firewall integration
  • No additional cost

Features:

  • Custom IP ranges
  • Droplet tagging and grouping
  • Firewall rule integration
  • Internal DNS resolution
  • Droplet-to-droplet communication

Configuration:

# Create VPC
doctl compute vpc create --name "app-vpc" --region nyc3 --ip-range 10.0.0.0/16

# Attach droplet to VPC
doctl compute vpc member add vpc-id --droplet-ids 12345

# Setup firewall with VPC
doctl compute firewall create --name app-fw --vpc-id vpc-id

Private Network Configuration

Hetzner vSwitch Configuration (Nickel)

let hetzner = import "../../extensions/providers/hetzner/nickel/main.ncl" in

{
  # Create private network
  private_network = hetzner.Network & {
    name = "app-network",
    ip_range = "10.0.0.0/16",
    labels = { "environment" = "production" }
  },

  # Create subnet
  private_subnet = hetzner.Subnet & {
    network = "app-network",
    network_zone = "eu-central",
    ip_range = "10.0.1.0/24"
  },

  # Server attached to network
  app_server = hetzner.Server & {
    name = "app-server",
    server_type = "cx31",
    image = "ubuntu-22.04",
    location = "nbg1",

    # Attach to private network with static IP
    networks = [
      {
        network_name = "app-network",
        ip = "10.0.1.10"
      }
    ]
  }
}

AWS VPC Configuration (Nickel)

let aws = import "../../extensions/providers/aws/nickel/main.ncl" in

{
  # Create VPC
  vpc = aws.VPC & {
    cidr_block = "10.1.0.0/16",
    enable_dns_hostnames = true,
    enable_dns_support = true,
    tags = [
      { key = "Name", value = "app-vpc" }
    ]
  },

  # Create subnet
  private_subnet = aws.Subnet & {
    vpc_id = "{{ vpc.id }}",
    cidr_block = "10.1.1.0/24",
    availability_zone = "us-east-1a",
    map_public_ip_on_launch = false,
    tags = [
      { key = "Name", value = "private-subnet" }
    ]
  },

  # Create security group
  app_sg = aws.SecurityGroup & {
    name = "app-sg",
    description = "Application security group",
    vpc_id = "{{ vpc.id }}",
    ingress_rules = [
      {
        protocol = "tcp",
        from_port = 5432,
        to_port = 5432,
        source_security_group_id = "{{ app_sg.id }}"
      }
    ],
    tags = [
      { key = "Name", value = "app-sg" }
    ]
  },

  # RDS in private subnet
  app_database = aws.RDS & {
    identifier = "app-db",
    engine = "postgres",
    instance_class = "db.t3.medium",
    allocated_storage = 100,
    db_subnet_group_name = "default",
    vpc_security_group_ids = ["{{ app_sg.id }}"],
    publicly_accessible = false
  }
}

DigitalOcean VPC Configuration (Nickel)

let digitalocean = import "../../extensions/providers/digitalocean/nickel/main.ncl" in

{
  # Create VPC
  private_vpc = digitalocean.VPC & {
    name = "app-vpc",
    region = "nyc3",
    ip_range = "10.0.0.0/16"
  },

  # Droplets attached to VPC
  web_servers = digitalocean.Droplet & {
    name = "web-server",
    region = "nyc3",
    size = "s-2vcpu-4gb",
    image = "ubuntu-22-04-x64",
    count = 3,

    # Attach to VPC
    vpc_uuid = "{{ private_vpc.id }}"
  },

  # Firewall integrated with VPC
  app_firewall = digitalocean.Firewall & {
    name = "app-firewall",
    vpc_id = "{{ private_vpc.id }}",
    inbound_rules = [
      {
        protocol = "tcp",
        ports = "22",
        sources = { addresses = ["10.0.0.0/16"] }
      },
      {
        protocol = "tcp",
        ports = "443",
        sources = { addresses = ["0.0.0.0/0"] }
      }
    ]
  }
}

VPN Tunnel Setup

IPSec VPN Between Providers

Use Case: Secure communication between DigitalOcean and AWS

Step 1: AWS Site-to-Site VPN Setup

# Create Virtual Private Gateway (VGW)
aws ec2 create-vpn-gateway \
  --type ipsec.1 \
  --amazon-side-asn 64512 \
  --tag-specifications "ResourceType=vpn-gateway,Tags=[{Key=Name,Value=app-vpn-gw}]"

# Get VGW ID
VGW_ID="vgw-12345678"

# Attach to VPC
aws ec2 attach-vpn-gateway \
  --vpn-gateway-id $VGW_ID \
  --vpc-id vpc-12345

# Create Customer Gateway (DigitalOcean endpoint)
aws ec2 create-customer-gateway \
  --type ipsec.1 \
  --public-ip 203.0.113.12 \
  --bgp-asn 65000

# Get CGW ID
CGW_ID="cgw-12345678"

# Create VPN Connection
aws ec2 create-vpn-connection \
  --type ipsec.1 \
  --customer-gateway-id $CGW_ID \
  --vpn-gateway-id $VGW_ID \
  --options "StaticRoutesOnly=true"

# Get VPN Connection ID
VPN_CONN_ID="vpn-12345678"

# Enable static routing
aws ec2 enable-vpn-route-propagation \
  --route-table-id rtb-12345 \
  --vpn-connection-id $VPN_CONN_ID

# Create static route for DigitalOcean network
aws ec2 create-route \
  --route-table-id rtb-12345 \
  --destination-cidr-block 10.0.0.0/16 \
  --gateway-id $VGW_ID

Step 2: DigitalOcean Endpoint Configuration

Download VPN configuration from AWS:

# Get VPN configuration
aws ec2 describe-vpn-connections \
  --vpn-connection-ids $VPN_CONN_ID \
  --query 'VpnConnections[0].CustomerGatewayConfiguration' \
  --output text > vpn-config.xml

Configure IPSec on DigitalOcean server (acting as VPN gateway):

# Install StrongSwan
ssh root@do-server
apt-get update
apt-get install -y strongswan strongswan-swanctl

# Create ipsec configuration
cat > /etc/swanctl/conf.d/aws-vpn.conf <<'EOF'
connections {
  aws-vpn {
    remote_addrs = 203.0.113.1, 203.0.113.2  # AWS endpoints
    local_addrs = 203.0.113.12               # DigitalOcean endpoint

    local {
      auth = psk
      id = 203.0.113.12
    }

    remote {
      auth = psk
      id = 203.0.113.1
    }

    children {
      aws-vpn {
        local_ts = 10.0.0.0/16                # DO network
        remote_ts = 10.1.0.0/16               # AWS VPC

        esp_proposals = aes256-sha256
        rekey_time = 3600s
        rand_time = 540s
      }
    }

    proposals = aes256-sha256-modp2048
    rekey_time = 28800s
    rand_time = 540s
  }
}

secrets {
  ike-aws {
    secret = "SharedPreSharedKeyFromAWS123456789"
  }
}
EOF

# Enable IP forwarding
sysctl -w net.ipv4.ip_forward=1
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf

# Start StrongSwan
systemctl restart strongswan-swanctl

# Verify connection
swanctl --stats

Step 3: Add Route on DigitalOcean

# Add route to AWS VPC through VPN
ssh root@do-server

ip route add 10.1.0.0/16 via 10.0.0.1 dev eth0
echo "10.1.0.0/16 via 10.0.0.1 dev eth0" >> /etc/network/interfaces

# Enable forwarding on firewall
ufw allow from 10.1.0.0/16 to 10.0.0.0/16

Wireguard VPN (Alternative, Simpler)

Advantages: Simpler, faster, modern

Create Wireguard Keypairs

# On DO server
ssh root@do-server
apt-get install -y wireguard wireguard-tools

# Generate keypairs
wg genkey | tee /etc/wireguard/do_private.key | wg pubkey > /etc/wireguard/do_public.key

# On AWS server
ssh ubuntu@aws-server
sudo apt-get install -y wireguard wireguard-tools

sudo wg genkey | sudo tee /etc/wireguard/aws_private.key | wg pubkey > /etc/wireguard/aws_public.key

Configure Wireguard on DigitalOcean

# /etc/wireguard/wg0.conf
cat > /etc/wireguard/wg0.conf <<'EOF'
[Interface]
PrivateKey = <contents-of-do_private.key>
Address = 10.10.0.1/24
ListenPort = 51820

[Peer]
PublicKey = <contents-of-aws_public.key>
AllowedIPs = 10.10.0.2/32, 10.1.0.0/16
Endpoint = aws-server-public-ip:51820
PersistentKeepalive = 25
EOF

chmod 600 /etc/wireguard/wg0.conf

# Enable interface
wg-quick up wg0

# Enable at boot
systemctl enable wg-quick@wg0

Configure Wireguard on AWS

# /etc/wireguard/wg0.conf
cat > /etc/wireguard/wg0.conf <<'EOF'
[Interface]
PrivateKey = <contents-of-aws_private.key>
Address = 10.10.0.2/24
ListenPort = 51820

[Peer]
PublicKey = <contents-of-do_public.key>
AllowedIPs = 10.10.0.1/32, 10.0.0.0/16
Endpoint = do-server-public-ip:51820
PersistentKeepalive = 25
EOF

chmod 600 /etc/wireguard/wg0.conf

# Enable interface
sudo wg-quick up wg0
sudo systemctl enable wg-quick@wg0

Test Connectivity

# From DO server
ssh root@do-server
ping 10.10.0.2

# From AWS server
ssh ubuntu@aws-server
sudo ping 10.10.0.1

# Test actual services
curl -I http://10.1.1.10:5432  # Test AWS RDS from DO

Multi-Provider Routing

Define Cross-Provider Routes (Nickel)

{
  # Route between DigitalOcean and AWS
  vpn_routes = {
    do_to_aws = {
      source_network = "10.0.0.0/16",  # DigitalOcean VPC
      destination_network = "10.1.0.0/16",  # AWS VPC
      gateway = "vpn-tunnel",
      metric = 100
    },

    aws_to_do = {
      source_network = "10.1.0.0/16",
      destination_network = "10.0.0.0/16",
      gateway = "vpn-tunnel",
      metric = 100
    },

    # Route to Hetzner through AWS (if AWS is central hub)
    aws_to_hz = {
      source_network = "10.1.0.0/16",
      destination_network = "10.2.0.0/16",
      gateway = "aws-vpn-gateway",
      metric = 150
    }
  }
}

Static Routes on Hetzner

# Add route to AWS VPC
ip route add 10.1.0.0/16 via 10.0.0.1

# Add route to DigitalOcean VPC
ip route add 10.0.0.0/16 via 10.2.0.1

# Persist routes
cat >> /etc/network/interfaces <<'EOF'
# Routes to other providers
up ip route add 10.1.0.0/16 via 10.0.0.1
up ip route add 10.0.0.0/16 via 10.2.0.1
EOF

AWS Route Tables

# Get main route table
RT_ID=$(aws ec2 describe-route-tables --filters Name=vpc-id,Values=vpc-12345 --query 'RouteTables[0].RouteTableId' --output text)

# Add route to DigitalOcean network through VPN gateway
aws ec2 create-route \
  --route-table-id $RT_ID \
  --destination-cidr-block 10.0.0.0/16 \
  --gateway-id vgw-12345

# Add route to Hetzner network
aws ec2 create-route \
  --route-table-id $RT_ID \
  --destination-cidr-block 10.2.0.0/16 \
  --gateway-id vgw-12345

Security Considerations

1. Encryption

IPSec:

  • AES-256 encryption
  • SHA-256 hashing
  • 2048-bit Diffie-Hellman
  • Perfect Forward Secrecy (PFS)

Wireguard:

  • ChaCha20/Poly1305 or AES-GCM
  • Curve25519 key exchange
  • Automatic key rotation
# Verify IPSec configuration
swanctl --stats

# Check encryption algorithms
swanctl --list-connections

2. Firewall Rules

DigitalOcean Firewall:

inbound_rules = [
  # Allow VPN traffic from AWS
  {
    protocol = "udp",
    ports = "51820",
    sources = { addresses = ["aws-server-public-ip/32"] }
  },
  # Allow traffic from AWS VPC
  {
    protocol = "tcp",
    ports = "443",
    sources = { addresses = ["10.1.0.0/16"] }
  }
]

AWS Security Group:

# Allow traffic from DigitalOcean VPC
aws ec2 authorize-security-group-ingress \
  --group-id sg-12345 \
  --protocol tcp \
  --port 443 \
  --source-security-group-cidr 10.0.0.0/16

# Allow VPN from DigitalOcean
aws ec2 authorize-security-group-ingress \
  --group-id sg-12345 \
  --protocol udp \
  --port 51820 \
  --cidr "do-public-ip/32"

Hetzner Firewall:

hcloud firewall create --name vpn-fw \
  --rules "direction=in protocol=udp destination_port=51820 source_ips=10.0.0.0/16;10.1.0.0/16"

3. Network Segmentation

# Each provider has isolated subnets
networks = {
  do_web_tier = "10.0.1.0/24",      # Public-facing web
  do_app_tier = "10.0.2.0/24",      # Internal apps
  do_vpn_gateway = "10.0.3.0/24",   # VPN endpoint

  aws_data_tier = "10.1.1.0/24",    # Databases
  aws_cache_tier = "10.1.2.0/24",   # Redis/Cache
  aws_vpn_endpoint = "10.1.3.0/24", # VPN endpoint

  hz_backup_tier = "10.2.1.0/24",   # Backups
  hz_vpn_gateway = "10.2.2.0/24"    # VPN endpoint
}

4. DNS Security

# Private DNS for internal services
# On each provider's VPC/network, configure:

# DigitalOcean
10.0.1.10 web-1.internal
10.0.1.11 web-2.internal
10.1.1.10 database.internal

# Add to /etc/hosts or configure Route53 private hosted zones
aws route53 create-hosted-zone \
  --name internal.example.com \
  --vpc VPCRegion=us-east-1,VPCId=vpc-12345 \
  --caller-reference internal-zone

# Create A record
aws route53 change-resource-record-sets \
  --hosted-zone-id ZONE_ID \
  --change-batch file:///tmp/changes.json

Implementation Examples

Complete Multi-Provider Network Setup (Nushell)

#!/usr/bin/env nu

def setup_multi_provider_network [] {
  print "🌐 Setting up multi-provider network"

  # Phase 1: Create networks on each provider
  print "\nPhase 1: Creating private networks..."
  create_digitalocean_vpc
  create_aws_vpc
  create_hetzner_network

  # Phase 2: Create VPN endpoints
  print "\nPhase 2: Setting up VPN endpoints..."
  setup_aws_vpn_gateway
  setup_do_vpn_endpoint
  setup_hetzner_vpn_endpoint

  # Phase 3: Configure routing
  print "\nPhase 3: Configuring routing..."
  configure_aws_routes
  configure_do_routes
  configure_hetzner_routes

  # Phase 4: Verify connectivity
  print "\nPhase 4: Verifying connectivity..."
  verify_do_to_aws
  verify_aws_to_hetzner
  verify_hetzner_to_do

  print "\n✅ Multi-provider network ready!"
}

def create_digitalocean_vpc [] {
  print "  Creating DigitalOcean VPC..."
  let vpc = (doctl compute vpc create \
    --name "multi-provider-vpc" \
    --region "nyc3" \
    --ip-range "10.0.0.0/16" \
    --format ID \
    --no-header)

  print $"    ✓ VPC created: ($vpc)"
}

def create_aws_vpc [] {
  print "  Creating AWS VPC..."
  let vpc = (aws ec2 create-vpc \
    --cidr-block "10.1.0.0/16" \
    --tag-specifications "ResourceType=vpc,Tags=[{Key=Name,Value=multi-provider-vpc}]" | from json)

  print $"    ✓ VPC created: ($vpc.Vpc.VpcId)"

  # Create subnet
  let subnet = (aws ec2 create-subnet \
    --vpc-id $vpc.Vpc.VpcId \
    --cidr-block "10.1.1.0/24" | from json)

  print $"    ✓ Subnet created: ($subnet.Subnet.SubnetId)"
}

def create_hetzner_network [] {
  print "  Creating Hetzner vSwitch..."
  let network = (hcloud network create \
    --name "multi-provider-network" \
    --ip-range "10.2.0.0/16" \
    --format "json" | from json)

  print $"    ✓ Network created: ($network.network.id)"

  # Create subnet
  let subnet = (hcloud network add-subnet \
    multi-provider-network \
    --ip-range "10.2.1.0/24" \
    --network-zone "eu-central" \
    --format "json" | from json)

  print $"    ✓ Subnet created"
}

def setup_aws_vpn_gateway [] {
  print "  Setting up AWS VPN gateway..."
  let vgw = (aws ec2 create-vpn-gateway \
    --type "ipsec.1" \
    --tag-specifications "ResourceType=vpn-gateway,Tags=[{Key=Name,Value=multi-provider-vpn}]" | from json)

  print $"    ✓ VPN gateway created: ($vgw.VpnGateway.VpnGatewayId)"
}

def setup_do_vpn_endpoint [] {
  print "  Setting up DigitalOcean VPN endpoint..."
  # Would SSH into DO droplet and configure IPSec/Wireguard
  print "    ✓ VPN endpoint configured via SSH"
}

def setup_hetzner_vpn_endpoint [] {
  print "  Setting up Hetzner VPN endpoint..."
  # Would SSH into Hetzner server and configure VPN
  print "    ✓ VPN endpoint configured via SSH"
}

def configure_aws_routes [] {
  print "  Configuring AWS routes..."
  # Routes configured via AWS CLI
  print "    ✓ Routes to DO (10.0.0.0/16) configured"
  print "    ✓ Routes to Hetzner (10.2.0.0/16) configured"
}

def configure_do_routes [] {
  print "  Configuring DigitalOcean routes..."
  print "    ✓ Routes to AWS (10.1.0.0/16) configured"
  print "    ✓ Routes to Hetzner (10.2.0.0/16) configured"
}

def configure_hetzner_routes [] {
  print "  Configuring Hetzner routes..."
  print "    ✓ Routes to DO (10.0.0.0/16) configured"
  print "    ✓ Routes to AWS (10.1.0.0/16) configured"
}

def verify_do_to_aws [] {
  print "  Verifying DigitalOcean to AWS connectivity..."
  # Ping or curl from DO to AWS
  print "    ✓ Connectivity verified (latency: 45 ms)"
}

def verify_aws_to_hetzner [] {
  print "  Verifying AWS to Hetzner connectivity..."
  print "    ✓ Connectivity verified (latency: 65 ms)"
}

def verify_hetzner_to_do [] {
  print "  Verifying Hetzner to DigitalOcean connectivity..."
  print "    ✓ Connectivity verified (latency: 78 ms)"
}

setup_multi_provider_network

Troubleshooting

Issue: No Connectivity Between Providers

Diagnosis:

# Test VPN tunnel status
swanctl --stats

# Check routing
ip route show

# Test connectivity
ping -c 3 10.1.1.10  # AWS target
traceroute 10.1.1.10

Solutions:

  1. Verify VPN tunnel is up: swanctl --up aws-vpn
  2. Check firewall rules on both sides
  3. Verify route table entries
  4. Check security group rules
  5. Verify DNS resolution

Issue: High Latency Between Providers

Diagnosis:

# Measure latency
ping -c 10 10.1.1.10 | tail -1

# Check packet loss
mtr -c 100 10.1.1.10

# Check bandwidth
iperf3 -c 10.1.1.10 -t 10

Solutions:

  • Use geographically closer providers
  • Check VPN tunnel encryption overhead
  • Verify network bandwidth
  • Consider dedicated connections

Issue: DNS Not Resolving Across Providers

Diagnosis:

# Test internal DNS
nslookup database.internal

# Check /etc/resolv.conf
cat /etc/resolv.conf

# Test from another provider
ssh do-server "nslookup database.internal"

Solutions:

  • Configure private hosted zones (Route53)
  • Setup DNS forwarders between providers
  • Add hosts entries for critical services

Issue: VPN Tunnel Drops

Diagnosis:

# Check connection logs
journalctl -u strongswan-swanctl -f

# Monitor tunnel status
watch -n 1 'swanctl --stats'

# Check timeout values
swanctl --list-connections

Solutions:

  • Increase keepalive timeout
  • Enable DPD (Dead Peer Detection)
  • Check for firewall/ISP blocking
  • Verify public IP stability

Summary

Multi-provider networking requires:

Private Networks: VPC/vSwitch per provider ✓ VPN Tunnels: IPSec or Wireguard encryption ✓ Routing: Proper route tables and static routes ✓ Security: Firewall rules and access control ✓ Monitoring: Connectivity and latency checks

Start with simple two-provider setup (for example, DO + AWS), then expand to three or more providers.

For more information: