# Multi-Cloud Deployment

Comprehensive guide to deploying and managing infrastructure across multiple cloud providers
using the Provisioning platform. This guide covers strategies, patterns, and real-world examples
for building resilient multi-cloud architectures.

## Overview

Multi-cloud deployment enables:

- **Vendor independence** - Avoid lock-in to single cloud provider
- **Geographic distribution** - Deploy closer to users worldwide
- **Resilience** - Survive provider outages or regional failures
- **Cost optimization** - Leverage competitive pricing across providers
- **Compliance** - Meet data residency and sovereignty requirements
- **Performance** - Optimize latency through strategic placement

## Multi-Cloud Strategies

### Strategy 1: Primary-Backup Architecture

One provider serves production traffic, another provides disaster recovery.

**Use cases:**
- Cost-conscious deployments
- Regulatory backup requirements
- Testing multi-cloud capabilities

**Example topology:**

```text
Primary (UpCloud EU)          Backup (AWS US)
├── Production workloads      ├── Standby replicas
├── Active databases          ├── Read-only databases
├── Live traffic              └── Failover ready
└── Real-time sync ────────────>
```

**Pros:** Simple management, lower costs, proven failover
**Cons:** Backup resources underutilized, sync lag

### Strategy 2: Active-Active Architecture

Multiple providers serve production traffic simultaneously.

**Use cases:**
- High availability requirements
- Global user base
- Zero-downtime deployments

**Example topology:**

```text
UpCloud (EU)                  AWS (US)                      Local (Development)
├── EU traffic                ├── US traffic                ├── Testing
├── Primary database          ├── Primary database          ├── CI/CD
└── Global load balancer ←────┴──────────────────────────────┘
```

**Pros:** Maximum availability, optimized latency, full utilization
**Cons:** Complex management, higher costs, data consistency challenges

### Strategy 3: Specialized Workload Distribution

Different providers for different workload types based on strengths.

**Use cases:**
- Heterogeneous workloads
- Cost optimization
- Leveraging provider-specific services

**Example topology:**

```text
UpCloud                       AWS                           Local
├── Compute-intensive         ├── Object storage (S3)       ├── Development
├── Kubernetes clusters       ├── Managed databases (RDS)   └── Testing
└── High-performance VMs      └── Serverless (Lambda)
```

**Pros:** Optimize for provider strengths, cost-effective, flexible
**Cons:** Complex integration, vendor-specific knowledge required

### Strategy 4: Compliance-Driven Architecture

Provider selection based on regulatory and data residency requirements.

**Use cases:**
- GDPR compliance
- Data sovereignty
- Industry regulations (HIPAA, PCI-DSS)

**Example topology:**

```text
UpCloud (EU - GDPR)           AWS (US - FedRAMP)            On-Premises (Sensitive)
├── EU customer data          ├── US customer data          ├── PII storage
├── GDPR-compliant            ├── US compliance             └── Encrypted backups
└── Regional processing       └── Federal workloads
```

**Pros:** Meets compliance requirements, data sovereignty
**Cons:** Geographic constraints, complex data management

## Infrastructure Definition

### Multi-Provider Server Configuration

Define servers across multiple providers using Nickel:

```nickel
# infra/multi-cloud-servers.ncl
{
  metadata = {
    name = "multi-cloud-infrastructure"
    environment = 'production
  }

  infrastructure = {
    servers = [
      # UpCloud servers (EU region)
      {
        name = "upcloud-web-01"
        provider = "upcloud"
        zone = "de-fra1"
        plan = "medium"
        role = 'web
        backup_enabled = true
        tags = ["frontend", "europe"]
      }
      {
        name = "upcloud-web-02"
        provider = "upcloud"
        zone = "fi-hel1"
        plan = "medium"
        role = 'web
        backup_enabled = true
        tags = ["frontend", "europe"]
      }

      # AWS servers (US region)
      {
        name = "aws-api-01"
        provider = "aws"
        zone = "us-east-1a"
        plan = "t3.large"
        role = 'api
        backup_enabled = true
        tags = ["backend", "americas"]
      }
      {
        name = "aws-api-02"
        provider = "aws"
        zone = "us-west-2a"
        plan = "t3.large"
        role = 'api
        backup_enabled = true
        tags = ["backend", "americas"]
      }

      # Local provider (development/testing)
      {
        name = "local-test-01"
        provider = "local"
        zone = "localhost"
        plan = "small"
        role = 'test
        backup_enabled = false
        tags = ["testing", "development"]
      }
    ]
  }

  networking = {
    vpn_mesh = true
    cross_provider_routing = true
    dns_strategy = 'geo_distributed
  }
}
```

### Batch Workflow for Multi-Cloud

Use batch workflows for orchestrated multi-cloud deployments:

```nickel
# infra/multi-cloud-batch.ncl
{
  batch_workflow = {
    name = "global-deployment"
    description = "Deploy infrastructure across three cloud providers"

    operations = [
      {
        id = "upcloud-eu"
        provider = "upcloud"
        region = "de-fra1"
        servers = [
          {name = "upcloud-web-01", plan = "medium", role = 'web}
          {name = "upcloud-db-01", plan = "large", role = 'database}
        ]
        taskservs = ["containerd", "nginx", "postgresql"]
        priority = 1
      }

      {
        id = "aws-us"
        provider = "aws"
        region = "us-east-1"
        servers = [
          {name = "aws-api-01", plan = "t3.large", role = 'api}
          {name = "aws-cache-01", plan = "t3.medium", role = 'cache}
        ]
        taskservs = ["containerd", "docker", "redis"]
        dependencies = ["upcloud-eu"]
        priority = 2
      }

      {
        id = "local-dev"
        provider = "local"
        region = "localhost"
        servers = [
          {name = "local-test-01", plan = "small", role = 'test}
        ]
        taskservs = ["containerd"]
        priority = 3
      }
    ]

    execution = {
      parallel_limit = 2
      retry_failed = true
      max_retries = 3
      checkpoint_enabled = true
    }
  }
}
```

## Deployment Patterns

### Pattern 1: Sequential Deployment

Deploy providers one at a time to minimize risk.

```bash
# Deploy to primary provider first
provisioning batch submit infra/upcloud-primary.ncl

# Verify primary deployment
provisioning server list --provider upcloud
provisioning server status upcloud-web-01

# Deploy to secondary provider
provisioning batch submit infra/aws-secondary.ncl

# Verify secondary deployment
provisioning server list --provider aws
```

**Advantages:**
- Controlled rollout
- Easy troubleshooting
- Clear rollback path

**Disadvantages:**
- Slower deployment
- Sequential dependencies

### Pattern 2: Parallel Deployment

Deploy to multiple providers simultaneously for speed.

```bash
# Submit multi-cloud batch workflow
provisioning batch submit infra/multi-cloud-batch.ncl

# Monitor all operations
provisioning batch status

# Check progress per provider
provisioning batch operations --filter provider=upcloud
provisioning batch operations --filter provider=aws
```

**Advantages:**
- Fast deployment
- Efficient resource usage
- Parallel testing

**Disadvantages:**
- Complex failure handling
- Resource contention
- Harder troubleshooting

### Pattern 3: Blue-Green Multi-Cloud

Deploy new infrastructure in parallel, then switch traffic.

```nickel
# infra/blue-green-multi-cloud.ncl
{
  deployment = {
    strategy = 'blue_green

    blue_environment = {
      upcloud = {servers = [{name = "upcloud-web-01-blue", role = 'web}]}
      aws = {servers = [{name = "aws-api-01-blue", role = 'api}]}
    }

    green_environment = {
      upcloud = {servers = [{name = "upcloud-web-01-green", role = 'web}]}
      aws = {servers = [{name = "aws-api-01-green", role = 'api}]}
    }

    traffic_switch = {
      type = 'dns
      validation_required = true
      rollback_timeout_seconds = 300
    }
  }
}
```

```bash
# Deploy green environment
provisioning deployment create --infra blue-green-multi-cloud --target green

# Validate green environment
provisioning deployment validate green

# Switch traffic to green
provisioning deployment switch-traffic green

# Decommission blue environment
provisioning deployment delete blue
```

## Network Configuration

### Cross-Provider VPN Mesh

Connect servers across providers using VPN mesh:

```nickel
# infra/vpn-mesh.ncl
{
  networking = {
    vpn_mesh = {
      enabled = true
      encryption = 'wireguard

      peers = [
        {
          name = "upcloud-gateway"
          provider = "upcloud"
          public_ip = "auto"
          private_subnet = "10.0.1.0/24"
        }
        {
          name = "aws-gateway"
          provider = "aws"
          public_ip = "auto"
          private_subnet = "10.0.2.0/24"
        }
        {
          name = "local-gateway"
          provider = "local"
          public_ip = "192.168.1.1"
          private_subnet = "10.0.3.0/24"
        }
      ]

      routing = {
        dynamic_routes = true
        bgp_enabled = false
        static_routes = [
          {from = "10.0.1.0/24", to = "10.0.2.0/24", via = "aws-gateway"}
          {from = "10.0.2.0/24", to = "10.0.1.0/24", via = "upcloud-gateway"}
        ]
      }
    }
  }
}
```

### Global DNS Configuration

Configure geo-distributed DNS for optimal routing:

```nickel
# infra/global-dns.ncl
{
  dns = {
    provider = 'cloudflare  # or 'route53, 'custom

    zones = [
      {
        name = "example.com"
        type = 'primary

        records = [
          {
            name = "eu"
            type = 'A
            ttl = 300
            values = ["upcloud-web-01.ip", "upcloud-web-02.ip"]
            geo_location = 'europe
          }
          {
            name = "us"
            type = 'A
            ttl = 300
            values = ["aws-api-01.ip", "aws-api-02.ip"]
            geo_location = 'americas
          }
          {
            name = "@"
            type = 'CNAME
            ttl = 60
            value = "global-lb.example.com"
            geo_routing = 'latency_based
          }
        ]
      }
    ]

    health_checks = [
      {target = "upcloud-web-01", interval_seconds = 30}
      {target = "aws-api-01", interval_seconds = 30}
    ]
  }
}
```

## Data Replication

### Database Replication Across Providers

Configure cross-provider database replication:

```nickel
# infra/database-replication.ncl
{
  databases = {
    postgresql = {
      primary = {
        provider = "upcloud"
        server = "upcloud-db-01"
        version = "15"
        replication_role = 'primary
      }

      replicas = [
        {
          provider = "aws"
          server = "aws-db-replica-01"
          version = "15"
          replication_role = 'replica
          replication_lag_max_seconds = 30
          failover_priority = 1
        }
        {
          provider = "local"
          server = "local-db-backup-01"
          version = "15"
          replication_role = 'replica
          replication_lag_max_seconds = 300
          failover_priority = 2
        }
      ]

      replication = {
        method = 'streaming
        ssl_required = true
        compression = true
        conflict_resolution = 'primary_wins
      }
    }
  }
}
```

### Object Storage Sync

Synchronize object storage across providers:

```bash
# Configure cross-provider storage sync
cat > infra/storage-sync.ncl <<'EOF'
{
  storage = {
    sync_policy = {
      source = {
        provider = "upcloud"
        bucket = "primary-storage"
        region = "de-fra1"
      }

      destinations = [
        {
          provider = "aws"
          bucket = "backup-storage"
          region = "us-east-1"
          sync_interval_minutes = 15
        }
      ]

      filters = {
        include_patterns = ["*.pdf", "*.jpg", "backups/*"]
        exclude_patterns = ["temp/*", "*.tmp"]
      }

      conflict_resolution = 'timestamp_wins
    }
  }
}
EOF
```

## Kubernetes Multi-Cloud

### Cluster Federation

Deploy Kubernetes clusters across providers with federation:

```nickel
# infra/k8s-federation.ncl
{
  kubernetes_federation = {
    clusters = [
      {
        name = "upcloud-eu-cluster"
        provider = "upcloud"
        region = "de-fra1"
        control_plane_count = 3
        worker_count = 5
        version = "1.29.0"
      }
      {
        name = "aws-us-cluster"
        provider = "aws"
        region = "us-east-1"
        control_plane_count = 3
        worker_count = 5
        version = "1.29.0"
      }
    ]

    federation = {
      enabled = true
      control_plane_cluster = "upcloud-eu-cluster"

      networking = {
        cluster_mesh = true
        service_discovery = 'dns
        cross_cluster_load_balancing = true
      }

      workload_distribution = {
        strategy = 'geo_aware
        prefer_local = true
        failover_enabled = true
      }
    }
  }
}
```

### Multi-Cluster Deployments

Deploy applications across multiple Kubernetes clusters:

```yaml
# k8s/multi-cluster-deployment.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: multi-cloud-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-frontend
  namespace: multi-cloud-app
  labels:
    app: frontend
    region: europe
spec:
  replicas: 3
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
```

```bash
# Deploy to multiple clusters
export UPCLOUD_KUBECONFIG=~/.kube/config-upcloud
export AWS_KUBECONFIG=~/.kube/config-aws

kubectl --kubeconfig $UPCLOUD_KUBECONFIG apply -f k8s/multi-cluster-deployment.yaml
kubectl --kubeconfig $AWS_KUBECONFIG apply -f k8s/multi-cluster-deployment.yaml

# Verify deployments
kubectl --kubeconfig $UPCLOUD_KUBECONFIG get pods -n multi-cloud-app
kubectl --kubeconfig $AWS_KUBECONFIG get pods -n multi-cloud-app
```

## Cost Optimization

### Provider Selection by Workload

Optimize costs by choosing the most cost-effective provider per workload:

```nickel
# infra/cost-optimized.ncl
{
  cost_optimization = {
    workloads = [
      {
        name = "compute-intensive"
        provider = "upcloud"  # Best compute pricing
        plan = "large"
        count = 10
      }
      {
        name = "storage-heavy"
        provider = "aws"  # Best storage pricing with S3
        plan = "medium"
        count = 5
        storage_type = 's3
      }
      {
        name = "development"
        provider = "local"  # Zero cost
        plan = "small"
        count = 3
      }
    ]

    budget_limits = {
      monthly_max_usd = 5000
      alerts = [
        {threshold_percent = 75, notify = "[ops-team@example.com](mailto:ops-team@example.com)"}
        {threshold_percent = 90, notify = "[finance@example.com](mailto:finance@example.com)"}
      ]
    }
  }
}
```

### Reserved Instance Strategy

Leverage reserved instances for predictable workloads:

```bash
# Configure reserved instances
cat > infra/reserved-instances.ncl <<'EOF'
{
  reserved_instances = {
    upcloud = {
      commitment = 'yearly
      instances = [
        {plan = "medium", count = 5}
        {plan = "large", count = 2}
      ]
    }

    aws = {
      commitment = 'yearly
      instances = [
        {type = "t3.large", count = 3}
        {type = "t3.xlarge", count = 1}
      ]
      savings_plan = true
    }
  }
}
EOF
```

## Monitoring Multi-Cloud

### Centralized Monitoring

Deploy unified monitoring across providers:

```nickel
# infra/monitoring.ncl
{
  monitoring = {
    prometheus = {
      enabled = true
      federation = true

      instances = [
        {provider = "upcloud", region = "de-fra1"}
        {provider = "aws", region = "us-east-1"}
      ]

      scrape_configs = [
        {
          job_name = "upcloud-nodes"
          static_configs = [{targets = ["upcloud-*.internal:9100"]}]
        }
        {
          job_name = "aws-nodes"
          static_configs = [{targets = ["aws-*.internal:9100"]}]
        }
      ]

      remote_write = {
        url = " [https://central-prometheus.example.com/api/v1/write"](https://central-prometheus.example.com/api/v1/write")
        compression = true
      }
    }

    grafana = {
      enabled = true
      dashboards = ["multi-cloud-overview", "per-provider", "cost-analysis"]
      alerts = ["high-latency", "provider-down", "budget-exceeded"]
    }
  }
}
```

## Disaster Recovery

### Cross-Provider Failover

Configure automatic failover between providers:

```nickel
# infra/disaster-recovery.ncl
{
  disaster_recovery = {
    primary_provider = "upcloud"
    secondary_provider = "aws"

    failover_triggers = [
      {condition = 'provider_unavailable, action = 'switch_to_secondary}
      {condition = 'health_check_failed, threshold = 3, action = 'switch_to_secondary}
      {condition = 'latency_exceeded, threshold_ms = 1000, action = 'switch_to_secondary}
    ]

    failover_process = {
      dns_ttl_seconds = 60
      health_check_interval_seconds = 10
      automatic = true
      notification_channels = ["email", "slack"]
    }

    backup_strategy = {
      frequency = 'daily
      retention_days = 30
      cross_region = true
      cross_provider = true
    }
  }
}
```

## Best Practices

### Configuration Management

- Use Nickel for all infrastructure definitions
- Version control all configuration files
- Use workspace per environment (dev/staging/prod)
- Implement configuration validation before deployment
- Maintain provider abstraction where possible

### Security

- Encrypt cross-provider communication (VPN, TLS)
- Use separate credentials per provider
- Implement RBAC consistently across providers
- Enable audit logging on all providers
- Encrypt data at rest and in transit

### Deployment

- Test in single-provider environment first
- Use batch workflows for complex multi-cloud deployments
- Enable checkpoints for long-running deployments
- Implement progressive rollout strategies
- Maintain rollback procedures

### Monitoring

- Centralize logs and metrics
- Monitor cross-provider network latency
- Track costs per provider
- Alert on provider-specific failures
- Measure failover readiness

### Cost Management

- Regular cost audits per provider
- Use reserved instances for predictable loads
- Implement budget alerts
- Optimize data transfer costs
- Consider spot instances for non-critical workloads

## Troubleshooting

### Provider Connectivity Issues

```bash
# Test provider connectivity
provisioning providers

# Test specific provider
provisioning provider test upcloud
provisioning provider test aws

# Debug network connectivity
provisioning network test --from upcloud-web-01 --to aws-api-01
```

### Cross-Provider Communication Failures

```bash
# Check VPN mesh status
provisioning network vpn-status

# Test cross-provider routes
provisioning network trace-route --from upcloud-web-01 --to aws-api-01

# Verify firewall rules
provisioning network firewall-check --provider upcloud
provisioning network firewall-check --provider aws
```

### Data Replication Lag

```bash
# Check replication status
provisioning database replication-status postgresql

# Force replication sync
provisioning database sync --source upcloud-db-01 --target aws-db-replica-01

# View replication lag metrics
provisioning database metrics --metric replication_lag
```

## See Also

- [Batch Workflows](../infrastructure/batch-workflows.md) - Multi-cloud orchestration
- [Providers](../infrastructure/providers.md) - Provider configuration
- [Disaster Recovery](disaster-recovery.md) - DR strategies
- [Workspace Management](workspace-management.md) - Multi-environment setup
