provisioning/examples/workspaces/multi-provider-web-app/README.md

Multi-Provider Web App Workspace

This workspace demonstrates a production-ready web application deployment spanning three cloud providers:

• DigitalOcean: Web servers and load balancing (NYC region)
• AWS: Managed PostgreSQL database with high availability (US-East region)
• Hetzner: Backup storage and disaster recovery (Germany region)

Why Three Providers?

This architecture optimizes cost, performance, and reliability:

• DigitalOcean (~$77/month): Cost-effective compute with simple management
• AWS RDS (~$75/month): Managed database with automatic failover
• Hetzner (~$13/month): Affordable backup storage
• Total: ~$165/month (vs $300+ for equivalent all-cloud setup)

Architecture Overview

┌─────────────────────────────────────────────┐
│            Client Requests                  │
└──────────────┬──────────────────────────────┘
               │ HTTPS/HTTP
       ┌───────▼─────────┐
       │ DigitalOcean LB │
       └───────┬─────────┘
      ┌────────┼────────┐
      │        │        │
    ┌─▼──┐  ┌─▼──┐  ┌─▼──┐
    │Web │  │Web │  │Web │ (DigitalOcean Droplets)
    │ 1  │  │ 2  │  │ 3  │
    └──┬─┘  └──┬─┘  └──┬─┘
       │       │       │
       └───────┼───────┘
               │ VPN Tunnel
       ┌───────▼────────────┐
       │   AWS RDS (PG)     │ (us-east-1)
       │  Multi-AZ Cluster  │
       └────────┬───────────┘
                │ Replication
         ┌──────▼──────────┐
         │ Hetzner Volume  │ (nbg1 - Germany)
         │    Backups      │
         └─────────────────┘

Prerequisites

1. Cloud Accounts

• DigitalOcean: Account with API token
• AWS: Account with access keys
• Hetzner: Account with API token

2. Environment Variables

Set these before deployment:

export DIGITALOCEAN_TOKEN="dop_v1_abc123def456ghi789jkl012mno"
export AWS_ACCESS_KEY_ID="AKIA1234567890ABCDEF"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG+j/zI0m1234567890ab"
export HCLOUD_TOKEN="MC4wNTI1YmE1M2E4YmE0YTQzMTQyZTdlODYy"

3. SSH Key Setup

DigitalOcean

# Upload your SSH public key
doctl compute ssh-key create provisioning-key 
  --public-key-from-file ~/.ssh/id_rsa.pub

# Note the key ID for workspace.ncl
doctl compute ssh-key list

AWS

# Create EC2 key pair (if needed)
aws ec2 create-key-pair --key-name provisioning-key 
  --query 'KeyMaterial' --output text > provisioning-key.pem
chmod 600 provisioning-key.pem

Hetzner

# Upload SSH key
hcloud ssh-key create --name provisioning-key 
  --public-key-from-file ~/.ssh/id_rsa.pub

# List keys
hcloud ssh-key list

4. DNS Setup

Update workspace.ncl with your domain:

• Replace your-certificate-id with actual AWS certificate ID
• Update load balancer CNAME to point to your domain

Deployment

Step 1: Configure the Workspace

Edit workspace.ncl to:

• Set your SSH key IDs
• Update certificate ID for HTTPS
• Set domain names
• Adjust instance counts if needed

Edit config.toml to:

• Set correct environment variable names
• Adjust thresholds and settings

Step 2: Validate Configuration

# Validate Nickel syntax
nickel export workspace.ncl | jq .

# Validate provider credentials
provisioning provider verify digitalocean
provisioning provider verify aws
provisioning provider verify hetzner

Step 3: Deploy

# Using provided deploy script
./deploy.nu

# Or manually via provisioning CLI
provisioning workspace deploy --config config.toml

Step 4: Verify Deployment

# List resources per provider
doctl compute droplet list
aws rds describe-db-instances
hcloud volume list

# Test load balancer
curl http://your-domain.com/health

Post-Deployment Configuration

1. Application Deployment

SSH into web servers and deploy application:

# Get web server IPs
doctl compute droplet list --format Name,PublicIPv4

# SSH to first server
ssh root@198.51.100.15

# Deploy application
cd /var/www
git clone https://github.com/your-org/web-app.git
cd web-app
./deploy.sh

2. Database Configuration

Connect to RDS database and initialize schema:

# Get RDS endpoint
aws rds describe-db-instances --query 'DBInstances[0].Endpoint.Address'

# Connect and initialize
psql -h webapp-db.c9akciq32.us-east-1.rds.amazonaws.com -U admin -d defaultdb < schema.sql

3. DNS Configuration

Point your domain to the load balancer:

# Get load balancer IP
doctl compute load-balancer list

# Update DNS CNAME
# Add CNAME record: app.example.com -> lb-123456789.nyc3.digitalocean.com

4. SSL/TLS Certificate

Use AWS Certificate Manager:

# Request certificate
aws acm request-certificate 
  --domain-name app.example.com 
  --validation-method DNS

# Validate and get certificate ID
aws acm list-certificates | grep app.example.com

# Update workspace.ncl with certificate ID

Monitoring

DigitalOcean Monitoring

• CPU usage tracked per droplet
• Memory usage alerts on Droplet greater than 85%
• Disk space alerts on greater than 90% full

AWS CloudWatch

• RDS database metrics (CPU, connections, disk)
• Automatic failover notifications
• Slow query logging

Hetzner Monitoring

• Volume usage tracking
• Manual monitoring script via cron

Application Monitoring

Implement application-level monitoring:

# SSH to web server
ssh root@198.51.100.15

# Check app logs
tail -f /var/www/app/logs/application.log

# Monitor system resources
top
iostat -x 1

# Check database connection pool
psql -h webapp-db.c9akciq32.us-east-1.rds.amazonaws.com -c "SELECT count(plus) FROM pg_stat_activity;"

Backup and Recovery

Automated Backups

• RDS: Daily backups retained for 30 days (AWS handles)
• Application Data: Weekly backups to Hetzner volume
• Configuration: Version control via Git

Manual Backup

# Backup RDS to Hetzner volume
ssh hetzner-backup-volume

# Mount Hetzner volume (if not mounted)
sudo mount /dev/sdb /mnt/backups

# Backup RDS database
pg_dump -h webapp-db.c9akciq32.us-east-1.rds.amazonaws.com -U admin -d defaultdb | 
  gzip > /mnt/backups/db-$(date +%Y%m%d).sql.gz

Recovery Procedure

1. Web Server Failure: Load balancer automatically redirects to healthy server
2. Database Failure: RDS Multi-AZ automatic failover
3. Complete Failure: Restore from Hetzner backup volume

Scaling

Add More Web Servers

Edit workspace.ncl:

droplets = digitalocean.Droplet & {
  name = "web-server",
  region = "nyc3",
  size = "s-2vcpu-4gb",
  count = 5
}

Redeploy:

./deploy.nu

Upgrade Database

Edit workspace.ncl:

database_tier = aws.RDS & {
  identifier = "webapp-db",
  instance_class = "db.t3.large"
}

Redeploy with minimal downtime (Multi-AZ handles switchover).

Cost Optimization

Reduce Costs

1. Droplets: Use smaller size or fewer instances
2. Database: Switch to smaller db.t3.small (approximately $30/month)
3. Storage: Reduce backup volume size
4. Data Transfer: Monitor and optimize outbound traffic

Monitor Costs

# DigitalOcean estimated bill
doctl billing get

# AWS Cost Explorer
aws ce get-cost-and-usage --time-period Start=2024-01-01,End=2024-01-31

# Hetzner manual tracking via console
# Navigate to https://console.hetzner.cloud/billing

Troubleshooting

Issue: Web Servers Unreachable

Diagnosis:

doctl compute droplet list
doctl compute firewall list-rules firewall-id

Solution:

• Check firewall allows ports 80, 443
• Verify droplets have public IPs
• Check web server application status

Issue: Database Connection Failure

Diagnosis:

aws rds describe-db-instances
aws security-group describe-security-groups

Solution:

• Verify RDS security group allows port 5432 from web servers
• Check RDS status is "available"
• Verify connection string in application

Issue: Backup Volume Not Mounted

Diagnosis:

hcloud volume list
ssh hetzner-volume
lsblk

Solution:

sudo mkfs.ext4 /dev/sdb
sudo mount /dev/sdb /mnt/backups
echo '/dev/sdb /mnt/backups ext4 defaults,nofail 0 0' | sudo tee -a /etc/fstab

Cleanup

To destroy all resources:

# This will delete everything - use carefully
provisioning workspace destroy --config config.toml

# Or manually
doctl compute droplet delete web-server-1 web-server-2 web-server-3
doctl compute load-balancer delete web-lb
aws rds delete-db-instance --db-instance-identifier webapp-db --skip-final-snapshot
hcloud volume delete webapp-backups

Next Steps

1. SSL/TLS: Update certificate and enable HTTPS
2. Auto-scaling: Add DigitalOcean autoscaling based on load
3. Multi-region: Add additional AWS RDS read replicas in other regions
4. Disaster Recovery: Test failover procedures
5. Cost Optimization: Review and optimize resource sizes

Support

For issues or questions:

• Review the multi-provider deployment guide
• Check provider-specific documentation
• Review workspace logs with debug flag: ./deploy.nu --debug

Files

• workspace.ncl: Infrastructure definition (Nickel)
• config.toml: Provider credentials and settings
• deploy.nu: Deployment automation script (Nushell)
• README.md: This file