provisioning/examples/workspaces/multi-provider-web-app/README.md

# Multi-Provider Web App Workspace

This workspace demonstrates a production-ready web application deployment spanning three cloud providers:

- **DigitalOcean**: Web servers and load balancing (NYC region)
- **AWS**: Managed PostgreSQL database with high availability (US-East region)
- **Hetzner**: Backup storage and disaster recovery (Germany region)

## Why Three Providers?

This architecture optimizes cost, performance, and reliability:

- **DigitalOcean** (~$77/month): Cost-effective compute with simple management
- **AWS RDS** (~$75/month): Managed database with automatic failover
- **Hetzner** (~$13/month): Affordable backup storage
- **Total**: ~$165/month (vs $300+ for equivalent all-cloud setup)

## Architecture Overview

```bash
┌─────────────────────────────────────────────┐
│            Client Requests                  │
└──────────────┬──────────────────────────────┘
               │ HTTPS/HTTP
       ┌───────▼─────────┐
       │ DigitalOcean LB │
       └───────┬─────────┘
      ┌────────┼────────┐
      │        │        │
    ┌─▼──┐  ┌─▼──┐  ┌─▼──┐
    │Web │  │Web │  │Web │ (DigitalOcean Droplets)
    │ 1  │  │ 2  │  │ 3  │
    └──┬─┘  └──┬─┘  └──┬─┘
       │       │       │
       └───────┼───────┘
               │ VPN Tunnel
       ┌───────▼────────────┐
       │   AWS RDS (PG)     │ (us-east-1)
       │  Multi-AZ Cluster  │
       └────────┬───────────┘
                │ Replication
         ┌──────▼──────────┐
         │ Hetzner Volume  │ (nbg1 - Germany)
         │    Backups      │
         └─────────────────┘
```

## Prerequisites

### 1. Cloud Accounts

- **DigitalOcean**: Account with API token
- **AWS**: Account with access keys
- **Hetzner**: Account with API token

### 2. Environment Variables

Set these before deployment:

```javascript
export DIGITALOCEAN_TOKEN="dop_v1_abc123def456ghi789jkl012mno"
export AWS_ACCESS_KEY_ID="AKIA1234567890ABCDEF"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG+j/zI0m1234567890ab"
export HCLOUD_TOKEN="MC4wNTI1YmE1M2E4YmE0YTQzMTQyZTdlODYy"
```

### 3. SSH Key Setup

#### DigitalOcean
```bash
# Upload your SSH public key
doctl compute ssh-key create provisioning-key 
  --public-key-from-file ~/.ssh/id_rsa.pub

# Note the key ID for workspace.ncl
doctl compute ssh-key list
```

#### AWS
```bash
# Create EC2 key pair (if needed)
aws ec2 create-key-pair --key-name provisioning-key 
  --query 'KeyMaterial' --output text > provisioning-key.pem
chmod 600 provisioning-key.pem
```

#### Hetzner
```bash
# Upload SSH key
hcloud ssh-key create --name provisioning-key 
  --public-key-from-file ~/.ssh/id_rsa.pub

# List keys
hcloud ssh-key list
```

### 4. DNS Setup

Update `workspace.ncl` with your domain:
- Replace `your-certificate-id` with actual AWS certificate ID
- Update load balancer CNAME to point to your domain

## Deployment

### Step 1: Configure the Workspace

Edit `workspace.ncl` to:
- Set your SSH key IDs
- Update certificate ID for HTTPS
- Set domain names
- Adjust instance counts if needed

Edit `config.toml` to:
- Set correct environment variable names
- Adjust thresholds and settings

### Step 2: Validate Configuration

```toml
# Validate Nickel syntax
nickel export workspace.ncl | jq .

# Validate provider credentials
provisioning provider verify digitalocean
provisioning provider verify aws
provisioning provider verify hetzner
```

### Step 3: Deploy

```bash
# Using provided deploy script
./deploy.nu

# Or manually via provisioning CLI
provisioning workspace deploy --config config.toml
```

### Step 4: Verify Deployment

```bash
# List resources per provider
doctl compute droplet list
aws rds describe-db-instances
hcloud volume list

# Test load balancer
curl http://your-domain.com/health
```

## Post-Deployment Configuration

### 1. Application Deployment

SSH into web servers and deploy application:

```bash
# Get web server IPs
doctl compute droplet list --format Name,PublicIPv4

# SSH to first server
ssh root@198.51.100.15

# Deploy application
cd /var/www
git clone https://github.com/your-org/web-app.git
cd web-app
./deploy.sh
```

### 2. Database Configuration

Connect to RDS database and initialize schema:

```bash
# Get RDS endpoint
aws rds describe-db-instances --query 'DBInstances[0].Endpoint.Address'

# Connect and initialize
psql -h webapp-db.c9akciq32.us-east-1.rds.amazonaws.com -U admin -d defaultdb < schema.sql
```

### 3. DNS Configuration

Point your domain to the load balancer:

```bash
# Get load balancer IP
doctl compute load-balancer list

# Update DNS CNAME
# Add CNAME record: app.example.com -> lb-123456789.nyc3.digitalocean.com
```

### 4. SSL/TLS Certificate

Use AWS Certificate Manager:

```bash
# Request certificate
aws acm request-certificate 
  --domain-name app.example.com 
  --validation-method DNS

# Validate and get certificate ID
aws acm list-certificates | grep app.example.com

# Update workspace.ncl with certificate ID
```

## Monitoring

### DigitalOcean Monitoring

- CPU usage tracked per droplet
- Memory usage alerts on Droplet greater than 85%
- Disk space alerts on greater than 90% full

### AWS CloudWatch

- RDS database metrics (CPU, connections, disk)
- Automatic failover notifications
- Slow query logging

### Hetzner Monitoring

- Volume usage tracking
- Manual monitoring script via cron

### Application Monitoring

Implement application-level monitoring:

```bash
# SSH to web server
ssh root@198.51.100.15

# Check app logs
tail -f /var/www/app/logs/application.log

# Monitor system resources
top
iostat -x 1

# Check database connection pool
psql -h webapp-db.c9akciq32.us-east-1.rds.amazonaws.com -c "SELECT count(plus) FROM pg_stat_activity;"
```

## Backup and Recovery

### Automated Backups

- **RDS**: Daily backups retained for 30 days (AWS handles)
- **Application Data**: Weekly backups to Hetzner volume
- **Configuration**: Version control via Git

### Manual Backup

```bash
# Backup RDS to Hetzner volume
ssh hetzner-backup-volume

# Mount Hetzner volume (if not mounted)
sudo mount /dev/sdb /mnt/backups

# Backup RDS database
pg_dump -h webapp-db.c9akciq32.us-east-1.rds.amazonaws.com -U admin -d defaultdb | 
  gzip > /mnt/backups/db-$(date +%Y%m%d).sql.gz
```

### Recovery Procedure

1. **Web Server Failure**: Load balancer automatically redirects to healthy server
2. **Database Failure**: RDS Multi-AZ automatic failover
3. **Complete Failure**: Restore from Hetzner backup volume

## Scaling

### Add More Web Servers

Edit `workspace.ncl`:

```nickel
droplets = digitalocean.Droplet & {
  name = "web-server",
  region = "nyc3",
  size = "s-2vcpu-4gb",
  count = 5
}
```

Redeploy:

```bash
./deploy.nu
```

### Upgrade Database

Edit `workspace.ncl`:

```nickel
database_tier = aws.RDS & {
  identifier = "webapp-db",
  instance_class = "db.t3.large"
}
```

Redeploy with minimal downtime (Multi-AZ handles switchover).

## Cost Optimization

### Reduce Costs

1. **Droplets**: Use smaller size or fewer instances
2. **Database**: Switch to smaller db.t3.small (approximately $30/month)
3. **Storage**: Reduce backup volume size
4. **Data Transfer**: Monitor and optimize outbound traffic

### Monitor Costs

```bash
# DigitalOcean estimated bill
doctl billing get

# AWS Cost Explorer
aws ce get-cost-and-usage --time-period Start=2024-01-01,End=2024-01-31

# Hetzner manual tracking via console
# Navigate to https://console.hetzner.cloud/billing
```

## Troubleshooting

### Issue: Web Servers Unreachable

**Diagnosis**:
```bash
doctl compute droplet list
doctl compute firewall list-rules firewall-id
```

**Solution**:
- Check firewall allows ports 80, 443
- Verify droplets have public IPs
- Check web server application status

### Issue: Database Connection Failure

**Diagnosis**:
```bash
aws rds describe-db-instances
aws security-group describe-security-groups
```

**Solution**:
- Verify RDS security group allows port 5432 from web servers
- Check RDS status is "available"
- Verify connection string in application

### Issue: Backup Volume Not Mounted

**Diagnosis**:
```bash
hcloud volume list
ssh hetzner-volume
lsblk
```

**Solution**:
```bash
sudo mkfs.ext4 /dev/sdb
sudo mount /dev/sdb /mnt/backups
echo '/dev/sdb /mnt/backups ext4 defaults,nofail 0 0' | sudo tee -a /etc/fstab
```

## Cleanup

To destroy all resources:

```bash
# This will delete everything - use carefully
provisioning workspace destroy --config config.toml

# Or manually
doctl compute droplet delete web-server-1 web-server-2 web-server-3
doctl compute load-balancer delete web-lb
aws rds delete-db-instance --db-instance-identifier webapp-db --skip-final-snapshot
hcloud volume delete webapp-backups
```

## Next Steps

1. **SSL/TLS**: Update certificate and enable HTTPS
2. **Auto-scaling**: Add DigitalOcean autoscaling based on load
3. **Multi-region**: Add additional AWS RDS read replicas in other regions
4. **Disaster Recovery**: Test failover procedures
5. **Cost Optimization**: Review and optimize resource sizes

## Support

For issues or questions:

- Review the multi-provider deployment guide
- Check provider-specific documentation
- Review workspace logs with debug flag: ./deploy.nu --debug

## Files

- `workspace.ncl`: Infrastructure definition (Nickel)
- `config.toml`: Provider credentials and settings
- `deploy.nu`: Deployment automation script (Nushell)
- `README.md`: This file