Multi-Provider Web App Workspace\n\nThis workspace demonstrates a production-ready web application deployment spanning three cloud providers:\n\n- DigitalOcean: Web servers and load balancing (NYC region)\n- AWS: Managed PostgreSQL database with high availability (US-East region)\n- Hetzner: Backup storage and disaster recovery (Germany region)\n\n## Why Three Providers?\n\nThis architecture optimizes cost, performance, and reliability:\n\n- DigitalOcean ($77/month): Cost-effective compute with simple management\n- AWS RDS ($75/month): Managed database with automatic failover\n- Hetzner (~$13/month): Affordable backup storage\n- Total: ~$165/month (vs $300+ for equivalent all-cloud setup)\n\n## Architecture Overview\n\n\n┌─────────────────────────────────────────────┐\n│ Client Requests │\n└──────────────┬──────────────────────────────┘\n │ HTTPS/HTTP\n ┌───────▼─────────┐\n │ DigitalOcean LB │\n └───────┬─────────┘\n ┌────────┼────────┐\n │ │ │\n ┌─▼──┐ ┌─▼──┐ ┌─▼──┐\n │Web │ │Web │ │Web │ (DigitalOcean Droplets)\n │ 1 │ │ 2 │ │ 3 │\n └──┬─┘ └──┬─┘ └──┬─┘\n │ │ │\n └───────┼───────┘\n │ VPN Tunnel\n ┌───────▼────────────┐\n │ AWS RDS (PG) │ (us-east-1)\n │ Multi-AZ Cluster │\n └────────┬───────────┘\n │ Replication\n ┌──────▼──────────┐\n │ Hetzner Volume │ (nbg1 - Germany)\n │ Backups │\n └─────────────────┘\n\n\n## Prerequisites\n\n### 1. Cloud Accounts\n\n- DigitalOcean: Account with API token\n- AWS: Account with access keys\n- Hetzner: Account with API token\n\n### 2. Environment Variables\n\nSet these before deployment:\n\n\nexport DIGITALOCEAN_TOKEN="dop_v1_abc123def456ghi789jkl012mno"\nexport AWS_ACCESS_KEY_ID="AKIA1234567890ABCDEF"\nexport AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG+j/zI0m1234567890ab"\nexport HCLOUD_TOKEN="MC4wNTI1YmE1M2E4YmE0YTQzMTQyZTdlODYy"\n\n\n### 3. SSH Key Setup\n\n#### DigitalOcean\n\n# Upload your SSH public key\ndoctl compute ssh-key create provisioning-key \\n --public-key-from-file ~/.ssh/id_rsa.pub\n\n# Note the key ID for workspace.ncl\ndoctl compute ssh-key list\n\n\n#### AWS\n\n# Create EC2 key pair (if needed)\naws ec2 create-key-pair --key-name provisioning-key \\n --query 'KeyMaterial' --output text > provisioning-key.pem\nchmod 600 provisioning-key.pem\n\n\n#### Hetzner\n\n# Upload SSH key\nhcloud ssh-key create --name provisioning-key \\n --public-key-from-file ~/.ssh/id_rsa.pub\n\n# List keys\nhcloud ssh-key list\n\n\n### 4. DNS Setup\n\nUpdate workspace.ncl with your domain:\n- Replace your-certificate-id with actual AWS certificate ID\n- Update load balancer CNAME to point to your domain\n\n## Deployment\n\n### Step 1: Configure the Workspace\n\nEdit workspace.ncl to:\n- Set your SSH key IDs\n- Update certificate ID for HTTPS\n- Set domain names\n- Adjust instance counts if needed\n\nEdit config.toml to:\n- Set correct environment variable names\n- Adjust thresholds and settings\n\n### Step 2: Validate Configuration\n\n\n# Validate Nickel syntax\nnickel export workspace.ncl | jq .\n\n# Validate provider credentials\nprovisioning provider verify digitalocean\nprovisioning provider verify aws\nprovisioning provider verify hetzner\n\n\n### Step 3: Deploy\n\n\n# Using provided deploy script\n./deploy.nu\n\n# Or manually via provisioning CLI\nprovisioning workspace deploy --config config.toml\n\n\n### Step 4: Verify Deployment\n\n\n# List resources per provider\ndoctl compute droplet list\naws rds describe-db-instances\nhcloud volume list\n\n# Test load balancer\ncurl http://your-domain.com/health\n\n\n## Post-Deployment Configuration\n\n### 1. Application Deployment\n\nSSH into web servers and deploy application:\n\n\n# Get web server IPs\ndoctl compute droplet list --format Name,PublicIPv4\n\n# SSH to first server\nssh root@198.51.100.15\n\n# Deploy application\ncd /var/www\ngit clone https://github.com/your-org/web-app.git\ncd web-app\n./deploy.sh\n\n\n### 2. Database Configuration\n\nConnect to RDS database and initialize schema:\n\n\n# Get RDS endpoint\naws rds describe-db-instances --query 'DBInstances[0].Endpoint.Address'\n\n# Connect and initialize\npsql -h webapp-db.c9akciq32.us-east-1.rds.amazonaws.com -U admin -d defaultdb < schema.sql\n\n\n### 3. DNS Configuration\n\nPoint your domain to the load balancer:\n\n\n# Get load balancer IP\ndoctl compute load-balancer list\n\n# Update DNS CNAME\n# Add CNAME record: app.example.com -> lb-123456789.nyc3.digitalocean.com\n\n\n### 4. SSL/TLS Certificate\n\nUse AWS Certificate Manager:\n\n\n# Request certificate\naws acm request-certificate \\n --domain-name app.example.com \\n --validation-method DNS\n\n# Validate and get certificate ID\naws acm list-certificates | grep app.example.com\n\n# Update workspace.ncl with certificate ID\n\n\n## Monitoring\n\n### DigitalOcean Monitoring\n\n- CPU usage tracked per droplet\n- Memory usage alerts on Droplet greater than 85%\n- Disk space alerts on greater than 90% full\n\n### AWS CloudWatch\n\n- RDS database metrics (CPU, connections, disk)\n- Automatic failover notifications\n- Slow query logging\n\n### Hetzner Monitoring\n\n- Volume usage tracking\n- Manual monitoring script via cron\n\n### Application Monitoring\n\nImplement application-level monitoring:\n\n\n# SSH to web server\nssh root@198.51.100.15\n\n# Check app logs\ntail -f /var/www/app/logs/application.log\n\n# Monitor system resources\ntop\niostat -x 1\n\n# Check database connection pool\npsql -h webapp-db.c9akciq32.us-east-1.rds.amazonaws.com -c "SELECT count(plus) FROM pg_stat_activity;"\n\n\n## Backup and Recovery\n\n### Automated Backups\n\n- RDS: Daily backups retained for 30 days (AWS handles)\n- Application Data: Weekly backups to Hetzner volume\n- Configuration: Version control via Git\n\n### Manual Backup\n\n\n# Backup RDS to Hetzner volume\nssh hetzner-backup-volume\n\n# Mount Hetzner volume (if not mounted)\nsudo mount /dev/sdb /mnt/backups\n\n# Backup RDS database\npg_dump -h webapp-db.c9akciq32.us-east-1.rds.amazonaws.com -U admin -d defaultdb | \\n gzip > /mnt/backups/db-$(date +%Y%m%d).sql.gz\n\n\n### Recovery Procedure\n\n1. Web Server Failure: Load balancer automatically redirects to healthy server\n2. Database Failure: RDS Multi-AZ automatic failover\n3. Complete Failure: Restore from Hetzner backup volume\n\n## Scaling\n\n### Add More Web Servers\n\nEdit workspace.ncl:\n\n\ndroplets = digitalocean.Droplet & {\n name = "web-server",\n region = "nyc3",\n size = "s-2vcpu-4gb",\n count = 5\n}\n\n\nRedeploy:\n\n\n./deploy.nu\n\n\n### Upgrade Database\n\nEdit workspace.ncl:\n\n\ndatabase_tier = aws.RDS & {\n identifier = "webapp-db",\n instance_class = "db.t3.large"\n}\n\n\nRedeploy with minimal downtime (Multi-AZ handles switchover).\n\n## Cost Optimization\n\n### Reduce Costs\n\n1. Droplets: Use smaller size or fewer instances\n2. Database: Switch to smaller db.t3.small (approximately $30/month)\n3. Storage: Reduce backup volume size\n4. Data Transfer: Monitor and optimize outbound traffic\n\n### Monitor Costs\n\n\n# DigitalOcean estimated bill\ndoctl billing get\n\n# AWS Cost Explorer\naws ce get-cost-and-usage --time-period Start=2024-01-01,End=2024-01-31\n\n# Hetzner manual tracking via console\n# Navigate to https://console.hetzner.cloud/billing\n\n\n## Troubleshooting\n\n### Issue: Web Servers Unreachable\n\nDiagnosis:\n\ndoctl compute droplet list\ndoctl compute firewall list-rules firewall-id\n\n\nSolution:\n- Check firewall allows ports 80, 443\n- Verify droplets have public IPs\n- Check web server application status\n\n### Issue: Database Connection Failure\n\nDiagnosis:\n\naws rds describe-db-instances\naws security-group describe-security-groups\n\n\nSolution:\n- Verify RDS security group allows port 5432 from web servers\n- Check RDS status is "available"\n- Verify connection string in application\n\n### Issue: Backup Volume Not Mounted\n\nDiagnosis:\n\nhcloud volume list\nssh hetzner-volume\nlsblk\n\n\nSolution:\n\nsudo mkfs.ext4 /dev/sdb\nsudo mount /dev/sdb /mnt/backups\necho '/dev/sdb /mnt/backups ext4 defaults,nofail 0 0' | sudo tee -a /etc/fstab\n\n\n## Cleanup\n\nTo destroy all resources:\n\n\n# This will delete everything - use carefully\nprovisioning workspace destroy --config config.toml\n\n# Or manually\ndoctl compute droplet delete web-server-1 web-server-2 web-server-3\ndoctl compute load-balancer delete web-lb\naws rds delete-db-instance --db-instance-identifier webapp-db --skip-final-snapshot\nhcloud volume delete webapp-backups\n\n\n## Next Steps\n\n1. SSL/TLS: Update certificate and enable HTTPS\n2. Auto-scaling: Add DigitalOcean autoscaling based on load\n3. Multi-region: Add additional AWS RDS read replicas in other regions\n4. Disaster Recovery: Test failover procedures\n5. Cost Optimization: Review and optimize resource sizes\n\n## Support\n\nFor issues or questions:\n\n- Review the multi-provider deployment guide\n- Check provider-specific documentation\n- Review workspace logs with debug flag: ./deploy.nu --debug\n\n## Files\n\n- workspace.ncl: Infrastructure definition (Nickel)\n- config.toml: Provider credentials and settings\n- deploy.nu: Deployment automation script (Nushell)\n- README.md: This file
