# Multi-Provider Web App Workspace\n\nThis workspace demonstrates a production-ready web application deployment spanning three cloud providers:\n\n- **DigitalOcean**: Web servers and load balancing (NYC region)\n- **AWS**: Managed PostgreSQL database with high availability (US-East region)\n- **Hetzner**: Backup storage and disaster recovery (Germany region)\n\n## Why Three Providers?\n\nThis architecture optimizes cost, performance, and reliability:\n\n- **DigitalOcean** (~$77/month): Cost-effective compute with simple management\n- **AWS RDS** (~$75/month): Managed database with automatic failover\n- **Hetzner** (~$13/month): Affordable backup storage\n- **Total**: ~$165/month (vs $300+ for equivalent all-cloud setup)\n\n## Architecture Overview\n\n```\n┌─────────────────────────────────────────────┐\n│            Client Requests                  │\n└──────────────┬──────────────────────────────┘\n               │ HTTPS/HTTP\n       ┌───────▼─────────┐\n       │ DigitalOcean LB │\n       └───────┬─────────┘\n      ┌────────┼────────┐\n      │        │        │\n    ┌─▼──┐  ┌─▼──┐  ┌─▼──┐\n    │Web │  │Web │  │Web │ (DigitalOcean Droplets)\n    │ 1  │  │ 2  │  │ 3  │\n    └──┬─┘  └──┬─┘  └──┬─┘\n       │       │       │\n       └───────┼───────┘\n               │ VPN Tunnel\n       ┌───────▼────────────┐\n       │   AWS RDS (PG)     │ (us-east-1)\n       │  Multi-AZ Cluster  │\n       └────────┬───────────┘\n                │ Replication\n         ┌──────▼──────────┐\n         │ Hetzner Volume  │ (nbg1 - Germany)\n         │    Backups      │\n         └─────────────────┘\n```\n\n## Prerequisites\n\n### 1. Cloud Accounts\n\n- **DigitalOcean**: Account with API token\n- **AWS**: Account with access keys\n- **Hetzner**: Account with API token\n\n### 2. Environment Variables\n\nSet these before deployment:\n\n```\nexport DIGITALOCEAN_TOKEN="dop_v1_abc123def456ghi789jkl012mno"\nexport AWS_ACCESS_KEY_ID="AKIA1234567890ABCDEF"\nexport AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG+j/zI0m1234567890ab"\nexport HCLOUD_TOKEN="MC4wNTI1YmE1M2E4YmE0YTQzMTQyZTdlODYy"\n```\n\n### 3. SSH Key Setup\n\n#### DigitalOcean\n```\n# Upload your SSH public key\ndoctl compute ssh-key create provisioning-key \\n  --public-key-from-file ~/.ssh/id_rsa.pub\n\n# Note the key ID for workspace.ncl\ndoctl compute ssh-key list\n```\n\n#### AWS\n```\n# Create EC2 key pair (if needed)\naws ec2 create-key-pair --key-name provisioning-key \\n  --query 'KeyMaterial' --output text > provisioning-key.pem\nchmod 600 provisioning-key.pem\n```\n\n#### Hetzner\n```\n# Upload SSH key\nhcloud ssh-key create --name provisioning-key \\n  --public-key-from-file ~/.ssh/id_rsa.pub\n\n# List keys\nhcloud ssh-key list\n```\n\n### 4. DNS Setup\n\nUpdate `workspace.ncl` with your domain:\n- Replace `your-certificate-id` with actual AWS certificate ID\n- Update load balancer CNAME to point to your domain\n\n## Deployment\n\n### Step 1: Configure the Workspace\n\nEdit `workspace.ncl` to:\n- Set your SSH key IDs\n- Update certificate ID for HTTPS\n- Set domain names\n- Adjust instance counts if needed\n\nEdit `config.toml` to:\n- Set correct environment variable names\n- Adjust thresholds and settings\n\n### Step 2: Validate Configuration\n\n```\n# Validate Nickel syntax\nnickel export workspace.ncl | jq .\n\n# Validate provider credentials\nprovisioning provider verify digitalocean\nprovisioning provider verify aws\nprovisioning provider verify hetzner\n```\n\n### Step 3: Deploy\n\n```\n# Using provided deploy script\n./deploy.nu\n\n# Or manually via provisioning CLI\nprovisioning workspace deploy --config config.toml\n```\n\n### Step 4: Verify Deployment\n\n```\n# List resources per provider\ndoctl compute droplet list\naws rds describe-db-instances\nhcloud volume list\n\n# Test load balancer\ncurl http://your-domain.com/health\n```\n\n## Post-Deployment Configuration\n\n### 1. Application Deployment\n\nSSH into web servers and deploy application:\n\n```\n# Get web server IPs\ndoctl compute droplet list --format Name,PublicIPv4\n\n# SSH to first server\nssh root@198.51.100.15\n\n# Deploy application\ncd /var/www\ngit clone https://github.com/your-org/web-app.git\ncd web-app\n./deploy.sh\n```\n\n### 2. Database Configuration\n\nConnect to RDS database and initialize schema:\n\n```\n# Get RDS endpoint\naws rds describe-db-instances --query 'DBInstances[0].Endpoint.Address'\n\n# Connect and initialize\npsql -h webapp-db.c9akciq32.us-east-1.rds.amazonaws.com -U admin -d defaultdb < schema.sql\n```\n\n### 3. DNS Configuration\n\nPoint your domain to the load balancer:\n\n```\n# Get load balancer IP\ndoctl compute load-balancer list\n\n# Update DNS CNAME\n# Add CNAME record: app.example.com -> lb-123456789.nyc3.digitalocean.com\n```\n\n### 4. SSL/TLS Certificate\n\nUse AWS Certificate Manager:\n\n```\n# Request certificate\naws acm request-certificate \\n  --domain-name app.example.com \\n  --validation-method DNS\n\n# Validate and get certificate ID\naws acm list-certificates | grep app.example.com\n\n# Update workspace.ncl with certificate ID\n```\n\n## Monitoring\n\n### DigitalOcean Monitoring\n\n- CPU usage tracked per droplet\n- Memory usage alerts on Droplet greater than 85%\n- Disk space alerts on greater than 90% full\n\n### AWS CloudWatch\n\n- RDS database metrics (CPU, connections, disk)\n- Automatic failover notifications\n- Slow query logging\n\n### Hetzner Monitoring\n\n- Volume usage tracking\n- Manual monitoring script via cron\n\n### Application Monitoring\n\nImplement application-level monitoring:\n\n```\n# SSH to web server\nssh root@198.51.100.15\n\n# Check app logs\ntail -f /var/www/app/logs/application.log\n\n# Monitor system resources\ntop\niostat -x 1\n\n# Check database connection pool\npsql -h webapp-db.c9akciq32.us-east-1.rds.amazonaws.com -c "SELECT count(plus) FROM pg_stat_activity;"\n```\n\n## Backup and Recovery\n\n### Automated Backups\n\n- **RDS**: Daily backups retained for 30 days (AWS handles)\n- **Application Data**: Weekly backups to Hetzner volume\n- **Configuration**: Version control via Git\n\n### Manual Backup\n\n```\n# Backup RDS to Hetzner volume\nssh hetzner-backup-volume\n\n# Mount Hetzner volume (if not mounted)\nsudo mount /dev/sdb /mnt/backups\n\n# Backup RDS database\npg_dump -h webapp-db.c9akciq32.us-east-1.rds.amazonaws.com -U admin -d defaultdb | \\n  gzip > /mnt/backups/db-$(date +%Y%m%d).sql.gz\n```\n\n### Recovery Procedure\n\n1. **Web Server Failure**: Load balancer automatically redirects to healthy server\n2. **Database Failure**: RDS Multi-AZ automatic failover\n3. **Complete Failure**: Restore from Hetzner backup volume\n\n## Scaling\n\n### Add More Web Servers\n\nEdit `workspace.ncl`:\n\n```\ndroplets = digitalocean.Droplet & {\n  name = "web-server",\n  region = "nyc3",\n  size = "s-2vcpu-4gb",\n  count = 5\n}\n```\n\nRedeploy:\n\n```\n./deploy.nu\n```\n\n### Upgrade Database\n\nEdit `workspace.ncl`:\n\n```\ndatabase_tier = aws.RDS & {\n  identifier = "webapp-db",\n  instance_class = "db.t3.large"\n}\n```\n\nRedeploy with minimal downtime (Multi-AZ handles switchover).\n\n## Cost Optimization\n\n### Reduce Costs\n\n1. **Droplets**: Use smaller size or fewer instances\n2. **Database**: Switch to smaller db.t3.small (approximately $30/month)\n3. **Storage**: Reduce backup volume size\n4. **Data Transfer**: Monitor and optimize outbound traffic\n\n### Monitor Costs\n\n```\n# DigitalOcean estimated bill\ndoctl billing get\n\n# AWS Cost Explorer\naws ce get-cost-and-usage --time-period Start=2024-01-01,End=2024-01-31\n\n# Hetzner manual tracking via console\n# Navigate to https://console.hetzner.cloud/billing\n```\n\n## Troubleshooting\n\n### Issue: Web Servers Unreachable\n\n**Diagnosis**:\n```\ndoctl compute droplet list\ndoctl compute firewall list-rules firewall-id\n```\n\n**Solution**:\n- Check firewall allows ports 80, 443\n- Verify droplets have public IPs\n- Check web server application status\n\n### Issue: Database Connection Failure\n\n**Diagnosis**:\n```\naws rds describe-db-instances\naws security-group describe-security-groups\n```\n\n**Solution**:\n- Verify RDS security group allows port 5432 from web servers\n- Check RDS status is "available"\n- Verify connection string in application\n\n### Issue: Backup Volume Not Mounted\n\n**Diagnosis**:\n```\nhcloud volume list\nssh hetzner-volume\nlsblk\n```\n\n**Solution**:\n```\nsudo mkfs.ext4 /dev/sdb\nsudo mount /dev/sdb /mnt/backups\necho '/dev/sdb /mnt/backups ext4 defaults,nofail 0 0' | sudo tee -a /etc/fstab\n```\n\n## Cleanup\n\nTo destroy all resources:\n\n```\n# This will delete everything - use carefully\nprovisioning workspace destroy --config config.toml\n\n# Or manually\ndoctl compute droplet delete web-server-1 web-server-2 web-server-3\ndoctl compute load-balancer delete web-lb\naws rds delete-db-instance --db-instance-identifier webapp-db --skip-final-snapshot\nhcloud volume delete webapp-backups\n```\n\n## Next Steps\n\n1. **SSL/TLS**: Update certificate and enable HTTPS\n2. **Auto-scaling**: Add DigitalOcean autoscaling based on load\n3. **Multi-region**: Add additional AWS RDS read replicas in other regions\n4. **Disaster Recovery**: Test failover procedures\n5. **Cost Optimization**: Review and optimize resource sizes\n\n## Support\n\nFor issues or questions:\n\n- Review the multi-provider deployment guide\n- Check provider-specific documentation\n- Review workspace logs with debug flag: ./deploy.nu --debug\n\n## Files\n\n- `workspace.ncl`: Infrastructure definition (Nickel)\n- `config.toml`: Provider credentials and settings\n- `deploy.nu`: Deployment automation script (Nushell)\n- `README.md`: This file
