# Cost-Optimized Multi-Provider Workspace\n\nThis workspace demonstrates cost optimization through intelligent provider specialization:\n\n- **Hetzner**: Compute tier (CPX21 servers at €20.90/month) - best price/performance\n- **AWS**: Managed services (RDS, ElastiCache, SQS) - reliability without ops overhead\n- **DigitalOcean**: CDN and object storage - affordable content delivery\n\n## Why This Architecture?\n\n### Cost Comparison\n\n```\nCost-Optimized Architecture:\n├── Hetzner compute: €72.70/month (~$78)\n├── AWS managed services: $115/month\n└── DigitalOcean CDN: $64/month\nTotal: ~$280/month\n\nAll-AWS Equivalent:\n├── EC2 instances: ~$200+\n├── RDS database: ~$150+\n├── ElastiCache: ~$50+\n├── CloudFront CDN: ~$100+\n└── Other services: ~$50+\nTotal: ~$600+/month\n\nSavings: ~$320/month (53% reduction)\n```\n\n### Architecture Benefits\n\n**Hetzner Advantages**:\n- Best price/performance for compute (€20.90/month for 4 vCPU/8GB)\n- Powerful Load Balancer (€10/month)\n- Fast networking (10Gbps)\n- EU data residency (GDPR compliant)\n\n**AWS Advantages**:\n- Managed RDS: Automatic backups, failover, patching\n- ElastiCache: Redis cluster with automatic failover\n- SQS: Scalable message queue (pay per message)\n- CloudWatch: Comprehensive monitoring\n\n**DigitalOcean Advantages**:\n- CDN: Cost-effective content delivery ($25/month)\n- Spaces: Object storage at scale ($15/month)\n- Simple pricing and management\n- Edge nodes for regional distribution\n\n## Architecture Overview\n\n```\n┌────────────────────────────────────────────────┐\n│         Client Requests                         │\n└─────────────────┬────────────────────────────────┘\n                  │ HTTPS/HTTP\n         ┌────────▼─────────┐\n         │  DigitalOcean    │\n         │    CDN / Spaces  │\n         └────────┬─────────┘\n                  │\n     ┌────────────┼────────────┐\n     │            │            │\n┌────▼──────┐ ┌──▼────────┐ ┌─▼──────┐\n│  Hetzner  │ │   AWS     │ │   DO   │\n│ Compute   │ │ Managed   │ │ CDN    │\n│ (Load LB) │ │ Services  │ │        │\n└────┬──────┘ └──┬────────┘ └────────┘\n     │VPN Tunnel │\n┌────▼──────────▼────┐\n│  Hetzner Network    │ AWS VPC        DO Spaces\n│  10.0.0.0/16 ◄──► 10.1.0.0/16 ◄──► nyc3\n│  3x CPX21 Servers  │ RDS + Cache     CDN +\n│                    │ + SQS           Backups\n└────────────────────┘\n```\n\n## Prerequisites\n\n### 1. Cloud Accounts\n\n- **Hetzner**: Account with API token\n- **AWS**: Account with access keys\n- **DigitalOcean**: Account with API token\n\n### 2. Environment Variables\n\n```\nexport HCLOUD_TOKEN="MC4wNTI1YmE1M2E4YmE0YTQzMTQyZTdlODYy"\nexport AWS_ACCESS_KEY_ID="AKIA1234567890ABCDEF"\nexport AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG+j/zI0m1234567890ab"\nexport DIGITALOCEAN_TOKEN="dop_v1_abc123def456ghi789jkl012mno"\n```\n\n### 3. CLI Tools\n\n```\n# Install and verify\nwhich hcloud && hcloud version\nwhich aws && aws --version\nwhich doctl && doctl version\nwhich nickel && nickel --version\n```\n\n### 4. SSH Keys\n\n```\n# Hetzner\nhcloud ssh-key create --name provisioning-key \\n  --public-key-from-file ~/.ssh/id_rsa.pub\n\n# AWS\naws ec2 create-key-pair --key-name provisioning-key \\n  --query 'KeyMaterial' --output text > provisioning-key.pem\nchmod 600 provisioning-key.pem\n\n# DigitalOcean\ndoctl compute ssh-key create provisioning-key \\n  --public-key-from-file ~/.ssh/id_rsa.pub\n```\n\n## Deployment\n\n### Step 1: Configure the Workspace\n\nEdit `workspace.ncl`:\n\n```\n# Update networking if needed\ncompute_tier.primary_servers = hetzner.Server & {\n  server_type = "cpx21",\n  count = 3,\n  location = "nbg1"\n}\n\n# Update AWS region if needed\nmanaged_services.database = aws.RDS & {\n  instance_class = "db.t3.small",\n  region = "us-east-1"\n}\n\n# Update CDN endpoints\ncdn_tier.cdn.endpoints = [{\n  name = "app-cdn",\n  origin = "content.example.com"\n}]\n```\n\nEdit `config.toml`:\n\n```\n[cost_tracking]\nmonthly_budget = 300\nbudget_alert_threshold = 280\n\n[application.cache]\nmax_memory = "250MB"\n```\n\n### Step 2: Validate Configuration\n\n```\n# Validate Nickel syntax\nnickel export workspace.ncl | jq . > /dev/null\n\n# Verify provider access\nhcloud context use default\naws sts get-caller-identity\ndoctl account get\n```\n\n### Step 3: Deploy\n\n```\nchmod +x deploy.nu\n./deploy.nu\n\n# Or with debug output\n./deploy.nu --debug\n```\n\n### Step 4: Verify Deployment\n\n```\n# Hetzner compute resources\nhcloud server list\nhcloud load-balancer list\n\n# AWS managed services\naws rds describe-db-instances --region us-east-1\naws elasticache describe-cache-clusters --region us-east-1\naws sqs list-queues --region us-east-1\n\n# DigitalOcean CDN\ndoctl compute cdn list\ndoctl compute spaces list\n```\n\n## Post-Deployment Configuration\n\n### 1. Connect Hetzner Compute to AWS Database\n\n```\n# Get Hetzner server IPs\nhcloud server list --format ID,PublicIPv4\n\n# Get RDS endpoint\naws rds describe-db-instances --region us-east-1 \\n  --query 'DBInstances[0].Endpoint.Address'\n\n# On Hetzner server, install PostgreSQL client\nssh root@hetzner-server\napt-get update && apt-get install postgresql-client\n\n# Test connection to RDS\npsql -h app-db.abc123.us-east-1.rds.amazonaws.com \\n  -U admin -d postgres -c "SELECT now();"\n```\n\n### 2. Configure Application for Services\n\n```\n# Application configuration file\ncat > /var/www/app/.env << EOF\nDATABASE_HOST=app-db.abc123.us-east-1.rds.amazonaws.com\nDATABASE_PORT=5432\nDATABASE_USER=admin\nDATABASE_PASSWORD=your_password\nDATABASE_NAME=app_db\n\nREDIS_HOST=app-cache.abc123.ng.0001.euc1.cache.amazonaws.com\nREDIS_PORT=6379\n\nSQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789/app-queue\n\nCDN_ENDPOINT=https://content.example.com\nSPACES_ENDPOINT=https://app-content.nyc3.digitaloceanspaces.com\nSPACES_KEY=your_spaces_key\nSPACES_SECRET=your_spaces_secret\n\nENVIRONMENT=production\nEOF\n```\n\n### 3. Setup CDN and Object Storage\n\n```\n# Configure Spaces bucket\ndoctl compute spaces create app-content --region nyc3\n\n# Get Spaces endpoint\ndoctl compute spaces list\n\n# Configure CDN endpoint\ndoctl compute cdn create --origin content.example.com\n\n# Upload test file\naws s3 cp test.html s3://app-content/\n```\n\n### 4. Configure Application Queue\n\n```\n# Get SQS queue URL\naws sqs list-queues --region us-east-1\n\n# Create queue if needed\naws sqs create-queue --queue-name app-queue --region us-east-1\n\n# Test queue\naws sqs send-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789/app-queue \\n  --message-body "test message" --region us-east-1\n```\n\n### 5. Deploy Application\n\nSSH to Hetzner servers:\n\n```\n# Get server IPs\nSERVERS=$(hcloud server list --format PublicIPv4 --no-header)\n\n# Deploy to each server\nfor server in $SERVERS; do\n  ssh -o StrictHostKeyChecking=no root@$server << 'DEPLOY'\n    cd /var/www\n    git clone https://github.com/your-org/app.git\n    cd app\n    cp .env.example .env\n    ./deploy.sh\n  DEPLOY\ndone\n```\n\n## Monitoring and Cost Control\n\n### Cost Monitoring\n\n```\n# Hetzner billing\n# Manual via console: https://console.hetzner.cloud/billing\n\n# AWS cost tracking\naws ce get-cost-and-usage \\n  --time-period Start=2024-01-01,End=2024-01-31 \\n  --granularity MONTHLY \\n  --metrics BlendedCost \\n  --group-by Type=DIMENSION,Key=SERVICE\n\n# DigitalOcean billing\ndoctl billing get\n\n# Real-time cost status\naws ce get-cost-and-usage \\n  --time-period Start=$(date -d '1 day ago' +%Y-%m-%d),End=$(date +%Y-%m-%d) \\n  --granularity DAILY \\n  --metrics BlendedCost\n```\n\n### Application Performance Monitoring\n\n```\n# RDS performance insights\naws pi describe-dimension-keys \\n  --service-type RDS \\n  --identifier arn:aws:rds:us-east-1:123456789:db:app-db \\n  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \\n  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \\n  --period-in-seconds 60 \\n  --metric db.load.avg \\n  --partition-by Dimension \\n  --dimension-group.group-by WAIT_EVENT\n\n# ElastiCache monitoring\naws cloudwatch get-metric-statistics \\n  --namespace AWS/ElastiCache \\n  --metric-name CPUUtilization \\n  --dimensions Name=CacheClusterId,Value=app-cache \\n  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \\n  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \\n  --period 300 \\n  --statistics Average\n\n# SQS monitoring\naws sqs get-queue-attributes \\n  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/app-queue \\n  --attribute-names All\n```\n\n### Alerts Configuration\n\n```\n# CPU threshold alert\naws cloudwatch put-metric-alarm \\n  --alarm-name hetzner-cpu-high \\n  --alarm-description "Alert when Hetzner CPU > 80%" \\n  --metric-name CPUUtilization \\n  --threshold 80 \\n  --comparison-operator GreaterThanThreshold\n\n# Queue depth alert\naws cloudwatch put-metric-alarm \\n  --alarm-name sqs-queue-depth-high \\n  --alarm-description "Alert when SQS queue depth > 1000" \\n  --metric-name ApproximateNumberOfMessagesVisible \\n  --threshold 1000 \\n  --comparison-operator GreaterThanThreshold\n\n# Cache eviction alert\naws cloudwatch put-metric-alarm \\n  --alarm-name elasticache-eviction-rate-high \\n  --alarm-description "Alert when cache eviction rate > 10%" \\n  --metric-name EvictionRate \\n  --namespace AWS/ElastiCache \\n  --threshold 10 \\n  --comparison-operator GreaterThanThreshold\n```\n\n## Scaling and Optimization\n\n### Scale Hetzner Compute\n\nEdit `workspace.ncl`:\n\n```\ncompute_tier.primary_servers = hetzner.Server & {\n  count = 5,  # Increase from 3\n  server_type = "cpx21"\n}\n```\n\nRedeploy:\n\n```\n./deploy.nu\n```\n\n### Upgrade Database\n\n```\n# Modify RDS instance class\naws rds modify-db-instance \\n  --db-instance-identifier app-db \\n  --db-instance-class db.t3.medium \\n  --apply-immediately \\n  --region us-east-1\n```\n\n### Add Caching Layer\n\nAlready configured with ElastiCache. Optimize by adjusting:\n\n```\n[application.cache]\nmax_memory = "512MB"\neviction_policy = "allkeys-lru"\n```\n\n### Increase Queue Throughput\n\nSQS automatically scales. Monitor with:\n\n```\naws sqs get-queue-attributes \\n  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/app-queue \\n  --attribute-names ApproximateNumberOfMessages\n```\n\n## Cost Optimization Tips\n\n1. **Hetzner Compute**: CPX21 is sweet spot. Consider CX21 for lower workloads\n2. **AWS RDS**: Use t3.small for dev, t3.medium for prod with burst capability\n3. **ElastiCache**: 2 nodes with auto-failover. Monitor eviction rates\n4. **SQS**: Pay per request, no fixed costs. Good for variable load\n5. **DigitalOcean CDN**: Cache more aggressively (86400s TTL for assets)\n6. **Spaces**: Use lifecycle rules to delete old files automatically\n\n### Cost Reduction Checklist\n\n- Reduce Hetzner servers from 3 to 2 (saves ~€21/month)\n- Downgrade RDS to db.t3.micro for dev (saves ~$40/month)\n- Reduce ElastiCache nodes from 2 to 1 (saves ~$12/month)\n- Archive old CDN content (savings from Spaces storage)\n- Use reserved capacity on AWS (20-30% discount)\n\nPotential total savings: ~$100+/month with right-sizing.\n\n## Troubleshooting\n\n### Issue: Hetzner Can't Connect to RDS\n\n**Diagnosis**:\n```\n# SSH to Hetzner server\nssh root@hetzner-server\n\n# Test connectivity\nnc -zv app-db.abc123.us-east-1.rds.amazonaws.com 5432\n```\n\n**Solution**:\n- Check VPN tunnel is active\n- Verify RDS security group allows port 5432 from Hetzner network\n- Check route table on both sides\n\n### Issue: High Database Latency\n\n**Diagnosis**:\n```\n# Check RDS performance\naws pi describe-dimension-keys --service-type RDS ...\n\n# Check network latency\nping -c 5 app-db.abc123.us-east-1.rds.amazonaws.com\n```\n\n**Solution**:\n- Upgrade RDS instance class\n- Increase ElastiCache size to reduce database queries\n- Check network bandwidth between providers\n\n### Issue: Queue Processing Slow\n\n**Diagnosis**:\n```\n# Check queue depth and age\naws sqs get-queue-attributes \\n  --queue-url <queue-url> \\n  --attribute-names All\n```\n\n**Solution**:\n- Scale up application servers processing queue\n- Reduce visibility timeout if messages are timing out\n- Check application logs for processing errors\n\n## Cleanup\n\n```\n# Hetzner\nhcloud server delete hetzner-app-1 hetzner-app-2 hetzner-app-3\nhcloud load-balancer delete app-lb\n\n# AWS\naws rds delete-db-instance --db-instance-identifier app-db --skip-final-snapshot\naws elasticache delete-cache-cluster --cache-cluster-id app-cache\naws sqs delete-queue --queue-url https://sqs.us-east-1.amazonaws.com/123456789/app-queue\n\n# DigitalOcean\ndoctl compute spaces delete app-content\ndoctl compute cdn delete cdn-app\ndoctl compute droplet delete edge-node-1 edge-node-2 edge-node-3\n```\n\n## Next Steps\n\n1. Implement application logging to CloudWatch\n2. Set up Hetzner monitoring dashboard\n3. Configure auto-scaling based on queue depth\n4. Implement database read replicas for read-heavy workloads\n5. Add WAF protection to Hetzner load balancer\n6. Implement cross-region backups to Spaces\n7. Set up cost anomaly detection alerts\n\n## Support\n\nFor issues or questions:\n\n- Review the cost-optimized deployment guide\n- Check provider-specific documentation\n- Monitor costs with: `aws ce get-cost-and-usage ...`\n- Review deployment logs: `./deploy.nu --debug`\n\n## Files\n\n- `workspace.ncl`: Infrastructure definition (Nickel)\n- `config.toml`: Provider credentials and settings\n- `deploy.nu`: Deployment orchestration (Nushell)\n- `README.md`: This file
