540 lines
14 KiB
Markdown
540 lines
14 KiB
Markdown
# Cost-Optimized Multi-Provider Workspace
|
|
|
|
This workspace demonstrates cost optimization through intelligent provider specialization:
|
|
|
|
- **Hetzner**: Compute tier (CPX21 servers at €20.90/month) - best price/performance
|
|
- **AWS**: Managed services (RDS, ElastiCache, SQS) - reliability without ops overhead
|
|
- **DigitalOcean**: CDN and object storage - affordable content delivery
|
|
|
|
## Why This Architecture?
|
|
|
|
### Cost Comparison
|
|
|
|
```bash
|
|
Cost-Optimized Architecture:
|
|
├── Hetzner compute: €72.70/month (~$78)
|
|
├── AWS managed services: $115/month
|
|
└── DigitalOcean CDN: $64/month
|
|
Total: ~$280/month
|
|
|
|
All-AWS Equivalent:
|
|
├── EC2 instances: ~$200+
|
|
├── RDS database: ~$150+
|
|
├── ElastiCache: ~$50+
|
|
├── CloudFront CDN: ~$100+
|
|
└── Other services: ~$50+
|
|
Total: ~$600+/month
|
|
|
|
Savings: ~$320/month (53% reduction)
|
|
```
|
|
|
|
### Architecture Benefits
|
|
|
|
**Hetzner Advantages**:
|
|
- Best price/performance for compute (€20.90/month for 4 vCPU/8GB)
|
|
- Powerful Load Balancer (€10/month)
|
|
- Fast networking (10Gbps)
|
|
- EU data residency (GDPR compliant)
|
|
|
|
**AWS Advantages**:
|
|
- Managed RDS: Automatic backups, failover, patching
|
|
- ElastiCache: Redis cluster with automatic failover
|
|
- SQS: Scalable message queue (pay per message)
|
|
- CloudWatch: Comprehensive monitoring
|
|
|
|
**DigitalOcean Advantages**:
|
|
- CDN: Cost-effective content delivery ($25/month)
|
|
- Spaces: Object storage at scale ($15/month)
|
|
- Simple pricing and management
|
|
- Edge nodes for regional distribution
|
|
|
|
## Architecture Overview
|
|
|
|
```bash
|
|
┌────────────────────────────────────────────────┐
|
|
│ Client Requests │
|
|
└─────────────────┬────────────────────────────────┘
|
|
│ HTTPS/HTTP
|
|
┌────────▼─────────┐
|
|
│ DigitalOcean │
|
|
│ CDN / Spaces │
|
|
└────────┬─────────┘
|
|
│
|
|
┌────────────┼────────────┐
|
|
│ │ │
|
|
┌────▼──────┐ ┌──▼────────┐ ┌─▼──────┐
|
|
│ Hetzner │ │ AWS │ │ DO │
|
|
│ Compute │ │ Managed │ │ CDN │
|
|
│ (Load LB) │ │ Services │ │ │
|
|
└────┬──────┘ └──┬────────┘ └────────┘
|
|
│VPN Tunnel │
|
|
┌────▼──────────▼────┐
|
|
│ Hetzner Network │ AWS VPC DO Spaces
|
|
│ 10.0.0.0/16 ◄──► 10.1.0.0/16 ◄──► nyc3
|
|
│ 3x CPX21 Servers │ RDS + Cache CDN +
|
|
│ │ + SQS Backups
|
|
└────────────────────┘
|
|
```
|
|
|
|
## Prerequisites
|
|
|
|
### 1. Cloud Accounts
|
|
|
|
- **Hetzner**: Account with API token
|
|
- **AWS**: Account with access keys
|
|
- **DigitalOcean**: Account with API token
|
|
|
|
### 2. Environment Variables
|
|
|
|
```javascript
|
|
export HCLOUD_TOKEN="MC4wNTI1YmE1M2E4YmE0YTQzMTQyZTdlODYy"
|
|
export AWS_ACCESS_KEY_ID="AKIA1234567890ABCDEF"
|
|
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG+j/zI0m1234567890ab"
|
|
export DIGITALOCEAN_TOKEN="dop_v1_abc123def456ghi789jkl012mno"
|
|
```
|
|
|
|
### 3. CLI Tools
|
|
|
|
```bash
|
|
# Install and verify
|
|
which hcloud && hcloud version
|
|
which aws && aws --version
|
|
which doctl && doctl version
|
|
which nickel && nickel --version
|
|
```
|
|
|
|
### 4. SSH Keys
|
|
|
|
```bash
|
|
# Hetzner
|
|
hcloud ssh-key create --name provisioning-key
|
|
--public-key-from-file ~/.ssh/id_rsa.pub
|
|
|
|
# AWS
|
|
aws ec2 create-key-pair --key-name provisioning-key
|
|
--query 'KeyMaterial' --output text > provisioning-key.pem
|
|
chmod 600 provisioning-key.pem
|
|
|
|
# DigitalOcean
|
|
doctl compute ssh-key create provisioning-key
|
|
--public-key-from-file ~/.ssh/id_rsa.pub
|
|
```
|
|
|
|
## Deployment
|
|
|
|
### Step 1: Configure the Workspace
|
|
|
|
Edit `workspace.ncl`:
|
|
|
|
```nickel
|
|
# Update networking if needed
|
|
compute_tier.primary_servers = hetzner.Server & {
|
|
server_type = "cpx21",
|
|
count = 3,
|
|
location = "nbg1"
|
|
}
|
|
|
|
# Update AWS region if needed
|
|
managed_services.database = aws.RDS & {
|
|
instance_class = "db.t3.small",
|
|
region = "us-east-1"
|
|
}
|
|
|
|
# Update CDN endpoints
|
|
cdn_tier.cdn.endpoints = [{
|
|
name = "app-cdn",
|
|
origin = "content.example.com"
|
|
}]
|
|
```
|
|
|
|
Edit `config.toml`:
|
|
|
|
```toml
|
|
[cost_tracking]
|
|
monthly_budget = 300
|
|
budget_alert_threshold = 280
|
|
|
|
[application.cache]
|
|
max_memory = "250MB"
|
|
```
|
|
|
|
### Step 2: Validate Configuration
|
|
|
|
```toml
|
|
# Validate Nickel syntax
|
|
nickel export workspace.ncl | jq . > /dev/null
|
|
|
|
# Verify provider access
|
|
hcloud context use default
|
|
aws sts get-caller-identity
|
|
doctl account get
|
|
```
|
|
|
|
### Step 3: Deploy
|
|
|
|
```bash
|
|
chmod +x deploy.nu
|
|
./deploy.nu
|
|
|
|
# Or with debug output
|
|
./deploy.nu --debug
|
|
```
|
|
|
|
### Step 4: Verify Deployment
|
|
|
|
```bash
|
|
# Hetzner compute resources
|
|
hcloud server list
|
|
hcloud load-balancer list
|
|
|
|
# AWS managed services
|
|
aws rds describe-db-instances --region us-east-1
|
|
aws elasticache describe-cache-clusters --region us-east-1
|
|
aws sqs list-queues --region us-east-1
|
|
|
|
# DigitalOcean CDN
|
|
doctl compute cdn list
|
|
doctl compute spaces list
|
|
```
|
|
|
|
## Post-Deployment Configuration
|
|
|
|
### 1. Connect Hetzner Compute to AWS Database
|
|
|
|
```bash
|
|
# Get Hetzner server IPs
|
|
hcloud server list --format ID,PublicIPv4
|
|
|
|
# Get RDS endpoint
|
|
aws rds describe-db-instances --region us-east-1
|
|
--query 'DBInstances[0].Endpoint.Address'
|
|
|
|
# On Hetzner server, install PostgreSQL client
|
|
ssh root@hetzner-server
|
|
apt-get update && apt-get install postgresql-client
|
|
|
|
# Test connection to RDS
|
|
psql -h app-db.abc123.us-east-1.rds.amazonaws.com
|
|
-U admin -d postgres -c "SELECT now();"
|
|
```
|
|
|
|
### 2. Configure Application for Services
|
|
|
|
```toml
|
|
# Application configuration file
|
|
cat > /var/www/app/.env << EOF
|
|
DATABASE_HOST=app-db.abc123.us-east-1.rds.amazonaws.com
|
|
DATABASE_PORT=5432
|
|
DATABASE_USER=admin
|
|
DATABASE_PASSWORD=your_password
|
|
DATABASE_NAME=app_db
|
|
|
|
REDIS_HOST=app-cache.abc123.ng.0001.euc1.cache.amazonaws.com
|
|
REDIS_PORT=6379
|
|
|
|
SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789/app-queue
|
|
|
|
CDN_ENDPOINT=https://content.example.com
|
|
SPACES_ENDPOINT=https://app-content.nyc3.digitaloceanspaces.com
|
|
SPACES_KEY=your_spaces_key
|
|
SPACES_SECRET=your_spaces_secret
|
|
|
|
ENVIRONMENT=production
|
|
EOF
|
|
```
|
|
|
|
### 3. Setup CDN and Object Storage
|
|
|
|
```bash
|
|
# Configure Spaces bucket
|
|
doctl compute spaces create app-content --region nyc3
|
|
|
|
# Get Spaces endpoint
|
|
doctl compute spaces list
|
|
|
|
# Configure CDN endpoint
|
|
doctl compute cdn create --origin content.example.com
|
|
|
|
# Upload test file
|
|
aws s3 cp test.html s3://app-content/
|
|
```
|
|
|
|
### 4. Configure Application Queue
|
|
|
|
```toml
|
|
# Get SQS queue URL
|
|
aws sqs list-queues --region us-east-1
|
|
|
|
# Create queue if needed
|
|
aws sqs create-queue --queue-name app-queue --region us-east-1
|
|
|
|
# Test queue
|
|
aws sqs send-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789/app-queue
|
|
--message-body "test message" --region us-east-1
|
|
```
|
|
|
|
### 5. Deploy Application
|
|
|
|
SSH to Hetzner servers:
|
|
|
|
```bash
|
|
# Get server IPs
|
|
SERVERS=$(hcloud server list --format PublicIPv4 --no-header)
|
|
|
|
# Deploy to each server
|
|
for server in $SERVERS; do
|
|
ssh -o StrictHostKeyChecking=no root@$server << 'DEPLOY'
|
|
cd /var/www
|
|
git clone https://github.com/your-org/app.git
|
|
cd app
|
|
cp .env.example .env
|
|
./deploy.sh
|
|
DEPLOY
|
|
done
|
|
```
|
|
|
|
## Monitoring and Cost Control
|
|
|
|
### Cost Monitoring
|
|
|
|
```bash
|
|
# Hetzner billing
|
|
# Manual via console: https://console.hetzner.cloud/billing
|
|
|
|
# AWS cost tracking
|
|
aws ce get-cost-and-usage
|
|
--time-period Start=2024-01-01,End=2024-01-31
|
|
--granularity MONTHLY
|
|
--metrics BlendedCost
|
|
--group-by Type=DIMENSION,Key=SERVICE
|
|
|
|
# DigitalOcean billing
|
|
doctl billing get
|
|
|
|
# Real-time cost status
|
|
aws ce get-cost-and-usage
|
|
--time-period Start=$(date -d '1 day ago' +%Y-%m-%d),End=$(date +%Y-%m-%d)
|
|
--granularity DAILY
|
|
--metrics BlendedCost
|
|
```
|
|
|
|
### Application Performance Monitoring
|
|
|
|
```bash
|
|
# RDS performance insights
|
|
aws pi describe-dimension-keys
|
|
--service-type RDS
|
|
--identifier arn:aws:rds:us-east-1:123456789:db:app-db
|
|
--start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S)
|
|
--end-time $(date -u +%Y-%m-%dT%H:%M:%S)
|
|
--period-in-seconds 60
|
|
--metric db.load.avg
|
|
--partition-by Dimension
|
|
--dimension-group.group-by WAIT_EVENT
|
|
|
|
# ElastiCache monitoring
|
|
aws cloudwatch get-metric-statistics
|
|
--namespace AWS/ElastiCache
|
|
--metric-name CPUUtilization
|
|
--dimensions Name=CacheClusterId,Value=app-cache
|
|
--start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S)
|
|
--end-time $(date -u +%Y-%m-%dT%H:%M:%S)
|
|
--period 300
|
|
--statistics Average
|
|
|
|
# SQS monitoring
|
|
aws sqs get-queue-attributes
|
|
--queue-url https://sqs.us-east-1.amazonaws.com/123456789/app-queue
|
|
--attribute-names All
|
|
```
|
|
|
|
### Alerts Configuration
|
|
|
|
```toml
|
|
# CPU threshold alert
|
|
aws cloudwatch put-metric-alarm
|
|
--alarm-name hetzner-cpu-high
|
|
--alarm-description "Alert when Hetzner CPU > 80%"
|
|
--metric-name CPUUtilization
|
|
--threshold 80
|
|
--comparison-operator GreaterThanThreshold
|
|
|
|
# Queue depth alert
|
|
aws cloudwatch put-metric-alarm
|
|
--alarm-name sqs-queue-depth-high
|
|
--alarm-description "Alert when SQS queue depth > 1000"
|
|
--metric-name ApproximateNumberOfMessagesVisible
|
|
--threshold 1000
|
|
--comparison-operator GreaterThanThreshold
|
|
|
|
# Cache eviction alert
|
|
aws cloudwatch put-metric-alarm
|
|
--alarm-name elasticache-eviction-rate-high
|
|
--alarm-description "Alert when cache eviction rate > 10%"
|
|
--metric-name EvictionRate
|
|
--namespace AWS/ElastiCache
|
|
--threshold 10
|
|
--comparison-operator GreaterThanThreshold
|
|
```
|
|
|
|
## Scaling and Optimization
|
|
|
|
### Scale Hetzner Compute
|
|
|
|
Edit `workspace.ncl`:
|
|
|
|
```nickel
|
|
compute_tier.primary_servers = hetzner.Server & {
|
|
count = 5, # Increase from 3
|
|
server_type = "cpx21"
|
|
}
|
|
```
|
|
|
|
Redeploy:
|
|
|
|
```bash
|
|
./deploy.nu
|
|
```
|
|
|
|
### Upgrade Database
|
|
|
|
```bash
|
|
# Modify RDS instance class
|
|
aws rds modify-db-instance
|
|
--db-instance-identifier app-db
|
|
--db-instance-class db.t3.medium
|
|
--apply-immediately
|
|
--region us-east-1
|
|
```
|
|
|
|
### Add Caching Layer
|
|
|
|
Already configured with ElastiCache. Optimize by adjusting:
|
|
|
|
```toml
|
|
[application.cache]
|
|
max_memory = "512MB"
|
|
eviction_policy = "allkeys-lru"
|
|
```
|
|
|
|
### Increase Queue Throughput
|
|
|
|
SQS automatically scales. Monitor with:
|
|
|
|
```bash
|
|
aws sqs get-queue-attributes
|
|
--queue-url https://sqs.us-east-1.amazonaws.com/123456789/app-queue
|
|
--attribute-names ApproximateNumberOfMessages
|
|
```
|
|
|
|
## Cost Optimization Tips
|
|
|
|
1. **Hetzner Compute**: CPX21 is sweet spot. Consider CX21 for lower workloads
|
|
2. **AWS RDS**: Use t3.small for dev, t3.medium for prod with burst capability
|
|
3. **ElastiCache**: 2 nodes with auto-failover. Monitor eviction rates
|
|
4. **SQS**: Pay per request, no fixed costs. Good for variable load
|
|
5. **DigitalOcean CDN**: Cache more aggressively (86400s TTL for assets)
|
|
6. **Spaces**: Use lifecycle rules to delete old files automatically
|
|
|
|
### Cost Reduction Checklist
|
|
|
|
- Reduce Hetzner servers from 3 to 2 (saves ~€21/month)
|
|
- Downgrade RDS to db.t3.micro for dev (saves ~$40/month)
|
|
- Reduce ElastiCache nodes from 2 to 1 (saves ~$12/month)
|
|
- Archive old CDN content (savings from Spaces storage)
|
|
- Use reserved capacity on AWS (20-30% discount)
|
|
|
|
Potential total savings: ~$100+/month with right-sizing.
|
|
|
|
## Troubleshooting
|
|
|
|
### Issue: Hetzner Can't Connect to RDS
|
|
|
|
**Diagnosis**:
|
|
```bash
|
|
# SSH to Hetzner server
|
|
ssh root@hetzner-server
|
|
|
|
# Test connectivity
|
|
nc -zv app-db.abc123.us-east-1.rds.amazonaws.com 5432
|
|
```
|
|
|
|
**Solution**:
|
|
- Check VPN tunnel is active
|
|
- Verify RDS security group allows port 5432 from Hetzner network
|
|
- Check route table on both sides
|
|
|
|
### Issue: High Database Latency
|
|
|
|
**Diagnosis**:
|
|
```bash
|
|
# Check RDS performance
|
|
aws pi describe-dimension-keys --service-type RDS ...
|
|
|
|
# Check network latency
|
|
ping -c 5 app-db.abc123.us-east-1.rds.amazonaws.com
|
|
```
|
|
|
|
**Solution**:
|
|
- Upgrade RDS instance class
|
|
- Increase ElastiCache size to reduce database queries
|
|
- Check network bandwidth between providers
|
|
|
|
### Issue: Queue Processing Slow
|
|
|
|
**Diagnosis**:
|
|
```bash
|
|
# Check queue depth and age
|
|
aws sqs get-queue-attributes
|
|
--queue-url <queue-url>
|
|
--attribute-names All
|
|
```
|
|
|
|
**Solution**:
|
|
- Scale up application servers processing queue
|
|
- Reduce visibility timeout if messages are timing out
|
|
- Check application logs for processing errors
|
|
|
|
## Cleanup
|
|
|
|
```bash
|
|
# Hetzner
|
|
hcloud server delete hetzner-app-1 hetzner-app-2 hetzner-app-3
|
|
hcloud load-balancer delete app-lb
|
|
|
|
# AWS
|
|
aws rds delete-db-instance --db-instance-identifier app-db --skip-final-snapshot
|
|
aws elasticache delete-cache-cluster --cache-cluster-id app-cache
|
|
aws sqs delete-queue --queue-url https://sqs.us-east-1.amazonaws.com/123456789/app-queue
|
|
|
|
# DigitalOcean
|
|
doctl compute spaces delete app-content
|
|
doctl compute cdn delete cdn-app
|
|
doctl compute droplet delete edge-node-1 edge-node-2 edge-node-3
|
|
```
|
|
|
|
## Next Steps
|
|
|
|
1. Implement application logging to CloudWatch
|
|
2. Set up Hetzner monitoring dashboard
|
|
3. Configure auto-scaling based on queue depth
|
|
4. Implement database read replicas for read-heavy workloads
|
|
5. Add WAF protection to Hetzner load balancer
|
|
6. Implement cross-region backups to Spaces
|
|
7. Set up cost anomaly detection alerts
|
|
|
|
## Support
|
|
|
|
For issues or questions:
|
|
|
|
- Review the cost-optimized deployment guide
|
|
- Check provider-specific documentation
|
|
- Monitor costs with: `aws ce get-cost-and-usage ...`
|
|
- Review deployment logs: `./deploy.nu --debug`
|
|
|
|
## Files
|
|
|
|
- `workspace.ncl`: Infrastructure definition (Nickel)
|
|
- `config.toml`: Provider credentials and settings
|
|
- `deploy.nu`: Deployment orchestration (Nushell)
|
|
- `README.md`: This file |