provisioning/examples/workspaces/cost-optimized/README.md

# Cost-Optimized Multi-Provider Workspace

This workspace demonstrates cost optimization through intelligent provider specialization:

- **Hetzner**: Compute tier (CPX21 servers at €20.90/month) - best price/performance
- **AWS**: Managed services (RDS, ElastiCache, SQS) - reliability without ops overhead
- **DigitalOcean**: CDN and object storage - affordable content delivery

## Why This Architecture?

### Cost Comparison

```bash
Cost-Optimized Architecture:
├── Hetzner compute: €72.70/month (~$78)
├── AWS managed services: $115/month
└── DigitalOcean CDN: $64/month
Total: ~$280/month

All-AWS Equivalent:
├── EC2 instances: ~$200+
├── RDS database: ~$150+
├── ElastiCache: ~$50+
├── CloudFront CDN: ~$100+
└── Other services: ~$50+
Total: ~$600+/month

Savings: ~$320/month (53% reduction)
```

### Architecture Benefits

**Hetzner Advantages**:
- Best price/performance for compute (€20.90/month for 4 vCPU/8GB)
- Powerful Load Balancer (€10/month)
- Fast networking (10Gbps)
- EU data residency (GDPR compliant)

**AWS Advantages**:
- Managed RDS: Automatic backups, failover, patching
- ElastiCache: Redis cluster with automatic failover
- SQS: Scalable message queue (pay per message)
- CloudWatch: Comprehensive monitoring

**DigitalOcean Advantages**:
- CDN: Cost-effective content delivery ($25/month)
- Spaces: Object storage at scale ($15/month)
- Simple pricing and management
- Edge nodes for regional distribution

## Architecture Overview

```bash
┌────────────────────────────────────────────────┐
│         Client Requests                         │
└─────────────────┬────────────────────────────────┘
                  │ HTTPS/HTTP
         ┌────────▼─────────┐
         │  DigitalOcean    │
         │    CDN / Spaces  │
         └────────┬─────────┘
                  │
     ┌────────────┼────────────┐
     │            │            │
┌────▼──────┐ ┌──▼────────┐ ┌─▼──────┐
│  Hetzner  │ │   AWS     │ │   DO   │
│ Compute   │ │ Managed   │ │ CDN    │
│ (Load LB) │ │ Services  │ │        │
└────┬──────┘ └──┬────────┘ └────────┘
     │VPN Tunnel │
┌────▼──────────▼────┐
│  Hetzner Network    │ AWS VPC        DO Spaces
│  10.0.0.0/16 ◄──► 10.1.0.0/16 ◄──► nyc3
│  3x CPX21 Servers  │ RDS + Cache     CDN +
│                    │ + SQS           Backups
└────────────────────┘
```

## Prerequisites

### 1. Cloud Accounts

- **Hetzner**: Account with API token
- **AWS**: Account with access keys
- **DigitalOcean**: Account with API token

### 2. Environment Variables

```javascript
export HCLOUD_TOKEN="MC4wNTI1YmE1M2E4YmE0YTQzMTQyZTdlODYy"
export AWS_ACCESS_KEY_ID="AKIA1234567890ABCDEF"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG+j/zI0m1234567890ab"
export DIGITALOCEAN_TOKEN="dop_v1_abc123def456ghi789jkl012mno"
```

### 3. CLI Tools

```bash
# Install and verify
which hcloud && hcloud version
which aws && aws --version
which doctl && doctl version
which nickel && nickel --version
```

### 4. SSH Keys

```bash
# Hetzner
hcloud ssh-key create --name provisioning-key 
  --public-key-from-file ~/.ssh/id_rsa.pub

# AWS
aws ec2 create-key-pair --key-name provisioning-key 
  --query 'KeyMaterial' --output text > provisioning-key.pem
chmod 600 provisioning-key.pem

# DigitalOcean
doctl compute ssh-key create provisioning-key 
  --public-key-from-file ~/.ssh/id_rsa.pub
```

## Deployment

### Step 1: Configure the Workspace

Edit `workspace.ncl`:

```nickel
# Update networking if needed
compute_tier.primary_servers = hetzner.Server & {
  server_type = "cpx21",
  count = 3,
  location = "nbg1"
}

# Update AWS region if needed
managed_services.database = aws.RDS & {
  instance_class = "db.t3.small",
  region = "us-east-1"
}

# Update CDN endpoints
cdn_tier.cdn.endpoints = [{
  name = "app-cdn",
  origin = "content.example.com"
}]
```

Edit `config.toml`:

```toml
[cost_tracking]
monthly_budget = 300
budget_alert_threshold = 280

[application.cache]
max_memory = "250MB"
```

### Step 2: Validate Configuration

```toml
# Validate Nickel syntax
nickel export workspace.ncl | jq . > /dev/null

# Verify provider access
hcloud context use default
aws sts get-caller-identity
doctl account get
```

### Step 3: Deploy

```bash
chmod +x deploy.nu
./deploy.nu

# Or with debug output
./deploy.nu --debug
```

### Step 4: Verify Deployment

```bash
# Hetzner compute resources
hcloud server list
hcloud load-balancer list

# AWS managed services
aws rds describe-db-instances --region us-east-1
aws elasticache describe-cache-clusters --region us-east-1
aws sqs list-queues --region us-east-1

# DigitalOcean CDN
doctl compute cdn list
doctl compute spaces list
```

## Post-Deployment Configuration

### 1. Connect Hetzner Compute to AWS Database

```bash
# Get Hetzner server IPs
hcloud server list --format ID,PublicIPv4

# Get RDS endpoint
aws rds describe-db-instances --region us-east-1 
  --query 'DBInstances[0].Endpoint.Address'

# On Hetzner server, install PostgreSQL client
ssh root@hetzner-server
apt-get update && apt-get install postgresql-client

# Test connection to RDS
psql -h app-db.abc123.us-east-1.rds.amazonaws.com 
  -U admin -d postgres -c "SELECT now();"
```

### 2. Configure Application for Services

```toml
# Application configuration file
cat > /var/www/app/.env << EOF
DATABASE_HOST=app-db.abc123.us-east-1.rds.amazonaws.com
DATABASE_PORT=5432
DATABASE_USER=admin
DATABASE_PASSWORD=your_password
DATABASE_NAME=app_db

REDIS_HOST=app-cache.abc123.ng.0001.euc1.cache.amazonaws.com
REDIS_PORT=6379

SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789/app-queue

CDN_ENDPOINT=https://content.example.com
SPACES_ENDPOINT=https://app-content.nyc3.digitaloceanspaces.com
SPACES_KEY=your_spaces_key
SPACES_SECRET=your_spaces_secret

ENVIRONMENT=production
EOF
```

### 3. Setup CDN and Object Storage

```bash
# Configure Spaces bucket
doctl compute spaces create app-content --region nyc3

# Get Spaces endpoint
doctl compute spaces list

# Configure CDN endpoint
doctl compute cdn create --origin content.example.com

# Upload test file
aws s3 cp test.html s3://app-content/
```

### 4. Configure Application Queue

```toml
# Get SQS queue URL
aws sqs list-queues --region us-east-1

# Create queue if needed
aws sqs create-queue --queue-name app-queue --region us-east-1

# Test queue
aws sqs send-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789/app-queue 
  --message-body "test message" --region us-east-1
```

### 5. Deploy Application

SSH to Hetzner servers:

```bash
# Get server IPs
SERVERS=$(hcloud server list --format PublicIPv4 --no-header)

# Deploy to each server
for server in $SERVERS; do
  ssh -o StrictHostKeyChecking=no root@$server << 'DEPLOY'
    cd /var/www
    git clone https://github.com/your-org/app.git
    cd app
    cp .env.example .env
    ./deploy.sh
  DEPLOY
done
```

## Monitoring and Cost Control

### Cost Monitoring

```bash
# Hetzner billing
# Manual via console: https://console.hetzner.cloud/billing

# AWS cost tracking
aws ce get-cost-and-usage 
  --time-period Start=2024-01-01,End=2024-01-31 
  --granularity MONTHLY 
  --metrics BlendedCost 
  --group-by Type=DIMENSION,Key=SERVICE

# DigitalOcean billing
doctl billing get

# Real-time cost status
aws ce get-cost-and-usage 
  --time-period Start=$(date -d '1 day ago' +%Y-%m-%d),End=$(date +%Y-%m-%d) 
  --granularity DAILY 
  --metrics BlendedCost
```

### Application Performance Monitoring

```bash
# RDS performance insights
aws pi describe-dimension-keys 
  --service-type RDS 
  --identifier arn:aws:rds:us-east-1:123456789:db:app-db 
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) 
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) 
  --period-in-seconds 60 
  --metric db.load.avg 
  --partition-by Dimension 
  --dimension-group.group-by WAIT_EVENT

# ElastiCache monitoring
aws cloudwatch get-metric-statistics 
  --namespace AWS/ElastiCache 
  --metric-name CPUUtilization 
  --dimensions Name=CacheClusterId,Value=app-cache 
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) 
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) 
  --period 300 
  --statistics Average

# SQS monitoring
aws sqs get-queue-attributes 
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/app-queue 
  --attribute-names All
```

### Alerts Configuration

```toml
# CPU threshold alert
aws cloudwatch put-metric-alarm 
  --alarm-name hetzner-cpu-high 
  --alarm-description "Alert when Hetzner CPU > 80%" 
  --metric-name CPUUtilization 
  --threshold 80 
  --comparison-operator GreaterThanThreshold

# Queue depth alert
aws cloudwatch put-metric-alarm 
  --alarm-name sqs-queue-depth-high 
  --alarm-description "Alert when SQS queue depth > 1000" 
  --metric-name ApproximateNumberOfMessagesVisible 
  --threshold 1000 
  --comparison-operator GreaterThanThreshold

# Cache eviction alert
aws cloudwatch put-metric-alarm 
  --alarm-name elasticache-eviction-rate-high 
  --alarm-description "Alert when cache eviction rate > 10%" 
  --metric-name EvictionRate 
  --namespace AWS/ElastiCache 
  --threshold 10 
  --comparison-operator GreaterThanThreshold
```

## Scaling and Optimization

### Scale Hetzner Compute

Edit `workspace.ncl`:

```nickel
compute_tier.primary_servers = hetzner.Server & {
  count = 5,  # Increase from 3
  server_type = "cpx21"
}
```

Redeploy:

```bash
./deploy.nu
```

### Upgrade Database

```bash
# Modify RDS instance class
aws rds modify-db-instance 
  --db-instance-identifier app-db 
  --db-instance-class db.t3.medium 
  --apply-immediately 
  --region us-east-1
```

### Add Caching Layer

Already configured with ElastiCache. Optimize by adjusting:

```toml
[application.cache]
max_memory = "512MB"
eviction_policy = "allkeys-lru"
```

### Increase Queue Throughput

SQS automatically scales. Monitor with:

```bash
aws sqs get-queue-attributes 
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/app-queue 
  --attribute-names ApproximateNumberOfMessages
```

## Cost Optimization Tips

1. **Hetzner Compute**: CPX21 is sweet spot. Consider CX21 for lower workloads
2. **AWS RDS**: Use t3.small for dev, t3.medium for prod with burst capability
3. **ElastiCache**: 2 nodes with auto-failover. Monitor eviction rates
4. **SQS**: Pay per request, no fixed costs. Good for variable load
5. **DigitalOcean CDN**: Cache more aggressively (86400s TTL for assets)
6. **Spaces**: Use lifecycle rules to delete old files automatically

### Cost Reduction Checklist

- Reduce Hetzner servers from 3 to 2 (saves ~€21/month)
- Downgrade RDS to db.t3.micro for dev (saves ~$40/month)
- Reduce ElastiCache nodes from 2 to 1 (saves ~$12/month)
- Archive old CDN content (savings from Spaces storage)
- Use reserved capacity on AWS (20-30% discount)

Potential total savings: ~$100+/month with right-sizing.

## Troubleshooting

### Issue: Hetzner Can't Connect to RDS

**Diagnosis**:
```bash
# SSH to Hetzner server
ssh root@hetzner-server

# Test connectivity
nc -zv app-db.abc123.us-east-1.rds.amazonaws.com 5432
```

**Solution**:
- Check VPN tunnel is active
- Verify RDS security group allows port 5432 from Hetzner network
- Check route table on both sides

### Issue: High Database Latency

**Diagnosis**:
```bash
# Check RDS performance
aws pi describe-dimension-keys --service-type RDS ...

# Check network latency
ping -c 5 app-db.abc123.us-east-1.rds.amazonaws.com
```

**Solution**:
- Upgrade RDS instance class
- Increase ElastiCache size to reduce database queries
- Check network bandwidth between providers

### Issue: Queue Processing Slow

**Diagnosis**:
```bash
# Check queue depth and age
aws sqs get-queue-attributes 
  --queue-url <queue-url> 
  --attribute-names All
```

**Solution**:
- Scale up application servers processing queue
- Reduce visibility timeout if messages are timing out
- Check application logs for processing errors

## Cleanup

```bash
# Hetzner
hcloud server delete hetzner-app-1 hetzner-app-2 hetzner-app-3
hcloud load-balancer delete app-lb

# AWS
aws rds delete-db-instance --db-instance-identifier app-db --skip-final-snapshot
aws elasticache delete-cache-cluster --cache-cluster-id app-cache
aws sqs delete-queue --queue-url https://sqs.us-east-1.amazonaws.com/123456789/app-queue

# DigitalOcean
doctl compute spaces delete app-content
doctl compute cdn delete cdn-app
doctl compute droplet delete edge-node-1 edge-node-2 edge-node-3
```

## Next Steps

1. Implement application logging to CloudWatch
2. Set up Hetzner monitoring dashboard
3. Configure auto-scaling based on queue depth
4. Implement database read replicas for read-heavy workloads
5. Add WAF protection to Hetzner load balancer
6. Implement cross-region backups to Spaces
7. Set up cost anomaly detection alerts

## Support

For issues or questions:

- Review the cost-optimized deployment guide
- Check provider-specific documentation
- Monitor costs with: `aws ce get-cost-and-usage ...`
- Review deployment logs: `./deploy.nu --debug`

## Files

- `workspace.ncl`: Infrastructure definition (Nickel)
- `config.toml`: Provider credentials and settings
- `deploy.nu`: Deployment orchestration (Nushell)
- `README.md`: This file