# Hetzner Provider Guide

This guide covers using Hetzner Cloud as a provider in the provisioning system. Hetzner is renowned for competitive pricing, powerful infrastructure, and outstanding performance, making it ideal for cost-conscious teams and performance-critical workloads.

## Table of Contents

- [Overview](#overview)
- [Why Hetzner](#why-hetzner)
- [Setup and Configuration](#setup-and-configuration)
- [Available Resources](#available-resources)
- [Nickel Schema Reference](#nickel-schema-reference)
- [Configuration Examples](#configuration-examples)
- [Best Practices](#best-practices)
- [Troubleshooting](#troubleshooting)

## Overview

Hetzner Cloud provides European cloud infrastructure with exceptional value. Key characteristics:

- **Best Price/Performance**: Lower cost than AWS, competitive with DigitalOcean
- **European Focus**: Primary datacenter in Germany with compliance emphasis
- **Powerful Hardware**: Modern CPUs, NVMe storage, 10Gbps networking
- **Flexible Billing**: Hourly or monthly, no long-term contracts
- **API-First**: Comprehensive RESTful API for automation

### Hetzner Pricing Model

Hetzner uses hourly billing with generous monthly rates (30.4 days):

- **Cloud Servers**: €0.003-0.072/hour (~€3-200/month depending on size)
- **Volumes**: €0.026/GB/month
- **Data Transfer**: €0.12/GB outbound (generous included traffic)
- **Floating IP**: Free (1 per server)

### Price Comparison (2 vCPU, 4 GB RAM)

| Provider | Monthly | Hourly | Notes |
|----------|---------|--------|-------|
| Hetzner CX31 | €6.90 | €0.003 | Best value |
| DigitalOcean | $24 | $0.0357 | 3.5x more expensive |
| AWS t3.medium | $60+ | $0.0896 | On-demand pricing |
| UpCloud | $15 | $0.0223 | Mid-range |

### Supported Resources

| Resource | Product Name | Status |
|----------|--------------|--------|
| Compute | Cloud Servers | ✓ Full support |
| Block Storage | Volumes | ✓ Full support |
| Object Storage | Object Storage | ✓ Full support |
| Load Balancer | Load Balancer | ✓ Full support |
| Network | vSwitch/Network | ✓ Full support |
| Firewall | Firewall | ✓ Full support |
| DNS | — | ✓ Via Hetzner DNS |
| Bare Metal | Dedicated Servers | ✓ Available |
| Floating IP | Floating IP | ✓ Full support |

## Why Hetzner

### When to Choose Hetzner

**Hetzner is ideal for**:

- **Cost-Conscious Teams**: 50-75% cheaper than AWS
- **European Operations**: Primary EU presence
- **Predictable Workloads**: Good for sustained compute
- **Performance-Critical**: Modern hardware, 10Gbps networking
- **Self-Managed Services**: Full control over infrastructure
- **Bulk Computing**: Good pricing for 10-100+ servers

**Hetzner is NOT ideal for**:

- **Managed Services**: Limited compared to AWS/DigitalOcean
- **Global Distribution**: Limited regions (mainly EU + US)
- **Windows Workloads**: Limited Windows support
- **Complex Compliance**: Fewer certifications than AWS
- **Hands-Off Operations**: Need to manage own infrastructure

### Cost Advantages

**Total Cost of Ownership Comparison** (5 servers, 100 GB storage):

| Provider | Compute | Storage | Data Transfer | Monthly |
|----------|---------|---------|----------------|---------|
| Hetzner | €34.50 | €2.60 | Included | **€37.10** |
| DigitalOcean | $120 | $10 | Included | **$130** |
| AWS | $300 | $100 | $450 | **$850** |

Hetzner is **3.5x cheaper** than DigitalOcean and **23x cheaper** than AWS for this scenario.

## Setup and Configuration

### Prerequisites

- Hetzner Cloud account at [Hetzner Console](https://console.hetzner.cloud/)
- API token from Cloud Console
- SSH key uploaded to Hetzner
- hcloud CLI installed (optional but recommended)
- Provisioning system with Hetzner provider plugin

### Step 1: Create Hetzner API Token

1. Log in to Hetzner Cloud Console
2. Go to **Projects** > **Your Project** > **Security** > **API Tokens**
3. Click **Generate Token**
4. Name it (for example, "provisioning")
5. Select **Read & Write** permission
6. Copy the token immediately (only shown once)

### Step 2: Configure Environment Variables

```bash
# Add to ~/.bashrc, ~/.zshrc, or env file
export HCLOUD_TOKEN="MC4wNTI1YmE1M2E4YmE0YTQzMTQ..."

# Optional: Set default location
export HCLOUD_LOCATION="nbg1"
```

### Step 3: Install hcloud CLI (Optional)

```bash
# macOS
brew install hcloud

# Linux
curl https://github.com/hetznercloud/cli/releases/download/v1.x.x/hcloud-linux-amd64.tar.gz | tar xz
sudo mv hcloud /usr/local/bin/

# Verify
hcloud version
```

### Step 4: Configure SSH Key

```bash
# Upload your SSH public key
hcloud ssh-key create --name "provisioning-key" \
  --public-key-from-file ~/.ssh/id_rsa.pub

# List keys
hcloud ssh-key list
```

### Step 5: Configure Workspace

Create or update `config.toml` in your workspace:

```toml
[providers.hetzner]
enabled = true
token_env = "HCLOUD_TOKEN"
default_location = "nbg1"
default_datacenter = "nbg1-dc8"

[workspace]
provider = "hetzner"
region = "nbg1"
```

## Available Resources

### 1. Cloud Servers (Compute)

Hetzner's core compute offering with outstanding performance.

**Available Server Types**:

| Type | vCPU | RAM | SSD Storage | Network | Monthly Price |
|------|------|-----|-------------|---------|---------------|
| CX11 | 1 | 1 GB | 25 GB | 1Gbps | €3.29 |
| CX21 | 2 | 4 GB | 40 GB | 1Gbps | €6.90 |
| CX31 | 2 | 8 GB | 80 GB | 1Gbps | €13.80 |
| CX41 | 4 | 16 GB | 160 GB | 1Gbps | €27.60 |
| CX51 | 8 | 32 GB | 240 GB | 10Gbps | €55.20 |
| CPX21 | 4 | 8 GB | 80 GB | 10Gbps | €20.90 |
| CPX31 | 8 | 16 GB | 160 GB | 10Gbps | €41.80 |
| CPX41 | 16 | 32 GB | 360 GB | 10Gbps | €83.60 |

**Key Features**:
- NVMe SSD storage
- Hourly or monthly billing
- Automatic backups
- SSH key management
- Floating IPs for high availability
- Network interfaces for multi-homing
- Cloud-init support
- IPMI/KVM console access

### 2. Volumes (Block Storage)

Persistent block storage that can be attached/detached.

**Characteristics**:
- €0.026/GB/month (highly affordable)
- SSD-based with good performance
- Up to 10 TB capacity
- Snapshots for backup
- Can attach to multiple servers (read-only)
- Automatic snapshots available

### 3. Object Storage

S3-compatible object storage.

**Characteristics**:
- €0.025/GB/month
- S3-compatible API
- Versioning and lifecycle policies
- Bucket policy support
- CORS configuration

### 4. Floating IPs

Static IP addresses that can be reassigned.

**Characteristics**:
- Free (1 per server, additional €0.50/month)
- IPv4 and IPv6 support
- Enable high availability and failover
- DNS pointing

### 5. Load Balancer

Layer 4/7 load balancing.

**Available Plans**:
- LB11: €5/month (100 Mbps)
- LB21: €10/month (1 Gbps)
- LB31: €20/month (10 Gbps)

**Features**:
- Health checks
- SSL/TLS termination
- Path/host-based routing
- Sticky sessions
- Algorithms: round robin, least connections

### 6. Network/vSwitch

Virtual switching for private networking.

**Characteristics**:
- Private networks between servers
- Subnets within networks
- Routes and gateways
- Firewall integration

### 7. Firewall

Network firewall rules.

**Features**:
- Per-server or per-network
- Stateful filtering
- Protocol-specific rules
- Source/destination filtering

## Nickel Schema Reference

### Cloud Server Configuration

```nickel
let hetzner = import "../../extensions/providers/hetzner/nickel/main.ncl" in

hetzner.Server & {
  # Required
  name = "my-server",
  server_type = "cx21",
  image = "ubuntu-22.04",

  # Optional
  location = "nbg1",  # nbg1, fsn1, hel1, ash
  datacenter = "nbg1-dc8",
  ssh_keys = ["key-name"],
  count = 1,
  public_net = {
    enable_ipv4 = true,
    enable_ipv6 = true
  },

  # Volumes to attach
  volumes = [
    {
      size = 100,
      format = "ext4",
      automount = true
    }
  ],

  # Network configuration
  networks = [
    {
      network_name = "private-net",
      ip = "10.0.1.5"
    }
  ],

  # Firewall rules
  firewall_rules = [
    {
      direction = "in",
      source_ips = ["0.0.0.0/0", "::/0"],
      destination_port = "22",
      protocol = "tcp"
    },
    {
      direction = "in",
      source_ips = ["0.0.0.0/0", "::/0"],
      destination_port = "80",
      protocol = "tcp"
    },
    {
      direction = "in",
      source_ips = ["0.0.0.0/0", "::/0"],
      destination_port = "443",
      protocol = "tcp"
    }
  ],

  # Labels for organization
  labels = {
    "environment" = "production",
    "application" = "web"
  },

  # Startup script
  user_data = "#!/bin/bash\napt-get update\napt-get install -y nginx"
}
```

### Volume Configuration

```nickel
hetzner.Volume & {
  name = "data-volume",
  size = 100,  # GB
  location = "nbg1",
  automount = true,
  format = "ext4",

  # Attach to server
  attachment = {
    server = "server-name",
    mount_point = "/data"
  }
}
```

### Load Balancer Configuration

```nickel
hetzner.LoadBalancer & {
  name = "web-lb",
  load_balancer_type = "lb11",
  network_zone = "eu-central",
  location = "nbg1",

  # Services (backend targets)
  services = [
    {
      protocol = "http",
      listen_port = 80,
      destination_port = 8080,
      health_check = {
        protocol = "http",
        port = 8080,
        interval = 15,
        timeout = 10,
        unhealthy_threshold = 3
      },
      http = {
        sticky_sessions = true,
        http_only = true,
        certificates = []
      }
    }
  ]
}
```

### Firewall Configuration

```nickel
hetzner.Firewall & {
  name = "web-firewall",
  labels = { "env" = "prod" },

  rules = [
    # Allow SSH from management network
    {
      direction = "in",
      source_ips = ["203.0.113.0/24"],
      destination_port = "22",
      protocol = "tcp"
    },
    # Allow HTTP/HTTPS from anywhere
    {
      direction = "in",
      source_ips = ["0.0.0.0/0", "::/0"],
      destination_port = "80",
      protocol = "tcp"
    },
    {
      direction = "in",
      source_ips = ["0.0.0.0/0", "::/0"],
      destination_port = "443",
      protocol = "tcp"
    },
    # Allow all outbound
    {
      direction = "out",
      destination_ips = ["0.0.0.0/0", "::/0"],
      protocol = "esp"
    }
  ]
}
```

## Configuration Examples

### Example 1: Single Server Web Server

```nickel
let hetzner = import "../../extensions/providers/hetzner/nickel/main.ncl" in

{
  workspace_name = "simple-web",

  web_server = hetzner.Server & {
    name = "web-01",
    server_type = "cx21",
    image = "ubuntu-22.04",
    location = "nbg1",
    ssh_keys = ["provisioning"],

    user_data = ''
      #!/bin/bash
      apt-get update
      apt-get install -y nginx
      systemctl start nginx
      systemctl enable nginx
    '',

    firewall_rules = [
      { direction = "in", source_ips = ["0.0.0.0/0"], destination_port = "22", protocol = "tcp" },
      { direction = "in", source_ips = ["0.0.0.0/0"], destination_port = "80", protocol = "tcp" },
      { direction = "in", source_ips = ["0.0.0.0/0"], destination_port = "443", protocol = "tcp" }
    ],

    labels = { "service" = "web" }
  }
}
```

### Example 2: Web Application with Load Balancer and Storage

```nickel
{
  # Backend servers
  app_servers = hetzner.Server & {
    name = "app",
    server_type = "cx31",
    image = "ubuntu-22.04",
    location = "nbg1",
    count = 3,
    ssh_keys = ["provisioning"],

    volumes = [
      {
        size = 100,
        format = "ext4",
        automount = true
      }
    ],

    firewall_rules = [
      { direction = "in", source_ips = ["0.0.0.0/0"], destination_port = "22", protocol = "tcp" },
      { direction = "in", source_ips = ["0.0.0.0/0"], destination_port = "8080", protocol = "tcp" }
    ],

    labels = { "tier" = "application" }
  },

  # Load balancer
  lb = hetzner.LoadBalancer & {
    name = "web-lb",
    load_balancer_type = "lb11",
    location = "nbg1",

    services = [
      {
        protocol = "http",
        listen_port = 80,
        destination_port = 8080,
        health_check = {
          protocol = "http",
          port = 8080,
          interval = 15
        }
      }
    ]
  },

  # Persistent storage
  shared_storage = hetzner.Volume & {
    name = "shared-data",
    size = 500,
    location = "nbg1",
    automount = false,
    format = "ext4"
  }
}
```

### Example 3: High-Performance Compute Cluster

```nickel
{
  # Compute nodes with 10Gbps networking
  compute_nodes = hetzner.Server & {
    name = "compute",
    server_type = "cpx41",  # 16 vCPU, 32 GB, 10Gbps
    image = "ubuntu-22.04",
    location = "nbg1",
    count = 5,

    volumes = [
      {
        size = 500,
        format = "ext4",
        automount = true
      }
    ],

    labels = { "tier" = "compute" }
  },

  # Storage node
  storage = hetzner.Server & {
    name = "storage",
    server_type = "cx41",
    image = "ubuntu-22.04",
    location = "nbg1",

    volumes = [
      {
        size = 2000,
        format = "ext4",
        automount = true
      }
    ],

    labels = { "tier" = "storage" }
  },

  # High-capacity volume for data
  data_volume = hetzner.Volume & {
    name = "compute-data",
    size = 5000,
    location = "nbg1"
  }
}
```

## Best Practices

### 1. Server Selection and Sizing

**Performance Tiers**:

- **CX Series** (Standard): Best value for most workloads
  - CX21: Default choice for 2-4 GB workloads
  - CX41: Good mid-range option

- **CPX Series** (ARM-based CPU-optimized): Better for CPU-intensive
  - CPX21: Outstanding value at €20.90/month
  - CPX31: Good for compute workloads

- **CCX Series** (AMD EPYC): High-performance options

**Selection Criteria**:
- Start with CX21 (€6.90/month) for testing
- Scale to CPX21 (€20.90/month) for CPU-bound workloads
- Use CX31+ (€13.80+) for balanced workloads with data

### 2. Network Architecture

**High Availability**:
```nickel
# Use Floating IPs for failover
floating_ip = hetzner.FloatingIP & {
  name = "web-ip",
  ip_type = "ipv4",
  location = "nbg1"
}

# Attach to primary server, reassign on failure
attachment = {
  server = "primary-server"
}
```

**Private Networking**:
```nickel
# Create private network for internal communication
private_network = hetzner.Network & {
  name = "private",
  ip_range = "10.0.0.0/8",
  labels = { "env" = "prod" }
}
```

### 3. Storage Strategy

**Volume Sizing**:
- Estimate storage needs: app + data + logs + backups
- Add 20% buffer for growth
- Monitor usage monthly

**Backup Strategy**:
- Enable automatic snapshots
- Regular manual snapshots for important data
- Test restore procedures
- Keep snapshots for minimum 30 days

### 4. Firewall Configuration

**Principle of Least Privilege**:
```nickel
# Only open necessary ports
firewall_rules = [
  # SSH from management IP only
  { direction = "in", source_ips = ["203.0.113.1/32"], destination_port = "22", protocol = "tcp" },

  # HTTP/HTTPS from anywhere
  { direction = "in", source_ips = ["0.0.0.0/0", "::/0"], destination_port = "80", protocol = "tcp" },
  { direction = "in", source_ips = ["0.0.0.0/0", "::/0"], destination_port = "443", protocol = "tcp" },

  # Database replication (internal only)
  { direction = "in", source_ips = ["10.0.0.0/8"], destination_port = "5432", protocol = "tcp" }
]
```

### 5. Monitoring and Health Checks

**Enable Monitoring**:
```bash
hcloud server update <server-id> --enable-rescue
```

**Health Check Patterns**:
- HTTP endpoint returning 200
- Custom health check scripts
- Regular resource verification

### 6. Cost Optimization

**Reserved Servers** (Pre-pay for 12 months):
- 25% discount vs hourly
- Good for predictable workloads

**Spot Pricing** (Coming):
- Watch for additional discounts
- Off-peak capacity

**Resource Cleanup**:
- Delete unused volumes
- Remove old snapshots
- Consolidate small servers

## Troubleshooting

### Issue: Cannot Connect to Server

**Symptoms**: SSH timeout or connection refused

**Diagnosis**:
```bash
# Check server status
hcloud server list

# Verify firewall allows port 22
hcloud firewall describe firewall-name

# Check if server has public IPv4
hcloud server describe server-name
```

**Solution**:
```bash
# Update firewall to allow SSH from your IP
hcloud firewall add-rules firewall-id \
  --rules "direction=in protocol=tcp source_ips=YOUR_IP/32 destination_port=22"

# Or reset SSH using rescue mode via console
hcloud server request-console server-id
```

### Issue: Volume Attachment Failed

**Symptoms**: Volume created but cannot attach, mount fails

**Diagnosis**:
```bash
# Check volume status
hcloud volume list

# Check server has available attachment slot
hcloud server describe server-name
```

**Solution**:
```bash
# Format volume (first time only)
sudo mkfs.ext4 /dev/sdb

# Mount manually
sudo mkdir -p /data
sudo mount /dev/sdb /data

# Make persistent
echo '/dev/sdb /data ext4 defaults,nofail 0 0' | sudo tee -a /etc/fstab
sudo mount -a
```

### Issue: High Data Transfer Costs

**Symptoms**: Unexpected egress charges

**Diagnosis**:
```bash
# Check server network traffic
sar -n DEV 1 100

# Monitor connection patterns
netstat -an | grep ESTABLISHED | wc -l
```

**Solution**:
- Use Hetzner Object Storage for static files
- Cache content locally
- Optimize data transfer patterns
- Consider using Content Delivery Network

### Issue: Load Balancer Not Routing Traffic

**Symptoms**: LB created but backends not receiving traffic

**Diagnosis**:
```bash
# Check LB status
hcloud load-balancer describe lb-name

# Test backend directly
curl -H "Host: example.com" http://backend-ip:8080/health
```

**Solution**:
- Ensure backends have firewall allowing LB traffic
- Verify health check endpoint works
- Check backend service is running
- Review health check configuration

## Summary

Hetzner provides exceptional value with modern infrastructure:

✓ Best price/performance ratio (50%+ cheaper than DigitalOcean)
✓ Excellent European presence
✓ Powerful hardware (NVMe, 10Gbps networking)
✓ Flexible deployment options
✓ Great API and CLI tools

Start with CX21 servers (€6.90/month) and scale based on needs.

For more information, visit: [Hetzner Cloud Documentation](https://docs.hetzner.cloud/)
