# Hetzner Provider Guide\n\nThis guide covers using Hetzner Cloud as a provider in the provisioning system. Hetzner is renowned for competitive pricing, powerful infrastructure,\nand outstanding performance, making it ideal for cost-conscious teams and performance-critical workloads.\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Why Hetzner](#why-hetzner)\n- [Setup and Configuration](#setup-and-configuration)\n- [Available Resources](#available-resources)\n- [Nickel Schema Reference](#nickel-schema-reference)\n- [Configuration Examples](#configuration-examples)\n- [Best Practices](#best-practices)\n- [Troubleshooting](#troubleshooting)\n\n## Overview\n\nHetzner Cloud provides European cloud infrastructure with exceptional value. Key characteristics:\n\n- **Best Price/Performance**: Lower cost than AWS, competitive with DigitalOcean\n- **European Focus**: Primary datacenter in Germany with compliance emphasis\n- **Powerful Hardware**: Modern CPUs, NVMe storage, 10Gbps networking\n- **Flexible Billing**: Hourly or monthly, no long-term contracts\n- **API-First**: Comprehensive RESTful API for automation\n\n### Hetzner Pricing Model\n\nHetzner uses hourly billing with generous monthly rates (30.4 days):\n\n- **Cloud Servers**: €0.003-0.072/hour (~€3-200/month depending on size)\n- **Volumes**: €0.026/GB/month\n- **Data Transfer**: €0.12/GB outbound (generous included traffic)\n- **Floating IP**: Free (1 per server)\n\n### Price Comparison (2 vCPU, 4 GB RAM)\n\n| Provider | Monthly | Hourly | Notes |\n| ---------- | --------- | -------- | ------- |\n| Hetzner CX31 | €6.90 | €0.003 | Best value |\n| DigitalOcean | $24 | $0.0357 | 3.5x more expensive |\n| AWS t3.medium | $60+ | $0.0896 | On-demand pricing |\n| UpCloud | $15 | $0.0223 | Mid-range |\n\n### Supported Resources\n\n| Resource | Product Name | Status |\n| ---------- | -------------- | -------- |\n| Compute | Cloud Servers | ✓ Full support |\n| Block Storage | Volumes | ✓ Full support |\n| Object Storage | Object Storage | ✓ Full support |\n| Load Balancer | Load Balancer | ✓ Full support |\n| Network | vSwitch/Network | ✓ Full support |\n| Firewall | Firewall | ✓ Full support |\n| DNS | — | ✓ Via Hetzner DNS |\n| Bare Metal | Dedicated Servers | ✓ Available |\n| Floating IP | Floating IP | ✓ Full support |\n\n## Why Hetzner\n\n### When to Choose Hetzner\n\n**Hetzner is ideal for**:\n\n- **Cost-Conscious Teams**: 50-75% cheaper than AWS\n- **European Operations**: Primary EU presence\n- **Predictable Workloads**: Good for sustained compute\n- **Performance-Critical**: Modern hardware, 10Gbps networking\n- **Self-Managed Services**: Full control over infrastructure\n- **Bulk Computing**: Good pricing for 10-100+ servers\n\n**Hetzner is NOT ideal for**:\n\n- **Managed Services**: Limited compared to AWS/DigitalOcean\n- **Global Distribution**: Limited regions (mainly EU + US)\n- **Windows Workloads**: Limited Windows support\n- **Complex Compliance**: Fewer certifications than AWS\n- **Hands-Off Operations**: Need to manage own infrastructure\n\n### Cost Advantages\n\n**Total Cost of Ownership Comparison** (5 servers, 100 GB storage):\n\n| Provider | Compute | Storage | Data Transfer | Monthly |\n| ---------- | --------- | --------- | ---------------- | --------- |\n| Hetzner | €34.50 | €2.60 | Included | **€37.10** |\n| DigitalOcean | $120 | $10 | Included | **$130** |\n| AWS | $300 | $100 | $450 | **$850** |\n\nHetzner is **3.5x cheaper** than DigitalOcean and **23x cheaper** than AWS for this scenario.\n\n## Setup and Configuration\n\n### Prerequisites\n\n- Hetzner Cloud account at [Hetzner Console](https://console.hetzner.cloud/)\n- API token from Cloud Console\n- SSH key uploaded to Hetzner\n- hcloud CLI installed (optional but recommended)\n- Provisioning system with Hetzner provider plugin\n\n### Step 1: Create Hetzner API Token\n\n1. Log in to Hetzner Cloud Console\n2. Go to **Projects** > **Your Project** > **Security** > **API Tokens**\n3. Click **Generate Token**\n4. Name it (for example, "provisioning")\n5. Select **Read & Write** permission\n6. Copy the token immediately (only shown once)\n\n### Step 2: Configure Environment Variables\n\n```\n# Add to ~/.bashrc, ~/.zshrc, or env file\nexport HCLOUD_TOKEN="MC4wNTI1YmE1M2E4YmE0YTQzMTQ..."\n\n# Optional: Set default location\nexport HCLOUD_LOCATION="nbg1"\n```\n\n### Step 3: Install hcloud CLI (Optional)\n\n```\n# macOS\nbrew install hcloud\n\n# Linux\ncurl https://github.com/hetznercloud/cli/releases/download/v1.x.x/hcloud-linux-amd64.tar.gz | tar xz\nsudo mv hcloud /usr/local/bin/\n\n# Verify\nhcloud version\n```\n\n### Step 4: Configure SSH Key\n\n```\n# Upload your SSH public key\nhcloud ssh-key create --name "provisioning-key" \\n  --public-key-from-file ~/.ssh/id_rsa.pub\n\n# List keys\nhcloud ssh-key list\n```\n\n### Step 5: Configure Workspace\n\nCreate or update `config.toml` in your workspace:\n\n```\n[providers.hetzner]\nenabled = true\ntoken_env = "HCLOUD_TOKEN"\ndefault_location = "nbg1"\ndefault_datacenter = "nbg1-dc8"\n\n[workspace]\nprovider = "hetzner"\nregion = "nbg1"\n```\n\n## Available Resources\n\n### 1. Cloud Servers (Compute)\n\nHetzner's core compute offering with outstanding performance.\n\n**Available Server Types**:\n\n| Type | vCPU | RAM | SSD Storage | Network | Monthly Price |\n| ------ | ------ | ----- | ------------- | --------- | --------------- |\n| CX11 | 1 | 1 GB | 25 GB | 1Gbps | €3.29 |\n| CX21 | 2 | 4 GB | 40 GB | 1Gbps | €6.90 |\n| CX31 | 2 | 8 GB | 80 GB | 1Gbps | €13.80 |\n| CX41 | 4 | 16 GB | 160 GB | 1Gbps | €27.60 |\n| CX51 | 8 | 32 GB | 240 GB | 10Gbps | €55.20 |\n| CPX21 | 4 | 8 GB | 80 GB | 10Gbps | €20.90 |\n| CPX31 | 8 | 16 GB | 160 GB | 10Gbps | €41.80 |\n| CPX41 | 16 | 32 GB | 360 GB | 10Gbps | €83.60 |\n\n**Key Features**:\n- NVMe SSD storage\n- Hourly or monthly billing\n- Automatic backups\n- SSH key management\n- Floating IPs for high availability\n- Network interfaces for multi-homing\n- Cloud-init support\n- IPMI/KVM console access\n\n### 2. Volumes (Block Storage)\n\nPersistent block storage that can be attached/detached.\n\n**Characteristics**:\n- €0.026/GB/month (highly affordable)\n- SSD-based with good performance\n- Up to 10 TB capacity\n- Snapshots for backup\n- Can attach to multiple servers (read-only)\n- Automatic snapshots available\n\n### 3. Object Storage\n\nS3-compatible object storage.\n\n**Characteristics**:\n- €0.025/GB/month\n- S3-compatible API\n- Versioning and lifecycle policies\n- Bucket policy support\n- CORS configuration\n\n### 4. Floating IPs\n\nStatic IP addresses that can be reassigned.\n\n**Characteristics**:\n- Free (1 per server, additional €0.50/month)\n- IPv4 and IPv6 support\n- Enable high availability and failover\n- DNS pointing\n\n### 5. Load Balancer\n\nLayer 4/7 load balancing.\n\n**Available Plans**:\n- LB11: €5/month (100 Mbps)\n- LB21: €10/month (1 Gbps)\n- LB31: €20/month (10 Gbps)\n\n**Features**:\n- Health checks\n- SSL/TLS termination\n- Path/host-based routing\n- Sticky sessions\n- Algorithms: round robin, least connections\n\n### 6. Network/vSwitch\n\nVirtual switching for private networking.\n\n**Characteristics**:\n- Private networks between servers\n- Subnets within networks\n- Routes and gateways\n- Firewall integration\n\n### 7. Firewall\n\nNetwork firewall rules.\n\n**Features**:\n- Per-server or per-network\n- Stateful filtering\n- Protocol-specific rules\n- Source/destination filtering\n\n## Nickel Schema Reference\n\n### Cloud Server Configuration\n\n```\nlet hetzner = import "../../extensions/providers/hetzner/nickel/main.ncl" in\n\nhetzner.Server & {\n  # Required\n  name = "my-server",\n  server_type = "cx21",\n  image = "ubuntu-22.04",\n\n  # Optional\n  location = "nbg1",  # nbg1, fsn1, hel1, ash\n  datacenter = "nbg1-dc8",\n  ssh_keys = ["key-name"],\n  count = 1,\n  public_net = {\n    enable_ipv4 = true,\n    enable_ipv6 = true\n  },\n\n  # Volumes to attach\n  volumes = [\n    {\n      size = 100,\n      format = "ext4",\n      automount = true\n    }\n  ],\n\n  # Network configuration\n  networks = [\n    {\n      network_name = "private-net",\n      ip = "10.0.1.5"\n    }\n  ],\n\n  # Firewall rules\n  firewall_rules = [\n    {\n      direction = "in",\n      source_ips = ["0.0.0.0/0", "::/0"],\n      destination_port = "22",\n      protocol = "tcp"\n    },\n    {\n      direction = "in",\n      source_ips = ["0.0.0.0/0", "::/0"],\n      destination_port = "80",\n      protocol = "tcp"\n    },\n    {\n      direction = "in",\n      source_ips = ["0.0.0.0/0", "::/0"],\n      destination_port = "443",\n      protocol = "tcp"\n    }\n  ],\n\n  # Labels for organization\n  labels = {\n    "environment" = "production",\n    "application" = "web"\n  },\n\n  # Startup script\n  user_data = "#!/bin/bash\napt-get update\napt-get install -y nginx"\n}\n```\n\n### Volume Configuration\n\n```\nhetzner.Volume & {\n  name = "data-volume",\n  size = 100,  # GB\n  location = "nbg1",\n  automount = true,\n  format = "ext4",\n\n  # Attach to server\n  attachment = {\n    server = "server-name",\n    mount_point = "/data"\n  }\n}\n```\n\n### Load Balancer Configuration\n\n```\nhetzner.LoadBalancer & {\n  name = "web-lb",\n  load_balancer_type = "lb11",\n  network_zone = "eu-central",\n  location = "nbg1",\n\n  # Services (backend targets)\n  services = [\n    {\n      protocol = "http",\n      listen_port = 80,\n      destination_port = 8080,\n      health_check = {\n        protocol = "http",\n        port = 8080,\n        interval = 15,\n        timeout = 10,\n        unhealthy_threshold = 3\n      },\n      http = {\n        sticky_sessions = true,\n        http_only = true,\n        certificates = []\n      }\n    }\n  ]\n}\n```\n\n### Firewall Configuration\n\n```\nhetzner.Firewall & {\n  name = "web-firewall",\n  labels = { "env" = "prod" },\n\n  rules = [\n    # Allow SSH from management network\n    {\n      direction = "in",\n      source_ips = ["203.0.113.0/24"],\n      destination_port = "22",\n      protocol = "tcp"\n    },\n    # Allow HTTP/HTTPS from anywhere\n    {\n      direction = "in",\n      source_ips = ["0.0.0.0/0", "::/0"],\n      destination_port = "80",\n      protocol = "tcp"\n    },\n    {\n      direction = "in",\n      source_ips = ["0.0.0.0/0", "::/0"],\n      destination_port = "443",\n      protocol = "tcp"\n    },\n    # Allow all outbound\n    {\n      direction = "out",\n      destination_ips = ["0.0.0.0/0", "::/0"],\n      protocol = "esp"\n    }\n  ]\n}\n```\n\n## Configuration Examples\n\n### Example 1: Single Server Web Server\n\n```\nlet hetzner = import "../../extensions/providers/hetzner/nickel/main.ncl" in\n\n{\n  workspace_name = "simple-web",\n\n  web_server = hetzner.Server & {\n    name = "web-01",\n    server_type = "cx21",\n    image = "ubuntu-22.04",\n    location = "nbg1",\n    ssh_keys = ["provisioning"],\n\n    user_data = ''\n      #!/bin/bash\n      apt-get update\n      apt-get install -y nginx\n      systemctl start nginx\n      systemctl enable nginx\n    '',\n\n    firewall_rules = [\n      { direction = "in", source_ips = ["0.0.0.0/0"], destination_port = "22", protocol = "tcp" },\n      { direction = "in", source_ips = ["0.0.0.0/0"], destination_port = "80", protocol = "tcp" },\n      { direction = "in", source_ips = ["0.0.0.0/0"], destination_port = "443", protocol = "tcp" }\n    ],\n\n    labels = { "service" = "web" }\n  }\n}\n```\n\n### Example 2: Web Application with Load Balancer and Storage\n\n```\n{\n  # Backend servers\n  app_servers = hetzner.Server & {\n    name = "app",\n    server_type = "cx31",\n    image = "ubuntu-22.04",\n    location = "nbg1",\n    count = 3,\n    ssh_keys = ["provisioning"],\n\n    volumes = [\n      {\n        size = 100,\n        format = "ext4",\n        automount = true\n      }\n    ],\n\n    firewall_rules = [\n      { direction = "in", source_ips = ["0.0.0.0/0"], destination_port = "22", protocol = "tcp" },\n      { direction = "in", source_ips = ["0.0.0.0/0"], destination_port = "8080", protocol = "tcp" }\n    ],\n\n    labels = { "tier" = "application" }\n  },\n\n  # Load balancer\n  lb = hetzner.LoadBalancer & {\n    name = "web-lb",\n    load_balancer_type = "lb11",\n    location = "nbg1",\n\n    services = [\n      {\n        protocol = "http",\n        listen_port = 80,\n        destination_port = 8080,\n        health_check = {\n          protocol = "http",\n          port = 8080,\n          interval = 15\n        }\n      }\n    ]\n  },\n\n  # Persistent storage\n  shared_storage = hetzner.Volume & {\n    name = "shared-data",\n    size = 500,\n    location = "nbg1",\n    automount = false,\n    format = "ext4"\n  }\n}\n```\n\n### Example 3: High-Performance Compute Cluster\n\n```\n{\n  # Compute nodes with 10Gbps networking\n  compute_nodes = hetzner.Server & {\n    name = "compute",\n    server_type = "cpx41",  # 16 vCPU, 32 GB, 10Gbps\n    image = "ubuntu-22.04",\n    location = "nbg1",\n    count = 5,\n\n    volumes = [\n      {\n        size = 500,\n        format = "ext4",\n        automount = true\n      }\n    ],\n\n    labels = { "tier" = "compute" }\n  },\n\n  # Storage node\n  storage = hetzner.Server & {\n    name = "storage",\n    server_type = "cx41",\n    image = "ubuntu-22.04",\n    location = "nbg1",\n\n    volumes = [\n      {\n        size = 2000,\n        format = "ext4",\n        automount = true\n      }\n    ],\n\n    labels = { "tier" = "storage" }\n  },\n\n  # High-capacity volume for data\n  data_volume = hetzner.Volume & {\n    name = "compute-data",\n    size = 5000,\n    location = "nbg1"\n  }\n}\n```\n\n## Best Practices\n\n### 1. Server Selection and Sizing\n\n**Performance Tiers**:\n\n- **CX Series** (Standard): Best value for most workloads\n  - CX21: Default choice for 2-4 GB workloads\n  - CX41: Good mid-range option\n\n- **CPX Series** (ARM-based CPU-optimized): Better for CPU-intensive\n  - CPX21: Outstanding value at €20.90/month\n  - CPX31: Good for compute workloads\n\n- **CCX Series** (AMD EPYC): High-performance options\n\n**Selection Criteria**:\n- Start with CX21 (€6.90/month) for testing\n- Scale to CPX21 (€20.90/month) for CPU-bound workloads\n- Use CX31+ (€13.80+) for balanced workloads with data\n\n### 2. Network Architecture\n\n**High Availability**:\n```\n# Use Floating IPs for failover\nfloating_ip = hetzner.FloatingIP & {\n  name = "web-ip",\n  ip_type = "ipv4",\n  location = "nbg1"\n}\n\n# Attach to primary server, reassign on failure\nattachment = {\n  server = "primary-server"\n}\n```\n\n**Private Networking**:\n```\n# Create private network for internal communication\nprivate_network = hetzner.Network & {\n  name = "private",\n  ip_range = "10.0.0.0/8",\n  labels = { "env" = "prod" }\n}\n```\n\n### 3. Storage Strategy\n\n**Volume Sizing**:\n- Estimate storage needs: app + data + logs + backups\n- Add 20% buffer for growth\n- Monitor usage monthly\n\n**Backup Strategy**:\n- Enable automatic snapshots\n- Regular manual snapshots for important data\n- Test restore procedures\n- Keep snapshots for minimum 30 days\n\n### 4. Firewall Configuration\n\n**Principle of Least Privilege**:\n```\n# Only open necessary ports\nfirewall_rules = [\n  # SSH from management IP only\n  { direction = "in", source_ips = ["203.0.113.1/32"], destination_port = "22", protocol = "tcp" },\n\n  # HTTP/HTTPS from anywhere\n  { direction = "in", source_ips = ["0.0.0.0/0", "::/0"], destination_port = "80", protocol = "tcp" },\n  { direction = "in", source_ips = ["0.0.0.0/0", "::/0"], destination_port = "443", protocol = "tcp" },\n\n  # Database replication (internal only)\n  { direction = "in", source_ips = ["10.0.0.0/8"], destination_port = "5432", protocol = "tcp" }\n]\n```\n\n### 5. Monitoring and Health Checks\n\n**Enable Monitoring**:\n```\nhcloud server update <server-id> --enable-rescue\n```\n\n**Health Check Patterns**:\n- HTTP endpoint returning 200\n- Custom health check scripts\n- Regular resource verification\n\n### 6. Cost Optimization\n\n**Reserved Servers** (Pre-pay for 12 months):\n- 25% discount vs hourly\n- Good for predictable workloads\n\n**Spot Pricing** (Coming):\n- Watch for additional discounts\n- Off-peak capacity\n\n**Resource Cleanup**:\n- Delete unused volumes\n- Remove old snapshots\n- Consolidate small servers\n\n## Troubleshooting\n\n### Issue: Cannot Connect to Server\n\n**Symptoms**: SSH timeout or connection refused\n\n**Diagnosis**:\n```\n# Check server status\nhcloud server list\n\n# Verify firewall allows port 22\nhcloud firewall describe firewall-name\n\n# Check if server has public IPv4\nhcloud server describe server-name\n```\n\n**Solution**:\n```\n# Update firewall to allow SSH from your IP\nhcloud firewall add-rules firewall-id \\n  --rules "direction=in protocol=tcp source_ips=YOUR_IP/32 destination_port=22"\n\n# Or reset SSH using rescue mode via console\nhcloud server request-console server-id\n```\n\n### Issue: Volume Attachment Failed\n\n**Symptoms**: Volume created but cannot attach, mount fails\n\n**Diagnosis**:\n```\n# Check volume status\nhcloud volume list\n\n# Check server has available attachment slot\nhcloud server describe server-name\n```\n\n**Solution**:\n```\n# Format volume (first time only)\nsudo mkfs.ext4 /dev/sdb\n\n# Mount manually\nsudo mkdir -p /data\nsudo mount /dev/sdb /data\n\n# Make persistent\necho '/dev/sdb /data ext4 defaults,nofail 0 0' | sudo tee -a /etc/fstab\nsudo mount -a\n```\n\n### Issue: High Data Transfer Costs\n\n**Symptoms**: Unexpected egress charges\n\n**Diagnosis**:\n```\n# Check server network traffic\nsar -n DEV 1 100\n\n# Monitor connection patterns\nnetstat -an | grep ESTABLISHED | wc -l\n```\n\n**Solution**:\n- Use Hetzner Object Storage for static files\n- Cache content locally\n- Optimize data transfer patterns\n- Consider using Content Delivery Network\n\n### Issue: Load Balancer Not Routing Traffic\n\n**Symptoms**: LB created but backends not receiving traffic\n\n**Diagnosis**:\n```\n# Check LB status\nhcloud load-balancer describe lb-name\n\n# Test backend directly\ncurl -H "Host: example.com" http://backend-ip:8080/health\n```\n\n**Solution**:\n- Ensure backends have firewall allowing LB traffic\n- Verify health check endpoint works\n- Check backend service is running\n- Review health check configuration\n\n## Summary\n\nHetzner provides exceptional value with modern infrastructure:\n\n✓ Best price/performance ratio (50%+ cheaper than DigitalOcean)\n✓ Excellent European presence\n✓ Powerful hardware (NVMe, 10Gbps networking)\n✓ Flexible deployment options\n✓ Great API and CLI tools\n\nStart with CX21 servers (€6.90/month) and scale based on needs.\n\nFor more information, visit: [Hetzner Cloud Documentation](https://docs.hetzner.cloud/)
