prvng_extensions/taskservs/networking/proxy

Proxy Task Service (HAProxy)

Overview

The Proxy task service provides a complete installation and configuration of HAProxy, a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. HAProxy is particularly suited for very high traffic web sites and is the de-facto standard open-source load balancer.

Features

Core Proxy Features

• Layer 4 & Layer 7 Load Balancing - TCP and HTTP/HTTPS traffic distribution
• High Availability - Active/passive and active/active configurations
• SSL/TLS Termination - SSL offloading and end-to-end encryption
• Content-Based Routing - Route based on URLs, headers, and other criteria
• Session Persistence - Sticky sessions and session affinity

Load Balancing Algorithms

• Round Robin - Distribute requests evenly across servers
• Least Connections - Route to server with fewest active connections
• Weighted Round Robin - Assign different weights to servers
• Source IP Hash - Route based on client IP hash
• URL Hash - Route based on URL hash for cache optimization

Health Checking & Monitoring

• Health Checks - TCP, HTTP, and custom health checks
• Server Status Monitoring - Real-time server status and metrics
• Statistics Interface - Built-in web statistics interface
• Prometheus Metrics - Native Prometheus metrics export
• Logging - Comprehensive access and error logging

Security Features

• DDoS Protection - Rate limiting and connection limits
• Access Control - IP-based access control lists
• SSL Security - Modern TLS configuration and cipher suites
• Request Filtering - Block malicious requests and patterns
• Security Headers - Automatic security header injection

Advanced Features

• Compression - HTTP response compression
• Caching - Basic HTTP caching capabilities
• Request Modification - Header manipulation and URL rewriting
• Multi-Process Mode - Multi-process for high concurrency
• Configuration Validation - Real-time configuration validation

Configuration

Basic HTTP Load Balancer

proxy: Proxy = {
    proxy_version: "2.8"
    proxy_lib: "/var/lib/haproxy"
    proxy_cfg_file: "haproxy.cfg"
    run_user: "haproxy"
    run_group: "haproxy"
    run_user_home: "/home/haproxy"
    https_in_binds: [
        {
            ip: "0.0.0.0"
            port: 80
        },
        {
            ip: "0.0.0.0"
            port: 443
        }
    ]
    https_options: ["tcplog", "dontlognull", "httplog"]
    https_log_format: "%H %ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"
    backends: [
        {
            name: "web_backend"
            ssl_sni: "example.com"
            mode: "http"
            balance: "roundrobin"
            option: "httpchk GET /health"
            server_name: "web1"
            server_host_ip: "10.0.1.10"
            server_port: 8080
            server_ops: "check fall 3 rise 2"
        }
    ]
}

Production HTTPS Load Balancer

proxy: Proxy = {
    proxy_version: "2.8"
    proxy_lib: "/var/lib/haproxy"
    proxy_cfg_file: "haproxy.cfg"
    run_user: "haproxy"
    run_group: "haproxy"
    run_user_home: "/var/lib/haproxy"
    https_in_binds: [
        {
            ip: "0.0.0.0"
            port: 80
        },
        {
            ip: "0.0.0.0"
            port: 443
        }
    ]
    https_options: ["tcplog", "dontlognull", "httplog", "log-health-checks"]
    https_log_format: "%H %ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
    ssl: {
        enabled: true
        certificate_path: "/etc/ssl/haproxy"
        certificate_file: "haproxy.pem"
        protocols: "TLSv1.2 TLSv1.3"
        ciphers: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305"
        redirect_http_to_https: true
        hsts_enabled: true
        hsts_max_age: 31536000
    }
    backends: [
        {
            name: "web_backend"
            ssl_sni: "api.company.com"
            mode: "http"
            balance: "leastconn"
            option: "httpchk GET /api/health HTTP/1.1\\r\\nHost:\\ api.company.com"
            server_name: "api1"
            server_host_ip: "10.0.1.10"
            server_port: 8080
            server_ops: "check fall 3 rise 2 weight 100"
        },
        {
            name: "web_backend"
            ssl_sni: "api.company.com"
            mode: "http"
            balance: "leastconn"
            option: "httpchk GET /api/health HTTP/1.1\\r\\nHost:\\ api.company.com"
            server_name: "api2"
            server_host_ip: "10.0.1.11"
            server_port: 8080
            server_ops: "check fall 3 rise 2 weight 100"
        },
        {
            name: "web_backend"
            ssl_sni: "api.company.com"
            mode: "http"
            balance: "leastconn"
            option: "httpchk GET /api/health HTTP/1.1\\r\\nHost:\\ api.company.com"
            server_name: "api3"
            server_host_ip: "10.0.1.12"
            server_port: 8080
            server_ops: "check fall 3 rise 2 weight 50 backup"
        }
    ]
    performance: {
        maxconn: 4096
        nbproc: 4
        cpu_map: "auto"
        tune_ssl_default_dh_param: 2048
        tune_bufsize: 32768
        tune_maxrewrite: 8192
    }
}

Multi-Service Load Balancer

proxy: Proxy = {
    proxy_version: "2.8"
    # ... base configuration
    https_in_binds: [
        {
            ip: "0.0.0.0"
            port: 80
        },
        {
            ip: "0.0.0.0"
            port: 443
        }
    ]
    backends: [
        {
            name: "api_backend"
            ssl_sni: "api.company.com"
            mode: "http"
            balance: "roundrobin"
            option: "httpchk GET /health"
            server_name: "api1"
            server_host_ip: "10.0.1.10"
            server_port: 3000
            server_ops: "check fall 3 rise 2"
        },
        {
            name: "api_backend"
            ssl_sni: "api.company.com"
            mode: "http"
            balance: "roundrobin"
            option: "httpchk GET /health"
            server_name: "api2"
            server_host_ip: "10.0.1.11"
            server_port: 3000
            server_ops: "check fall 3 rise 2"
        },
        {
            name: "web_backend"
            ssl_sni: "www.company.com"
            mode: "http"
            balance: "source"
            option: "httpchk GET /"
            server_name: "web1"
            server_host_ip: "10.0.2.10"
            server_port: 80
            server_ops: "check fall 3 rise 2"
        },
        {
            name: "web_backend"
            ssl_sni: "www.company.com"
            mode: "http"
            balance: "source"
            option: "httpchk GET /"
            server_name: "web2"
            server_host_ip: "10.0.2.11"
            server_port: 80
            server_ops: "check fall 3 rise 2"
        }
    ]
    routing_rules: [
        {
            condition: "hdr(host) -i api.company.com"
            backend: "api_backend"
        },
        {
            condition: "hdr(host) -i www.company.com"
            backend: "web_backend"
        },
        {
            condition: "path_beg /api/"
            backend: "api_backend"
        }
    ]
}

TCP Load Balancer for Databases

proxy: Proxy = {
    proxy_version: "2.8"
    # ... base configuration
    https_in_binds: [
        {
            ip: "0.0.0.0"
            port: 5432
        },
        {
            ip: "0.0.0.0"
            port: 3306
        }
    ]
    https_options: ["tcplog", "dontlognull"]
    backends: [
        {
            name: "postgres_backend"
            ssl_sni: "postgres.company.com"
            mode: "tcp"
            balance: "leastconn"
            option: "tcp-check"
            server_name: "postgres1"
            server_host_ip: "10.0.3.10"
            server_port: 5432
            server_ops: "check fall 3 rise 2"
        },
        {
            name: "postgres_backend"
            ssl_sni: "postgres.company.com"
            mode: "tcp"
            balance: "leastconn"
            option: "tcp-check"
            server_name: "postgres2"
            server_host_ip: "10.0.3.11"
            server_port: 5432
            server_ops: "check fall 3 rise 2 backup"
        },
        {
            name: "mysql_backend"
            ssl_sni: "mysql.company.com"
            mode: "tcp"
            balance: "source"
            option: "mysql-check user haproxy"
            server_name: "mysql1"
            server_host_ip: "10.0.4.10"
            server_port: 3306
            server_ops: "check fall 3 rise 2"
        }
    ]
    tcp_services: [
        {
            bind_port: 5432
            backend: "postgres_backend"
        },
        {
            bind_port: 3306
            backend: "mysql_backend"
        }
    ]
}

High-Availability Configuration

proxy: Proxy = {
    proxy_version: "2.8"
    # ... base configuration
    ha_config: {
        keepalived: {
            enabled: true
            virtual_ip: "10.0.0.100"
            interface: "eth0"
            priority: 100  # Master: 100, Backup: 90
            advert_int: 1
            auth_pass: "haproxy_vip_password"
        }
        peers: [
            {
                name: "haproxy1"
                ip: "10.0.0.10"
            },
            {
                name: "haproxy2"
                ip: "10.0.0.11"
            }
        ]
        stick_tables: true
        session_synchronization: true
    }
    monitoring: {
        stats: {
            enabled: true
            bind_ip: "127.0.0.1"
            bind_port: 8404
            uri: "/stats"
            username: "admin"
            password: "admin123"
            refresh: 30
        }
        prometheus: {
            enabled: true
            bind_ip: "127.0.0.1"
            bind_port: 8405
            uri: "/metrics"
        }
        health_checks: {
            enabled: true
            log_health_checks: true
            email_alerts: "admin@company.com"
        }
    }
}

Usage

Deploy HAProxy

./core/nulib/provisioning taskserv create proxy --infra <infrastructure-name>

List Available Task Services

./core/nulib/provisioning taskserv list

SSH to Proxy Server

./core/nulib/provisioning server ssh <proxy-server>

Service Management

# Check HAProxy status
systemctl status haproxy

# Start/stop HAProxy
systemctl start haproxy
systemctl stop haproxy
systemctl restart haproxy

# Reload configuration without downtime
systemctl reload haproxy

# Check HAProxy version
haproxy -v

Configuration Management

# Test configuration syntax
haproxy -c -f /etc/haproxy/haproxy.cfg

# Check configuration with detailed output
haproxy -c -V -f /etc/haproxy/haproxy.cfg

# Reload configuration gracefully
sudo haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

# View current configuration
cat /etc/haproxy/haproxy.cfg

Statistics and Monitoring

# Access statistics via command line
echo "show info; show stat" | socat stdio /var/lib/haproxy/stats

# View current sessions
echo "show sess" | socat stdio /var/lib/haproxy/stats

# Show backend servers status
echo "show servers state" | socat stdio /var/lib/haproxy/stats

# Disable/enable server
echo "disable server backend/server1" | socat stdio /var/lib/haproxy/stats
echo "enable server backend/server1" | socat stdio /var/lib/haproxy/stats

SSL Certificate Management

# Create combined certificate file
cat /etc/ssl/certs/company.crt /etc/ssl/private/company.key > /etc/ssl/haproxy/haproxy.pem

# Set proper permissions
chmod 600 /etc/ssl/haproxy/haproxy.pem
chown haproxy:haproxy /etc/ssl/haproxy/haproxy.pem

# Test SSL configuration
openssl s_client -connect localhost:443 -servername company.com

# Check certificate expiration
openssl x509 -in /etc/ssl/haproxy/haproxy.pem -noout -dates

Performance Tuning

# Check current connections
echo "show info" | socat stdio /var/lib/haproxy/stats | grep -E "(CurrConns|MaxConns)"

# Monitor connection rates
echo "show info" | socat stdio /var/lib/haproxy/stats | grep -E "(ConnRate|SessRate)"

# Check memory usage
ps aux | grep haproxy
cat /proc/$(pgrep haproxy)/status | grep -E "(VmSize|VmRSS)"

# Monitor network I/O
iftop -i eth0 -f "port 80 or port 443"

Log Analysis

# View real-time access logs
tail -f /var/log/haproxy/access.log

# Analyze response times
awk '{print $10}' /var/log/haproxy/access.log | sort -n | tail -10

# Count status codes
awk '{print $11}' /var/log/haproxy/access.log | sort | uniq -c

# Top client IPs
awk '{print $6}' /var/log/haproxy/access.log | cut -d: -f1 | sort | uniq -c | sort -nr | head -10

Architecture

System Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Clients       │────│   HAProxy        │────│   Backend       │
│                 │    │                  │    │   Servers       │
│ • Web Browsers  │    │ • Load Balancer  │    │                 │
│ • Mobile Apps   │────│ • SSL Termination│────│ • Web Servers   │
│ • API Clients   │    │ • Health Checks  │    │ • App Servers   │
│ • Load Testing  │    │ • Rate Limiting  │    │ • Databases     │
└─────────────────┘    └──────────────────┘    └─────────────────┘

High-Availability Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Virtual IP    │    │   HAProxy        │    │   Backend       │
│   (Keepalived)  │    │   Cluster        │    │   Pool          │
│                 │    │                  │    │                 │
│ • 10.0.0.100    │────│ • Master (Active)│────│ • Server 1      │
│ • Failover      │    │ • Backup (Standby│    │ • Server 2      │
│ • Health Check  │    │ • Sync Sessions  │    │ • Server 3      │
└─────────────────┘    └──────────────────┘    └─────────────────┘

Request Flow Architecture

Client Request → Frontend → ACL Rules → Backend Selection → Health Check → Server Selection → Response
     ↓              ↓           ↓             ↓               ↓              ↓           ↓
SSL Termination → Routing → Load Balancing → Failover → Server Response → SSL → Client
     ↓              ↓           ↓             ↓               ↓              ↓           ↓
Certificate    → Headers → Session Persistence → Backup Server → Compression → Headers → Browser

File Structure

/etc/haproxy/               # Configuration directory
├── haproxy.cfg            # Main configuration file
├── errors/               # Custom error pages
│   ├── 400.http         # Bad request error page
│   ├── 403.http         # Forbidden error page
│   ├── 408.http         # Request timeout
│   ├── 500.http         # Internal server error
│   ├── 502.http         # Bad gateway
│   ├── 503.http         # Service unavailable
│   └── 504.http         # Gateway timeout
└── certs/               # SSL certificates
    └── haproxy.pem     # Combined certificate file

/var/lib/haproxy/          # Runtime directory
├── stats                 # Statistics socket
└── info                 # Runtime information

/var/log/haproxy/         # Log directory
├── access.log           # Access logs
├── error.log            # Error logs
└── haproxy.log          # Combined logs

/run/haproxy/             # Process runtime
└── haproxy.pid          # Process ID file

Supported Operating Systems

• Ubuntu 20.04+ / Debian 11+
• CentOS 8+ / RHEL 8+ / Fedora 35+
• Amazon Linux 2+
• SUSE Linux Enterprise 15+

System Requirements

Minimum Requirements

• RAM: 1GB (2GB+ recommended)
• Storage: 10GB (20GB+ for logs)
• CPU: 2 cores (4+ cores recommended)
• Network: 100Mbps (1Gbps+ for high load)

Production Requirements

• RAM: 4GB+ (8GB+ for high concurrency)
• Storage: 50GB+ SSD
• CPU: 4+ cores (16+ cores for very high load)
• Network: 1Gbps+ with low latency

Performance Requirements

• Network Bandwidth: Adequate for peak traffic
• CPU Performance: High single-thread performance
• Memory: Sufficient for connection state and SSL
• Disk I/O: Fast storage for logging

Troubleshooting

Service Issues

# Check HAProxy status
systemctl status haproxy

# Test configuration
haproxy -c -f /etc/haproxy/haproxy.cfg

# View error logs
tail -f /var/log/haproxy/error.log

# Check process information
ps aux | grep haproxy

Connection Issues

# Check listening ports
netstat -tlnp | grep haproxy
ss -tlnp | grep haproxy

# Test frontend connectivity
curl -I http://localhost/
telnet localhost 80

# Check backend connectivity
curl -I http://backend-server:8080/health

# Monitor active connections
echo "show info" | socat stdio /var/lib/haproxy/stats

SSL Issues

# Test SSL connectivity
openssl s_client -connect localhost:443

# Check certificate validity
openssl x509 -in /etc/ssl/haproxy/haproxy.pem -noout -text

# Verify certificate chain
openssl verify -CApath /etc/ssl/certs /etc/ssl/haproxy/haproxy.pem

# Check SSL logs
grep -i ssl /var/log/haproxy/error.log

Performance Issues

# Check HAProxy statistics
echo "show info; show stat" | socat stdio /var/lib/haproxy/stats

# Monitor system resources
htop
iostat -x 1
iftop -i eth0

# Check connection limits
ulimit -n
cat /proc/sys/net/core/somaxconn

# Analyze access patterns
tail -f /var/log/haproxy/access.log | awk '{print $6, $11, $10}'

Backend Health Issues

# Check backend server status
echo "show servers state" | socat stdio /var/lib/haproxy/stats

# Test backend health checks
curl -I http://backend-server:8080/health

# Enable/disable servers
echo "enable server backend/server1" | socat stdio /var/lib/haproxy/stats
echo "disable server backend/server1" | socat stdio /var/lib/haproxy/stats

# Check health check logs
grep "Health check" /var/log/haproxy/error.log

Security Considerations

SSL/TLS Security

• Strong Ciphers - Use modern, secure cipher suites
• Protocol Versions - Disable older TLS versions
• Certificate Management - Regular certificate renewal
• Perfect Forward Secrecy - Enable PFS for all connections

Access Control

• IP Whitelisting - Restrict admin access by IP
• Rate Limiting - Implement request rate limiting
• DDoS Protection - Configure connection and rate limits
• Firewall Rules - Limit access to necessary ports

Configuration Security

• Secure Headers - Add security headers to responses
• Error Page Security - Don't expose internal information
• Log Security - Secure log files and prevent log injection
• Process Security - Run with minimum required privileges

Network Security

• Network Segmentation - Isolate proxy and backend networks
• Monitoring - Monitor for suspicious traffic patterns
• Regular Updates - Keep HAProxy updated to latest version
• Security Audits - Regular security configuration reviews

Performance Optimization

Hardware Optimization

• CPU - High single-thread performance for SSL termination
• Memory - Adequate RAM for connection state and buffers
• Network - High-bandwidth, low-latency network interfaces
• Storage - Fast storage for logging and certificates

Configuration Optimization

• Connection Limits - Optimize maxconn and server limits
• Buffer Sizes - Tune buffer sizes for your workload
• SSL Optimization - Optimize SSL session caching
• Health Check Intervals - Balance responsiveness and overhead

System Optimization

• Kernel Parameters - Tune TCP/IP stack parameters
• File Descriptors - Increase ulimit for connections
• CPU Affinity - Bind processes to specific CPU cores
• Memory Management - Optimize memory allocation

Load Balancing Optimization

• Algorithm Selection - Choose optimal load balancing algorithm
• Health Checks - Efficient health check configuration
• Session Persistence - Optimize sticky session handling
• Backend Weights - Balance load based on server capacity

Integration Examples

Nginx Integration (Frontend Proxy)

upstream haproxy_backend {
    server 10.0.1.10:80;
    server 10.0.1.11:80;
}

server {
    listen 80;
    server_name company.com;
    
    location / {
        proxy_pass http://haproxy_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Keepalived Configuration

# /etc/keepalived/keepalived.conf
vrrp_script chk_haproxy {
    script "/bin/kill -0 `cat /var/run/haproxy.pid`"
    interval 2
    weight 2
    fall 3
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass haproxy_pass
    }
    virtual_ipaddress {
        10.0.0.100
    }
    track_script {
        chk_haproxy
    }
}

Prometheus Monitoring

# prometheus.yml
scrape_configs:
  - job_name: 'haproxy'
    static_configs:
      - targets: ['localhost:8405']
    scrape_interval: 30s
    metrics_path: '/metrics'

Resources

• Official Documentation: docs.haproxy.org
• HAProxy Community: discourse.haproxy.org
• Configuration Generator: haproxytech.github.io/haproxy-dconv
• Best Practices: haproxy.com/blog
• GitHub Repository: haproxy/haproxy