2025-10-07 11:05:08 +01:00

760 lines
22 KiB
Markdown

# Proxy Task Service (HAProxy)
## Overview
The Proxy task service provides a complete installation and configuration of [HAProxy](https://www.haproxy.org/), a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. HAProxy is particularly suited for very high traffic web sites and is the de-facto standard open-source load balancer.
## Features
### Core Proxy Features
- **Layer 4 & Layer 7 Load Balancing** - TCP and HTTP/HTTPS traffic distribution
- **High Availability** - Active/passive and active/active configurations
- **SSL/TLS Termination** - SSL offloading and end-to-end encryption
- **Content-Based Routing** - Route based on URLs, headers, and other criteria
- **Session Persistence** - Sticky sessions and session affinity
### Load Balancing Algorithms
- **Round Robin** - Distribute requests evenly across servers
- **Least Connections** - Route to server with fewest active connections
- **Weighted Round Robin** - Assign different weights to servers
- **Source IP Hash** - Route based on client IP hash
- **URL Hash** - Route based on URL hash for cache optimization
### Health Checking & Monitoring
- **Health Checks** - TCP, HTTP, and custom health checks
- **Server Status Monitoring** - Real-time server status and metrics
- **Statistics Interface** - Built-in web statistics interface
- **Prometheus Metrics** - Native Prometheus metrics export
- **Logging** - Comprehensive access and error logging
### Security Features
- **DDoS Protection** - Rate limiting and connection limits
- **Access Control** - IP-based access control lists
- **SSL Security** - Modern TLS configuration and cipher suites
- **Request Filtering** - Block malicious requests and patterns
- **Security Headers** - Automatic security header injection
### Advanced Features
- **Compression** - HTTP response compression
- **Caching** - Basic HTTP caching capabilities
- **Request Modification** - Header manipulation and URL rewriting
- **Multi-Process Mode** - Multi-process for high concurrency
- **Configuration Validation** - Real-time configuration validation
## Configuration
### Basic HTTP Load Balancer
```kcl
proxy: Proxy = {
proxy_version: "2.8"
proxy_lib: "/var/lib/haproxy"
proxy_cfg_file: "haproxy.cfg"
run_user: "haproxy"
run_group: "haproxy"
run_user_home: "/home/haproxy"
https_in_binds: [
{
ip: "0.0.0.0"
port: 80
},
{
ip: "0.0.0.0"
port: 443
}
]
https_options: ["tcplog", "dontlognull", "httplog"]
https_log_format: "%H %ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"
backends: [
{
name: "web_backend"
ssl_sni: "example.com"
mode: "http"
balance: "roundrobin"
option: "httpchk GET /health"
server_name: "web1"
server_host_ip: "10.0.1.10"
server_port: 8080
server_ops: "check fall 3 rise 2"
}
]
}
```
### Production HTTPS Load Balancer
```kcl
proxy: Proxy = {
proxy_version: "2.8"
proxy_lib: "/var/lib/haproxy"
proxy_cfg_file: "haproxy.cfg"
run_user: "haproxy"
run_group: "haproxy"
run_user_home: "/var/lib/haproxy"
https_in_binds: [
{
ip: "0.0.0.0"
port: 80
},
{
ip: "0.0.0.0"
port: 443
}
]
https_options: ["tcplog", "dontlognull", "httplog", "log-health-checks"]
https_log_format: "%H %ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
ssl: {
enabled: true
certificate_path: "/etc/ssl/haproxy"
certificate_file: "haproxy.pem"
protocols: "TLSv1.2 TLSv1.3"
ciphers: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305"
redirect_http_to_https: true
hsts_enabled: true
hsts_max_age: 31536000
}
backends: [
{
name: "web_backend"
ssl_sni: "api.company.com"
mode: "http"
balance: "leastconn"
option: "httpchk GET /api/health HTTP/1.1\\r\\nHost:\\ api.company.com"
server_name: "api1"
server_host_ip: "10.0.1.10"
server_port: 8080
server_ops: "check fall 3 rise 2 weight 100"
},
{
name: "web_backend"
ssl_sni: "api.company.com"
mode: "http"
balance: "leastconn"
option: "httpchk GET /api/health HTTP/1.1\\r\\nHost:\\ api.company.com"
server_name: "api2"
server_host_ip: "10.0.1.11"
server_port: 8080
server_ops: "check fall 3 rise 2 weight 100"
},
{
name: "web_backend"
ssl_sni: "api.company.com"
mode: "http"
balance: "leastconn"
option: "httpchk GET /api/health HTTP/1.1\\r\\nHost:\\ api.company.com"
server_name: "api3"
server_host_ip: "10.0.1.12"
server_port: 8080
server_ops: "check fall 3 rise 2 weight 50 backup"
}
]
performance: {
maxconn: 4096
nbproc: 4
cpu_map: "auto"
tune_ssl_default_dh_param: 2048
tune_bufsize: 32768
tune_maxrewrite: 8192
}
}
```
### Multi-Service Load Balancer
```kcl
proxy: Proxy = {
proxy_version: "2.8"
# ... base configuration
https_in_binds: [
{
ip: "0.0.0.0"
port: 80
},
{
ip: "0.0.0.0"
port: 443
}
]
backends: [
{
name: "api_backend"
ssl_sni: "api.company.com"
mode: "http"
balance: "roundrobin"
option: "httpchk GET /health"
server_name: "api1"
server_host_ip: "10.0.1.10"
server_port: 3000
server_ops: "check fall 3 rise 2"
},
{
name: "api_backend"
ssl_sni: "api.company.com"
mode: "http"
balance: "roundrobin"
option: "httpchk GET /health"
server_name: "api2"
server_host_ip: "10.0.1.11"
server_port: 3000
server_ops: "check fall 3 rise 2"
},
{
name: "web_backend"
ssl_sni: "www.company.com"
mode: "http"
balance: "source"
option: "httpchk GET /"
server_name: "web1"
server_host_ip: "10.0.2.10"
server_port: 80
server_ops: "check fall 3 rise 2"
},
{
name: "web_backend"
ssl_sni: "www.company.com"
mode: "http"
balance: "source"
option: "httpchk GET /"
server_name: "web2"
server_host_ip: "10.0.2.11"
server_port: 80
server_ops: "check fall 3 rise 2"
}
]
routing_rules: [
{
condition: "hdr(host) -i api.company.com"
backend: "api_backend"
},
{
condition: "hdr(host) -i www.company.com"
backend: "web_backend"
},
{
condition: "path_beg /api/"
backend: "api_backend"
}
]
}
```
### TCP Load Balancer for Databases
```kcl
proxy: Proxy = {
proxy_version: "2.8"
# ... base configuration
https_in_binds: [
{
ip: "0.0.0.0"
port: 5432
},
{
ip: "0.0.0.0"
port: 3306
}
]
https_options: ["tcplog", "dontlognull"]
backends: [
{
name: "postgres_backend"
ssl_sni: "postgres.company.com"
mode: "tcp"
balance: "leastconn"
option: "tcp-check"
server_name: "postgres1"
server_host_ip: "10.0.3.10"
server_port: 5432
server_ops: "check fall 3 rise 2"
},
{
name: "postgres_backend"
ssl_sni: "postgres.company.com"
mode: "tcp"
balance: "leastconn"
option: "tcp-check"
server_name: "postgres2"
server_host_ip: "10.0.3.11"
server_port: 5432
server_ops: "check fall 3 rise 2 backup"
},
{
name: "mysql_backend"
ssl_sni: "mysql.company.com"
mode: "tcp"
balance: "source"
option: "mysql-check user haproxy"
server_name: "mysql1"
server_host_ip: "10.0.4.10"
server_port: 3306
server_ops: "check fall 3 rise 2"
}
]
tcp_services: [
{
bind_port: 5432
backend: "postgres_backend"
},
{
bind_port: 3306
backend: "mysql_backend"
}
]
}
```
### High-Availability Configuration
```kcl
proxy: Proxy = {
proxy_version: "2.8"
# ... base configuration
ha_config: {
keepalived: {
enabled: true
virtual_ip: "10.0.0.100"
interface: "eth0"
priority: 100 # Master: 100, Backup: 90
advert_int: 1
auth_pass: "haproxy_vip_password"
}
peers: [
{
name: "haproxy1"
ip: "10.0.0.10"
},
{
name: "haproxy2"
ip: "10.0.0.11"
}
]
stick_tables: true
session_synchronization: true
}
monitoring: {
stats: {
enabled: true
bind_ip: "127.0.0.1"
bind_port: 8404
uri: "/stats"
username: "admin"
password: "admin123"
refresh: 30
}
prometheus: {
enabled: true
bind_ip: "127.0.0.1"
bind_port: 8405
uri: "/metrics"
}
health_checks: {
enabled: true
log_health_checks: true
email_alerts: "admin@company.com"
}
}
}
```
## Usage
### Deploy HAProxy
```bash
./core/nulib/provisioning taskserv create proxy --infra <infrastructure-name>
```
### List Available Task Services
```bash
./core/nulib/provisioning taskserv list
```
### SSH to Proxy Server
```bash
./core/nulib/provisioning server ssh <proxy-server>
```
### Service Management
```bash
# Check HAProxy status
systemctl status haproxy
# Start/stop HAProxy
systemctl start haproxy
systemctl stop haproxy
systemctl restart haproxy
# Reload configuration without downtime
systemctl reload haproxy
# Check HAProxy version
haproxy -v
```
### Configuration Management
```bash
# Test configuration syntax
haproxy -c -f /etc/haproxy/haproxy.cfg
# Check configuration with detailed output
haproxy -c -V -f /etc/haproxy/haproxy.cfg
# Reload configuration gracefully
sudo haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
# View current configuration
cat /etc/haproxy/haproxy.cfg
```
### Statistics and Monitoring
```bash
# Access statistics via command line
echo "show info; show stat" | socat stdio /var/lib/haproxy/stats
# View current sessions
echo "show sess" | socat stdio /var/lib/haproxy/stats
# Show backend servers status
echo "show servers state" | socat stdio /var/lib/haproxy/stats
# Disable/enable server
echo "disable server backend/server1" | socat stdio /var/lib/haproxy/stats
echo "enable server backend/server1" | socat stdio /var/lib/haproxy/stats
```
### SSL Certificate Management
```bash
# Create combined certificate file
cat /etc/ssl/certs/company.crt /etc/ssl/private/company.key > /etc/ssl/haproxy/haproxy.pem
# Set proper permissions
chmod 600 /etc/ssl/haproxy/haproxy.pem
chown haproxy:haproxy /etc/ssl/haproxy/haproxy.pem
# Test SSL configuration
openssl s_client -connect localhost:443 -servername company.com
# Check certificate expiration
openssl x509 -in /etc/ssl/haproxy/haproxy.pem -noout -dates
```
### Performance Tuning
```bash
# Check current connections
echo "show info" | socat stdio /var/lib/haproxy/stats | grep -E "(CurrConns|MaxConns)"
# Monitor connection rates
echo "show info" | socat stdio /var/lib/haproxy/stats | grep -E "(ConnRate|SessRate)"
# Check memory usage
ps aux | grep haproxy
cat /proc/$(pgrep haproxy)/status | grep -E "(VmSize|VmRSS)"
# Monitor network I/O
iftop -i eth0 -f "port 80 or port 443"
```
### Log Analysis
```bash
# View real-time access logs
tail -f /var/log/haproxy/access.log
# Analyze response times
awk '{print $10}' /var/log/haproxy/access.log | sort -n | tail -10
# Count status codes
awk '{print $11}' /var/log/haproxy/access.log | sort | uniq -c
# Top client IPs
awk '{print $6}' /var/log/haproxy/access.log | cut -d: -f1 | sort | uniq -c | sort -nr | head -10
```
## Architecture
### System Architecture
```
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Clients │────│ HAProxy │────│ Backend │
│ │ │ │ │ Servers │
│ • Web Browsers │ │ • Load Balancer │ │ │
│ • Mobile Apps │────│ • SSL Termination│────│ • Web Servers │
│ • API Clients │ │ • Health Checks │ │ • App Servers │
│ • Load Testing │ │ • Rate Limiting │ │ • Databases │
└─────────────────┘ └──────────────────┘ └─────────────────┘
```
### High-Availability Architecture
```
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Virtual IP │ │ HAProxy │ │ Backend │
│ (Keepalived) │ │ Cluster │ │ Pool │
│ │ │ │ │ │
│ • 10.0.0.100 │────│ • Master (Active)│────│ • Server 1 │
│ • Failover │ │ • Backup (Standby│ │ • Server 2 │
│ • Health Check │ │ • Sync Sessions │ │ • Server 3 │
└─────────────────┘ └──────────────────┘ └─────────────────┘
```
### Request Flow Architecture
```
Client Request → Frontend → ACL Rules → Backend Selection → Health Check → Server Selection → Response
↓ ↓ ↓ ↓ ↓ ↓ ↓
SSL Termination → Routing → Load Balancing → Failover → Server Response → SSL → Client
↓ ↓ ↓ ↓ ↓ ↓ ↓
Certificate → Headers → Session Persistence → Backup Server → Compression → Headers → Browser
```
### File Structure
```
/etc/haproxy/ # Configuration directory
├── haproxy.cfg # Main configuration file
├── errors/ # Custom error pages
│ ├── 400.http # Bad request error page
│ ├── 403.http # Forbidden error page
│ ├── 408.http # Request timeout
│ ├── 500.http # Internal server error
│ ├── 502.http # Bad gateway
│ ├── 503.http # Service unavailable
│ └── 504.http # Gateway timeout
└── certs/ # SSL certificates
└── haproxy.pem # Combined certificate file
/var/lib/haproxy/ # Runtime directory
├── stats # Statistics socket
└── info # Runtime information
/var/log/haproxy/ # Log directory
├── access.log # Access logs
├── error.log # Error logs
└── haproxy.log # Combined logs
/run/haproxy/ # Process runtime
└── haproxy.pid # Process ID file
```
## Supported Operating Systems
- Ubuntu 20.04+ / Debian 11+
- CentOS 8+ / RHEL 8+ / Fedora 35+
- Amazon Linux 2+
- SUSE Linux Enterprise 15+
## System Requirements
### Minimum Requirements
- **RAM**: 1GB (2GB+ recommended)
- **Storage**: 10GB (20GB+ for logs)
- **CPU**: 2 cores (4+ cores recommended)
- **Network**: 100Mbps (1Gbps+ for high load)
### Production Requirements
- **RAM**: 4GB+ (8GB+ for high concurrency)
- **Storage**: 50GB+ SSD
- **CPU**: 4+ cores (16+ cores for very high load)
- **Network**: 1Gbps+ with low latency
### Performance Requirements
- **Network Bandwidth**: Adequate for peak traffic
- **CPU Performance**: High single-thread performance
- **Memory**: Sufficient for connection state and SSL
- **Disk I/O**: Fast storage for logging
## Troubleshooting
### Service Issues
```bash
# Check HAProxy status
systemctl status haproxy
# Test configuration
haproxy -c -f /etc/haproxy/haproxy.cfg
# View error logs
tail -f /var/log/haproxy/error.log
# Check process information
ps aux | grep haproxy
```
### Connection Issues
```bash
# Check listening ports
netstat -tlnp | grep haproxy
ss -tlnp | grep haproxy
# Test frontend connectivity
curl -I http://localhost/
telnet localhost 80
# Check backend connectivity
curl -I http://backend-server:8080/health
# Monitor active connections
echo "show info" | socat stdio /var/lib/haproxy/stats
```
### SSL Issues
```bash
# Test SSL connectivity
openssl s_client -connect localhost:443
# Check certificate validity
openssl x509 -in /etc/ssl/haproxy/haproxy.pem -noout -text
# Verify certificate chain
openssl verify -CApath /etc/ssl/certs /etc/ssl/haproxy/haproxy.pem
# Check SSL logs
grep -i ssl /var/log/haproxy/error.log
```
### Performance Issues
```bash
# Check HAProxy statistics
echo "show info; show stat" | socat stdio /var/lib/haproxy/stats
# Monitor system resources
htop
iostat -x 1
iftop -i eth0
# Check connection limits
ulimit -n
cat /proc/sys/net/core/somaxconn
# Analyze access patterns
tail -f /var/log/haproxy/access.log | awk '{print $6, $11, $10}'
```
### Backend Health Issues
```bash
# Check backend server status
echo "show servers state" | socat stdio /var/lib/haproxy/stats
# Test backend health checks
curl -I http://backend-server:8080/health
# Enable/disable servers
echo "enable server backend/server1" | socat stdio /var/lib/haproxy/stats
echo "disable server backend/server1" | socat stdio /var/lib/haproxy/stats
# Check health check logs
grep "Health check" /var/log/haproxy/error.log
```
## Security Considerations
### SSL/TLS Security
- **Strong Ciphers** - Use modern, secure cipher suites
- **Protocol Versions** - Disable older TLS versions
- **Certificate Management** - Regular certificate renewal
- **Perfect Forward Secrecy** - Enable PFS for all connections
### Access Control
- **IP Whitelisting** - Restrict admin access by IP
- **Rate Limiting** - Implement request rate limiting
- **DDoS Protection** - Configure connection and rate limits
- **Firewall Rules** - Limit access to necessary ports
### Configuration Security
- **Secure Headers** - Add security headers to responses
- **Error Page Security** - Don't expose internal information
- **Log Security** - Secure log files and prevent log injection
- **Process Security** - Run with minimum required privileges
### Network Security
- **Network Segmentation** - Isolate proxy and backend networks
- **Monitoring** - Monitor for suspicious traffic patterns
- **Regular Updates** - Keep HAProxy updated to latest version
- **Security Audits** - Regular security configuration reviews
## Performance Optimization
### Hardware Optimization
- **CPU** - High single-thread performance for SSL termination
- **Memory** - Adequate RAM for connection state and buffers
- **Network** - High-bandwidth, low-latency network interfaces
- **Storage** - Fast storage for logging and certificates
### Configuration Optimization
- **Connection Limits** - Optimize maxconn and server limits
- **Buffer Sizes** - Tune buffer sizes for your workload
- **SSL Optimization** - Optimize SSL session caching
- **Health Check Intervals** - Balance responsiveness and overhead
### System Optimization
- **Kernel Parameters** - Tune TCP/IP stack parameters
- **File Descriptors** - Increase ulimit for connections
- **CPU Affinity** - Bind processes to specific CPU cores
- **Memory Management** - Optimize memory allocation
### Load Balancing Optimization
- **Algorithm Selection** - Choose optimal load balancing algorithm
- **Health Checks** - Efficient health check configuration
- **Session Persistence** - Optimize sticky session handling
- **Backend Weights** - Balance load based on server capacity
## Integration Examples
### Nginx Integration (Frontend Proxy)
```nginx
upstream haproxy_backend {
server 10.0.1.10:80;
server 10.0.1.11:80;
}
server {
listen 80;
server_name company.com;
location / {
proxy_pass http://haproxy_backend;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
```
### Keepalived Configuration
```bash
# /etc/keepalived/keepalived.conf
vrrp_script chk_haproxy {
script "/bin/kill -0 `cat /var/run/haproxy.pid`"
interval 2
weight 2
fall 3
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 101
advert_int 1
authentication {
auth_type PASS
auth_pass haproxy_pass
}
virtual_ipaddress {
10.0.0.100
}
track_script {
chk_haproxy
}
}
```
### Prometheus Monitoring
```yaml
# prometheus.yml
scrape_configs:
- job_name: 'haproxy'
static_configs:
- targets: ['localhost:8405']
scrape_interval: 30s
metrics_path: '/metrics'
```
## Resources
- **Official Documentation**: [docs.haproxy.org](https://docs.haproxy.org/)
- **HAProxy Community**: [discourse.haproxy.org](https://discourse.haproxy.org/)
- **Configuration Generator**: [haproxytech.github.io/haproxy-dconv](https://haproxytech.github.io/haproxy-dconv/)
- **Best Practices**: [haproxy.com/blog](https://www.haproxy.com/blog/)
- **GitHub Repository**: [haproxy/haproxy](https://github.com/haproxy/haproxy)