Rustelo/info/deployment.md

Deployment Guide

This guide covers the deployment of the Rustelo application with comprehensive monitoring, health checks, and CI/CD pipeline setup.

Table of Contents

1. Overview
2. Prerequisites
3. Docker Setup
4. Health Checks
5. Metrics and Monitoring
6. CI/CD Pipeline
7. Deployment Commands
8. Production Deployment
9. Troubleshooting

Overview

The Rustelo application includes the following deployment features:

• Docker Containerization: Multi-stage builds with optimized production images
• Health Check Endpoints: Kubernetes-compatible liveness and readiness probes
• Prometheus Metrics: Comprehensive application and system metrics
• GitHub Actions CI/CD: Automated testing, building, and deployment
• Grafana Dashboards: Pre-configured monitoring dashboards

Prerequisites

System Requirements

• Docker: Version 20.0+ with Docker Compose
• Node.js: Version 18+ (for frontend builds)
• Git: For version control and CI/CD
• curl: For health checks and API testing

Optional (for monitoring)

• Prometheus: For metrics collection
• Grafana: For visualization
• PostgreSQL: For production database
• Redis: For caching and sessions

Docker Setup

1. Basic Development Setup

# Clone the repository
git clone <repository-url>
cd rustelo

# Start development environment
docker-compose up -d

# View logs
docker-compose logs -f

2. Production Setup

# Build and deploy production
./deploy.sh deploy -e production --migrate --backup

# Check deployment status
./deploy.sh status

# View application logs
./deploy.sh logs -f

3. Environment-Specific Configurations

Development

# Use development profile
docker-compose --profile dev up -d

# Or use the deploy script
./deploy.sh deploy -e development

Staging

# Deploy to staging
./deploy.sh deploy -e staging --migrate

Production

# Deploy to production with full monitoring
./deploy.sh deploy -e production --migrate --backup

Health Checks

The application provides comprehensive health check endpoints:

Endpoints

Endpoint	Purpose	Description
/health	Comprehensive health check	Checks all components (database, auth, content, email, system)
/health/live	Liveness probe	Simple check if application is running
/health/ready	Readiness probe	Checks if application can handle traffic

Example Health Check Response

{
  "status": "healthy",
  "timestamp": "2024-01-15T10:30:00Z",
  "version": "0.1.0",
  "environment": "production",
  "uptime_seconds": 3600,
  "components": [
    {
      "name": "database",
      "status": "healthy",
      "message": "Database connection successful",
      "response_time_ms": 25,
      "metadata": {
        "pool_size": 10,
        "idle_connections": 8
      }
    },
    {
      "name": "auth_service",
      "status": "healthy",
      "message": "Authentication service operational",
      "response_time_ms": 12,
      "metadata": {}
    }
  ],
  "summary": {
    "healthy": 5,
    "degraded": 0,
    "unhealthy": 0
  }
}

Kubernetes Health Check Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rustelo-app
spec:
  template:
    spec:
      containers:
      - name: rustelo
        image: rustelo:latest
        ports:
        - containerPort: 3030
        livenessProbe:
          httpGet:
            path: /health/live
            port: 3030
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 3030
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3

Metrics and Monitoring

Prometheus Metrics

The application exposes metrics at /metrics endpoint with the following categories:

HTTP Metrics

• rustelo_http_requests_total: Total HTTP requests by method, path, status
• rustelo_http_request_duration_seconds: Request duration histogram
• rustelo_http_requests_in_flight: Current number of active requests

Database Metrics

• rustelo_db_connections_active: Active database connections
• rustelo_db_connections_idle: Idle database connections
• rustelo_db_queries_total: Total database queries by operation and table
• rustelo_db_query_duration_seconds: Database query duration histogram

Authentication Metrics

• rustelo_auth_requests_total: Authentication requests by type and status
• rustelo_auth_failures_total: Authentication failures by type and reason
• rustelo_auth_sessions_active: Number of active sessions
• rustelo_auth_token_generations_total: Total tokens generated

Content Metrics

• rustelo_content_requests_total: Content requests by type and status
• rustelo_content_cache_hits_total: Content cache hits
• rustelo_content_cache_misses_total: Content cache misses
• rustelo_content_processing_duration_seconds: Content processing time

System Metrics

• rustelo_memory_usage_bytes: Memory usage in bytes
• rustelo_cpu_usage_percent: CPU usage percentage
• rustelo_disk_usage_bytes: Disk usage by path
• rustelo_uptime_seconds: Application uptime

Grafana Setup

1. Start Monitoring Stack

# Start with monitoring services
docker-compose --profile monitoring up -d

# Access Grafana at http://localhost:3000
# Default credentials: admin/admin

2. Pre-configured Dashboards

• Rustelo Application Overview: Main application metrics
• System Resources: CPU, memory, disk usage
• Database Performance: Connection pool, query metrics
• Authentication Analytics: Login patterns, failures
• Content Management: Cache performance, processing times

3. Custom Metrics

You can add custom business metrics in your application:

// Record custom events
metrics.record_user_registration();
metrics.record_content_view();
metrics.record_rate_limit_hit();

Alerting Rules

Example Prometheus alerting rules:

groups:
  - name: rustelo
    rules:
      - alert: HighErrorRate
        expr: rate(rustelo_http_requests_total{status_code=~"5.."}[5m]) > 0.1
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value }} requests per second"

      - alert: DatabaseConnectionPoolExhausted
        expr: rustelo_db_connections_idle == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Database connection pool exhausted"
          description: "No idle database connections available"

      - alert: HighMemoryUsage
        expr: rustelo_memory_usage_bytes > 1000000000  # 1GB
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage"
          description: "Memory usage is {{ $value }} bytes"

CI/CD Pipeline

GitHub Actions Workflow

The CI/CD pipeline includes:

1. Test Suite: Runs on every push and PR
2. Security Audit: Vulnerability scanning
3. Docker Build: Multi-platform image building
4. Deployment: Automated deployment to staging/production

Pipeline Stages

1. Testing Stage

- name: Run tests
  run: cargo test --all-features
  env:
    DATABASE_URL: postgresql://postgres:postgres@localhost:5432/rustelo_test
    REDIS_URL: redis://localhost:6379

2. Security Stage

- name: Run security audit
  run: cargo audit

- name: Run cargo-deny
  uses: EmbarkStudios/cargo-deny-action@v1

3. Build Stage

- name: Build and push Docker image
  uses: docker/build-push-action@v5
  with:
    context: .
    platforms: linux/amd64,linux/arm64
    push: ${{ github.event_name == 'release' }}
    tags: ${{ steps.meta.outputs.tags }}
    cache-from: type=gha
    cache-to: type=gha,mode=max

4. Deployment Stage

- name: Deploy to production
  run: |
    ./deploy.sh deploy -e production --migrate --backup
    ./deploy.sh health

Required Secrets

Set these secrets in your GitHub repository:

DOCKER_USERNAME=your_docker_username
DOCKER_PASSWORD=your_docker_password
PRODUCTION_SSH_KEY=your_production_server_ssh_key
DATABASE_URL=your_production_database_url

Deployment Commands

Using the Deploy Script

# Basic deployment
./deploy.sh deploy

# Deploy with options
./deploy.sh deploy -e staging --migrate --backup

# Scale application
./deploy.sh scale -s 3

# Check status
./deploy.sh status

# View logs
./deploy.sh logs -f

# Health check
./deploy.sh health

# Update to latest
./deploy.sh update

# Stop application
./deploy.sh stop

# Clean up
./deploy.sh clean

Manual Docker Compose Commands

# Build and start services
docker-compose up -d --build

# Scale specific service
docker-compose up -d --scale app=3

# View logs
docker-compose logs -f app

# Check service status
docker-compose ps

# Stop all services
docker-compose down

# Remove volumes (WARNING: destroys data)
docker-compose down -v

Production Deployment

1. Server Preparation

# Update system
sudo apt update && sudo apt upgrade -y

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Install Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/download/v2.23.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

# Create application directory
sudo mkdir -p /opt/rustelo
cd /opt/rustelo

2. Environment Configuration

# Create production environment file
cat > .env.production << EOF
ENVIRONMENT=production
DATABASE_URL=postgresql://username:password@localhost:5432/rustelo_prod
REDIS_URL=redis://localhost:6379
RUST_LOG=info
ENABLE_METRICS=true
ENABLE_HEALTH_CHECK=true
EOF

3. SSL/TLS Setup

# Generate certificates (using Let's Encrypt)
sudo apt install certbot
sudo certbot certonly --standalone -d yourdomain.com

# Update docker-compose.yml to use certificates
# Mount certificates in nginx service

4. Database Setup

# Create production database
sudo -u postgres createdb rustelo_prod
sudo -u postgres createuser rustelo_user

# Run migrations
./deploy.sh migrate -e production

5. Monitoring Setup

# Start monitoring stack
docker-compose --profile monitoring up -d

# Configure Grafana
# 1. Open http://your-server:3000
# 2. Login with admin/admin
# 3. Import dashboards from monitoring/grafana/dashboards/

6. Backup Strategy

# Create backup script
cat > backup.sh << 'EOF'
#!/bin/bash
DATE=$(date +%Y%m%d_%H%M%S)
docker-compose exec -T db pg_dump -U postgres rustelo_prod > /opt/backups/rustelo_backup_$DATE.sql
find /opt/backups -name "rustelo_backup_*.sql" -mtime +7 -delete
EOF

# Add to crontab
crontab -e
# Add: 0 2 * * * /opt/rustelo/backup.sh

Troubleshooting

Common Issues

1. Application Won't Start

# Check container logs
docker-compose logs app

# Check system resources
docker stats

# Verify environment variables
docker-compose config

2. Database Connection Issues

# Test database connectivity
docker-compose exec app psql $DATABASE_URL -c "SELECT 1"

# Check database logs
docker-compose logs db

# Verify connection pool settings
curl http://localhost:3030/health | jq '.components[] | select(.name == "database")'

3. Health Check Failures

# Check detailed health status
curl -s http://localhost:3030/health | jq .

# Test individual components
curl -s http://localhost:3030/health/live
curl -s http://localhost:3030/health/ready

4. Performance Issues

# Check metrics
curl -s http://localhost:3030/metrics | grep rustelo_

# Monitor resource usage
docker stats --no-stream

# Check for slow queries
docker-compose exec db psql -U postgres -c "SELECT query, calls, total_time FROM pg_stat_statements ORDER BY total_time DESC LIMIT 10;"

Debugging Commands

# Enter container shell
docker-compose exec app bash

# Check application logs
docker-compose logs -f app

# Test endpoints
curl -v http://localhost:3030/health
curl -v http://localhost:3030/metrics

# Check database
docker-compose exec db psql -U postgres rustelo_prod

# Monitor in real-time
watch -n 5 'docker stats --no-stream'

Recovery Procedures

1. Database Recovery

# Restore from backup
docker-compose exec -T db psql -U postgres rustelo_prod < backup_file.sql

# Reset database (WARNING: destroys data)
docker-compose down
docker volume rm rustelo_postgres_data
docker-compose up -d db
./deploy.sh migrate -e production

2. Application Recovery

# Restart application
docker-compose restart app

# Full restart
docker-compose down
docker-compose up -d

# Rollback to previous version
docker-compose down
docker-compose pull
docker-compose up -d

Security Considerations

1. Container Security

• Use non-root user in containers
• Scan images for vulnerabilities
• Keep base images updated
• Use multi-stage builds to reduce attack surface

2. Network Security

• Use internal networks for service communication
• Expose only necessary ports
• Implement rate limiting
• Use TLS for all external communications

3. Data Protection

• Encrypt sensitive data at rest
• Use encrypted database connections
• Implement proper backup encryption
• Regular security audits

4. Access Control

• Use strong authentication
• Implement role-based access control
• Regular password rotation
• Monitor access logs

Performance Tuning

1. Application Optimization

# config.toml optimizations
[database]
max_connections = 20
min_connections = 5
connect_timeout = 10

[server]
worker_threads = 4
max_request_size = 1048576

[app]
enable_compression = true
cache_size = 1000

2. Database Optimization

-- Create indexes for frequently queried columns
CREATE INDEX idx_users_email ON users(email);
CREATE INDEX idx_content_created_at ON content(created_at);

-- Analyze query performance
EXPLAIN ANALYZE SELECT * FROM users WHERE email = 'user@example.com';

3. Container Resource Limits

services:
  app:
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 1G
        reservations:
          cpus: '1.0'
          memory: 512M

Maintenance

Regular Tasks

1. Daily: Check application logs and health status
2. Weekly: Review metrics and performance
3. Monthly: Update dependencies and security patches
4. Quarterly: Review and update monitoring alerts

Maintenance Commands

# Update system packages
sudo apt update && sudo apt upgrade -y

# Update Docker images
docker-compose pull
docker-compose up -d

# Clean up unused resources
docker system prune -f

# Backup database
./deploy.sh backup

# Check for security updates
cargo audit

For more detailed information, refer to the individual documentation files:

• Configuration Guide
• Email Configuration
• Security Guide
• API Documentation