provisioning/provisioning-platform
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
provisioning-platform/prov-ecosystem/crates/observability/INTEGRATION.md

14 KiB
Raw Permalink Blame History

Observability Integration Guide

This guide covers integrating the observability crate with other prov-ecosystem components and external systems.

Table of Contents

  1. Integration with Other Crates
  2. Docker Integration
  3. Kubernetes Integration
  4. Grafana Setup
  5. OpenTelemetry Collector
  6. Init Systems
  7. Cloud Providers

Integration with Other Crates

Backup Module

Monitor backup operations and integrate with observability:

use observability::metrics;
use backup::client::BackupClient;

async fn monitor_backup() -> Result<(), Box<dyn std::error::Error>> {
    let backup_client = BackupClient::from_config(config)?;

    let start_time = std::time::Instant::now();

    match backup_client.backup().await {
        Ok(result) => {
            let duration = start_time.elapsed().as_secs_f64();

            observability::metrics::counter!("backup_success_total").increment();
            observability::metrics::gauge!("backup_data_bytes").set(result.data_size as f64);
            observability::metrics::histogram!("backup_duration_seconds").record(duration);

            tracing::info!(
                duration_secs = duration,
                data_bytes = result.data_size,
                "Backup completed successfully"
            );
        }
        Err(e) => {
            observability::metrics::counter!("backup_failure_total").increment();
            tracing::error!(error = %e, "Backup failed");
        }
    }

    Ok(())
}

Init-Servs Module

Integrate service management with observability:

use observability::health::checks::{HealthCheckRegistry, HealthStatus};
use init_servs::ServiceManager;

fn setup_service_health_checks(
    manager: &ServiceManager,
    registry: &HealthCheckRegistry,
) -> Result<(), Box<dyn std::error::Error>> {
    let manager_clone = manager.clone();

    registry.register("services_running", move || {
        match manager_clone.list_services() {
            Ok(services) => {
                let running = services.iter().filter(|s| s.is_running).count();
                if running == services.len() {
                    HealthStatus::Healthy
                } else {
                    HealthStatus::Degraded
                }
            }
            Err(_) => HealthStatus::Unhealthy,
        }
    });

    Ok(())
}

Runtime Module

Monitor container runtime metrics:

use observability::metrics;
use runtime::ContainerRuntime;

async fn monitor_containers() -> Result<(), Box<dyn std::error::Error>> {
    let runtime = ContainerRuntime::new()?;

    for container in runtime.list_containers()? {
        observability::metrics::gauge!(
            "container_memory_bytes"
        ).set(container.memory_usage as f64);

        observability::metrics::gauge!(
            "container_cpu_shares"
        ).set(container.cpu_shares as f64);
    }

    Ok(())
}

Encryption Module

Track encryption operations:

use observability::metrics;
use encrypt::EncryptionClient;

async fn encrypt_with_monitoring(
    client: &EncryptionClient,
    data: &[u8],
) -> Result<Vec<u8>, Box<dyn std::error::Error>> {
    let start = std::time::Instant::now();

    let encrypted = client.encrypt(data).await?;

    let duration = start.elapsed().as_secs_f64();
    observability::metrics::histogram!("encryption_duration_seconds").record(duration);
    observability::metrics::counter!("encryption_operations_total").increment();

    Ok(encrypted)
}

Docker Integration

Dockerfile with Health Checks

FROM rust:1.75

WORKDIR /app

COPY . .

RUN cargo build --release

# Health check using observability endpoints
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
    CMD curl -f http://localhost:8080/healthz || exit 1

EXPOSE 8080 9090

CMD ["./target/release/my-app"]

Docker Compose

version: '3.8'

services:
  app:
    build: .
    ports:
      - "8080:8080"  # Health checks
      - "9090:9090"  # Prometheus metrics
    environment:
      RUST_LOG: info
      OBS_ENVIRONMENT: docker
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/healthz"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9091:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - ./dashboards:/etc/grafana/provisioning/dashboards

Prometheus Configuration

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'app'
    static_configs:
      - targets: ['localhost:9090']
    scrape_interval: 10s
    scrape_timeout: 5s

Kubernetes Integration

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: my-app
        image: my-app:latest
        ports:
        - containerPort: 8080
          name: health
        - containerPort: 9090
          name: metrics
        env:
        - name: OBS_ENVIRONMENT
          value: "kubernetes"
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace

        # Health checks
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3

        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 2

        startupProbe:
          httpGet:
            path: /startup
            port: 8080
          initialDelaySeconds: 0
          periodSeconds: 2
          timeoutSeconds: 3
          failureThreshold: 30

        # Resource limits
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi

ServiceMonitor for Prometheus Operator

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics

Configuration via ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-observability
  namespace: default
data:
  observability.toml: |
    [context]
    service_name = "my-app"
    service_version = "1.0.0"
    environment = "kubernetes"
    namespace = "default"

    [logging]
    level = "info"
    format = "json"
    include_spans = false
    outputs = ["stdout"]

    [metrics]
    enabled = true
    export_interval_secs = 60

    [health]
    enabled = true
    port = 8080
    liveness_path = "/healthz"
    readiness_path = "/ready"
    startup_path = "/startup"
---
apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: my-app
    image: my-app:latest
    volumeMounts:
    - name: config
      mountPath: /etc/my-app
  volumes:
  - name: config
    configMap:
      name: my-app-observability

Grafana Setup

Adding Prometheus Data Source

{
  "name": "Prometheus",
  "type": "prometheus",
  "url": "http://prometheus:9090",
  "access": "proxy",
  "isDefault": true
}

Provisioning Dashboards

Create provisioning/dashboards/provider.yaml:

apiVersion: 1

providers:
  - name: 'Observability Dashboards'
    orgId: 1
    folder: 'Services'
    type: file
    disableDeletion: false
    editable: true
    options:
      path: /etc/grafana/provisioning/dashboards

Copy dashboard JSON files to /etc/grafana/provisioning/dashboards/:

cp dashboards/*.json /etc/grafana/provisioning/dashboards/

Alerting Rules

Create rules.yml:

groups:
  - name: app_alerts
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: rate(http_errors_total[5m]) > 0.05
        for: 5m
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }}"

      - alert: ServiceDown
        expr: up == 0
        for: 1m
        annotations:
          summary: "Service is down"
          description: "{{ $labels.job }} on {{ $labels.instance }} is down"

      - alert: HighLatency
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        annotations:
          summary: "High latency detected"
          description: "p95 latency is {{ $value }}s"

OpenTelemetry Collector

Docker Compose with OTEL Collector

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    ports:
      - "4317:4317"   # gRPC
      - "4318:4318"   # HTTP
      - "8888:8888"   # Prometheus metrics
    volumes:
      - ./otel-config.yaml:/etc/otel/config.yaml
    command: ["--config=/etc/otel/config.yaml"]

OpenTelemetry Collector Configuration

otel-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 10s
          static_configs:
            - targets: ['localhost:8888']

processors:
  batch:
    send_batch_size: 1024
    timeout: 10s

  memory_limiter:
    check_interval: 1s
    limit_mib: 1024

exporters:
  prometheus:
    endpoint: "0.0.0.0:8888"

  jaeger:
    endpoint: "jaeger:14250"

  otlp:
    client:
      endpoint: "datadog-agent:4317"

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, batch]
      exporters: [prometheus, otlp]

    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger, otlp]

Init Systems

Systemd Service with Health Checks

[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/my-app
ExecReload=/bin/kill -HUP $MAINPID
Environment="RUST_LOG=info"
Environment="OBS_ENVIRONMENT=systemd"
Environment="OBS_CONFIG=/etc/my-app/observability.toml"

# Health check via systemd
ExecStartPost=/usr/bin/curl -f http://localhost:8080/healthz

# Restart policy
Restart=on-failure
RestartSec=10

# Resource limits
MemoryMax=512M
CPUQuota=50%

[Install]
WantedBy=multi-user.target

Monitoring with journalctl

# View recent logs
journalctl -u my-app -n 100 -f

# Parse JSON structured logs
journalctl -u my-app -o json | jq '.MESSAGE | fromjson?'

Cloud Providers

AWS CloudWatch Integration

Use CloudWatch exporter or send logs to CloudWatch:

// Configuration for CloudWatch
[logging]
outputs = ["stdout", "cloudwatch"]

[logging.cloudwatch]
region = "us-east-1"
log_group = "/aws/services/my-app"
log_stream = "production"

Google Cloud Monitoring

Configure for Google Cloud logging:

[logging]
format = "json"
outputs = ["stdout"]

[context]
environment = "gcp"
attributes = { "gcp_project" = "my-project" }

Azure Monitor

Send metrics to Azure Monitor:

[metrics.otlp]
endpoint = "https://<workspace-id>.ods.opinsights.azure.com/api/logs"
protocol = "http"
headers = { "Authorization" = "Bearer <token>" }

Performance Tuning

For High-Throughput Scenarios

[logging]
# Reduce overhead
level = "warn"
include_spans = false

[metrics]
export_interval_secs = 120

[metrics.prometheus]
# Histogram buckets optimized for typical latencies
histogram_buckets = [0.001, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0]

[health]
# Disable if not needed in high-throughput
enabled = false

For Development

[logging]
level = "debug"
format = "pretty"
include_spans = true
include_target = true

[metrics]
export_interval_secs = 10

[health]
enabled = true
port = 8080

Troubleshooting

Metrics Not Appearing

  1. Check Prometheus targets at http://localhost:9091/targets
  2. Verify metrics endpoint: curl http://localhost:9090/metrics
  3. Check logs for initialization errors

Health Checks Timing Out

  1. Verify port binding: curl http://localhost:8080/healthz
  2. Check firewall rules
  3. Review probe configuration in Kubernetes

High Memory Usage

  1. Reduce histogram buckets
  2. Decrease log verbosity
  3. Increase batch timeout for OTLP

Logs Not Appearing

  1. Check RUST_LOG environment variable
  2. Verify configured outputs (stdout/file)
  3. Check file permissions for file output

Best Practices

  1. Development: Use pretty format with debug logging
  2. Staging: Use JSON format with info logging
  3. Production: Use JSON format with warn logging
  4. Health Checks: Always implement for K8s deployments
  5. Metrics: Use consistent naming conventions
  6. Dashboards: Create dashboards for critical metrics
  7. Alerts: Set up alerts for error rates and latency