provisioning-platform/prov-ecosystem/crates/observability/INTEGRATION.md

# Observability Integration Guide

This guide covers integrating the observability crate with other prov-ecosystem components and external systems.

## Table of Contents

1. [Integration with Other Crates](#integration-with-other-crates)
2. [Docker Integration](#docker-integration)
3. [Kubernetes Integration](#kubernetes-integration)
4. [Grafana Setup](#grafana-setup)
5. [OpenTelemetry Collector](#opentelemetry-collector)
6. [Init Systems](#init-systems)
7. [Cloud Providers](#cloud-providers)

## Integration with Other Crates

### Backup Module

Monitor backup operations and integrate with observability:

```rust
use observability::metrics;
use backup::client::BackupClient;

async fn monitor_backup() -> Result<(), Box<dyn std::error::Error>> {
    let backup_client = BackupClient::from_config(config)?;

    let start_time = std::time::Instant::now();

    match backup_client.backup().await {
        Ok(result) => {
            let duration = start_time.elapsed().as_secs_f64();

            observability::metrics::counter!("backup_success_total").increment();
            observability::metrics::gauge!("backup_data_bytes").set(result.data_size as f64);
            observability::metrics::histogram!("backup_duration_seconds").record(duration);

            tracing::info!(
                duration_secs = duration,
                data_bytes = result.data_size,
                "Backup completed successfully"
            );
        }
        Err(e) => {
            observability::metrics::counter!("backup_failure_total").increment();
            tracing::error!(error = %e, "Backup failed");
        }
    }

    Ok(())
}
```

### Init-Servs Module

Integrate service management with observability:

```rust
use observability::health::checks::{HealthCheckRegistry, HealthStatus};
use init_servs::ServiceManager;

fn setup_service_health_checks(
    manager: &ServiceManager,
    registry: &HealthCheckRegistry,
) -> Result<(), Box<dyn std::error::Error>> {
    let manager_clone = manager.clone();

    registry.register("services_running", move || {
        match manager_clone.list_services() {
            Ok(services) => {
                let running = services.iter().filter(|s| s.is_running).count();
                if running == services.len() {
                    HealthStatus::Healthy
                } else {
                    HealthStatus::Degraded
                }
            }
            Err(_) => HealthStatus::Unhealthy,
        }
    });

    Ok(())
}
```

### Runtime Module

Monitor container runtime metrics:

```rust
use observability::metrics;
use runtime::ContainerRuntime;

async fn monitor_containers() -> Result<(), Box<dyn std::error::Error>> {
    let runtime = ContainerRuntime::new()?;

    for container in runtime.list_containers()? {
        observability::metrics::gauge!(
            "container_memory_bytes"
        ).set(container.memory_usage as f64);

        observability::metrics::gauge!(
            "container_cpu_shares"
        ).set(container.cpu_shares as f64);
    }

    Ok(())
}
```

### Encryption Module

Track encryption operations:

```rust
use observability::metrics;
use encrypt::EncryptionClient;

async fn encrypt_with_monitoring(
    client: &EncryptionClient,
    data: &[u8],
) -> Result<Vec<u8>, Box<dyn std::error::Error>> {
    let start = std::time::Instant::now();

    let encrypted = client.encrypt(data).await?;

    let duration = start.elapsed().as_secs_f64();
    observability::metrics::histogram!("encryption_duration_seconds").record(duration);
    observability::metrics::counter!("encryption_operations_total").increment();

    Ok(encrypted)
}
```

## Docker Integration

### Dockerfile with Health Checks

```dockerfile
FROM rust:1.75

WORKDIR /app

COPY . .

RUN cargo build --release

# Health check using observability endpoints
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
    CMD curl -f http://localhost:8080/healthz || exit 1

EXPOSE 8080 9090

CMD ["./target/release/my-app"]
```

### Docker Compose

```yaml
version: '3.8'

services:
  app:
    build: .
    ports:
      - "8080:8080"  # Health checks
      - "9090:9090"  # Prometheus metrics
    environment:
      RUST_LOG: info
      OBS_ENVIRONMENT: docker
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/healthz"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9091:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - ./dashboards:/etc/grafana/provisioning/dashboards
```

### Prometheus Configuration

```yaml
# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'app'
    static_configs:
      - targets: ['localhost:9090']
    scrape_interval: 10s
    scrape_timeout: 5s
```

## Kubernetes Integration

### Kubernetes Deployment

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: my-app
        image: my-app:latest
        ports:
        - containerPort: 8080
          name: health
        - containerPort: 9090
          name: metrics
        env:
        - name: OBS_ENVIRONMENT
          value: "kubernetes"
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace

        # Health checks
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3

        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 2

        startupProbe:
          httpGet:
            path: /startup
            port: 8080
          initialDelaySeconds: 0
          periodSeconds: 2
          timeoutSeconds: 3
          failureThreshold: 30

        # Resource limits
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi
```

### ServiceMonitor for Prometheus Operator

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
```

### Configuration via ConfigMap

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-observability
  namespace: default
data:
  observability.toml: |
    [context]
    service_name = "my-app"
    service_version = "1.0.0"
    environment = "kubernetes"
    namespace = "default"

    [logging]
    level = "info"
    format = "json"
    include_spans = false
    outputs = ["stdout"]

    [metrics]
    enabled = true
    export_interval_secs = 60

    [health]
    enabled = true
    port = 8080
    liveness_path = "/healthz"
    readiness_path = "/ready"
    startup_path = "/startup"
---
apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: my-app
    image: my-app:latest
    volumeMounts:
    - name: config
      mountPath: /etc/my-app
  volumes:
  - name: config
    configMap:
      name: my-app-observability
```

## Grafana Setup

### Adding Prometheus Data Source

```json
{
  "name": "Prometheus",
  "type": "prometheus",
  "url": "http://prometheus:9090",
  "access": "proxy",
  "isDefault": true
}
```

### Provisioning Dashboards

Create `provisioning/dashboards/provider.yaml`:

```yaml
apiVersion: 1

providers:
  - name: 'Observability Dashboards'
    orgId: 1
    folder: 'Services'
    type: file
    disableDeletion: false
    editable: true
    options:
      path: /etc/grafana/provisioning/dashboards
```

Copy dashboard JSON files to `/etc/grafana/provisioning/dashboards/`:

```bash
cp dashboards/*.json /etc/grafana/provisioning/dashboards/
```

### Alerting Rules

Create `rules.yml`:

```yaml
groups:
  - name: app_alerts
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: rate(http_errors_total[5m]) > 0.05
        for: 5m
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }}"

      - alert: ServiceDown
        expr: up == 0
        for: 1m
        annotations:
          summary: "Service is down"
          description: "{{ $labels.job }} on {{ $labels.instance }} is down"

      - alert: HighLatency
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        annotations:
          summary: "High latency detected"
          description: "p95 latency is {{ $value }}s"
```

## OpenTelemetry Collector

### Docker Compose with OTEL Collector

```yaml
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    ports:
      - "4317:4317"   # gRPC
      - "4318:4318"   # HTTP
      - "8888:8888"   # Prometheus metrics
    volumes:
      - ./otel-config.yaml:/etc/otel/config.yaml
    command: ["--config=/etc/otel/config.yaml"]
```

### OpenTelemetry Collector Configuration

`otel-config.yaml`:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 10s
          static_configs:
            - targets: ['localhost:8888']

processors:
  batch:
    send_batch_size: 1024
    timeout: 10s

  memory_limiter:
    check_interval: 1s
    limit_mib: 1024

exporters:
  prometheus:
    endpoint: "0.0.0.0:8888"

  jaeger:
    endpoint: "jaeger:14250"

  otlp:
    client:
      endpoint: "datadog-agent:4317"

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, batch]
      exporters: [prometheus, otlp]

    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger, otlp]
```

## Init Systems

### Systemd Service with Health Checks

```ini
[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/my-app
ExecReload=/bin/kill -HUP $MAINPID
Environment="RUST_LOG=info"
Environment="OBS_ENVIRONMENT=systemd"
Environment="OBS_CONFIG=/etc/my-app/observability.toml"

# Health check via systemd
ExecStartPost=/usr/bin/curl -f http://localhost:8080/healthz

# Restart policy
Restart=on-failure
RestartSec=10

# Resource limits
MemoryMax=512M
CPUQuota=50%

[Install]
WantedBy=multi-user.target
```

### Monitoring with journalctl

```bash
# View recent logs
journalctl -u my-app -n 100 -f

# Parse JSON structured logs
journalctl -u my-app -o json | jq '.MESSAGE | fromjson?'
```

## Cloud Providers

### AWS CloudWatch Integration

Use CloudWatch exporter or send logs to CloudWatch:

```rust
// Configuration for CloudWatch
[logging]
outputs = ["stdout", "cloudwatch"]

[logging.cloudwatch]
region = "us-east-1"
log_group = "/aws/services/my-app"
log_stream = "production"
```

### Google Cloud Monitoring

Configure for Google Cloud logging:

```rust
[logging]
format = "json"
outputs = ["stdout"]

[context]
environment = "gcp"
attributes = { "gcp_project" = "my-project" }
```

### Azure Monitor

Send metrics to Azure Monitor:

```toml
[metrics.otlp]
endpoint = "https://<workspace-id>.ods.opinsights.azure.com/api/logs"
protocol = "http"
headers = { "Authorization" = "Bearer <token>" }
```

## Performance Tuning

### For High-Throughput Scenarios

```toml
[logging]
# Reduce overhead
level = "warn"
include_spans = false

[metrics]
export_interval_secs = 120

[metrics.prometheus]
# Histogram buckets optimized for typical latencies
histogram_buckets = [0.001, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0]

[health]
# Disable if not needed in high-throughput
enabled = false
```

### For Development

```toml
[logging]
level = "debug"
format = "pretty"
include_spans = true
include_target = true

[metrics]
export_interval_secs = 10

[health]
enabled = true
port = 8080
```

## Troubleshooting

### Metrics Not Appearing

1. Check Prometheus targets at `http://localhost:9091/targets`
2. Verify metrics endpoint: `curl http://localhost:9090/metrics`
3. Check logs for initialization errors

### Health Checks Timing Out

1. Verify port binding: `curl http://localhost:8080/healthz`
2. Check firewall rules
3. Review probe configuration in Kubernetes

### High Memory Usage

1. Reduce histogram buckets
2. Decrease log verbosity
3. Increase batch timeout for OTLP

### Logs Not Appearing

1. Check `RUST_LOG` environment variable
2. Verify configured outputs (stdout/file)
3. Check file permissions for file output

## Best Practices

1. **Development**: Use pretty format with debug logging
2. **Staging**: Use JSON format with info logging
3. **Production**: Use JSON format with warn logging
4. **Health Checks**: Always implement for K8s deployments
5. **Metrics**: Use consistent naming conventions
6. **Dashboards**: Create dashboards for critical metrics
7. **Alerts**: Set up alerts for error rates and latency