provisioning-platform/prov-ecosystem/crates/observability

Observability Crate

Multi-backend observability framework providing unified support for logging, metrics, distributed tracing, health checks, and dashboard generation.

Features

• Logging: Structured logging via tracing with configurable output formats (JSON, Pretty, Compact)
• Metrics: Prometheus pull-based metrics (default) + optional OpenTelemetry push
• Distributed Tracing: Optional OpenTelemetry support for distributed tracing
• Health Checks: Kubernetes-style liveness, readiness, and startup probes
• Dashboards: Predefined Grafana dashboard templates + programmatic dashboard generation
• Multi-Backend: Flexible architecture supporting multiple exporters simultaneously

Cargo Features

[dependencies]
observability = { version = "0.1", features = ["full"] }

Available Features

• logging (default): Structured logging with tracing-subscriber
• metrics-prometheus (default): Prometheus metrics exporter
• metrics-otlp: OpenTelemetry metrics push support
• tracing-otlp: Distributed tracing via OpenTelemetry
• health (default): HTTP health check endpoints
• dashboards: Grafana dashboard generation and templates
• full: All features enabled

Quick Start

Basic Initialization

use observability::init_from_env;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize observability with defaults
    let _guard = observability::init_from_env("my-app", "1.0.0")?;

    // Logging via tracing macros
    tracing::info!("Application started");
    tracing::warn!("Something warning-worthy happened");

    // Metrics
    observability::metrics::counter!("requests_total").increment();
    observability::metrics::gauge!("active_connections").set(42.0);
    observability::metrics::histogram!("request_duration_seconds").record(0.125);

    // Health checks
    // GET /healthz (liveness)
    // GET /ready (readiness)
    // GET /startup (startup)

    Ok(())
}

Configuration from TOML

use observability::init_from_toml;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let _guard = observability::init_from_toml("config/observability.toml")?;
    Ok(())
}

Configuration Structure

[context]
service_name = "my-app"
service_version = "1.0.0"
environment = "production"

[logging]
level = "info"
format = "json"
include_spans = true
outputs = ["stdout"]

[metrics]
enabled = true
export_interval_secs = 60

[metrics.prometheus]
port = 9090
path = "/metrics"

[health]
enabled = true
port = 8080
liveness_path = "/healthz"
readiness_path = "/ready"

[dashboards]
auto_generate = true
output_dir = "./dashboards"

Logging

Structured logging via the tracing crate:

// Info level
tracing::info!("Starting service", port = 8080, host = "0.0.0.0");

// With structured data
tracing::warn!(
    event = "high_latency",
    duration_ms = 250,
    threshold_ms = 100,
    "Request exceeded latency threshold"
);

// Conditional logging
tracing::debug!("Debug information");

Output Formats

• JSON: Structured JSON output for parsing/indexing
• Pretty: Human-readable colored output for development
• Compact: Single-line output with essential information

Metrics

Collect and export application metrics:

use observability::metrics;

// Counter - monotonically increasing
metrics::counter!("http_requests_total").increment();
metrics::counter!("http_requests_total").add(10);

// Gauge - can increase or decrease
metrics::gauge!("active_connections").set(42.0);

// Histogram - measure distribution
metrics::histogram!("request_duration_seconds").record(0.250);

Prometheus Metrics

Default exporter exposes metrics at http://localhost:9090/metrics in Prometheus text format:

# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",path="/api/users"} 150.0
http_requests_total{method="POST",path="/api/users"} 42.0

# HELP request_duration_seconds Request duration in seconds
# TYPE request_duration_seconds histogram
request_duration_seconds_bucket{le="0.005"} 10
request_duration_seconds_bucket{le="0.01"} 25
request_duration_seconds_bucket{le="0.025"} 50

Optional: OpenTelemetry Push

Enable feature and configure to push metrics:

[metrics.otlp]
endpoint = "http://localhost:4317"
protocol = "grpc"
timeout_secs = 10

Health Checks

Kubernetes-style health check endpoints:

• Liveness (/healthz): Indicates if service should be restarted
• Readiness (/ready): Indicates if service can handle traffic
• Startup (/startup): Indicates if initialization completed

Register custom health checks:

use observability::health::checks::{HealthCheckRegistry, HealthStatus};

let registry = HealthCheckRegistry::new();

// Register database health check
registry.register("database", || {
    if db.is_connected() {
        HealthStatus::Healthy
    } else {
        HealthStatus::Unhealthy
    }
});

// Register cache health check with degradation
registry.register("cache", || {
    if cache.is_responsive() {
        HealthStatus::Healthy
    } else {
        HealthStatus::Degraded
    }
});

// Run all checks
let response = registry.run_all();
// response.status: Healthy, Degraded, or Unhealthy
// response.checks: HashMap of individual results

Dashboards

Predefined Dashboards

Three predefined Grafana dashboards included:

1. Overview (dashboards/overview.json)

   • Request rate, error rate
   • Response times (p95)
   • Active connections
   • Memory and CPU usage

2. Backup Operations (dashboards/backup.json)

   • Backup success rate
   • Last backup status
   • Duration trends
   • Data processed
   • Retention policy status

3. Runtime Performance (dashboards/runtime.json)

   • Memory allocation
   • CPU usage
   • Goroutines count
   • GC pause duration
   • File descriptors

Programmatic Dashboard Generation

Generate custom dashboards:

use observability::dashboards::{
    DashboardBuilder, Panel, PanelType, GridPos, Target
};

let dashboard = DashboardBuilder::new("Custom Dashboard")
    .with_description("Custom application monitoring")
    .with_tag("custom")
    .with_panel(Panel {
        id: 1,
        title: "Custom Metric".to_string(),
        panel_type: PanelType::Graph,
        targets: vec![
            Target {
                expr: "custom_metric".to_string(),
                ref_id: "A".to_string(),
                legend_format: None,
            }
        ],
        gridpos: GridPos { h: 8, w: 12, x: 0, y: 0 },
        description: Some("My custom metric".to_string()),
        unit: Some("short".to_string()),
        decimals: Some(2),
    })
    .build();

// Serialize to JSON
let json = serde_json::to_string_pretty(&dashboard)?;
std::fs::write("custom-dashboard.json", json)?;

Configuration File Examples

Development

[context]
service_name = "my-app"
service_version = "0.1.0"
environment = "development"

[logging]
level = "debug"
format = "pretty"
include_spans = true

[metrics]
enabled = true
export_interval_secs = 30

[health]
enabled = true
port = 8080

[dashboards]
auto_generate = true
output_dir = "./dashboards"

Production

[context]
service_name = "my-app"
service_version = "1.0.0"
environment = "production"
namespace = "production"

[logging]
level = "info"
format = "json"
include_spans = false
outputs = ["stdout"]

[metrics]
enabled = true
export_interval_secs = 60

[metrics.prometheus]
port = 9090
path = "/metrics"

[metrics.otlp]
endpoint = "http://otel-collector:4317"
protocol = "grpc"
timeout_secs = 30

[tracing]
enabled = true
sampling_rate = 0.1
otlp_endpoint = "http://otel-collector:4317"

[health]
enabled = true
port = 8080
liveness_path = "/healthz"
readiness_path = "/ready"
startup_path = "/startup"

[dashboards]
auto_generate = true
output_dir = "/etc/grafana/provisioning/dashboards"
grafana_url = "http://grafana:3000"

Best Practices

1. Initialization: Initialize observability early in your application startup
2. Logging: Use structured logging with context-aware fields
3. Metrics: Use meaningful metric names and labels
4. Health Checks: Register checks for critical dependencies
5. Dashboards: Customize dashboards for your specific metrics
6. Production: Use JSON logging format in production for parsing/indexing

Integration with Other Crates

With Backup Module

Monitor backup operations via observability:

use observability::metrics;
use backup::BackupClient;

let client = BackupClient::new(config)?;
let result = client.backup()?;

if result.success {
    metrics::counter!("backup_success_total").increment();
    metrics::gauge!("backup_data_bytes").set(result.data_size as f64);
} else {
    metrics::counter!("backup_failure_total").increment();
}

With Init-Servs Module

Monitor service management:

use observability::metrics;
use init_servs::ServiceManager;

let manager = ServiceManager::new()?;
manager.start_service("my-service")?;

metrics::counter!("service_start_total").increment();

Environment Variables

• RUST_LOG: Control logging level (e.g., RUST_LOG=debug)
• OBS_CONFIG: Path to observability configuration file
• OBS_SERVICE_NAME: Override service name
• OBS_ENVIRONMENT: Override environment (dev/staging/prod)

Performance Considerations

• Logging: Minimal overhead with async I/O
• Metrics: Pre-aggregated, no cardinality explosion
• Health Checks: Lightweight, typically < 1ms per check
• Dashboards: Pure data structures, no runtime overhead

License

MIT