provisioning-platform/prov-ecosystem/crates/observability/README.md

# Observability Crate

Multi-backend observability framework providing unified support for logging, metrics, distributed tracing, health checks, and dashboard generation.

## Features

- **Logging**: Structured logging via `tracing` with configurable output formats (JSON, Pretty, Compact)
- **Metrics**: Prometheus pull-based metrics (default) + optional OpenTelemetry push
- **Distributed Tracing**: Optional OpenTelemetry support for distributed tracing
- **Health Checks**: Kubernetes-style liveness, readiness, and startup probes
- **Dashboards**: Predefined Grafana dashboard templates + programmatic dashboard generation
- **Multi-Backend**: Flexible architecture supporting multiple exporters simultaneously

## Cargo Features

```toml
[dependencies]
observability = { version = "0.1", features = ["full"] }
```

### Available Features

- **`logging`** (default): Structured logging with `tracing-subscriber`
- **`metrics-prometheus`** (default): Prometheus metrics exporter
- **`metrics-otlp`**: OpenTelemetry metrics push support
- **`tracing-otlp`**: Distributed tracing via OpenTelemetry
- **`health`** (default): HTTP health check endpoints
- **`dashboards`**: Grafana dashboard generation and templates
- **`full`**: All features enabled

## Quick Start

### Basic Initialization

```rust
use observability::init_from_env;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize observability with defaults
    let _guard = observability::init_from_env("my-app", "1.0.0")?;

    // Logging via tracing macros
    tracing::info!("Application started");
    tracing::warn!("Something warning-worthy happened");

    // Metrics
    observability::metrics::counter!("requests_total").increment();
    observability::metrics::gauge!("active_connections").set(42.0);
    observability::metrics::histogram!("request_duration_seconds").record(0.125);

    // Health checks
    // GET /healthz (liveness)
    // GET /ready (readiness)
    // GET /startup (startup)

    Ok(())
}
```

### Configuration from TOML

```rust
use observability::init_from_toml;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let _guard = observability::init_from_toml("config/observability.toml")?;
    Ok(())
}
```

### Configuration Structure

```toml
[context]
service_name = "my-app"
service_version = "1.0.0"
environment = "production"

[logging]
level = "info"
format = "json"
include_spans = true
outputs = ["stdout"]

[metrics]
enabled = true
export_interval_secs = 60

[metrics.prometheus]
port = 9090
path = "/metrics"

[health]
enabled = true
port = 8080
liveness_path = "/healthz"
readiness_path = "/ready"

[dashboards]
auto_generate = true
output_dir = "./dashboards"
```

## Logging

Structured logging via the `tracing` crate:

```rust
// Info level
tracing::info!("Starting service", port = 8080, host = "0.0.0.0");

// With structured data
tracing::warn!(
    event = "high_latency",
    duration_ms = 250,
    threshold_ms = 100,
    "Request exceeded latency threshold"
);

// Conditional logging
tracing::debug!("Debug information");
```

### Output Formats

- **JSON**: Structured JSON output for parsing/indexing
- **Pretty**: Human-readable colored output for development
- **Compact**: Single-line output with essential information

## Metrics

Collect and export application metrics:

```rust
use observability::metrics;

// Counter - monotonically increasing
metrics::counter!("http_requests_total").increment();
metrics::counter!("http_requests_total").add(10);

// Gauge - can increase or decrease
metrics::gauge!("active_connections").set(42.0);

// Histogram - measure distribution
metrics::histogram!("request_duration_seconds").record(0.250);
```

### Prometheus Metrics

Default exporter exposes metrics at `http://localhost:9090/metrics` in Prometheus text format:

```
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",path="/api/users"} 150.0
http_requests_total{method="POST",path="/api/users"} 42.0

# HELP request_duration_seconds Request duration in seconds
# TYPE request_duration_seconds histogram
request_duration_seconds_bucket{le="0.005"} 10
request_duration_seconds_bucket{le="0.01"} 25
request_duration_seconds_bucket{le="0.025"} 50
```

### Optional: OpenTelemetry Push

Enable feature and configure to push metrics:

```toml
[metrics.otlp]
endpoint = "http://localhost:4317"
protocol = "grpc"
timeout_secs = 10
```

## Health Checks

Kubernetes-style health check endpoints:

- **Liveness** (`/healthz`): Indicates if service should be restarted
- **Readiness** (`/ready`): Indicates if service can handle traffic
- **Startup** (`/startup`): Indicates if initialization completed

Register custom health checks:

```rust
use observability::health::checks::{HealthCheckRegistry, HealthStatus};

let registry = HealthCheckRegistry::new();

// Register database health check
registry.register("database", || {
    if db.is_connected() {
        HealthStatus::Healthy
    } else {
        HealthStatus::Unhealthy
    }
});

// Register cache health check with degradation
registry.register("cache", || {
    if cache.is_responsive() {
        HealthStatus::Healthy
    } else {
        HealthStatus::Degraded
    }
});

// Run all checks
let response = registry.run_all();
// response.status: Healthy, Degraded, or Unhealthy
// response.checks: HashMap of individual results
```

## Dashboards

### Predefined Dashboards

Three predefined Grafana dashboards included:

1. **Overview** (`dashboards/overview.json`)
   - Request rate, error rate
   - Response times (p95)
   - Active connections
   - Memory and CPU usage

2. **Backup Operations** (`dashboards/backup.json`)
   - Backup success rate
   - Last backup status
   - Duration trends
   - Data processed
   - Retention policy status

3. **Runtime Performance** (`dashboards/runtime.json`)
   - Memory allocation
   - CPU usage
   - Goroutines count
   - GC pause duration
   - File descriptors

### Programmatic Dashboard Generation

Generate custom dashboards:

```rust
use observability::dashboards::{
    DashboardBuilder, Panel, PanelType, GridPos, Target
};

let dashboard = DashboardBuilder::new("Custom Dashboard")
    .with_description("Custom application monitoring")
    .with_tag("custom")
    .with_panel(Panel {
        id: 1,
        title: "Custom Metric".to_string(),
        panel_type: PanelType::Graph,
        targets: vec![
            Target {
                expr: "custom_metric".to_string(),
                ref_id: "A".to_string(),
                legend_format: None,
            }
        ],
        gridpos: GridPos { h: 8, w: 12, x: 0, y: 0 },
        description: Some("My custom metric".to_string()),
        unit: Some("short".to_string()),
        decimals: Some(2),
    })
    .build();

// Serialize to JSON
let json = serde_json::to_string_pretty(&dashboard)?;
std::fs::write("custom-dashboard.json", json)?;
```

## Configuration File Examples

### Development

```toml
[context]
service_name = "my-app"
service_version = "0.1.0"
environment = "development"

[logging]
level = "debug"
format = "pretty"
include_spans = true

[metrics]
enabled = true
export_interval_secs = 30

[health]
enabled = true
port = 8080

[dashboards]
auto_generate = true
output_dir = "./dashboards"
```

### Production

```toml
[context]
service_name = "my-app"
service_version = "1.0.0"
environment = "production"
namespace = "production"

[logging]
level = "info"
format = "json"
include_spans = false
outputs = ["stdout"]

[metrics]
enabled = true
export_interval_secs = 60

[metrics.prometheus]
port = 9090
path = "/metrics"

[metrics.otlp]
endpoint = "http://otel-collector:4317"
protocol = "grpc"
timeout_secs = 30

[tracing]
enabled = true
sampling_rate = 0.1
otlp_endpoint = "http://otel-collector:4317"

[health]
enabled = true
port = 8080
liveness_path = "/healthz"
readiness_path = "/ready"
startup_path = "/startup"

[dashboards]
auto_generate = true
output_dir = "/etc/grafana/provisioning/dashboards"
grafana_url = "http://grafana:3000"
```

## Best Practices

1. **Initialization**: Initialize observability early in your application startup
2. **Logging**: Use structured logging with context-aware fields
3. **Metrics**: Use meaningful metric names and labels
4. **Health Checks**: Register checks for critical dependencies
5. **Dashboards**: Customize dashboards for your specific metrics
6. **Production**: Use JSON logging format in production for parsing/indexing

## Integration with Other Crates

### With Backup Module

Monitor backup operations via observability:

```rust
use observability::metrics;
use backup::BackupClient;

let client = BackupClient::new(config)?;
let result = client.backup()?;

if result.success {
    metrics::counter!("backup_success_total").increment();
    metrics::gauge!("backup_data_bytes").set(result.data_size as f64);
} else {
    metrics::counter!("backup_failure_total").increment();
}
```

### With Init-Servs Module

Monitor service management:

```rust
use observability::metrics;
use init_servs::ServiceManager;

let manager = ServiceManager::new()?;
manager.start_service("my-service")?;

metrics::counter!("service_start_total").increment();
```

## Environment Variables

- `RUST_LOG`: Control logging level (e.g., `RUST_LOG=debug`)
- `OBS_CONFIG`: Path to observability configuration file
- `OBS_SERVICE_NAME`: Override service name
- `OBS_ENVIRONMENT`: Override environment (dev/staging/prod)

## Performance Considerations

- **Logging**: Minimal overhead with async I/O
- **Metrics**: Pre-aggregated, no cardinality explosion
- **Health Checks**: Lightweight, typically < 1ms per check
- **Dashboards**: Pure data structures, no runtime overhead

## License

MIT