# ADR-008: Unified Observability Stack (Metrics, Logs, Traces)

**Status**: Accepted | **Date**: 2025-01-16 | **Supersedes**: None

## Context

Distributed 12-microservice architecture requires observability to understand system behavior, diagnose failures, and optimize performance.

## Decision

Implement unified observability using three pillars:
1. **Metrics** - Prometheus/VictoriaMetrics for time-series data
2. **Logs** - ELK Stack (Elasticsearch, Logstash, Kibana) or Loki
3. **Traces** - Jaeger for distributed request tracing

## Rationale

1. **Prometheus Metrics**: Industry standard, minimal overhead, powerful querying
2. **Structured Logging**: JSON logs for machine parsing, full-text search in Kibana
3. **Distributed Traces**: End-to-end request tracking across all 12 services
4. **Correlation**: Unified correlation IDs linking metrics, logs, traces

## Implementation

### Metrics Layer

**Prometheus** (all services expose `/metrics` endpoint):

```rust
// Every service exports metrics
use prometheus::{Counter, Histogram, Registry};

lazy_static::lazy_static! {
    static ref HTTP_REQUESTS: Counter = Counter::new("http_requests_total", "Total HTTP requests").unwrap();
    static ref RESPONSE_TIME: Histogram = Histogram::new("http_response_time_seconds", "HTTP response time").unwrap();
}

#[get("/api/v1/workspaces")]
async fn list_workspaces() -> HttpResponse {
    let timer = RESPONSE_TIME.start_timer();
    HTTP_REQUESTS.inc();

    // Business logic
    let workspaces = db.list_workspaces().await;

    timer.observe_duration();
    HttpResponse::Ok().json(workspaces)
}
```

**Key Metrics** (per service):

| Metric | Type | Purpose |
| -------- | ------ | --------- |
| `http_requests_total` | Counter | API call volume |
| `http_response_time_seconds` | Histogram | API latency distribution |
| `workflow_executions_total` | Counter | Workflow count |
| `workflow_duration_seconds` | Histogram | Workflow execution time |
| `database_query_duration_seconds` | Histogram | DB query performance |
| `cache_hits_total` | Counter | Cache effectiveness |
| `secrets_decryption_duration_seconds` | Histogram | Vault latency |

**Alerting Rules** (Prometheus alerts):

```yaml
# provisioning/monitoring/prometheus-rules.yaml
groups:
- name: provisioning
  rules:
  - alert: ServiceDown
    expr: up{job="provisioning"} == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Service {{ $labels.service }} is down"

  - alert: HighLatency
    expr: histogram_quantile(0.99, http_response_time_seconds) > 1
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "High API latency: {{ $value }}s"

  - alert: WorkflowFailureRate
    expr: (increase(workflow_failures_total[5m]) / increase(workflow_executions_total[5m])) > 0.05
    labels:
      severity: critical
    annotations:
      summary: "Workflow failure rate exceeds 5%"
```

### Logging Layer

**Structured Logging** (JSON, machine-parseable):

```rust
// Every service logs in JSON with context
use slog::{Logger, o, info, warn, error};
use slog_json_compact::JsonCompact;

let logger = Logger::root(
    JsonCompact::new(io::stdout()).fuse(),
    o!("service" => "control-center", "version" => "1.0.0")
);

info!(logger, "Workspace created";
    "workspace_id" => "ws-123",
    "user_id" => "user-456",
    "region" => "us-east-1",
    "duration_ms" => 234,
    "correlation_id" => "corr-789"
);
```

**Log Aggregation** (Loki):

```yaml
# Loki config: labels for efficient querying
scrape_configs:
- job_name: provisioning
  static_configs:
  - targets:
    - localhost
    labels:
      service: control-center
      environment: production
      region: us-east-1
```

**Log Analysis**:

```bash
# LogQL queries in Kibana/Grafana
# Find errors in last 5 minutes
{service="control-center", level="error"} | json | level="error"

# Latency distribution by endpoint
{service="control-center"} | json | histogram(duration_ms)

# Error rate by user
{service="vault-service"} | json | errors_by_user(user_id)
```

### Tracing Layer

**Distributed Tracing** (Jaeger):

```rust
use opentelemetry::{global, sdk::trace as sdktrace};
use opentelemetry_jaeger::new_pipeline;

// Initialize tracing
let tracer = new_pipeline()
    .install_simple()
    .unwrap();

// Instrument requests with spans
#[tracing::instrument(skip(req))]
async fn create_workspace(req: CreateWorkspaceRequest) -> Result<Workspace> {
    let span = global::tracer("control-center").start("create_workspace");

    // Each internal call creates child span
    let config = fetch_config().await?;  // Traced automatically

    let workspace = db.create(req).await?;  // Traced automatically

    tracing::info_span!("post_creation_hook").in_scope( | | {
        send_notification(&workspace)?;
    });

    Ok(workspace)
}
```

**Trace Visualization** (Jaeger UI):

```text
Request: POST /api/v1/workspaces
├─ span: api_handler (10ms)
│  ├─ span: validate_input (2ms)
│  ├─ span: fetch_config (100ms)
│  │  ├─ span: control-center_api_call (100ms) [service: control-center]
│  ├─ span: db_create (50ms)
│  └─ span: post_creation_hook (200ms)
│     ├─ span: notification_send (150ms) [service: notification-service]
│     └─ span: webhook_call (50ms)
└─ Total: 362ms
```

## Correlation ID

All requests traced by correlation ID:

```text
Client Request
  → Generate: correlation_id = "corr-abc123"
  → Pass in X-Correlation-ID header
  ↓
Control-Center
  → Receive: correlation_id from header
  → Log: {"correlation_id": "corr-abc123", ...}
  → Call Orchestrator with X-Correlation-ID header
  ↓
Orchestrator
  → Inherit correlation_id from header
  → Create spans: correlation_id = "corr-abc123"
  → Call Vault-Service with X-Correlation-ID header
  ↓
All logs, metrics, traces tagged with same correlation_id
→ Easy to correlate across services
```

## Dashboard Queries

**Real-Time Health Dashboard**:

```text
Prometheus metrics
- Service health (up/down)
- Request rate (req/sec)
- Error rate (errors/sec)
- P99 latency (milliseconds)
- CPU/Memory per service
- Cache hit rate

Grafana visualizations
- Time-series graphs
- Heatmaps for latency distribution
- Error rate alerts
- Dependency graph (which services call which)
```

**SLO Monitoring**:

```yaml
# Service Level Objectives
objectives:
- name: API Availability
  expr: up{service="control-center"} > 0.9995
  target: 99.95%
  window: 30d

- name: API Latency (P99)
  expr: histogram_quantile(0.99, http_response_time_seconds) < 1
  target: <1 second
  window: 5m

- name: Workflow Success Rate
  expr: (1 - (increase(workflow_failures_total[5m]) / increase(workflow_executions_total[5m]))) > 0.999
  target: 99.9%
  window: 5m
```

## Performance Considerations

| Overhead | Cost | Mitigation |
| ---------- | ------ | ----------- |
| Metrics collection | 2-5% CPU | Sampling (10% requests) |
| Logging to ELK | 5-10% latency | Async logging |
| Trace sampling | Variable | 10% sample rate default |
| Disk storage | 100GB/day | Retention: 7 days (metrics), 30 days (logs) |

## Related ADRs

- [ADR-004: Microservice Distribution](./adr-004-microservice-distribution.md) - Multiple services need observability
- [ADR-009: SLO and Error Budgets](./adr-009-slo-error-budgets.md)
