provisioning-platform/daemon-cli/ARCHITECTURE.md

# daemon-cli Architecture

**daemon-cli** is the orchestration hub of the prov-ecosystem, providing unified daemon and CLI interfaces for all ecosystem services.

## Table of Contents

1. [Architecture Overview](#architecture-overview)
2. [Stable API Contracts](#stable-api-contracts)
3. [Module Organization](#module-organization)
4. [Core Concepts](#core-concepts)
5. [Extension Points](#extension-points)
6. [Best Practices](#best-practices)
7. [Integration Examples](#integration-examples)

---

## Architecture Overview

### Design Pattern: Plugin + Hub Architecture

```
┌─────────────────────────────────────────────────────────┐
│                    daemon-cli                            │
│                  (Orchestration Hub)                      │
├─────────────────────────────────────────────────────────┤
│  ┌──────────────────────────────────────────────────┐  │
│  │          Core Infrastructure                      │  │
│  │  ├─ DaemonConfig (Configuration)                 │  │
│  │  ├─ EventBus (Pub-Sub)                           │  │
│  │  └─ OperationRegistry (Plugin Discovery)         │  │
│  └──────────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────────┤
│  ┌──────────────────────────────────────────────────┐  │
│  │          Execution Layers                         │  │
│  │  ├─ HTTP API (provd daemon)                      │  │
│  │  ├─ CLI Client (provctl)                         │  │
│  │  └─ Dual-mode Execution (daemon or direct)       │  │
│  └──────────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────────┤
│  ┌──────────────────────────────────────────────────┐  │
│  │          Observability & Integration              │  │
│  │  ├─ Health Probes (Kubernetes-ready)             │  │
│  │  ├─ Webhooks (Event Ingestion)                   │  │
│  │  ├─ Hierarchical Caching (3-layer LRU)           │  │
│  │  ├─ Internationalization (i18n)                  │  │
│  │  └─ Config Rendering (KCL/Nickel/Tera)          │  │
│  └──────────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────────┤
│  ┌──────────────────────────────────────────────────┐  │
│  │          Ecosystem Operations (Plugins)           │  │
│  │  ├─ valida (config validation)                   │  │
│  │  ├─ encrypt (encryption)                         │  │
│  │  ├─ runtime (container runtime detection)        │  │
│  │  ├─ init-servs (service management)              │  │
│  │  ├─ observability (monitoring)                   │  │
│  │  └─ audit (security scanning)                    │  │
│  └──────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────┘
```

### Execution Flow

```
Client Request
    ↓
┌─ provctl (CLI) or HTTP API endpoint
├─ Parse input and create Operation request
├─ EventBus publishes "operation_requested" event
├─ OperationRegistry selects matching EcosystemOperation
├─ Check HierarchicalCache for cached result
├─ If miss: Execute operation (e.g., valida::ValidaOperation::execute)
├─ Cache result with TTL
├─ EventBus publishes "operation_completed" event
└─ Return result to client
```

---

## Stable API Contracts

### Why Contracts Matter

The daemon-cli module is heavily depended on by other crates (52+ dependents). To maintain compatibility:

- **Stable APIs** are guaranteed to remain compatible across minor version updates
- **Internal modules** may change between versions without notice
- **Contracts module** explicitly defines stability boundaries

### Accessing Stable APIs

**Always prefer importing from the `contracts` module:**

```rust
// ✅ GOOD - Stable contract
use daemon_cli::contracts::stable::{EcosystemOperation, OperationRegistry};
use daemon_cli::contracts::observability::HealthProbe;

// ⚠️  OK but less clear - Direct import (same types, but unstable guarantee)
use daemon_cli::{EcosystemOperation, HealthProbe};

// ❌ AVOID - Internal implementation details
use daemon_cli::cache::HierarchicalCache;  // Not in contracts
use daemon_cli::rendering::TemplateEngine;  // Not in contracts
```

### Stable API Categories

#### 1. **Core/Stable** - Orchestration Contract
```rust
pub mod contracts::stable {
    pub use crate::core::{DaemonConfig, DaemonError, Result};
    pub use crate::orchestration::{EcosystemOperation, OperationRegistry};
    pub use crate::api::AppState;
    pub use crate::events::{Event, EventBus, EventType, EventPayload};
}
```

**Use When:**
- Implementing new operations (extend `EcosystemOperation`)
- Registering with `OperationRegistry`
- Publishing events to `EventBus`
- Accessing shared `AppState`

#### 2. **Execution/Stable** - CLI and Output
```rust
pub mod contracts::execution {
    pub use crate::cli::{DaemonClient, OfflineMode, OutputFormat};
    pub use crate::config_renderer::{ConfigRenderer, RenderRequest};
}
```

**Use When:**
- Running in CLI mode (use `DaemonClient` or `OfflineMode`)
- Rendering configuration (KCL, Nickel, Tera templates)
- Formatting output (JSON, YAML, Table, etc.)

#### 3. **Observability/Stable** - Monitoring
```rust
pub mod contracts::observability {
    pub use crate::health::{HealthMetrics, HealthProbe, ProbeStatus};
}
```

**Use When:**
- Implementing health check endpoints
- Collecting metrics
- Exposing readiness/liveness probes for Kubernetes

#### 4. **I18n/Stable** - Internationalization
```rust
pub mod contracts::i18n {
    pub use crate::i18n::{
        I18nManager, TranslationRequest, TranslationResponse,
        TranslationContext,
    };
}
```

**Use When:**
- Supporting multiple languages in messages
- Translating output
- Managing locale-aware formatting

#### 5. **Integration/Stable** - External Systems
```rust
pub mod contracts::integration {
    pub use crate::webhooks::{WebHookEvent, WebHookHandler, WebHookStore};
}
```

**Use When:**
- Ingesting webhook events
- Storing webhook configurations
- Integrating with external CI/CD systems

---

## Module Organization

### Tier 1: Core Infrastructure (Stable)
| Module | Purpose | Stability |
|--------|---------|-----------|
| `core` | Configuration, errors, types | **STABLE** |
| `orchestration` | Operation trait, registry | **STABLE** |
| `events` | Event bus, pub-sub | **STABLE** |

**What to know:**
- `DaemonConfig` loads from YAML/TOML/environment
- `EcosystemOperation` is the extension point for ecosystem crates
- `EventBus` uses async-broadcast for real-time events

### Tier 2: Execution Layers (Stable via Contracts)
| Module | Purpose | Stability |
|--------|---------|-----------|
| `api` | Axum HTTP server | **STABLE** (via contracts) |
| `cli` | DaemonClient + OfflineMode | **STABLE** (via contracts) |
| `config_renderer` | Multi-format rendering | **STABLE** (via contracts) |

**What to know:**
- `DaemonClient` connects to HTTP daemon over network
- `OfflineMode` executes operations directly (no daemon)
- Choose mode based on deployment: cloud (daemon) vs embedded (offline)

### Tier 3: Observability & Integration (Stable via Contracts)
| Module | Purpose | Stability |
|--------|---------|-----------|
| `health` | K8s probes, metrics | **STABLE** (via contracts) |
| `webhooks` | Event ingestion | **STABLE** (via contracts) |
| `i18n` | Internationalization | **STABLE** (via contracts) |

**What to know:**
- `HealthProbe` exposes readiness/liveness for Kubernetes
- `WebHookStore` persists webhook configurations
- `I18nManager` handles message translation

### Tier 4: Internal Implementation (Implementation Details - May Change)
| Module | Purpose | Note |
|--------|---------|------|
| `cache` | 3-layer LRU caching | Internal, use `contracts` instead |
| `rendering` | Template engine | Internal, use `config_renderer` |
| `nushell` | Shell integration | Feature-gated internal |

**Important:** These modules are NOT in `contracts` because they are implementation details and subject to change.

---

## Core Concepts

### 1. EcosystemOperation - The Extension Point

The `EcosystemOperation` trait is how ecosystem crates (valida, encrypt, runtime, etc.) integrate:

```rust
#[async_trait]
pub trait EcosystemOperation: Send + Sync {
    fn name(&self) -> &'static str;

    async fn execute(&self, params: serde_json::Value)
        -> Result<OperationResult, OperationError>;

    fn cache_key(&self, params: &serde_json::Value) -> String;
}
```

**Pattern: Self-Registration**
```rust
// In valida crate:
pub struct ValidaOperation;

#[async_trait]
impl EcosystemOperation for ValidaOperation {
    fn name(&self) -> &'static str { "valida" }

    async fn execute(&self, params: serde_json::Value)
        -> Result<OperationResult, OperationError> {
        let config_path = params["config_path"].as_str()?;
        let rules = load_rules(config_path)?;
        validate_config(&rules)?;
        Ok(OperationResult { status: "success", data: json!({}) })
    }

    fn cache_key(&self, params: &serde_json::Value) -> String {
        format!("valida:{}", params["config_path"])
    }
}

// Registration happens via OperationRegistry::register()
registry.register(Arc::new(ValidaOperation));
```

### 2. Hierarchical Caching - Three Levels

**Not exposed in stable API (implementation detail)**, but important to understand:

```
Level 1: Command Cache (1 hour TTL)
  └─ Full operation results cached by cache_key
  └─ E.g., "valida:path/to/config.yaml" → validation result

Level 2: Config Cache (5 minutes TTL)
  └─ Parsed configuration objects
  └─ Shared across operations

Level 3: Module Cache (Permanent)
  └─ Loaded modules, plugins, binaries
  └─ Only evicted on explicit clear()
```

**Performance Impact:** Operations execute in <100ms when cached.

### 3. Event Bus - Pub-Sub Architecture

**Stable API:**
```rust
pub use daemon_cli::contracts::stable::EventBus;

// Publishing events
bus.publish(Event {
    event_type: EventType::OperationStarted,
    payload: EventPayload::Operation { name: "valida".into(), params: json!({}) },
});

// Subscribing to events
let mut subscriber = bus.subscribe();
while let Some(event) = subscriber.recv().await {
    println!("Event: {:?}", event);
}
```

**Use Cases:**
- Audit logging: publish every operation for compliance
- Real-time UI updates: push results to web dashboards
- Workflow coordination: trigger dependent operations

### 4. Dual-Mode Execution

**HTTP Daemon Mode (Default):**
```
provctl --daemon
  ↓
HTTP request to local daemon (provd)
  ↓
Daemon executes via OperationRegistry
  ↓
Response returned to client
```

**Offline Mode (Embedded):**
```
provctl --offline
  ↓
Create OfflineMode executor directly
  ↓
Execute operations in-process
  ↓
Return results immediately
```

---

## Extension Points

### Adding a New Operation

**Step 1: Implement `EcosystemOperation`**

```rust
// In your crate's main lib.rs
use daemon_cli::contracts::stable::EcosystemOperation;
use async_trait::async_trait;

pub struct MyCustomOperation;

#[async_trait]
impl EcosystemOperation for MyCustomOperation {
    fn name(&self) -> &'static str { "my-custom" }

    async fn execute(&self, params: serde_json::Value)
        -> daemon_cli::Result<OperationResult> {
        // Your implementation
        Ok(OperationResult {
            status: "success".into(),
            data: json!({ "result": "data" }),
        })
    }

    fn cache_key(&self, params: &serde_json::Value) -> String {
        format!("my-custom:{}", params["input"])
    }
}
```

**Step 2: Register with Registry**

```rust
// In daemon-cli's setup code
use daemon_cli::contracts::stable::OperationRegistry;

let registry = OperationRegistry::new();
registry.register(Arc::new(MyCustomOperation));
```

**Step 3: Access via API or CLI**

```bash
# Via REST API
curl -X POST http://localhost:9999/execute \
  -H "Content-Type: application/json" \
  -d '{"operation": "my-custom", "params": {"input": "value"}}'

# Via CLI
provctl execute my-custom --input value
```

### Adding Health Checks

**Implement custom health probe:**

```rust
use daemon_cli::contracts::observability::{HealthProbe, ProbeStatus};

struct MyServiceProbe;

#[async_trait]
impl HealthProbe for MyServiceProbe {
    fn name(&self) -> &str { "my-service" }

    async fn check(&self) -> ProbeStatus {
        if my_service_is_healthy().await {
            ProbeStatus::Healthy
        } else {
            ProbeStatus::Unhealthy("Service unavailable".into())
        }
    }
}

// Register probe
daemon.register_probe(Box::new(MyServiceProbe));
```

**Kubernetes Usage:**

```yaml
livenessProbe:
  httpGet:
    path: /health/live
    port: 9999
  initialDelaySeconds: 10
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /health/ready
    port: 9999
  initialDelaySeconds: 5
  periodSeconds: 5
```

### Adding Webhook Handlers

**Implement webhook processor:**

```rust
use daemon_cli::contracts::integration::WebHookEvent;

async fn handle_github_webhook(event: WebHookEvent) {
    match event.event_type.as_str() {
        "push" => {
            let branch = event.payload["ref"].as_str().unwrap_or("unknown");
            println!("Push to {}", branch);
            // Trigger deployment, run tests, etc.
        }
        "pull_request" => {
            println!("PR event: {:?}", event.payload["action"]);
        }
        _ => {}
    }
}

// Register webhook
daemon.register_webhook_handler("github", Box::new(handle_github_webhook)).await?;
```

---

## Best Practices

### 1. Use Contracts for Imports

```rust
// ✅ Good: Explicit stability contract
use daemon_cli::contracts::stable::EcosystemOperation;
use daemon_cli::contracts::observability::HealthProbe;

// ⚠️  Acceptable: Direct import (less clear about stability)
use daemon_cli::EcosystemOperation;

// ❌ Avoid: Implementation details
use daemon_cli::cache::CacheEntry;
```

### 2. Handle Errors Properly

```rust
use daemon_cli::contracts::stable::Result;

#[async_trait]
impl EcosystemOperation for MyOp {
    async fn execute(&self, params: serde_json::Value) -> Result<OperationResult> {
        // Use ? operator for propagation
        let value = params["required_field"]
            .as_str()
            .ok_or(DaemonError::validation("required_field is required"))?;

        Ok(OperationResult { status: "success".into(), data: json!({}) })
    }
}
```

### 3. Implement Caching for Performance

```rust
fn cache_key(&self, params: &serde_json::Value) -> String {
    // Include all inputs that affect output
    format!(
        "my-op:{}:{}",
        params["input_a"],
        params["input_b"]
    )
}
```

### 4. Publish Events for Observability

```rust
#[async_trait]
impl EcosystemOperation for MyOp {
    async fn execute(&self, params: serde_json::Value) -> Result<OperationResult> {
        // Publish start event
        bus.publish(Event {
            event_type: EventType::OperationStarted,
            payload: EventPayload::Operation { name: self.name().into(), params },
        });

        let result = self.do_work().await?;

        // Publish completion event
        bus.publish(Event {
            event_type: EventType::OperationCompleted,
            payload: EventPayload::OperationResult { name: self.name().into(), result: json!(result) },
        });

        Ok(result)
    }
}
```

### 5. Support Configuration Rendering

```rust
use daemon_cli::contracts::execution::{ConfigRenderer, RenderRequest};

async fn render_config(&self, params: serde_json::Value) -> Result<String> {
    let renderer = ConfigRenderer::new();
    let request = RenderRequest {
        language: ConfigLanguage::Tera,
        template: params["template"].as_str().unwrap().into(),
        context: params["context"].clone(),
    };

    let response = renderer.render(&request).await?;
    Ok(response.rendered_output)
}
```

---

## Integration Examples

### Example 1: Implementing a Custom Validator

**Goal:** Add a new validation operation for Helm charts.

```rust
// my_validator crate
use daemon_cli::contracts::stable::EcosystemOperation;
use async_trait::async_trait;

pub struct HelmValidatorOp;

#[async_trait]
impl EcosystemOperation for HelmValidatorOp {
    fn name(&self) -> &'static str { "helm-validator" }

    async fn execute(&self, params: serde_json::Value) -> Result<OperationResult> {
        let chart_path = params["chart_path"].as_str()?;

        // Validate Helm chart
        let is_valid = validate_helm_chart(chart_path).await?;

        Ok(OperationResult {
            status: if is_valid { "success" } else { "failed" }.into(),
            data: json!({
                "valid": is_valid,
                "chart": chart_path,
            }),
        })
    }

    fn cache_key(&self, params: &serde_json::Value) -> String {
        format!("helm-validator:{}", params["chart_path"])
    }
}
```

**Register in daemon-cli:**
```rust
registry.register(Arc::new(HelmValidatorOp));
```

**Use via CLI:**
```bash
provctl execute helm-validator --chart-path ./my-chart/
```

### Example 2: Publishing Events to External System

**Goal:** Send operation results to a monitoring system.

```rust
use daemon_cli::contracts::stable::EventBus;

pub struct MonitoringBridge {
    bus: Arc<EventBus>,
    remote_url: String,
}

impl MonitoringBridge {
    pub async fn start(self) {
        let mut subscriber = self.bus.subscribe();

        while let Some(event) = subscriber.recv().await {
            // Forward to remote monitoring system
            self.send_to_monitoring(&event).await.ok();
        }
    }

    async fn send_to_monitoring(&self, event: &Event) -> Result<()> {
        let client = reqwest::Client::new();
        client
            .post(&format!("{}/events", self.remote_url))
            .json(event)
            .send()
            .await?;
        Ok(())
    }
}
```

### Example 3: Health Check for Custom Service

```rust
use daemon_cli::contracts::observability::{HealthProbe, ProbeStatus};

pub struct DatabaseProbe {
    connection_string: String,
}

#[async_trait]
impl HealthProbe for DatabaseProbe {
    fn name(&self) -> &str { "database" }

    async fn check(&self) -> ProbeStatus {
        match connect_to_db(&self.connection_string).await {
            Ok(_) => ProbeStatus::Healthy,
            Err(e) => ProbeStatus::Unhealthy(format!("DB connection failed: {}", e)),
        }
    }
}
```

---

## Versioning & Compatibility

### Semantic Versioning

- **Major (0.x.y → 1.0.0)**: Breaking changes to stable API
- **Minor (1.x.y → 1.1.0)**: New features (backwards compatible)
- **Patch (1.0.x → 1.0.1)**: Bug fixes

### Stable API Guarantees

Within a major version:
- ✅ New fields may be added to structs (with defaults)
- ✅ New methods may be added to traits
- ❌ Existing fields/methods will not be removed or changed
- ❌ Stable types will not move to different modules

### Internal Module Changes

Between any versions:
- Internal modules (not in `contracts`) may:
  - Change function signatures
  - Reorganize code
  - Be replaced with different implementations

---

## Troubleshooting

### Operation Not Found

**Problem:** `OperationError::OperationNotFound("my-op")`

**Solution:** Ensure operation is registered before use.
```rust
registry.register(Arc::new(MyOp));  // Must happen before daemon starts
```

### Cache Stale Data

**Problem:** Operation returns old results.

**Solution:** Clear cache or use different cache key.
```rust
fn cache_key(&self, params: &serde_json::Value) -> String {
    // Include version or timestamp to invalidate when needed
    format!("my-op:{}:v{}", params["input"], params["version"])
}
```

### Webhook Events Not Received

**Problem:** WebHook handler never called.

**Solution:** Verify webhook is registered and endpoint is reachable.
```rust
daemon.register_webhook("github", handler).await?;
// Then configure webhook in GitHub to point to: http://daemon-host:9999/webhooks/github
```

### Health Check Always Fails

**Problem:** Readiness probe always returning Unhealthy.

**Solution:** Verify probe logic and dependencies are available.
```rust
async fn check(&self) -> ProbeStatus {
    // Must complete within timeout
    match timeout(Duration::from_secs(5), self.check_impl()).await {
        Ok(result) => result,
        Err(_) => ProbeStatus::Unhealthy("Check timed out".into()),
    }
}
```

---

## References

- **EcosystemOperation**: Extension point for operations
- **EventBus**: Real-time event distribution
- **OperationRegistry**: Operation discovery and execution
- **AppState**: Shared state across daemon
- **contracts module**: Stable API definitions