provisioning-platform/daemon-cli/ARCHITECTURE.md

daemon-cli Architecture

daemon-cli is the orchestration hub of the prov-ecosystem, providing unified daemon and CLI interfaces for all ecosystem services.

Table of Contents

1. Architecture Overview
2. Stable API Contracts
3. Module Organization
4. Core Concepts
5. Extension Points
6. Best Practices
7. Integration Examples

---

Architecture Overview

Design Pattern: Plugin + Hub Architecture

┌─────────────────────────────────────────────────────────┐
│                    daemon-cli                            │
│                  (Orchestration Hub)                      │
├─────────────────────────────────────────────────────────┤
│  ┌──────────────────────────────────────────────────┐  │
│  │          Core Infrastructure                      │  │
│  │  ├─ DaemonConfig (Configuration)                 │  │
│  │  ├─ EventBus (Pub-Sub)                           │  │
│  │  └─ OperationRegistry (Plugin Discovery)         │  │
│  └──────────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────────┤
│  ┌──────────────────────────────────────────────────┐  │
│  │          Execution Layers                         │  │
│  │  ├─ HTTP API (provd daemon)                      │  │
│  │  ├─ CLI Client (provctl)                         │  │
│  │  └─ Dual-mode Execution (daemon or direct)       │  │
│  └──────────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────────┤
│  ┌──────────────────────────────────────────────────┐  │
│  │          Observability & Integration              │  │
│  │  ├─ Health Probes (Kubernetes-ready)             │  │
│  │  ├─ Webhooks (Event Ingestion)                   │  │
│  │  ├─ Hierarchical Caching (3-layer LRU)           │  │
│  │  ├─ Internationalization (i18n)                  │  │
│  │  └─ Config Rendering (KCL/Nickel/Tera)          │  │
│  └──────────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────────┤
│  ┌──────────────────────────────────────────────────┐  │
│  │          Ecosystem Operations (Plugins)           │  │
│  │  ├─ valida (config validation)                   │  │
│  │  ├─ encrypt (encryption)                         │  │
│  │  ├─ runtime (container runtime detection)        │  │
│  │  ├─ init-servs (service management)              │  │
│  │  ├─ observability (monitoring)                   │  │
│  │  └─ audit (security scanning)                    │  │
│  └──────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────┘

Execution Flow

Client Request
    ↓
┌─ provctl (CLI) or HTTP API endpoint
├─ Parse input and create Operation request
├─ EventBus publishes "operation_requested" event
├─ OperationRegistry selects matching EcosystemOperation
├─ Check HierarchicalCache for cached result
├─ If miss: Execute operation (e.g., valida::ValidaOperation::execute)
├─ Cache result with TTL
├─ EventBus publishes "operation_completed" event
└─ Return result to client

---

Stable API Contracts

Why Contracts Matter

The daemon-cli module is heavily depended on by other crates (52+ dependents). To maintain compatibility:

• Stable APIs are guaranteed to remain compatible across minor version updates
• Internal modules may change between versions without notice
• Contracts module explicitly defines stability boundaries

Accessing Stable APIs

Always prefer importing from the contracts module:

// ✅ GOOD - Stable contract
use daemon_cli::contracts::stable::{EcosystemOperation, OperationRegistry};
use daemon_cli::contracts::observability::HealthProbe;

// ⚠️  OK but less clear - Direct import (same types, but unstable guarantee)
use daemon_cli::{EcosystemOperation, HealthProbe};

// ❌ AVOID - Internal implementation details
use daemon_cli::cache::HierarchicalCache;  // Not in contracts
use daemon_cli::rendering::TemplateEngine;  // Not in contracts

Stable API Categories

1. Core/Stable - Orchestration Contract

pub mod contracts::stable {
    pub use crate::core::{DaemonConfig, DaemonError, Result};
    pub use crate::orchestration::{EcosystemOperation, OperationRegistry};
    pub use crate::api::AppState;
    pub use crate::events::{Event, EventBus, EventType, EventPayload};
}

Use When:

• Implementing new operations (extend EcosystemOperation)
• Registering with OperationRegistry
• Publishing events to EventBus
• Accessing shared AppState

2. Execution/Stable - CLI and Output

pub mod contracts::execution {
    pub use crate::cli::{DaemonClient, OfflineMode, OutputFormat};
    pub use crate::config_renderer::{ConfigRenderer, RenderRequest};
}

Use When:

• Running in CLI mode (use DaemonClient or OfflineMode)
• Rendering configuration (KCL, Nickel, Tera templates)
• Formatting output (JSON, YAML, Table, etc.)

3. Observability/Stable - Monitoring

pub mod contracts::observability {
    pub use crate::health::{HealthMetrics, HealthProbe, ProbeStatus};
}

Use When:

• Implementing health check endpoints
• Collecting metrics
• Exposing readiness/liveness probes for Kubernetes

4. I18n/Stable - Internationalization

pub mod contracts::i18n {
    pub use crate::i18n::{
        I18nManager, TranslationRequest, TranslationResponse,
        TranslationContext,
    };
}

Use When:

• Supporting multiple languages in messages
• Translating output
• Managing locale-aware formatting

5. Integration/Stable - External Systems

pub mod contracts::integration {
    pub use crate::webhooks::{WebHookEvent, WebHookHandler, WebHookStore};
}

Use When:

• Ingesting webhook events
• Storing webhook configurations
• Integrating with external CI/CD systems

---

Module Organization

Tier 1: Core Infrastructure (Stable)

Module	Purpose	Stability
core	Configuration, errors, types	STABLE
orchestration	Operation trait, registry	STABLE
events	Event bus, pub-sub	STABLE

What to know:

• DaemonConfig loads from YAML/TOML/environment
• EcosystemOperation is the extension point for ecosystem crates
• EventBus uses async-broadcast for real-time events

Tier 2: Execution Layers (Stable via Contracts)

Module	Purpose	Stability
api	Axum HTTP server	STABLE (via contracts)
cli	DaemonClient + OfflineMode	STABLE (via contracts)
config_renderer	Multi-format rendering	STABLE (via contracts)

What to know:

• DaemonClient connects to HTTP daemon over network
• OfflineMode executes operations directly (no daemon)
• Choose mode based on deployment: cloud (daemon) vs embedded (offline)

Tier 3: Observability & Integration (Stable via Contracts)

Module	Purpose	Stability
health	K8s probes, metrics	STABLE (via contracts)
webhooks	Event ingestion	STABLE (via contracts)
i18n	Internationalization	STABLE (via contracts)

What to know:

• HealthProbe exposes readiness/liveness for Kubernetes
• WebHookStore persists webhook configurations
• I18nManager handles message translation

Tier 4: Internal Implementation (Implementation Details - May Change)

Module	Purpose	Note
cache	3-layer LRU caching	Internal, use contracts instead
rendering	Template engine	Internal, use config_renderer
nushell	Shell integration	Feature-gated internal

Important: These modules are NOT in contracts because they are implementation details and subject to change.

---

Core Concepts

1. EcosystemOperation - The Extension Point

The EcosystemOperation trait is how ecosystem crates (valida, encrypt, runtime, etc.) integrate:

#[async_trait]
pub trait EcosystemOperation: Send + Sync {
    fn name(&self) -> &'static str;

    async fn execute(&self, params: serde_json::Value)
        -> Result<OperationResult, OperationError>;

    fn cache_key(&self, params: &serde_json::Value) -> String;
}

Pattern: Self-Registration

// In valida crate:
pub struct ValidaOperation;

#[async_trait]
impl EcosystemOperation for ValidaOperation {
    fn name(&self) -> &'static str { "valida" }

    async fn execute(&self, params: serde_json::Value)
        -> Result<OperationResult, OperationError> {
        let config_path = params["config_path"].as_str()?;
        let rules = load_rules(config_path)?;
        validate_config(&rules)?;
        Ok(OperationResult { status: "success", data: json!({}) })
    }

    fn cache_key(&self, params: &serde_json::Value) -> String {
        format!("valida:{}", params["config_path"])
    }
}

// Registration happens via OperationRegistry::register()
registry.register(Arc::new(ValidaOperation));

2. Hierarchical Caching - Three Levels

Not exposed in stable API (implementation detail), but important to understand:

Level 1: Command Cache (1 hour TTL)
  └─ Full operation results cached by cache_key
  └─ E.g., "valida:path/to/config.yaml" → validation result

Level 2: Config Cache (5 minutes TTL)
  └─ Parsed configuration objects
  └─ Shared across operations

Level 3: Module Cache (Permanent)
  └─ Loaded modules, plugins, binaries
  └─ Only evicted on explicit clear()

Performance Impact: Operations execute in <100ms when cached.

3. Event Bus - Pub-Sub Architecture

Stable API:

pub use daemon_cli::contracts::stable::EventBus;

// Publishing events
bus.publish(Event {
    event_type: EventType::OperationStarted,
    payload: EventPayload::Operation { name: "valida".into(), params: json!({}) },
});

// Subscribing to events
let mut subscriber = bus.subscribe();
while let Some(event) = subscriber.recv().await {
    println!("Event: {:?}", event);
}

Use Cases:

• Audit logging: publish every operation for compliance
• Real-time UI updates: push results to web dashboards
• Workflow coordination: trigger dependent operations

4. Dual-Mode Execution

HTTP Daemon Mode (Default):

provctl --daemon
  ↓
HTTP request to local daemon (provd)
  ↓
Daemon executes via OperationRegistry
  ↓
Response returned to client

Offline Mode (Embedded):

provctl --offline
  ↓
Create OfflineMode executor directly
  ↓
Execute operations in-process
  ↓
Return results immediately

---

Extension Points

Adding a New Operation

Step 1: Implement EcosystemOperation

// In your crate's main lib.rs
use daemon_cli::contracts::stable::EcosystemOperation;
use async_trait::async_trait;

pub struct MyCustomOperation;

#[async_trait]
impl EcosystemOperation for MyCustomOperation {
    fn name(&self) -> &'static str { "my-custom" }

    async fn execute(&self, params: serde_json::Value)
        -> daemon_cli::Result<OperationResult> {
        // Your implementation
        Ok(OperationResult {
            status: "success".into(),
            data: json!({ "result": "data" }),
        })
    }

    fn cache_key(&self, params: &serde_json::Value) -> String {
        format!("my-custom:{}", params["input"])
    }
}

Step 2: Register with Registry

// In daemon-cli's setup code
use daemon_cli::contracts::stable::OperationRegistry;

let registry = OperationRegistry::new();
registry.register(Arc::new(MyCustomOperation));

Step 3: Access via API or CLI

# Via REST API
curl -X POST http://localhost:9999/execute \
  -H "Content-Type: application/json" \
  -d '{"operation": "my-custom", "params": {"input": "value"}}'

# Via CLI
provctl execute my-custom --input value

Adding Health Checks

Implement custom health probe:

use daemon_cli::contracts::observability::{HealthProbe, ProbeStatus};

struct MyServiceProbe;

#[async_trait]
impl HealthProbe for MyServiceProbe {
    fn name(&self) -> &str { "my-service" }

    async fn check(&self) -> ProbeStatus {
        if my_service_is_healthy().await {
            ProbeStatus::Healthy
        } else {
            ProbeStatus::Unhealthy("Service unavailable".into())
        }
    }
}

// Register probe
daemon.register_probe(Box::new(MyServiceProbe));

Kubernetes Usage:

livenessProbe:
  httpGet:
    path: /health/live
    port: 9999
  initialDelaySeconds: 10
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /health/ready
    port: 9999
  initialDelaySeconds: 5
  periodSeconds: 5

Adding Webhook Handlers

Implement webhook processor:

use daemon_cli::contracts::integration::WebHookEvent;

async fn handle_github_webhook(event: WebHookEvent) {
    match event.event_type.as_str() {
        "push" => {
            let branch = event.payload["ref"].as_str().unwrap_or("unknown");
            println!("Push to {}", branch);
            // Trigger deployment, run tests, etc.
        }
        "pull_request" => {
            println!("PR event: {:?}", event.payload["action"]);
        }
        _ => {}
    }
}

// Register webhook
daemon.register_webhook_handler("github", Box::new(handle_github_webhook)).await?;

---

Best Practices

1. Use Contracts for Imports

// ✅ Good: Explicit stability contract
use daemon_cli::contracts::stable::EcosystemOperation;
use daemon_cli::contracts::observability::HealthProbe;

// ⚠️  Acceptable: Direct import (less clear about stability)
use daemon_cli::EcosystemOperation;

// ❌ Avoid: Implementation details
use daemon_cli::cache::CacheEntry;

2. Handle Errors Properly

use daemon_cli::contracts::stable::Result;

#[async_trait]
impl EcosystemOperation for MyOp {
    async fn execute(&self, params: serde_json::Value) -> Result<OperationResult> {
        // Use ? operator for propagation
        let value = params["required_field"]
            .as_str()
            .ok_or(DaemonError::validation("required_field is required"))?;

        Ok(OperationResult { status: "success".into(), data: json!({}) })
    }
}

3. Implement Caching for Performance

fn cache_key(&self, params: &serde_json::Value) -> String {
    // Include all inputs that affect output
    format!(
        "my-op:{}:{}",
        params["input_a"],
        params["input_b"]
    )
}

4. Publish Events for Observability

#[async_trait]
impl EcosystemOperation for MyOp {
    async fn execute(&self, params: serde_json::Value) -> Result<OperationResult> {
        // Publish start event
        bus.publish(Event {
            event_type: EventType::OperationStarted,
            payload: EventPayload::Operation { name: self.name().into(), params },
        });

        let result = self.do_work().await?;

        // Publish completion event
        bus.publish(Event {
            event_type: EventType::OperationCompleted,
            payload: EventPayload::OperationResult { name: self.name().into(), result: json!(result) },
        });

        Ok(result)
    }
}

5. Support Configuration Rendering

use daemon_cli::contracts::execution::{ConfigRenderer, RenderRequest};

async fn render_config(&self, params: serde_json::Value) -> Result<String> {
    let renderer = ConfigRenderer::new();
    let request = RenderRequest {
        language: ConfigLanguage::Tera,
        template: params["template"].as_str().unwrap().into(),
        context: params["context"].clone(),
    };

    let response = renderer.render(&request).await?;
    Ok(response.rendered_output)
}

---

Integration Examples

Example 1: Implementing a Custom Validator

Goal: Add a new validation operation for Helm charts.

// my_validator crate
use daemon_cli::contracts::stable::EcosystemOperation;
use async_trait::async_trait;

pub struct HelmValidatorOp;

#[async_trait]
impl EcosystemOperation for HelmValidatorOp {
    fn name(&self) -> &'static str { "helm-validator" }

    async fn execute(&self, params: serde_json::Value) -> Result<OperationResult> {
        let chart_path = params["chart_path"].as_str()?;

        // Validate Helm chart
        let is_valid = validate_helm_chart(chart_path).await?;

        Ok(OperationResult {
            status: if is_valid { "success" } else { "failed" }.into(),
            data: json!({
                "valid": is_valid,
                "chart": chart_path,
            }),
        })
    }

    fn cache_key(&self, params: &serde_json::Value) -> String {
        format!("helm-validator:{}", params["chart_path"])
    }
}

Register in daemon-cli:

registry.register(Arc::new(HelmValidatorOp));

Use via CLI:

provctl execute helm-validator --chart-path ./my-chart/

Example 2: Publishing Events to External System

Goal: Send operation results to a monitoring system.

use daemon_cli::contracts::stable::EventBus;

pub struct MonitoringBridge {
    bus: Arc<EventBus>,
    remote_url: String,
}

impl MonitoringBridge {
    pub async fn start(self) {
        let mut subscriber = self.bus.subscribe();

        while let Some(event) = subscriber.recv().await {
            // Forward to remote monitoring system
            self.send_to_monitoring(&event).await.ok();
        }
    }

    async fn send_to_monitoring(&self, event: &Event) -> Result<()> {
        let client = reqwest::Client::new();
        client
            .post(&format!("{}/events", self.remote_url))
            .json(event)
            .send()
            .await?;
        Ok(())
    }
}

Example 3: Health Check for Custom Service

use daemon_cli::contracts::observability::{HealthProbe, ProbeStatus};

pub struct DatabaseProbe {
    connection_string: String,
}

#[async_trait]
impl HealthProbe for DatabaseProbe {
    fn name(&self) -> &str { "database" }

    async fn check(&self) -> ProbeStatus {
        match connect_to_db(&self.connection_string).await {
            Ok(_) => ProbeStatus::Healthy,
            Err(e) => ProbeStatus::Unhealthy(format!("DB connection failed: {}", e)),
        }
    }
}

---

Versioning & Compatibility

Semantic Versioning

• Major (0.x.y → 1.0.0): Breaking changes to stable API
• Minor (1.x.y → 1.1.0): New features (backwards compatible)
• Patch (1.0.x → 1.0.1): Bug fixes

Stable API Guarantees

Within a major version:

• ✅ New fields may be added to structs (with defaults)
• ✅ New methods may be added to traits
• ❌ Existing fields/methods will not be removed or changed
• ❌ Stable types will not move to different modules

Internal Module Changes

Between any versions:

• Internal modules (not in contracts) may:
  • Change function signatures
  • Reorganize code
  • Be replaced with different implementations

---

Troubleshooting

Operation Not Found

Problem: OperationError::OperationNotFound("my-op")

Solution: Ensure operation is registered before use.

registry.register(Arc::new(MyOp));  // Must happen before daemon starts

Cache Stale Data

Problem: Operation returns old results.

Solution: Clear cache or use different cache key.

fn cache_key(&self, params: &serde_json::Value) -> String {
    // Include version or timestamp to invalidate when needed
    format!("my-op:{}:v{}", params["input"], params["version"])
}

Webhook Events Not Received

Problem: WebHook handler never called.

Solution: Verify webhook is registered and endpoint is reachable.

daemon.register_webhook("github", handler).await?;
// Then configure webhook in GitHub to point to: http://daemon-host:9999/webhooks/github

Health Check Always Fails

Problem: Readiness probe always returning Unhealthy.

Solution: Verify probe logic and dependencies are available.

async fn check(&self) -> ProbeStatus {
    // Must complete within timeout
    match timeout(Duration::from_secs(5), self.check_impl()).await {
        Ok(result) => result,
        Err(_) => ProbeStatus::Unhealthy("Check timed out".into()),
    }
}

---

References

• EcosystemOperation: Extension point for operations
• EventBus: Real-time event distribution
• OperationRegistry: Operation discovery and execution
• AppState: Shared state across daemon
• contracts module: Stable API definitions