chore: update README and CHANGELOG with ops control plane, audit-mirror, contract-tests

.cargo/config.toml

@@ -0,0 +1,2 @@
+[build]
+target-dir = "/Volumes/Devel/provisioning/platform/target"

CHANGELOG.md

@@ -0,0 +1,138 @@
+# Platform Changelog
+
+## [Unreleased]
+
+### Added
+
+#### ops control plane (`crates/ops-keeper`, `crates/ops-controller`)
+
+New two-component ops control plane implementing secure, auditable operation dispatch.
+
+**ops-keeper** — policy gate before any operation reaches the orchestrator:
+- Glob-based `PolicyDef` matching on `op_type`, `image_patterns`, `target_patterns`
+- `Signer` issues compact Ed25519 JWTs (`OpsClaims`) on policy approval
+- NATS JetStream pull consumer on `ops.pending.*`
+- `AuditEvent` emission to `ops.audit.*` on every terminal decision
+
+**ops-controller** — durable executor with at-least-once delivery:
+- Verifies keeper-signed JWT before dispatching to orchestrator HTTP API
+- SurrealDB idempotency store; `reconcile_pending` reconciles stale ops on startup
+- Structured `AckResult` (Ack / Nak / Term) per JetStream semantics
+- Emits audit events to `ops.audit.*` on every terminal outcome
+
+**ADR**: ADR-038 (ops control plane design)
+
+#### audit-mirror (`crates/audit-mirror`)
+
+Sidecar that consumes `ops.audit.*` and creates a signed git commit per event in a local Radicle repository, then announces the repo. JTI deduplication prevents double-commits on consumer redelivery.
+
+**ADR**: ADR-038
+
+#### contract-tests (`crates/contract-tests`)
+
+G3 contract test suite verifying semantic equivalence across three tiers:
+- **Tier A**: direct registry invocation (reference baseline)
+- **Tier B**: axum HTTP daemon on ephemeral port
+- **Tier C**: in-process MCP `handle_request`
+
+Normaliser strips volatile fields; JSON schema validates every tier output against `listing_output_schema`. Regression guard for CLI↔HTTP↔MCP drift.
+
+#### control-center: NATS→WebSocket bridge (`src/handlers/nats_bridge.rs`)
+
+Eliminates long-polling between control-center UI and orchestrator. Durable JetStream consumer `cc-task-status-bridge` subscribes to `TASKS` stream and re-broadcasts each `TaskStatusPayload` to all connected WebSocket clients via `WebSocketManager`.
+
+#### API catalog endpoints (`crates/ai-service`, `crates/control-center`)
+
+`GET /api-catalog` added to ai-service and control-center. Each handler uses `inventory::iter::<ApiRouteEntry>()` to emit a sorted JSON catalog of all registered routes — self-describing API surface without a separate OpenAPI toolchain.
+
+#### SOLID boundary pre-commit check (`.pre-commit-hooks/solid-boundary-check.sh`)
+
+Shell hook enforced in `.pre-commit-config.yaml`. Fails the commit if a crate imports across declared SOLID layer boundaries, keeping dependency direction consistent before code reaches CI.
+
+### Removed / Lifted
+
+- **`crates/backup-manager`** — lifted to `cloudatasave` (LibreCloud/cloudDataSave). **ADR-041**.
+- **`crates/buildkit-launcher`** — lifted to `lian-build`. **ADR-040**.
+- **`crates/daemon`** — `Cargo.toml` removed; binary consolidated into `provisioning-daemon`.
+- **`prov-ecosystem/crates/nu-daemon`** / **`daemon-cli`** — excluded from workspace; `rustls=0.23.28` pin conflicts with surrealdb@3 (`^0.23.36`). Build standalone with `cargo build -p nu-daemon` until `nu-command` relaxes the pin.
+
+---
+
+## [3.6.0] — 2026-04-17
+
+### Added
+
+#### ncl-sync daemon (`crates/ncl-sync/`)
+
+New Rust daemon that eliminates `nickel export` latency from CLI commands by pre-compiling NCL files and maintaining a shared cache.
+
+- **File watching**: `notify` watcher on workspace NCL directories; re-exports automatically on file change
+- **Warm-up**: on `prvng platform start`, scans all NCL files in the workspace and exports stale entries concurrently (configurable concurrency)
+- **Shared cache**: `~/.cache/provisioning/config-cache/` — same directory and key strategy as `nu_plugin_nickel`
+- **Content-addressed keys**: `SHA256(file_content + sorted_import_paths + format)` — zero coordination overhead between daemon and plugin
+- **Sync requests**: Nu processes write `.sync-<pid>.json` sidecars after mutations; daemon drains them within 500 ms
+- **CLI subcommands**: `daemon`, `warm`, `invalidate`, `key`, `stats`
+- **Config-driven** via `platform-config`: `platform/config/ncl-sync.ncl` — idle timeout, concurrency, poll interval, extra import paths
+- **No platform dependencies**: intentionally avoids NATS, SurrealDB, and orchestrator to prevent bootstrap circularity
+
+#### Platform lifecycle integration
+
+- `prvng platform start` now launches `ncl-sync daemon` (via `ncl-sync-start` in `service-manager.nu`)
+- `prvng platform stop` stops ncl-sync (via PID file)
+- `prvng platform status` shows ncl-sync running state
+
+#### Nu cache layer
+
+- `lib_provisioning/config/cache/core.nu`: implemented `cache-lookup`, `cache-write`, `write-sync-request` (previously no-ops)
+- `lib_provisioning/config/cache/nickel.nu`: implemented `lookup-nickel-cache`, `derive-ncl-cache-key`, `request-ncl-sync`
+
+#### nickel_processor wrappers
+
+New `lib_provisioning/utils/nickel_processor.nu` functions:
+- `ncl-eval`: drop-in for `^nickel export ... | from json` — checks plugin cache, propagates error on failure
+- `ncl-eval-soft`: soft-failure variant with configurable fallback value
+
+#### nu_plugin_nickel updates
+
+- `nickel-eval` and `nickel-export` now accept `--import-path LIST` flag (`-I`)
+- New command `nickel-cache-key`: prints the cache key for a file (parity testing)
+- Cache key derivation updated to `SHA256(file_content + sorted_import_paths + format)` — aligned with ncl-sync
+- Cache key now includes import paths: same file with different import paths produces different cache entries
+
+#### Hot-path migration (C1)
+
+Replaced `^nickel export ... | from json` with `ncl-eval`/`ncl-eval-soft` in the four highest-frequency call sites:
+- `main_provisioning/dispatcher.nu` — commands-registry load
+- `main_provisioning/components.nu` — workspace settings export
+- `main_provisioning/workflow.nu` — workflow NCL exports
+- `main_provisioning/extensions.nu` — per-extension metadata
+
+#### Extended migration (C2 + C3)
+
+Migrated remaining high-value call sites (~55 additional sites across 30+ files) including `config/export.nu`, `taskservs/discover.nu`, `servers/create.nu`, `ontoref-queries.nu`, `service-manager.nu`, `dag.nu`, and others.
+
+#### ADRs
+
+- `adrs/adr-022-ncl-sync-daemon.ncl`: daemon design decisions, key strategy rationale, constraint enforcement
+- `adrs/adr-023-ncl-export-wrapper.ncl`: `ncl-eval`/`ncl-eval-soft` wrapper design, migration strategy
+
+#### Tests
+
+- `tests/cache/test_key_parity.nu`: validates that ncl-sync and nu_plugin_nickel produce identical keys for the same `(file, import_paths, format)` triple
+
+### Performance
+
+| Command | Before | After (warm cache) |
+|---------|--------|--------------------|
+| `prvng component list` | 3–7 s | ~1.5 s |
+| `prvng workflow list` | 3–5 s | ~1.5 s |
+| `prvng deploy` | 15–30 s | ~3–5 s |
+| Multi-export commands (ontoref) | 12–30 s | ~1.5 s |
+
+Nu module parse startup (~1.2 s) is unaffected — separate concern.
+
+---
+
+## [3.5.0] — 2025-10-07
+
+Initial platform version with orchestrator, control center, installer, MCP server, vault service, and extension registry.

Cargo.toml

@@ -11,24 +11,39 @@ members = [
   "crates/control-center",
   "crates/control-center-ui",
   "crates/vault-service",
-  "crates/detector",
   "crates/mcp-server",
-  "crates/daemon",
+  # lifted: "crates/backup-manager" → cloudatasave (LibreCloud/cloudDataSave) — adr-041
-  "prov-ecosystem/crates/daemon-cli",
+  # archived: "crates/detector" → archive/detector (no dependents, stale since Jan 2026)
   "prov-ecosystem/crates/machines",
   "prov-ecosystem/crates/encrypt",
   "prov-ecosystem/crates/backup",
   "prov-ecosystem/crates/observability",
+  "crates/ncl-sync",
+  "crates/prvng-cli",
+  "crates/provisioning-core",
+  "crates/provisioning-tool",
+  "crates/provisioning-daemon",
+  "crates/contract-tests",
+  "crates/extension-manager",
+  "crates/ops-keeper",
+  "crates/audit-mirror",
+  "crates/ops-controller",
+  # lifted: "crates/buildkit-launcher" → lian-build — adr-040
 ]
 
 exclude = [
-  "syntaxis",
+  # archived: syntaxis/ → archive/syntaxis (Jan 2026, all refs commented out)
-  "syntaxis/core",
+  # archived: stratumiops/ → archive/stratumiops (Jan 2026, workspace uses canonical ../../../Development/stratumiops)
   "prov-ecosystem/crates/syntaxis-integration",
   "prov-ecosystem/crates/audit",
   "prov-ecosystem/crates/valida",
   "prov-ecosystem/crates/runtime",
   "prov-ecosystem/crates/gitops",
+  # nu-daemon + daemon-cli are excluded: nu-command@0.110.0 (via nushell feature) pins
+  # rustls=0.23.28, hard conflict with surrealdb@3 (requires ^0.23.36). Not resolvable
+  # until nu-command relaxes its rustls pin. Build standalone: cargo build -p nu-daemon
+  "crates/nu-daemon",
+  "prov-ecosystem/crates/daemon-cli",
 ]
 
 resolver = "2"
@@ -97,12 +112,16 @@ resolver = "2"
   # DATABASE AND STORAGE
   # ============================================================================
   sqlx = { version = "0.8", features = ["runtime-tokio-rustls", "sqlite", "chrono", "uuid"] }
-  surrealdb = { version = "2.6", features = ["kv-mem", "protocol-ws", "protocol-http"] }
+  # kv-surrealkv: core relational/graph (orchestrator state, control-center)
+  # kv-rocksdb:   hot data (embeddings cache, audit logs) — via platform-db embedded-rocksdb feature
+  # rustls excluded: nu-command@0.110.0 pins rustls=0.23.28, SurrealDB 3 requires ^0.23.36 (conflict)
+  # TLS for remote connections is handled at the proxy layer (nginx/Caddy) in production.
+  surrealdb = { version = "3", features = ["kv-mem", "kv-surrealkv", "protocol-ws"], default-features = false }
 
   # ============================================================================
   # MESSAGING (NATS)
   # ============================================================================
-  async-nats = "0.40"
+  async-nats = "0.46"
 
   # ============================================================================
   # SECURITY AND CRYPTOGRAPHY
@@ -112,7 +131,7 @@ resolver = "2"
   base64 = "0.22"
   git2 = { version = "0.20", default-features = false, features = ["https", "ssh"] }
   hmac = "0.12"
-  jsonwebtoken = { version = "10.3", features = ["rust_crypto"] }
+  jsonwebtoken = { version = "10.3", default-features = false, features = ["aws_lc_rs"] }
   rand = { version = "0.9", features = ["std_rng", "os_rng"] }
   ring = "0.17"
   sha2 = "0.10"
@@ -127,12 +146,18 @@ resolver = "2"
   # ============================================================================
   regex = "1.12"
   validator = { version = "0.20", features = ["derive"] }
+  globset = "0.4"
 
   # ============================================================================
   # GRAPH ALGORITHMS AND UTILITIES
   # ============================================================================
   petgraph = "0.8"
 
+  # ============================================================================
+  # CONCURRENT DATA STRUCTURES
+  # ============================================================================
+  dashmap = "6"
+
   # ============================================================================
   # ADDITIONAL SHARED DEPENDENCIES
   # ============================================================================
@@ -250,10 +275,8 @@ resolver = "2"
   xxhash-rust = { version = "0.8", features = ["xxh3"] }
 
   # ============================================================================
-  # RAG FRAMEWORK DEPENDENCIES (Rig)
+  # RAG AND TEXT PROCESSING
   # ============================================================================
-  rig-core = "0.30"
-  rig-surrealdb = "0.1"
   tokenizers = "0.22"
 
   # ============================================================================
@@ -261,8 +284,8 @@ resolver = "2"
   # ============================================================================
   moka = { version = "0.12", features = ["future"] }
   sled = "0.34"
-  fastembed = "5.8"
+  fastembed = "5.11"
-  lancedb = "0.23"
+  lancedb = "0.26"
   arrow = "=56"
 
   # ============================================================================
@@ -271,30 +294,38 @@ resolver = "2"
   platform-config = { path = "./crates/platform-config" }
   platform-nats = { path = "./crates/platform-nats" }
   platform-db = { path = "./crates/platform-db" }
-  service-clients = { path = "./crates/service-clients" }
+  platform-clients = { path = "./crates/service-clients" }
-  rag = { path = "./crates/rag" }
+  platform-rag = { path = "./crates/rag" }
-  mcp-server = { path = "./crates/mcp-server" }
+  provisioning-mcp = { path = "./crates/mcp-server" }
   ai-service = { path = "./crates/ai-service" }
 
   # ============================================================================
   # PROV-ECOSYSTEM (Now members of workspace)
   # ============================================================================
   daemon-cli = { path = "./prov-ecosystem/crates/daemon-cli" }
-  machines = { path = "./prov-ecosystem/crates/machines" }
+  platform-machines = { path = "./prov-ecosystem/crates/machines" }
-  encrypt = { path = "./prov-ecosystem/crates/encrypt" }
+  platform-encrypt = { path = "./prov-ecosystem/crates/encrypt" }
-  backup = { path = "./prov-ecosystem/crates/backup" }
+  platform-backup = { path = "./prov-ecosystem/crates/backup" }
-  observability = { path = "./prov-ecosystem/crates/observability" }
+  platform-observability = { path = "./prov-ecosystem/crates/observability" }
   init-servs = { path = "./prov-ecosystem/crates/init-servs" }
 
-  # stratum-embeddings and stratum-llm are built in isolated Docker context for RAG
+  # ============================================================================
-  # See: crates/rag/docker/Dockerfile
+  # ONTOREF PROTOCOL ADOPTION (API catalog surface — Phase 4)
-  stratum-embeddings = { path = "./stratumiops/crates/stratum-embeddings", features = ["openai-provider", "ollama-provider", "fastembed-provider", "memory-cache"] }
+  # ============================================================================
-  stratum-llm = { path = "./stratumiops/crates/stratum-llm", features = ["anthropic", "openai", "ollama"] }
+  ontoref-ontology = { path = "../../ontoref/crates/ontoref-ontology", features = ["derive"] }
+  ontoref-derive   = { path = "../../ontoref/crates/ontoref-derive" }
+  inventory        = "0.3"
+
+  # Stratum ecosystem — sourced from canonical stratumiops repo (SurrealDB v3 throughout)
+  stratum-embeddings = { path = "../../stratumiops/crates/stratum-embeddings", features = ["openai-provider", "ollama-provider", "fastembed-provider", "memory-cache"] }
+  stratum-llm = { path = "../../stratumiops/crates/stratum-llm", features = ["anthropic", "openai", "ollama"] }
+  stratum-graph = { path = "../../stratumiops/crates/stratum-graph" }
+  stratum-state = { path = "../../stratumiops/crates/stratum-state" }
 
   # ============================================================================
   # SECRETUMVAULT (Enterprise Secrets Management - canonical source)
   # ============================================================================
-  secretumvault = { path = "../../../Development/secretumvault", features = ["surrealdb-storage", "filesystem", "server", "cedar"] }
+  secretumvault = { path = "../../secretumvault", features = ["surrealdb-storage", "filesystem", "server", "cedar"] }
 
   # ============================================================================
   # WASM/WEB-SPECIFIC DEPENDENCIES

README.md

@@ -204,7 +204,71 @@ OCI-compliant registry for extension distribution and versioning.
 
 ---
 
-### 7. **API Gateway** (`infrastructure/api-gateway/`)
+### 7. **ops-keeper** (`crates/ops-keeper/`)
+
+Policy-based operation gate — signs approved operations with Ed25519 keys before forwarding to the control plane.
+
+**Language**: Rust
+
+**Purpose**: Operation approval, policy enforcement, and keeper-signed JWT emission
+
+**Key Features**:
+
+- Glob-based `PolicyDef` matching against op type, image patterns, and target patterns
+- `Signer` wraps an Ed25519 key pair; emits compact JWTs (`OpsClaims`) on approval
+- `PendingOp` tracking with NATS JetStream durable consumer (`ops.pending.*`)
+- `AuditEvent` emission to `ops.audit.*` stream on approval or rejection
+- Nickel-driven policy config (`keeper_policy.ncl`)
+
+**Status**: ✅ Active Development
+
+---
+
+### 8. **ops-controller** (`crates/ops-controller/`)
+
+NATS JetStream consumer that processes keeper-approved operations, calls the orchestrator, and enforces idempotency via SurrealDB.
+
+**Language**: Rust
+
+**Purpose**: Durable control plane execution with at-least-once delivery guarantees
+
+**Key Features**:
+
+- Pull consumer on `ops.pending.*` JetStream stream
+- Ed25519 JWT verification of keeper-signed claims before dispatch
+- Idempotency check via SurrealDB; reconciles stale pending ops on startup
+- Orchestrator HTTP dispatch with structured `AckResult` (Ack/Nak/Term)
+- Audit emission (`ops.audit.*`) on every terminal outcome
+
+**Status**: ✅ Active Development
+
+**ADR**: ADR-038 (ops control plane design)
+
+---
+
+### 9. **audit-mirror** (`crates/audit-mirror/`)
+
+Sidecar that consumes `ops.audit.*` NATS events and mirrors each event as a signed git commit into a Radicle repository.
+
+**Language**: Rust
+
+**Purpose**: Immutable, content-addressed audit trail via Radicle git storage
+
+**Key Features**:
+
+- NATS JetStream pull consumer on `ops.audit.*`
+- JTI deduplication — skips already-committed event IDs via `git log` scan
+- `commit_writer` creates signed commits with the audit payload as the blob
+- `radicle_publish` announces the repo to the Radicle network after each commit
+- Configurable via CLI flags (NATS URL, workspace, Radicle repo path, key path)
+
+**Status**: ✅ Active Development
+
+**ADR**: ADR-038
+
+---
+
+### 10. **API Gateway** (`infrastructure/api-gateway/`)
 
 Unified REST API gateway for external integration.
 
@@ -218,46 +282,113 @@ Unified REST API gateway for external integration.
 - Authentication and authorization
 - Rate limiting and throttling
 - API versioning
-- Request validation
-- Metrics and monitoring
 
 **Status**: 🔄 Planned
 
-**Endpoints** (Planned):
+---
 
-- `/api/v1/servers/*` - Server management
+### 11. **Extension Registry** (`crates/extension-registry/`)
-- `/api/v1/taskservs/*` - Task service operations
+
-- `/api/v1/clusters/*` - Cluster operations
+Registry and catalog for browsing, discovering, and distributing extensions.
-- `/api/v1/workflows/*` - Workflow management
+
+**Language**: Rust
+
+**Purpose**: Extension discovery, metadata management, and OCI/Forgejo-backed distribution
+
+**Status**: ✅ Active Development
 
 ---
 
-### 8. **Extension Registry** (`extension-registry/`)
+### 12. **contract-tests** (`crates/contract-tests/`)
 
-Registry and catalog for browsing and discovering extensions.
+G3 contract test suite — verifies semantic equivalence across the CLI↔HTTP↔MCP tier stack.
 
-**Purpose**: Extension discovery and metadata management
+**Language**: Rust (test crate)
+
+**Purpose**: Prevent drift between registry, HTTP daemon, and MCP server response shapes
 
 **Key Features**:
 
-- Extension catalog
+- Tier A: direct registry invocation (reference baseline)
-- Search and filtering
+- Tier B: axum HTTP server on `127.0.0.1:0` (ephemeral port)
-- Version history
+- Tier C: in-process MCP `handle_request`
-- Dependency information
+- Normaliser strips volatile fields (`trace_id`, `timestamp`) — asserts semantic, not byte-for-byte equality
-- Documentation links
+- JSON schema validation against `listing_output_schema` on every tier
-- Community ratings (future)
 
-**Status**: 🔄 Planned
+**Status**: ✅ Active Development
 
 ---
 
-### 9. **Provisioning Server** (`provisioning-server/`)
+### 13. **ncl-sync** (`crates/ncl-sync/`)
 
-Alternative provisioning service implementation.
+Nickel configuration sync daemon — compiles NCL to JSON proactively and maintains a shared cache for all Nu processes.
 
-**Purpose**: Additional provisioning service capabilities
+**Language**: Rust
 
-**Status**: 🔄 In Development
+**Purpose**: Eliminate `nickel export` latency (~2–5s per call) from CLI commands by pre-compiling NCL files and serving results from an in-memory-backed file cache.
+
+**Key Features**:
+
+- File watcher (`notify`) on workspace NCL directories — re-exports on change automatically
+- Warm-up on `prvng platform start` — first command of the day already finds cache hot
+- Shared cache at `~/.cache/provisioning/config-cache/` used by both this daemon and `nu_plugin_nickel`
+- Content-addressed keys: `SHA256(file_content + sorted_import_paths + format)` — identical to plugin key strategy, zero coordination overhead
+- Post-operation sync: Nu writes `.sync-<pid>.json` sidecar after mutations; daemon re-exports within 500 ms
+- Configurable via `platform/config/ncl-sync.ncl` (idle timeout, concurrency, poll interval)
+- No NATS, no SurrealDB, no platform service dependencies — intentional to avoid bootstrap circularity
+
+**Status**: ✅ Production Ready
+
+**Install**:
+
+```bash
+cargo build --release --package ncl-sync
+install -m 0755 target/release/provisioning-ncl-sync ~/.local/bin/provisioning-ncl-sync
+```
+
+**Usage**:
+
+```bash
+# Start daemon for a workspace
+ncl-sync daemon --workspace ~/workspaces/libre-daoshi
+
+# One-shot warm-up
+ncl-sync warm ~/workspaces/libre-daoshi
+
+# Evict a specific file from cache
+ncl-sync invalidate settings.ncl
+
+# Print cache key (parity testing)
+ncl-sync key settings.ncl --import-path /ws --import-path /prov
+
+# Cache statistics
+ncl-sync stats
+```
+
+**Lifecycle integration**: Started automatically by `prvng platform start`, stopped by `prvng platform stop`. Status visible in `prvng platform status`.
+
+**Performance impact** (with warm cache):
+
+| Command | Before | After |
+|---------|--------|-------|
+| `prvng component list` | ~3–7 s | ~1.5 s |
+| `prvng workflow list` | ~3–5 s | ~1.5 s |
+| `prvng deploy` | ~15–30 s | ~3–5 s |
+
+**Configuration** (`platform/config/ncl-sync.ncl`):
+
+```nickel
+{
+  ncl_sync = {
+    idle_timeout_secs = 600,     # daemon auto-shutdown after N seconds idle
+    sync_poll_interval_ms = 500, # how often to check for sync-request sidecars
+    warm_concurrency = 4,        # max parallel nickel export during warm-up
+    extra_import_paths = [],     # additional import paths beyond workspace + $PROVISIONING
+  }
+}
+```
+
+**ADRs**: [ADR-022](../adrs/adr-022-ncl-sync-daemon.ncl) (daemon design), [ADR-023](../adrs/adr-023-ncl-export-wrapper.ncl) (Nu wrapper strategy)
 
 ---
 
@@ -345,16 +476,27 @@ Systemd service units for platform services.
 │   └──────────────┘  └──────────────┘  └──────────────┘       │
 │                                                              │
 │   ┌──────────────┐  ┌──────────────┐  ┌──────────────┐       │
-│   │  Installer   │  │ OCI Registry │  │  Extension   │       │
+│   │  Installer   │  │ Extension    │  │  ops-keeper  │       │
-│   │  (Rust/Nu)   │  │              │  │  Registry    │       │
+│   │  (Rust/Nu)   │  │  Registry    │  │  (Rust)      │       │
 │   └──────────────┘  └──────────────┘  └──────────────┘       │
+│                                                              │
+│   ┌──────────────┐  ┌──────────────┐                         │
+│   │ops-controller│  │ audit-mirror │                         │
+│   │   (Rust)     │  │   (Rust)     │                         │
+│   └──────────────┘  └──────────────┘                         │
+│                                                              │
+│   ┌──────────────────────────────────────────────────────┐   │
+│   │  ncl-sync daemon  (Rust)                             │   │
+│   │  ~/.cache/provisioning/config-cache/  ←→  Nu procs  │   │
+│   └──────────────────────────────────────────────────────┘   │
 └──────────────────────────────────────────────────────────────┘
                              ↓
 ┌──────────────────────────────────────────────────────────────┐
 │                Data & State Layer                            │
-│  • SurrealDB (State Management)                              │
+│  • NATS JetStream (ops.pending.*, ops.audit.*, TASKS)        │
+│  • SurrealDB (State Management, Idempotency)                 │
+│  • Radicle (Immutable Audit Log via git)                     │
 │  • File-based Persistence (Checkpoints)                      │
-│  • Configuration Storage                                     │
 └──────────────────────────────────────────────────────────────┘
 ```
 
@@ -373,11 +515,13 @@ Systemd service units for platform services.
 ### Key Dependencies
 
 - **tokio** - Async runtime for Rust services
-- **axum** / **actix-web** - Web frameworks
+- **axum** - Web framework (control-center, orchestrator, provisioning-daemon)
+- **async-nats** - NATS JetStream client (ops-keeper, ops-controller, audit-mirror, control-center)
+- **surrealdb** - State management and idempotency store
 - **serde** - Serialization/deserialization
-- **bollard** - Docker API client (test environments)
 - **ratatui** - Terminal UI framework (installer)
-- **SurrealDB** - State management database
+- **git2** - Radicle git integration (audit-mirror)
+- **jsonwebtoken** - Ed25519 JWT signing/verification (ops-keeper, ops-controller)
 
 ---
 
@@ -477,19 +621,23 @@ nu run.nu
 
 ```bash
 platform/
-├── orchestrator/       # Rust orchestrator service
+├── crates/
-├── control-center/     # Rust control center backend
+│   ├── orchestrator/       # Rust orchestrator service
-├── control-center-ui/  # Web frontend
+│   ├── control-center/     # Rust control center backend
-├── installer/          # Rust/Nushell installer
+│   ├── control-center-ui/  # Web frontend
-├── mcp-server/         # Nushell MCP server
+│   ├── mcp-server/         # Nushell MCP server
-├── infrastructure/api-gateway/        # Rust API gateway (planned)
+│   ├── ncl-sync/           # Nickel config sync daemon
-├── infrastructure/oci-registry/       # OCI registry (planned)
+│   └── ...
-├── extension-registry/ # Extension catalog (planned)
+├── config/
-├── provisioning-server/# Alternative service
+│   ├── ncl-sync.ncl        # ncl-sync daemon configuration
-├── infrastructure/docker/     # Docker Compose configs
+│   └── external-services.ncl
-├── k8s/                # Kubernetes manifests
+├── infrastructure/
-├── infrastructure/systemd/            # Systemd units
+│   ├── api-gateway/        # Rust API gateway (planned)
-└── docs/               # Platform documentation
+│   ├── oci-registry/       # OCI registry (planned)
+│   ├── docker/             # Docker Compose configs
+│   ├── systemd/            # Systemd units
+│   └── ...
+└── docs/                   # Platform documentation
 ```
 
 ### Adding New Services
@@ -569,5 +717,5 @@ For platform service issues:
 ---
 
 **Maintained By**: Platform Team
-**Last Updated**: 2025-10-07
+**Last Updated**: 2026-05-12
-**Platform Version**: 3.5.0
+**Platform Version**: 3.6.0

crates/ai-service/Cargo.toml

@@ -20,6 +20,11 @@ axum = { workspace = true }
 tower = { workspace = true, features = ["full"] }
 tower-http = { workspace = true, features = ["cors", "trace"] }
 
+# Ontoref API catalog
+ontoref-ontology = { workspace = true }
+ontoref-derive   = { workspace = true }
+inventory        = { workspace = true }
+
 # Serialization
 serde = { workspace = true, features = ["derive"] }
 serde_json = { workspace = true }
@@ -29,7 +34,7 @@ toml = { workspace = true }
 platform-config = { workspace = true }
 
 # Centralized observability (logging, metrics, health, tracing)
-observability = { workspace = true, features = ["logging", "metrics-prometheus", "health"] }
+platform-observability = { workspace = true, features = ["logging", "metrics-prometheus", "health"] }
 
 # Error handling
 anyhow = { workspace = true }
@@ -47,10 +52,10 @@ uuid = { workspace = true, features = ["v4", "serde"] }
 clap = { workspace = true, features = ["derive"] }
 
 # RAG crate for AI capabilities
-rag = { workspace = true }
+platform-rag = { workspace = true }
 
 # MCP server tools for real implementations
-mcp-server = { workspace = true }
+provisioning-mcp = { workspace = true }
 
 # Graph operations for DAG
 petgraph = { workspace = true }

crates/ai-service/src/api_catalog.rs

@@ -0,0 +1,12 @@
+use axum::{extract::State, response::IntoResponse, Json};
+use ontoref_ontology::api::ApiRouteEntry;
+use serde_json::json;
+use std::sync::Arc;
+
+use crate::service::AiService;
+
+pub async fn api_catalog(State(_state): State<Arc<AiService>>) -> impl IntoResponse {
+    let mut routes: Vec<&'static ApiRouteEntry> = inventory::iter::<ApiRouteEntry>().collect();
+    routes.sort_by(|a, b| a.path.cmp(b.path).then(a.method.cmp(b.method)));
+    Json(json!({ "service": "ai-service", "routes": routes }))
+}

crates/ai-service/src/handlers.rs

@@ -9,6 +9,7 @@ use axum::{
     routing::{get, post},
     Json, Router,
 };
+use ontoref_derive::onto_api;
 use serde_json::json;
 use tracing::debug;
 
@@ -27,10 +28,19 @@ pub fn create_routes(state: Arc<AiService>) -> Router {
             get(get_best_practices_handler),
         )
         .route("/health", get(health_check_handler))
+        .route("/api/catalog", get(crate::api_catalog::api_catalog))
         .with_state(state)
 }
 
-/// Call an MCP tool
+#[onto_api(
+    method = "POST",
+    path = "/api/v1/ai/mcp/tool",
+    description = "Invoke an MCP tool by name with arguments",
+    auth = "bearer",
+    actors = "developer, agent",
+    tags = "ai, mcp",
+    feature = ""
+)]
 async fn call_mcp_tool_handler(
     State(service): State<Arc<AiService>>,
     Json(req): Json<McpToolRequest>,
@@ -45,7 +55,15 @@ async fn call_mcp_tool_handler(
     Ok(Json(response))
 }
 
-/// Ask AI a question (RAG-powered)
+#[onto_api(
+    method = "POST",
+    path = "/api/v1/ai/ask",
+    description = "Ask a RAG-powered question grounded in Nickel schemas and deployment history",
+    auth = "bearer",
+    actors = "developer, agent",
+    tags = "ai, rag",
+    feature = ""
+)]
 async fn ask_handler(
     State(service): State<Arc<AiService>>,
     Json(req): Json<AskRequest>,
@@ -57,7 +75,15 @@ async fn ask_handler(
     Ok(Json(response))
 }
 
-/// Get extension dependency DAG
+#[onto_api(
+    method = "GET",
+    path = "/api/v1/ai/dag/extensions",
+    description = "Get the extension dependency DAG used by AI for schema-aware config generation",
+    auth = "bearer",
+    actors = "developer, agent",
+    tags = "ai, dag, extensions",
+    feature = ""
+)]
 async fn get_extension_dag_handler(
     State(service): State<Arc<AiService>>,
 ) -> Result<Json<DagResponse>, InternalError> {
@@ -71,7 +97,15 @@ async fn get_extension_dag_handler(
     Ok(Json(dag))
 }
 
-/// Get best practices for a category
+#[onto_api(
+    method = "GET",
+    path = "/api/v1/ai/knowledge/best-practices",
+    description = "Get best practices for a given category from the knowledge base",
+    auth = "bearer",
+    actors = "developer, agent",
+    tags = "ai, knowledge",
+    feature = ""
+)]
 async fn get_best_practices_handler(
     State(service): State<Arc<AiService>>,
     axum::extract::Query(params): axum::extract::Query<std::collections::HashMap<String, String>>,
@@ -91,7 +125,15 @@ async fn get_best_practices_handler(
     Ok(Json(practices))
 }
 
-/// Health check endpoint
+#[onto_api(
+    method = "GET",
+    path = "/health",
+    description = "AI service health check",
+    auth = "none",
+    actors = "developer, agent, ci",
+    tags = "health",
+    feature = ""
+)]
 async fn health_check_handler(
     State(service): State<Arc<AiService>>,
 ) -> Result<StatusCode, InternalError> {

crates/ai-service/src/lib.rs

@@ -4,6 +4,7 @@
 //! Exposes Claude-based question answering, MCP tool execution, extension
 //! dependency graphs, and best practice recommendations via HTTP API.
 
+pub mod api_catalog;
 pub mod config;
 pub mod dag;
 pub mod handlers;

crates/ai-service/src/main.rs

@@ -31,6 +31,11 @@ struct Args {
     /// Service bind port
     #[arg(short = 'p', long, default_value_t = DEFAULT_PORT)]
     port: u16,
+
+    /// Print all #[onto_api] registered routes as JSON and exit.
+    /// Pipe to api-catalog-ai-service.json: `just export-api-catalog`
+    #[arg(long)]
+    dump_api_catalog: bool,
 }
 
 #[tokio::main]
@@ -38,6 +43,11 @@ async fn main() -> anyhow::Result<()> {
     // Parse CLI arguments FIRST (so --help works before any other processing)
     let args = Args::parse();
 
+    if args.dump_api_catalog {
+        println!("{}", ontoref_ontology::api::dump_catalog_json());
+        return Ok(());
+    }
+
     // Initialize centralized observability (logging, metrics, health checks)
     let _guard = observability::init_from_env("ai-service", env!("CARGO_PKG_VERSION"))?;
 

crates/audit-mirror/Cargo.toml

@@ -0,0 +1,30 @@
+[package]
+authors.workspace = true
+edition.workspace = true
+license.workspace = true
+name = "audit-mirror"
+version.workspace = true
+description = "NATS ops.audit.* → Radicle git commit sidecar with jti idempotency (ADR-038)"
+
+[[bin]]
+name = "audit-mirror"
+path = "src/main.rs"
+
+[dependencies]
+async-nats     = { workspace = true }
+tokio          = { workspace = true, features = ["full"] }
+serde          = { workspace = true }
+serde_json     = { workspace = true }
+thiserror      = { workspace = true }
+anyhow         = { workspace = true }
+tracing        = { workspace = true }
+tracing-subscriber = { workspace = true }
+chrono         = { workspace = true }
+clap           = { workspace = true }
+futures        = { workspace = true }
+git2           = { workspace = true }
+bytes          = { workspace = true }
+reqwest        = { workspace = true }
+
+[dev-dependencies]
+tempfile       = { workspace = true }

crates/audit-mirror/src/commit_writer.rs

@@ -0,0 +1,99 @@
+use git2::{Repository, Signature};
+
+use crate::error::MirrorError;
+
+/// Writes a single commit into the HEAD of `repo` embedding `jti` in the message.
+///
+/// The commit message format is:
+///   audit: jti=<jti>
+///
+///   <json_payload>
+///
+/// This is the canonical format that `jti_check::already_committed` searches for.
+/// An empty tree is used because the audit repo contains only commit-message blobs;
+/// no working-tree files are modified.
+pub fn write_audit_commit(
+    repo: &Repository,
+    jti: &str,
+    json_payload: &str,
+    author_name: &str,
+    author_email: &str,
+) -> Result<git2::Oid, MirrorError> {
+    let sig = Signature::now(author_name, author_email)
+        .map_err(|e| MirrorError::Git(e.to_string()))?;
+
+    let tree_id = {
+        let mut index = repo.index().map_err(|e| MirrorError::Git(e.to_string()))?;
+        // Write tree from current index (may be empty — no working-tree blobs needed)
+        index.write_tree().map_err(|e| MirrorError::Git(e.to_string()))?
+    };
+
+    let tree = repo
+        .find_tree(tree_id)
+        .map_err(|e| MirrorError::Git(e.to_string()))?;
+
+    let message = format!("audit: jti={jti}\n\n{json_payload}");
+
+    // Parent is HEAD if it exists; first commit for an empty (unborn) branch has no parent.
+    let parent_commit;
+    let parents: Vec<&git2::Commit<'_>>;
+    match repo.head() {
+        Ok(head_ref) => {
+            parent_commit = head_ref
+                .peel_to_commit()
+                .map_err(|e| MirrorError::Git(e.to_string()))?;
+            parents = vec![&parent_commit];
+        }
+        Err(e) if e.code() == git2::ErrorCode::UnbornBranch => {
+            parents = vec![];
+        }
+        Err(e) => return Err(MirrorError::Git(e.to_string())),
+    }
+
+    let oid = repo
+        .commit(Some("HEAD"), &sig, &sig, &message, &tree, &parents)
+        .map_err(|e| MirrorError::Git(e.to_string()))?;
+
+    Ok(oid)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use tempfile::TempDir;
+
+    fn init_repo() -> (TempDir, Repository) {
+        let dir = TempDir::new().unwrap();
+        let repo = Repository::init(dir.path()).unwrap();
+        let mut cfg = repo.config().unwrap();
+        cfg.set_str("user.name", "test").unwrap();
+        cfg.set_str("user.email", "test@test.com").unwrap();
+        (dir, repo)
+    }
+
+    #[test]
+    fn writes_commit_to_empty_repo() {
+        let (_dir, repo) = init_repo();
+        let oid = write_audit_commit(&repo, "jti-001", r#"{"event":"test"}"#, "ci", "ci@ci.local").unwrap();
+        let commit = repo.find_commit(oid).unwrap();
+        assert!(commit.message().unwrap().contains("jti=jti-001"));
+        assert!(commit.message().unwrap().contains(r#"{"event":"test"}"#));
+    }
+
+    #[test]
+    fn writes_second_commit_with_correct_parent() {
+        let (_dir, repo) = init_repo();
+        let first = write_audit_commit(&repo, "jti-001", "{}", "ci", "ci@ci.local").unwrap();
+        let second = write_audit_commit(&repo, "jti-002", "{}", "ci", "ci@ci.local").unwrap();
+        let commit = repo.find_commit(second).unwrap();
+        assert_eq!(commit.parent_id(0).unwrap(), first);
+    }
+
+    #[test]
+    fn message_contains_jti_marker() {
+        let (_dir, repo) = init_repo();
+        let oid = write_audit_commit(&repo, "abc-xyz", r#"{"op":"deploy"}"#, "keeper", "keeper@ops").unwrap();
+        let msg = repo.find_commit(oid).unwrap().message().unwrap().to_string();
+        assert!(msg.starts_with("audit: jti=abc-xyz\n\n"));
+    }
+}

crates/audit-mirror/src/error.rs

@@ -0,0 +1,19 @@
+use thiserror::Error;
+
+#[derive(Debug, Error)]
+pub enum MirrorError {
+    #[error("git error: {0}")]
+    Git(String),
+    #[error("NATS error: {0}")]
+    Nats(String),
+    #[error("json error: {0}")]
+    Json(#[from] serde_json::Error),
+    #[error("io error: {0}")]
+    Io(#[from] std::io::Error),
+}
+
+impl From<git2::Error> for MirrorError {
+    fn from(e: git2::Error) -> Self {
+        MirrorError::Git(e.to_string())
+    }
+}

crates/audit-mirror/src/jti_check.rs

@@ -0,0 +1,92 @@
+use git2::Repository;
+
+use crate::error::MirrorError;
+
+/// Returns true if a commit with the given jti already exists in the HEAD ancestor chain.
+/// Searches commit messages for the marker `jti=<value>` added by commit_writer.
+///
+/// This is the idempotency guard required by ADR-038 constraint audit-mirror-idempotent-on-jti.
+pub fn already_committed(repo: &Repository, jti: &str) -> Result<bool, MirrorError> {
+    let head = match repo.head() {
+        Ok(r) => r,
+        Err(e) if e.code() == git2::ErrorCode::UnbornBranch => return Ok(false),
+        Err(e) => return Err(MirrorError::Git(e.to_string())),
+    };
+
+    let head_commit = head
+        .peel_to_commit()
+        .map_err(|e| MirrorError::Git(e.to_string()))?;
+
+    let marker = format!("jti={jti}");
+
+    let mut revwalk = repo
+        .revwalk()
+        .map_err(|e| MirrorError::Git(e.to_string()))?;
+    revwalk
+        .push(head_commit.id())
+        .map_err(|e| MirrorError::Git(e.to_string()))?;
+    revwalk.set_sorting(git2::Sort::TOPOLOGICAL | git2::Sort::TIME)
+        .map_err(|e| MirrorError::Git(e.to_string()))?;
+
+    for oid in revwalk {
+        let oid = oid.map_err(|e| MirrorError::Git(e.to_string()))?;
+        let commit = repo
+            .find_commit(oid)
+            .map_err(|e| MirrorError::Git(e.to_string()))?;
+        if commit.message().unwrap_or("").contains(&marker) {
+            return Ok(true);
+        }
+    }
+
+    Ok(false)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use tempfile::TempDir;
+
+    fn init_bare_repo_with_commit(message: &str) -> (TempDir, Repository) {
+        let dir = TempDir::new().unwrap();
+        let repo = Repository::init(dir.path()).unwrap();
+
+        {
+            let mut config = repo.config().unwrap();
+            config.set_str("user.name", "test").unwrap();
+            config.set_str("user.email", "test@test.com").unwrap();
+        }
+
+        let sig = git2::Signature::now("test", "test@test.com").unwrap();
+        {
+            let tree_id = {
+                let mut index = repo.index().unwrap();
+                index.write_tree().unwrap()
+            };
+            let tree = repo.find_tree(tree_id).unwrap();
+            repo.commit(Some("HEAD"), &sig, &sig, message, &tree, &[])
+                .unwrap();
+            // tree drops here so repo can be moved below
+        }
+
+        (dir, repo)
+    }
+
+    #[test]
+    fn detects_existing_jti_in_head_commit() {
+        let (_dir, repo) = init_bare_repo_with_commit("audit: jti=abc-123\n\npayload");
+        assert!(already_committed(&repo, "abc-123").unwrap());
+    }
+
+    #[test]
+    fn returns_false_for_unknown_jti() {
+        let (_dir, repo) = init_bare_repo_with_commit("audit: jti=abc-123\n\npayload");
+        assert!(!already_committed(&repo, "unknown-jti").unwrap());
+    }
+
+    #[test]
+    fn returns_false_for_empty_repo() {
+        let dir = TempDir::new().unwrap();
+        let repo = Repository::init(dir.path()).unwrap();
+        assert!(!already_committed(&repo, "any-jti").unwrap());
+    }
+}

crates/audit-mirror/src/main.rs

@@ -0,0 +1,188 @@
+mod commit_writer;
+mod error;
+mod jti_check;
+mod radicle_publish;
+
+use std::path::PathBuf;
+
+use async_nats::jetstream::{self, consumer::PullConsumer};
+use bytes::Bytes;
+use clap::Parser;
+use futures::StreamExt;
+use git2::Repository;
+use tracing::{error, info, warn};
+
+use commit_writer::write_audit_commit;
+use jti_check::already_committed;
+use radicle_publish::announce_repo;
+
+#[derive(Parser)]
+#[command(name = "audit-mirror", about = "NATS ops.audit.* → Radicle git commit sidecar (ADR-038)")]
+struct Cli {
+    /// NATS server URL
+    #[arg(long, default_value = "nats://127.0.0.1:4222")]
+    nats_url: String,
+
+    /// Workspace name (used to derive stream and consumer names)
+    #[arg(long)]
+    workspace: String,
+
+    /// Path to the local Radicle storage repo for this workspace's state
+    /// (e.g. ~/.radicle/storage/<rid>)
+    #[arg(long)]
+    state_repo_path: PathBuf,
+
+    /// Radicle Repository ID (RID) for gossip announce
+    #[arg(long)]
+    rid: String,
+
+    /// Author name embedded in git commits
+    #[arg(long, default_value = "audit-mirror")]
+    author_name: String,
+
+    /// Author email embedded in git commits
+    #[arg(long, default_value = "audit-mirror@ops.local")]
+    author_email: String,
+
+    /// Radicle local HTTP API base URL
+    #[arg(long, default_value = "http://127.0.0.1:8776")]
+    radicle_api_url: String,
+}
+
+#[tokio::main]
+async fn main() -> Result<(), Box<dyn std::error::Error>> {
+    tracing_subscriber::fmt()
+        .with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
+        .init();
+
+    let cli = Cli::parse();
+
+    let nats = async_nats::connect(&cli.nats_url).await?;
+    let js = jetstream::new(nats);
+
+    let stream_name = format!("OPS_AUDIT_{}", cli.workspace.to_uppercase());
+    let consumer_name = format!("{}-audit-mirror", cli.workspace);
+
+    // Ensure stream exists (idempotent — ADR-038 requires the stream be pre-provisioned
+    // by ops-controller, so we only access it here, not create it).
+    let stream = js.get_stream(&stream_name).await.map_err(|e| {
+        format!("stream '{stream_name}' not found — ensure ops-controller has provisioned it: {e}")
+    })?;
+
+    let consumer: PullConsumer = stream
+        .get_or_create_consumer(
+            &consumer_name,
+            jetstream::consumer::pull::Config {
+                durable_name: Some(consumer_name.clone()),
+                ack_policy: jetstream::consumer::AckPolicy::Explicit,
+                filter_subject: format!("ops.audit.{}.>", cli.workspace),
+                ..Default::default()
+            },
+        )
+        .await?;
+
+    info!(
+        stream = stream_name,
+        consumer = consumer_name,
+        repo = ?cli.state_repo_path,
+        "audit-mirror ready"
+    );
+
+    let repo = Repository::open(&cli.state_repo_path).map_err(|e| {
+        format!("cannot open state repo at {:?}: {e}", cli.state_repo_path)
+    })?;
+
+    let mut messages = consumer.messages().await?;
+
+    while let Some(msg_result) = messages.next().await {
+        let msg = match msg_result {
+            Ok(m) => m,
+            Err(e) => {
+                error!("NATS message error: {e}");
+                continue;
+            }
+        };
+
+        let jti = match extract_jti(&msg.payload) {
+            Some(j) => j,
+            None => {
+                warn!(subject = %msg.subject, "no jti in audit payload; acking and skipping");
+                let _ = msg.ack().await;
+                continue;
+            }
+        };
+
+        match already_committed(&repo, &jti) {
+            Ok(true) => {
+                info!(jti, "jti already committed — idempotent skip");
+                let _ = msg.ack().await;
+                continue;
+            }
+            Ok(false) => {}
+            Err(e) => {
+                error!(jti, error = %e, "jti_check failed; leaving in queue for retry");
+                continue;
+            }
+        }
+
+        let payload_str = String::from_utf8_lossy(&msg.payload);
+
+        match write_audit_commit(
+            &repo,
+            &jti,
+            &payload_str,
+            &cli.author_name,
+            &cli.author_email,
+        ) {
+            Ok(oid) => {
+                info!(jti, commit = %oid, "audit commit written");
+            }
+            Err(e) => {
+                error!(jti, error = %e, "commit_writer failed; leaving in queue for retry");
+                continue;
+            }
+        }
+
+        // Best-effort gossip announce — non-fatal, commit is durable.
+        if let Err(e) = announce_repo(&cli.rid, &cli.radicle_api_url).await {
+            warn!(jti, error = %e, "announce_repo failed (non-fatal)");
+        }
+
+        if let Err(e) = msg.ack().await {
+            warn!(jti, error = %e, "ack failed (JetStream will redeliver; jti_check will deduplicate)");
+        }
+    }
+
+    warn!("NATS message stream ended; audit-mirror exiting");
+    Ok(())
+}
+
+/// Extracts the `jti` field from a JSON payload.
+fn extract_jti(payload: &Bytes) -> Option<String> {
+    let v: serde_json::Value = serde_json::from_slice(payload).ok()?;
+    v.get("jti")?.as_str().map(str::to_owned)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use bytes::Bytes;
+
+    #[test]
+    fn extracts_jti_from_payload() {
+        let payload = Bytes::from(r#"{"jti":"abc-123","event":"deploy"}"#);
+        assert_eq!(extract_jti(&payload), Some("abc-123".to_string()));
+    }
+
+    #[test]
+    fn returns_none_for_missing_jti() {
+        let payload = Bytes::from(r#"{"event":"deploy"}"#);
+        assert_eq!(extract_jti(&payload), None);
+    }
+
+    #[test]
+    fn returns_none_for_invalid_json() {
+        let payload = Bytes::from(b"not-json" as &[u8]);
+        assert_eq!(extract_jti(&payload), None);
+    }
+}

crates/audit-mirror/src/radicle_publish.rs

@@ -0,0 +1,35 @@
+use tracing::{info, warn};
+
+/// Signals the local Radicle node to announce updated refs for `rid`.
+///
+/// Radicle Heartwood exposes an HTTP API at `http://127.0.0.1:8776`. After
+/// writing a new audit commit to the Radicle storage repo, we POST to the node
+/// sync endpoint so gossip propagates immediately rather than waiting for the
+/// node's background scan interval.
+///
+/// Failure here is non-fatal: the commit is already durable in the local git
+/// store and radicled will eventually pick it up via its storage-directory
+/// watcher. We log a warning and return Ok(()) so the caller can ack the
+/// JetStream message.
+pub async fn announce_repo(rid: &str, node_api_url: &str) -> Result<(), crate::error::MirrorError> {
+    let url = format!("{node_api_url}/v1/repos/{rid}/sync");
+
+    let client = reqwest::Client::builder()
+        .timeout(std::time::Duration::from_secs(5))
+        .build()
+        .map_err(|e| crate::error::MirrorError::Nats(format!("HTTP client init: {e}")))?;
+
+    match client.post(&url).send().await {
+        Ok(resp) if resp.status().is_success() => {
+            info!(rid, "Radicle repo sync triggered successfully");
+        }
+        Ok(resp) => {
+            warn!(rid, status = %resp.status(), "Radicle sync returned non-success; node will sync on next scan");
+        }
+        Err(e) => {
+            warn!(rid, error = %e, "Radicle sync request failed (non-fatal; local commit is durable)");
+        }
+    }
+
+    Ok(())
+}

crates/backup-manager/Cargo.toml

@@ -0,0 +1,39 @@
+[package]
+authors.workspace = true
+edition.workspace = true
+license.workspace = true
+name = "backup-manager"
+repository.workspace = true
+version.workspace = true
+
+description = "Multi-context backup orchestrator: restic-first with kopia opt-in, multi-destination replication, ontoref-driven docs. One-shot, daemon, standalone, or coordinator modes. Distinct from platform-backup (../prov-ecosystem/crates/backup) which is a low-level multi-backend library; backup-manager is the orchestrator binary."
+keywords = ["backup", "restic", "kopia", "orchestrator", "ontoref"]
+categories = ["command-line-utilities", "development-tools"]
+
+[lib]
+name = "backup_manager"
+path = "src/lib.rs"
+
+[[bin]]
+name = "prvng-backup"
+path = "src/main.rs"
+
+[dependencies]
+platform-config = { workspace = true }
+
+anyhow      = { workspace = true }
+async-trait = { workspace = true }
+chrono      = { workspace = true }
+clap        = { version = "4", features = ["derive", "env"] }
+serde       = { workspace = true }
+serde_json  = { workspace = true }
+thiserror   = { workspace = true }
+tokio       = { workspace = true, features = ["full"] }
+toml        = { workspace = true }
+tracing     = { workspace = true }
+tracing-subscriber = { version = "0.3", features = ["env-filter"] }
+uuid        = { workspace = true }
+
+[dev-dependencies]
+tempfile    = { workspace = true }
+tokio-test  = { workspace = true }

crates/backup-manager/src/config.rs

@@ -0,0 +1,329 @@
+//! Configuration loading for the backup-manager via [`platform_config::ConfigLoader`].
+//!
+//! Configuration files are searched along the standard hierarchy
+//! (`~/.config/provisioning/platform/config/backup-manager.ncl` on Linux,
+//! `~/Library/Application Support/...` on macOS) and merged with environment
+//! variable overrides (`BACKUP_*` mapped to JSON paths). Nickel contracts
+//! validate the merged configuration BEFORE serde deserialisation, so type
+//! errors surface as Nickel error messages, not serde "expected struct field".
+//!
+//! See ADR-015 (`adrs/adr-015-config-driven-platform-config.ncl`) for the
+//! mandate that every platform crate uses this loader.
+
+use std::path::Path;
+
+use platform_config::ConfigLoader;
+use serde::{Deserialize, Serialize};
+
+/// Top-level configuration for the backup-manager binary across all four modes.
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+pub struct BackupManagerConfig {
+    /// Daemon-mode runtime parameters. Ignored in `one-shot` and `standalone`.
+    #[serde(default)]
+    pub daemon: DaemonConfig,
+
+    /// Provider configuration (default provider, version pinning).
+    #[serde(default)]
+    pub providers: ProvidersConfig,
+
+    /// Path (or glob) to the directory holding policy Nickel files.
+    /// Resolved at runtime by invoking `nickel export` on each match.
+    #[serde(default)]
+    pub policies: PoliciesRef,
+
+    /// secretumvault client configuration.
+    #[serde(default)]
+    pub vault: VaultClientConfig,
+
+    /// platform-nats client configuration.
+    #[serde(default)]
+    pub nats: NatsClientConfig,
+
+    /// Prometheus metrics emission configuration.
+    #[serde(default)]
+    pub metrics: MetricsConfig,
+}
+
+/// Daemon-mode parameters: priority queue, concurrency, API endpoint.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct DaemonConfig {
+    /// HTTP API listen address (e.g. `127.0.0.1:9099`).
+    #[serde(default = "default_daemon_http_addr")]
+    pub http_addr: String,
+
+    /// Unix socket path for local CLI communication.
+    #[serde(default = "default_daemon_unix_socket")]
+    pub unix_socket: String,
+
+    /// Maximum concurrent backup operations across all destinations.
+    #[serde(default = "default_max_parallel_backups")]
+    pub max_parallel_backups: usize,
+
+    /// Maximum concurrent operations per destination.
+    #[serde(default = "default_max_parallel_per_destination")]
+    pub max_parallel_per_destination: usize,
+
+    /// Path where the daemon persists its state (queue, last-run timestamps).
+    #[serde(default = "default_state_path")]
+    pub state_path: String,
+}
+
+impl Default for DaemonConfig {
+    fn default() -> Self {
+        Self {
+            http_addr: default_daemon_http_addr(),
+            unix_socket: default_daemon_unix_socket(),
+            max_parallel_backups: default_max_parallel_backups(),
+            max_parallel_per_destination: default_max_parallel_per_destination(),
+            state_path: default_state_path(),
+        }
+    }
+}
+
+fn default_daemon_http_addr() -> String { "127.0.0.1:9099".to_string() }
+fn default_daemon_unix_socket() -> String { "/run/prvng-backup.sock".to_string() }
+fn default_max_parallel_backups() -> usize { 4 }
+fn default_max_parallel_per_destination() -> usize { 2 }
+fn default_state_path() -> String { "/var/lib/prvng-backup/state".to_string() }
+
+/// Provider defaults and version pinning.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ProvidersConfig {
+    /// Provider used when a `BackupPolicy` does not specify one explicitly.
+    #[serde(default = "default_provider_name")]
+    pub default: String,
+
+    /// Minimum acceptable restic CLI version. The daemon emits a warning if
+    /// the installed binary is older. Format: `semver` ("0.16.0").
+    #[serde(default)]
+    pub restic_min_version: Option<String>,
+
+    /// Minimum acceptable kopia CLI version, when kopia is referenced.
+    #[serde(default)]
+    pub kopia_min_version: Option<String>,
+}
+
+impl Default for ProvidersConfig {
+    fn default() -> Self {
+        Self {
+            default: default_provider_name(),
+            restic_min_version: None,
+            kopia_min_version: None,
+        }
+    }
+}
+
+fn default_provider_name() -> String { "restic".to_string() }
+
+/// Reference to the directory tree where `BackupPolicy` / `BackupGroup` /
+/// `SystemBackupDef` Nickel files are committed.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct PoliciesRef {
+    /// Root directory containing component policies (`infra/<workspace>/components/`).
+    #[serde(default = "default_components_path")]
+    pub components_path: String,
+
+    /// Path to the BackupGroup declarations file.
+    #[serde(default = "default_groups_path")]
+    pub groups_path: String,
+
+    /// Path to the SystemBackupDef declarations file.
+    #[serde(default = "default_system_backups_path")]
+    pub system_backups_path: String,
+
+    /// Nickel `--import-path` value used when exporting policies.
+    #[serde(default = "default_nickel_import_path")]
+    pub nickel_import_path: String,
+}
+
+impl Default for PoliciesRef {
+    fn default() -> Self {
+        Self {
+            components_path: default_components_path(),
+            groups_path: default_groups_path(),
+            system_backups_path: default_system_backups_path(),
+            nickel_import_path: default_nickel_import_path(),
+        }
+    }
+}
+
+fn default_components_path() -> String {
+    "infra/libre-wuji/components".to_string()
+}
+fn default_groups_path() -> String {
+    "infra/libre-wuji/backup-groups.ncl".to_string()
+}
+fn default_system_backups_path() -> String {
+    "infra/libre-wuji/system-backups.ncl".to_string()
+}
+fn default_nickel_import_path() -> String { "provisioning".to_string() }
+
+/// secretumvault connection parameters.
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+pub struct VaultClientConfig {
+    /// Vault HTTP endpoint (e.g. `https://vault.libre-wuji.svc.cluster.local:8200`).
+    #[serde(default)]
+    pub endpoint: Option<String>,
+
+    /// Path inside vault where the manager's NKey/JWT lives for auth.
+    #[serde(default)]
+    pub auth_path: Option<String>,
+
+    /// Audit log target (subject prefix or file path).
+    #[serde(default)]
+    pub audit_target: Option<String>,
+}
+
+/// platform-nats connection parameters (mirrors `NatsConnectionConfig`).
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+pub struct NatsClientConfig {
+    /// NATS URL (e.g. `nats://nats.ops-system.svc:4222`).
+    #[serde(default)]
+    pub url: Option<String>,
+
+    /// Path inside vault where the NKey seed is stored.
+    #[serde(default)]
+    pub nkey_seed_vault_path: Option<String>,
+
+    /// Require signed messages on subscribed subjects.
+    #[serde(default)]
+    pub require_signed_messages: bool,
+}
+
+/// Prometheus metrics configuration.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MetricsConfig {
+    /// Metrics emission mode. Daemon mode uses HTTP; one-shot uses textfile.
+    #[serde(default)]
+    pub mode: MetricsMode,
+
+    /// HTTP listen address when `mode = "http"`.
+    #[serde(default = "default_metrics_http_addr")]
+    pub http_addr: String,
+
+    /// Textfile path when `mode = "textfile"` (consumed by node_exporter).
+    #[serde(default = "default_metrics_textfile_path")]
+    pub textfile_path: String,
+}
+
+impl Default for MetricsConfig {
+    fn default() -> Self {
+        Self {
+            mode: MetricsMode::default(),
+            http_addr: default_metrics_http_addr(),
+            textfile_path: default_metrics_textfile_path(),
+        }
+    }
+}
+
+/// Metrics emission mode.
+#[derive(Debug, Clone, Copy, Serialize, Deserialize, Default, PartialEq, Eq)]
+#[serde(rename_all = "lowercase")]
+pub enum MetricsMode {
+    /// HTTP endpoint scraped by Prometheus (used by daemon mode).
+    #[default]
+    Http,
+    /// Textfile written to a directory scraped by node_exporter (used by one-shot).
+    Textfile,
+}
+
+fn default_metrics_http_addr() -> String { "127.0.0.1:9100".to_string() }
+fn default_metrics_textfile_path() -> String {
+    "/var/lib/node-exporter/textfile_collector/prvng-backup.prom".to_string()
+}
+
+impl ConfigLoader for BackupManagerConfig {
+    fn service_name() -> &'static str {
+        "backup-manager"
+    }
+
+    fn collect_env_overrides() -> serde_json::Value {
+        // Map `BACKUP_*` environment variables to nested JSON paths matching
+        // the structure of `BackupManagerConfig`. Only set keys are included;
+        // the merge happens inside `platform_config::format::load_config_with_overrides`.
+        let mut overrides = serde_json::Map::new();
+
+        let mut daemon = serde_json::Map::new();
+        if let Ok(v) = std::env::var("BACKUP_DAEMON_HTTP_ADDR") {
+            daemon.insert("http_addr".into(), serde_json::Value::String(v));
+        }
+        if let Ok(v) = std::env::var("BACKUP_DAEMON_UNIX_SOCKET") {
+            daemon.insert("unix_socket".into(), serde_json::Value::String(v));
+        }
+        if !daemon.is_empty() {
+            overrides.insert("daemon".into(), serde_json::Value::Object(daemon));
+        }
+
+        let mut vault = serde_json::Map::new();
+        if let Ok(v) = std::env::var("BACKUP_VAULT_ENDPOINT") {
+            vault.insert("endpoint".into(), serde_json::Value::String(v));
+        }
+        if !vault.is_empty() {
+            overrides.insert("vault".into(), serde_json::Value::Object(vault));
+        }
+
+        let mut nats = serde_json::Map::new();
+        if let Ok(v) = std::env::var("BACKUP_NATS_URL") {
+            nats.insert("url".into(), serde_json::Value::String(v));
+        }
+        if !nats.is_empty() {
+            overrides.insert("nats".into(), serde_json::Value::Object(nats));
+        }
+
+        let mut providers = serde_json::Map::new();
+        if let Ok(v) = std::env::var("BACKUP_PROVIDER_DEFAULT") {
+            providers.insert("default".into(), serde_json::Value::String(v));
+        }
+        if !providers.is_empty() {
+            overrides.insert("providers".into(), serde_json::Value::Object(providers));
+        }
+
+        serde_json::Value::Object(overrides)
+    }
+
+    fn from_path<P: AsRef<Path>>(
+        path: P,
+    ) -> std::result::Result<Self, Box<dyn std::error::Error + Send + Sync>> {
+        let path = path.as_ref();
+        let overrides = Self::collect_env_overrides();
+        let json_value = platform_config::format::load_config_with_overrides(path, &overrides)
+            .map_err(|e| -> Box<dyn std::error::Error + Send + Sync> { Box::new(e) })?;
+
+        serde_json::from_value(json_value).map_err(|e| {
+            let err_msg = format!(
+                "Failed to deserialize backup-manager config from {:?}: {}",
+                path, e
+            );
+            Box::new(std::io::Error::new(std::io::ErrorKind::InvalidData, err_msg))
+                as Box<dyn std::error::Error + Send + Sync>
+        })
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn defaults_are_consistent() {
+        let cfg = BackupManagerConfig::default();
+        assert_eq!(cfg.daemon.http_addr, "127.0.0.1:9099");
+        assert_eq!(cfg.providers.default, "restic");
+        assert_eq!(cfg.metrics.mode, MetricsMode::Http);
+        assert!(!cfg.nats.require_signed_messages);
+    }
+
+    #[test]
+    fn service_name_matches_practice_node_artifact_path() {
+        assert_eq!(BackupManagerConfig::service_name(), "backup-manager");
+    }
+
+    #[test]
+    fn env_overrides_emit_only_set_keys() {
+        // Without env vars set, the override map should be empty.
+        // (The test runs in an isolated process; if BACKUP_* leaks from the
+        // host shell the assertion is loose: just ensure it's a JSON object.)
+        let v = BackupManagerConfig::collect_env_overrides();
+        assert!(v.is_object());
+    }
+}

crates/backup-manager/src/error.rs

@@ -0,0 +1,72 @@
+//! Error types for the backup-manager crate.
+//!
+//! All public APIs return [`Result<T>`] which is shorthand for
+//! `std::result::Result<T, BackupError>`.
+
+use thiserror::Error;
+
+/// Top-level error type for the backup-manager.
+#[derive(Debug, Error)]
+pub enum BackupError {
+    /// Configuration could not be loaded or validated.
+    #[error("configuration error: {0}")]
+    Config(String),
+
+    /// A policy file could not be deserialised or violated a contract.
+    #[error("policy error: {0}")]
+    Policy(String),
+
+    /// A backup provider (restic, kopia) failed.
+    #[error("provider '{provider}' error: {message}")]
+    Provider {
+        /// Provider identifier (`restic`, `kopia`, ...).
+        provider: String,
+        /// Underlying error message from the provider invocation.
+        message: String,
+    },
+
+    /// A destination was unreachable or rejected the operation.
+    #[error("destination '{destination}' error: {message}")]
+    Destination {
+        /// Destination identifier (matches `BackupPolicy.destinations[].name`).
+        destination: String,
+        /// Underlying error.
+        message: String,
+    },
+
+    /// The vault (secretumvault) returned an error or was unreachable.
+    #[error("vault error: {0}")]
+    Vault(String),
+
+    /// NATS publish/subscribe error.
+    #[error("nats error: {0}")]
+    Nats(String),
+
+    /// A Kubernetes API call failed.
+    #[error("kubernetes error: {0}")]
+    Kubernetes(String),
+
+    /// Filesystem I/O error.
+    #[error("io error: {0}")]
+    Io(#[from] std::io::Error),
+
+    /// JSON (de)serialisation error.
+    #[error("json error: {0}")]
+    Json(#[from] serde_json::Error),
+
+    /// A pre/post hook exited non-zero.
+    #[error("hook '{hook}' failed: exit {exit_code}")]
+    HookFailed {
+        /// Hook identifier (`pre`, `post`, or a named hook).
+        hook: String,
+        /// Process exit code.
+        exit_code: i32,
+    },
+
+    /// Any other anyhow-flavoured failure that does not fit the categories above.
+    #[error("{0}")]
+    Other(#[from] anyhow::Error),
+}
+
+/// Crate-local result alias.
+pub type Result<T> = std::result::Result<T, BackupError>;

crates/backup-manager/src/lib.rs

@@ -0,0 +1,60 @@
+//! Multi-context backup orchestrator: restic-first with kopia opt-in, multi-destination replication, ontoref-driven docs.
+//!
+//! `backup-manager` runs in four modes that share a single binary, configuration
+//! loader, schema set, and event surface:
+//!
+//! - **one-shot** — invoked by a scheduler (Kubernetes CronJob, system cron,
+//!   systemd timer). Loads the policy, executes the requested operation
+//!   (`backup`, `restore`, `verify`, `prune`, `list`), reports through metrics
+//!   and NATS events, terminates.
+//! - **daemon** — long-running orchestrator. Holds a priority queue, enforces
+//!   concurrency budgets, reloads policies via [`platform_config`] watcher,
+//!   exposes a local HTTP/Unix socket API, and emits Prometheus metrics on a
+//!   dedicated endpoint.
+//! - **standalone** — workstation mode. Connects to a destination directly with
+//!   the operator's age key and runs `mount`, `restore`, or `list` without
+//!   needing the cluster, the daemon, or even a Nickel runtime (the binary
+//!   embeds the schema subset required to deserialise policies).
+//! - **coordinator** — orchestrates `BackupGroup` consistent cuts across
+//!   members. Tags every member's snapshot with the same `group_id` and
+//!   `group_ts` so a single timestamp recovers the whole group.
+//!
+//! ## Schema integration
+//!
+//! Policies are declared in Nickel (`provisioning/schemas/lib/backup_policy.ncl`,
+//! `backup_group.ncl`, `system_backup.ncl`, `verify_policy.ncl`, `concerns.ncl`).
+//! The Nickel layer enforces three invariants at export time:
+//!
+//! - `EncryptionRequired` — every [`BackupProvider`](policy::BackupProvider)
+//!   must declare `features.encryption = true`.
+//! - `MultiDestinationRequired` — every enabled
+//!   [`BackupPolicy`](policy::BackupPolicy) must declare ≥2 destinations,
+//!   at least one with `role = 'primary`.
+//! - `NonEmptyScopes` — every enabled `BackupPolicy` must list at least one
+//!   `BackupScope`.
+//!
+//! ## Custody model (ADR-011)
+//!
+//! Four custody boxes, never co-dependent: keys live in `secretumvault`,
+//! definitions in git, data in ≥2 destinations, production in the cluster.
+//! Standalone restore needs only one custody box plus the bootstrap key.
+//!
+//! ## ontoref integration
+//!
+//! This crate is registered as the `backup-manager` Practice node in
+//! `.ontology/core.ncl`. `ontoref generate-mdbook` extracts this `//!` and
+//! produces architecture documentation automatically. The first sentence above
+//! is Jaccard-aligned with the node `description` and validated by
+//! `ontoref sync diff --docs`.
+
+#![forbid(unsafe_code)]
+#![warn(missing_docs)]
+#![warn(rust_2018_idioms)]
+
+pub mod config;
+pub mod error;
+pub mod modes;
+pub mod policy;
+
+pub use config::BackupManagerConfig;
+pub use error::{BackupError, Result};

crates/backup-manager/src/main.rs

@@ -0,0 +1,348 @@
+//! `prvng-backup` — entry point for the backup-manager binary.
+//!
+//! Subcommands map to the four modes described in the crate-level docs.
+//! This first cut implements the `policy` subcommand end-to-end (validate
+//! and render Nickel-exported policies); the other modes are stubs whose
+//! signatures and CLI surface are wired so the contract is visible to users
+//! and downstream code while the implementations land in subsequent commits.
+
+use std::path::{Path, PathBuf};
+use std::process::ExitCode;
+
+use clap::{Args, Parser, Subcommand};
+
+use backup_manager::modes::policy_cmds;
+
+/// CLI for the backup-manager.
+#[derive(Debug, Parser)]
+#[command(
+    name = "prvng-backup",
+    version,
+    about = "Multi-context backup orchestrator (restic/kopia, ontoref-integrated)"
+)]
+struct Cli {
+    /// Path to a configuration file. When absent, the standard hierarchy
+    /// (`~/.config/provisioning/platform/config/backup-manager.ncl`) is used.
+    #[arg(short, long, env = "BACKUP_CONFIG")]
+    config: Option<PathBuf>,
+
+    /// Subcommand selecting the mode of operation.
+    #[command(subcommand)]
+    command: Mode,
+}
+
+/// Top-level mode dispatch.
+#[derive(Debug, Subcommand)]
+enum Mode {
+    /// Render and validate Nickel policy declarations.
+    Policy(PolicyArgs),
+    /// One-shot operation invoked by a scheduler (cron, systemd timer).
+    OneShot(OneShotArgs),
+    /// Long-running daemon orchestrator.
+    Daemon(DaemonArgs),
+    /// Standalone workstation mode (mount/restore/list without cluster).
+    Standalone(StandaloneArgs),
+    /// BackupGroup coordinator.
+    Coordinator(CoordinatorArgs),
+}
+
+// ── policy ──────────────────────────────────────────────────────────────────
+
+#[derive(Debug, Args)]
+struct PolicyArgs {
+    /// Sub-action to perform.
+    #[command(subcommand)]
+    action: PolicyAction,
+}
+
+#[derive(Debug, Subcommand)]
+enum PolicyAction {
+    /// Validate one or more Nickel policy files (export + deserialise).
+    Validate {
+        /// Files or directories to validate.
+        #[arg(required = true)]
+        paths: Vec<PathBuf>,
+
+        /// Nickel `--import-path` value.
+        #[arg(long, default_value = "provisioning")]
+        import_path: String,
+    },
+    /// Render a policy file to JSON on stdout.
+    Render {
+        /// Policy file to render.
+        path: PathBuf,
+        /// Nickel `--import-path` value.
+        #[arg(long, default_value = "provisioning")]
+        import_path: String,
+    },
+}
+
+// ── one-shot ────────────────────────────────────────────────────────────────
+
+#[derive(Debug, Args)]
+struct OneShotArgs {
+    /// Operation to perform.
+    #[command(subcommand)]
+    op: OneShotOp,
+}
+
+#[derive(Debug, Subcommand)]
+enum OneShotOp {
+    /// Run a backup for a component, group, or system target.
+    Backup {
+        /// Target identifier.
+        target: String,
+        /// Optional scope filter.
+        #[arg(long)]
+        scope: Option<String>,
+    },
+    /// Restore a snapshot to a directory.
+    Restore {
+        /// Target identifier.
+        target: String,
+        /// Snapshot ID or `latest`.
+        #[arg(long)]
+        snapshot: String,
+        /// Restore destination directory.
+        #[arg(long)]
+        to: PathBuf,
+    },
+    /// Verify snapshots (provider-level integrity checks).
+    Verify {
+        /// Target identifier.
+        target: String,
+        /// Verify level.
+        #[arg(long, default_value = "quick")]
+        level: String,
+    },
+    /// Apply retention policy and prune old snapshots.
+    Prune {
+        /// Target identifier.
+        target: String,
+        /// Dry run (do not delete).
+        #[arg(long)]
+        dry_run: bool,
+    },
+    /// List snapshots.
+    List {
+        /// Target identifier.
+        target: String,
+        /// Optional scope filter.
+        #[arg(long)]
+        scope: Option<String>,
+    },
+}
+
+// ── daemon ──────────────────────────────────────────────────────────────────
+
+#[derive(Debug, Args)]
+struct DaemonArgs {
+    /// Daemon control action.
+    #[command(subcommand)]
+    action: DaemonAction,
+}
+
+#[derive(Debug, Subcommand)]
+enum DaemonAction {
+    /// Start the daemon (foreground).
+    Start,
+    /// Reload policies (sends SIGHUP equivalent through the local API).
+    Reload,
+    /// Drain the queue and shut down cleanly.
+    Drain,
+    /// Print status as JSON.
+    Status,
+}
+
+// ── standalone ──────────────────────────────────────────────────────────────
+
+#[derive(Debug, Args)]
+struct StandaloneArgs {
+    /// Standalone operation.
+    #[command(subcommand)]
+    op: StandaloneOp,
+}
+
+#[derive(Debug, Subcommand)]
+enum StandaloneOp {
+    /// Mount a snapshot via FUSE.
+    Mount {
+        /// Component name.
+        component: String,
+        /// Snapshot ID or `latest`.
+        #[arg(long)]
+        snapshot: String,
+        /// Mountpoint directory.
+        #[arg(long)]
+        at: PathBuf,
+        /// Tag filters (`key=value`).
+        #[arg(long)]
+        tag: Vec<String>,
+    },
+    /// Restore directly to a directory without going through the daemon.
+    Restore {
+        /// Component name.
+        component: String,
+        /// Snapshot ID or `latest`.
+        #[arg(long)]
+        snapshot: String,
+        /// Restore destination directory.
+        #[arg(long)]
+        to: PathBuf,
+    },
+    /// List snapshots (no daemon).
+    List {
+        /// Component name.
+        component: String,
+        /// Optional scope filter.
+        #[arg(long)]
+        scope: Option<String>,
+    },
+}
+
+// ── coordinator ─────────────────────────────────────────────────────────────
+
+#[derive(Debug, Args)]
+struct CoordinatorArgs {
+    /// Coordinator action.
+    #[command(subcommand)]
+    action: CoordinatorAction,
+}
+
+#[derive(Debug, Subcommand)]
+enum CoordinatorAction {
+    /// Run a BackupGroup.
+    GroupRun {
+        /// Group name.
+        name: String,
+    },
+    /// Restore a BackupGroup at a given timestamp.
+    GroupRestore {
+        /// Group name.
+        name: String,
+        /// RFC3339 timestamp.
+        #[arg(long)]
+        at: String,
+        /// Restore destination directory.
+        #[arg(long)]
+        to: PathBuf,
+    },
+}
+
+#[tokio::main]
+async fn main() -> ExitCode {
+    tracing_subscriber::fmt()
+        .with_env_filter(
+            tracing_subscriber::EnvFilter::try_from_env("BACKUP_LOG")
+                .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")),
+        )
+        .init();
+
+    let cli = Cli::parse();
+
+    match cli.command {
+        Mode::Policy(args) => run_policy(args),
+        Mode::OneShot(_) | Mode::Daemon(_) | Mode::Standalone(_) | Mode::Coordinator(_) => {
+            eprintln!(
+                "this mode is wired in the CLI surface but its implementation \
+                 lands in a subsequent commit. The `policy` subcommand is fully \
+                 functional today (validate / render Nickel-exported policies)."
+            );
+            ExitCode::from(2)
+        }
+    }
+}
+
+fn run_policy(args: PolicyArgs) -> ExitCode {
+    match args.action {
+        PolicyAction::Validate { paths, import_path } => run_policy_validate(&paths, &import_path),
+        PolicyAction::Render { path, import_path } => run_policy_render(&path, &import_path),
+    }
+}
+
+fn run_policy_validate(paths: &[PathBuf], import_path: &str) -> ExitCode {
+    let mut total_failures = 0usize;
+    for path in paths {
+        total_failures += validate_path(path, import_path);
+    }
+
+    if total_failures == 0 {
+        ExitCode::SUCCESS
+    } else {
+        eprintln!("{} validation failure(s)", total_failures);
+        ExitCode::from(1)
+    }
+}
+
+fn validate_path(path: &Path, import_path: &str) -> usize {
+    if path.is_dir() {
+        validate_directory(path, import_path)
+    } else {
+        let v = policy_cmds::validate_policy(path, import_path);
+        print_validation(&v);
+        usize::from(!v.ok)
+    }
+}
+
+fn validate_directory(path: &Path, import_path: &str) -> usize {
+    let entries = match std::fs::read_dir(path) {
+        Ok(e) => e,
+        Err(e) => {
+            eprintln!("could not read directory {:?}: {}", path, e);
+            return 1;
+        }
+    };
+
+    entries
+        .flatten()
+        .filter_map(|entry| {
+            let p = entry.path();
+            if p.extension().is_some_and(|e| e == "ncl") {
+                Some(p)
+            } else {
+                None
+            }
+        })
+        .map(|p| {
+            let v = policy_cmds::validate_policy(&p, import_path);
+            print_validation(&v);
+            usize::from(!v.ok)
+        })
+        .sum()
+}
+
+fn run_policy_render(path: &Path, import_path: &str) -> ExitCode {
+    let value = match policy_cmds::nickel_export_to_json(path, import_path) {
+        Ok(v) => v,
+        Err(e) => {
+            eprintln!("{}", e);
+            return ExitCode::from(1);
+        }
+    };
+
+    match serde_json::to_string_pretty(&value) {
+        Ok(rendered) => {
+            println!("{}", rendered);
+            ExitCode::SUCCESS
+        }
+        Err(e) => {
+            eprintln!("failed to render JSON: {}", e);
+            ExitCode::from(1)
+        }
+    }
+}
+
+fn print_validation(v: &policy_cmds::PolicyValidation) {
+    if v.ok {
+        println!(
+            "OK    {}  components={}  backup-enabled={}",
+            v.source, v.component_count, v.enabled_backup_components.len()
+        );
+    } else {
+        println!(
+            "FAIL  {}  error={}",
+            v.source,
+            v.error.as_deref().unwrap_or("unknown")
+        );
+    }
+}

crates/backup-manager/src/modes/mod.rs

@@ -0,0 +1,7 @@
+//! Execution modes — `one-shot`, `daemon`, `standalone`, `coordinator`.
+//!
+//! Each mode is a distinct subcommand of the [`prvng-backup`](../../bin/main.rs)
+//! binary. The mode is selected at process start; the configuration loaded
+//! from [`crate::BackupManagerConfig`] is shared across all modes.
+
+pub mod policy_cmds;

crates/backup-manager/src/modes/policy_cmds.rs

@@ -0,0 +1,170 @@
+//! `policy` subcommand — render and validate `BackupPolicy` declarations.
+//!
+//! This is the first end-to-end mode wired into the binary. It exercises:
+//!
+//! - Reading workspace declarations via `nickel export --import-path …`.
+//! - Deserialising the resulting JSON into [`crate::policy`] structs.
+//! - Reporting structured information about policies, groups, and system
+//!   backup definitions without touching destinations or vault.
+//!
+//! Subsequent modes (`backup`, `restore`, `mount`, etc.) build on the same
+//! deserialisation pipeline implemented here.
+
+use std::path::Path;
+use std::process::Command;
+
+use crate::error::{BackupError, Result};
+use crate::policy::BackupPolicy;
+
+/// Render a Nickel file to JSON via the `nickel` CLI.
+///
+/// Honours the `nickel_import_path` provided in [`crate::config::PoliciesRef`].
+pub fn nickel_export_to_json(path: &Path, import_path: &str) -> Result<serde_json::Value> {
+    let output = Command::new("nickel")
+        .args(["export", "--format", "json", "--import-path", import_path])
+        .arg(path)
+        .output()
+        .map_err(|e| BackupError::Policy(format!(
+            "failed to invoke `nickel export` on {:?}: {}", path, e
+        )))?;
+
+    if !output.status.success() {
+        let stderr = String::from_utf8_lossy(&output.stderr);
+        return Err(BackupError::Policy(format!(
+            "nickel export failed for {:?}: {}", path, stderr
+        )));
+    }
+
+    serde_json::from_slice::<serde_json::Value>(&output.stdout).map_err(BackupError::from)
+}
+
+/// Result of validating a single policy file.
+#[derive(Debug)]
+pub struct PolicyValidation {
+    /// Path to the source Nickel file.
+    pub source: String,
+    /// Whether the file successfully exported and deserialised.
+    pub ok: bool,
+    /// Error message if validation failed.
+    pub error: Option<String>,
+    /// Component count (top-level keys in the JSON object).
+    pub component_count: usize,
+    /// Names of components for which `concerns.backup.kind == "enabled"`.
+    pub enabled_backup_components: Vec<String>,
+}
+
+/// Validate a single Nickel policy file by exporting and deserialising it.
+pub fn validate_policy(path: &Path, import_path: &str) -> PolicyValidation {
+    let source = path.display().to_string();
+    let json = match nickel_export_to_json(path, import_path) {
+        Ok(v) => v,
+        Err(e) => {
+            return PolicyValidation {
+                source,
+                ok: false,
+                error: Some(e.to_string()),
+                component_count: 0,
+                enabled_backup_components: Vec::new(),
+            };
+        }
+    };
+
+    let object = match json.as_object() {
+        Some(map) => map,
+        None => {
+            return PolicyValidation {
+                source,
+                ok: false,
+                error: Some("policy file did not export a JSON object".to_string()),
+                component_count: 0,
+                enabled_backup_components: Vec::new(),
+            };
+        }
+    };
+
+    let mut enabled = Vec::new();
+    for (component_name, component_value) in object {
+        // Best-effort: locate concerns.backup.kind inside the component.
+        // Components produced by `ext.make_X` helpers may nest the policy in
+        // different ways; we don't fail on missing concerns here, we just
+        // skip when not found. Strict enforcement is the Nickel layer's job.
+        if let Some(kind) = component_value
+            .pointer("/concerns/backup/kind")
+            .and_then(serde_json::Value::as_str)
+        {
+            if kind == "enabled" {
+                enabled.push(component_name.clone());
+            }
+        }
+    }
+
+    PolicyValidation {
+        source,
+        ok: true,
+        error: None,
+        component_count: object.len(),
+        enabled_backup_components: enabled,
+    }
+}
+
+/// Try to deserialise a `BackupPolicy` directly from its JSON representation.
+/// Used by tests and by future scope-expansion code; not exposed via CLI yet.
+pub fn deserialize_backup_policy(value: serde_json::Value) -> Result<BackupPolicy> {
+    serde_json::from_value(value).map_err(BackupError::from)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use serde_json::json;
+
+    #[test]
+    fn deserialize_minimal_backup_policy() {
+        let value = json!({
+            "provider": { "name": "restic" },
+            "destinations": [
+                {
+                    "name": "primary",
+                    "kind": "s3",
+                    "uri": "s3:fsn1.hetzner/test",
+                    "cred_ref": { "path": "creds/test", "kind": "s3" },
+                    "role": "primary"
+                },
+                {
+                    "name": "replica",
+                    "kind": "b2",
+                    "uri": "b2:test-replica",
+                    "cred_ref": { "path": "creds/replica", "kind": "b2" },
+                    "role": "replica"
+                }
+            ],
+            "encryption": {
+                "path": "backup-manager/master-key",
+                "algorithm": "age_x25519"
+            },
+            "schedule": { "kind": "cron", "cron_expr": "0 2 * * *" },
+            "retention": {
+                "keep_last": 7,
+                "keep_daily": 7,
+                "keep_weekly": 4,
+                "keep_monthly": 6,
+                "keep_yearly": 0
+            },
+            "scopes": [
+                {
+                    "kind": "service_full",
+                    "name": "main",
+                    "paths": ["/data"]
+                }
+            ],
+            "tag_strategy": {
+                "component_label": "test-component"
+            }
+        });
+
+        let policy = deserialize_backup_policy(value).expect("deserialise");
+        assert_eq!(policy.destinations.len(), 2);
+        assert_eq!(policy.scopes.len(), 1);
+        assert_eq!(policy.provider.name, "restic");
+    }
+}

crates/backup-manager/src/policy/component.rs

@@ -0,0 +1,324 @@
+//! Component-level policy: `BackupPolicy`, `Destination`, `RetentionPolicy`,
+//! `Schedule`, `BackupProviderRef`, plus the surrounding `ServiceConcerns`
+//! umbrella.
+
+use std::collections::BTreeMap;
+
+use serde::{Deserialize, Serialize};
+
+use super::scope::BackupScope;
+
+// ============================================================================
+// ServiceConcerns umbrella (ADR-008)
+// ============================================================================
+
+/// One of `enabled`, `disabled`, `pending`, `inherited`.
+#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
+#[serde(rename_all = "lowercase")]
+pub enum ConcernStateKind {
+    /// Concern is implemented; the impl payload field is populated.
+    Enabled,
+    /// Concern is explicitly opted out.
+    Disabled,
+    /// Concern is deferred; `backlog_ref` points to the tracking item.
+    Pending,
+    /// Concern is inherited from a parent component.
+    Inherited,
+}
+
+/// State of one concern within [`ServiceConcerns`].
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ConcernState {
+    /// Discriminator.
+    pub kind: ConcernStateKind,
+
+    /// `disabled` reason.
+    #[serde(default)]
+    pub reason: Option<String>,
+
+    /// `disabled` since (ISO date).
+    #[serde(default)]
+    pub since: Option<String>,
+
+    /// `pending` backlog reference.
+    #[serde(default)]
+    pub backlog_ref: Option<String>,
+
+    /// `pending` target iteration.
+    #[serde(default)]
+    pub target_iteration: Option<String>,
+
+    /// `inherited` parent component.
+    #[serde(default)]
+    pub from: Option<String>,
+
+    /// `enabled` payload — only one of these is populated based on which
+    /// concern this state belongs to.
+    #[serde(default)]
+    pub backup_impl: Option<BackupPolicy>,
+
+    /// `enabled` payload for non-backup concerns kept opaque at this layer
+    /// (TLS/DNS/certs/observability/security details are not consumed by the
+    /// backup-manager binary).
+    #[serde(default, flatten)]
+    pub other_impl: BTreeMap<String, serde_json::Value>,
+}
+
+/// Mandatory umbrella in `ComponentDef.concerns`.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ServiceConcerns {
+    /// TLS concern state (opaque to the backup binary).
+    pub tls: ConcernState,
+    /// DNS concern state.
+    pub dns: ConcernState,
+    /// Certificates concern state.
+    pub certs: ConcernState,
+    /// Backup concern state — when `kind = Enabled`, `backup_impl` is set.
+    pub backup: ConcernState,
+    /// Observability concern state.
+    pub observability: ConcernState,
+    /// Security concern state.
+    pub security: ConcernState,
+}
+
+// ============================================================================
+// BackupPolicy
+// ============================================================================
+
+/// Reference to a [`BackupProvider`] declared under
+/// `provisioning/extensions/providers/backup/<name>/`.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct BackupProviderRef {
+    /// Provider directory name (e.g. `"restic"`, `"kopia"`).
+    pub name: String,
+
+    /// Pinned CLI version. The daemon emits a warning if the installed
+    /// binary is older.
+    #[serde(default)]
+    pub version: Option<String>,
+}
+
+/// Schedule discriminated union (cron / interval / NATS-event).
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ScheduleSpec {
+    /// `cron`, `interval`, or `on_event`.
+    pub kind: String,
+
+    /// Cron expression when `kind = "cron"`.
+    #[serde(default)]
+    pub cron_expr: Option<String>,
+
+    /// Jitter seconds applied to cron firing.
+    #[serde(default)]
+    pub jitter_sec: Option<u64>,
+
+    /// Period when `kind = "interval"`.
+    #[serde(default)]
+    pub every: Option<String>,
+
+    /// Random jitter for interval scheduling.
+    #[serde(default)]
+    pub jitter: Option<String>,
+
+    /// NATS subject when `kind = "on_event"`.
+    #[serde(default)]
+    pub subject: Option<String>,
+
+    /// Debounce duration for event-driven schedules.
+    #[serde(default)]
+    pub debounce: Option<String>,
+}
+
+/// Retention preset. Mirrors restic's `--keep-*` flags.
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+pub struct RetentionPolicy {
+    /// `--keep-last`.
+    #[serde(default)]
+    pub keep_last: u32,
+    /// `--keep-daily`.
+    #[serde(default)]
+    pub keep_daily: u32,
+    /// `--keep-weekly`.
+    #[serde(default)]
+    pub keep_weekly: u32,
+    /// `--keep-monthly`.
+    #[serde(default)]
+    pub keep_monthly: u32,
+    /// `--keep-yearly`.
+    #[serde(default)]
+    pub keep_yearly: u32,
+    /// Hard upper bound on snapshot age (deleted regardless of keep_*).
+    #[serde(default)]
+    pub prune_after: Option<String>,
+}
+
+/// Destination protocol kind.
+#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
+#[serde(rename_all = "snake_case")]
+pub enum DestinationKind {
+    /// S3-compatible object storage (Hetzner, MinIO, AWS S3, etc.).
+    S3,
+    /// Backblaze B2.
+    B2,
+    /// Local filesystem (only valid for archive role).
+    Local,
+    /// SFTP target.
+    Sftp,
+    /// restic REST server.
+    RestServer,
+}
+
+/// Role of a destination within a policy. The `MultiDestinationRequired`
+/// Nickel contract requires at least one `Primary` and a total of ≥2 entries.
+#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
+#[serde(rename_all = "lowercase")]
+pub enum DestinationRole {
+    /// First-write target. Backups land here directly.
+    Primary,
+    /// Replicated copy of `Primary` for off-site durability.
+    Replica,
+    /// Cold archive (typically slower or cheaper storage).
+    Archive,
+}
+
+/// Backup destination specification.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct Destination {
+    /// Stable identifier used in metric labels and tags.
+    pub name: String,
+    /// Protocol kind.
+    pub kind: DestinationKind,
+    /// Provider URI (e.g. `s3:fsn1.hetzner/libre-wuji-backups`).
+    pub uri: String,
+    /// Reference to credentials in `secretumvault`.
+    pub cred_ref: VaultCredRef,
+    /// Role within the policy.
+    pub role: DestinationRole,
+    /// Region (for cloud providers that support it).
+    #[serde(default)]
+    pub region: Option<String>,
+}
+
+/// Reference to a credentials entry in vault.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct VaultCredRef {
+    /// Path under vault (e.g. `backup-manager/destinations/hetzner-primary`).
+    pub path: String,
+    /// Type of credential payload at the path.
+    pub kind: String,
+}
+
+/// Reference to an encryption key in vault.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct VaultKeyRef {
+    /// Path under vault.
+    pub path: String,
+    /// Algorithm (`age_x25519`, `aes_gcm_256`, etc.).
+    #[serde(default)]
+    pub algorithm: String,
+    /// HKDF derivation parameters when applicable.
+    #[serde(default)]
+    pub derivation: Option<HkdfDerivation>,
+}
+
+/// HKDF derivation parameters.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct HkdfDerivation {
+    /// Method (`none` or `hkdf_sha256`).
+    pub method: String,
+    /// HKDF info parameter.
+    pub info: String,
+}
+
+/// Tag generation strategy for snapshots.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct TagStrategy {
+    /// Used as the `component=<value>` tag.
+    pub component_label: String,
+    /// Additional static tags as `k=v` strings.
+    #[serde(default)]
+    pub extra: Vec<String>,
+}
+
+/// Pre/post hooks executed by the manager around the backup run.
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+pub struct Hooks {
+    /// Commands run before backup; non-zero exit aborts when `abort_on_failure = true`.
+    #[serde(default)]
+    pub pre: Vec<String>,
+    /// Commands run after backup, regardless of outcome.
+    #[serde(default)]
+    pub post: Vec<String>,
+    /// Hook timeout (e.g. `"5m"`).
+    #[serde(default = "default_hook_timeout")]
+    pub timeout: String,
+    /// Whether a non-zero pre hook aborts the backup.
+    #[serde(default = "default_abort_on_failure")]
+    pub abort_on_failure: bool,
+}
+
+fn default_hook_timeout() -> String { "5m".to_string() }
+fn default_abort_on_failure() -> bool { true }
+
+/// Bandwidth throttle (passed to provider as `--limit-upload`/`--limit-download`).
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+pub struct Throttle {
+    /// Upload limit in KB/s.
+    #[serde(default)]
+    pub upload_kbps: Option<u32>,
+    /// Download limit in KB/s.
+    #[serde(default)]
+    pub download_kbps: Option<u32>,
+}
+
+/// Reference to a verify policy (drill spec lives separately to keep policies
+/// focused).
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct VerifyPolicyRef {
+    /// Verify schedule.
+    #[serde(default)]
+    pub schedule: Option<ScheduleSpec>,
+    /// Level: `quick`, `deep`, `restore_drill`, `full_dr`.
+    #[serde(default = "default_verify_level")]
+    pub level: String,
+    /// Reference to a `DrillSpec` by name.
+    #[serde(default)]
+    pub drill_ref: Option<String>,
+}
+
+fn default_verify_level() -> String { "quick".to_string() }
+
+/// Component-level backup policy.
+///
+/// The Nickel contracts `EncryptionRequired`, `MultiDestinationRequired`, and
+/// `NonEmptyScopes` are enforced at policy export time, so values reaching
+/// this struct already satisfy the invariants.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct BackupPolicy {
+    /// Provider reference.
+    pub provider: BackupProviderRef,
+    /// ≥2 destinations, at least one with `role = Primary`.
+    pub destinations: Vec<Destination>,
+    /// Encryption key reference.
+    pub encryption: VaultKeyRef,
+    /// Schedule.
+    pub schedule: ScheduleSpec,
+    /// Retention policy.
+    pub retention: RetentionPolicy,
+    /// Scopes (≥1).
+    pub scopes: Vec<BackupScope>,
+    /// Tag strategy.
+    pub tag_strategy: TagStrategy,
+    /// Optional hooks.
+    #[serde(default)]
+    pub hooks: Option<Hooks>,
+    /// Optional verify reference.
+    #[serde(default)]
+    pub verify: Option<VerifyPolicyRef>,
+    /// Optional throttle.
+    #[serde(default)]
+    pub throttle: Option<Throttle>,
+    /// Optional consistency-group name (this policy participates in a `BackupGroup`).
+    #[serde(default)]
+    pub consistency_group: Option<String>,
+}

crates/backup-manager/src/policy/group.rs

@@ -0,0 +1,68 @@
+//! BackupGroup — multi-component consistency points.
+
+use serde::{Deserialize, Serialize};
+
+use super::component::{Destination, RetentionPolicy, ScheduleSpec, TagStrategy, VaultKeyRef,
+                        VerifyPolicyRef};
+
+/// Member of a [`BackupGroup`].
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct GroupMember {
+    /// Component name.
+    pub component: String,
+    /// Optional scope filter; omitted means all scopes of the component.
+    #[serde(default)]
+    pub scope: Option<String>,
+}
+
+/// Coordination strategy discriminator.
+#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
+#[serde(rename_all = "snake_case")]
+pub enum CoordinationKind {
+    /// Snapshots fired in parallel; consistency is tag-based (group_id).
+    BestEffort,
+    /// Ordered pre-hooks pause writers, snapshot, unquiesce. Bounded downtime.
+    QuiesceWindow,
+    /// Atomic at CSI layer via Longhorn VolumeSnapshotGroup.
+    CsiConsistentGroup,
+}
+
+/// Coordination strategy applied during a group run.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct CoordinationStrategy {
+    /// Variant.
+    pub kind: CoordinationKind,
+    /// Quiesce sequence (`QuiesceWindow`).
+    #[serde(default)]
+    pub quiesce_seq: Vec<String>,
+    /// Maximum downtime for `QuiesceWindow`.
+    #[serde(default)]
+    pub max_downtime: Option<String>,
+    /// CSI VolumeSnapshotClass for `CsiConsistentGroup`.
+    #[serde(default)]
+    pub snapshot_class: Option<String>,
+}
+
+/// A group of components/scopes captured atomically (à la Chandy-Lamport).
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct BackupGroup {
+    /// Group identifier (used in CLI: `--group <name>`).
+    pub name: String,
+    /// Members participating in the consistent cut.
+    pub members: Vec<GroupMember>,
+    /// Group schedule (independent of per-member policies).
+    pub schedule: ScheduleSpec,
+    /// Coordination strategy.
+    pub coordination: CoordinationStrategy,
+    /// Retention shared across all members for the group's snapshots.
+    pub retention: RetentionPolicy,
+    /// Destinations (same `MultiDestinationRequired` invariant).
+    pub destinations: Vec<Destination>,
+    /// Encryption key.
+    pub encryption: VaultKeyRef,
+    /// Tag strategy.
+    pub tag_strategy: TagStrategy,
+    /// Optional verify reference.
+    #[serde(default)]
+    pub verify: Option<VerifyPolicyRef>,
+}

crates/backup-manager/src/policy/mod.rs

@@ -0,0 +1,26 @@
+//! Policy types deserialised from Nickel-exported JSON.
+//!
+//! Nickel files under `provisioning/schemas/lib/` define the canonical schema
+//! (BackupPolicy, BackupGroup, SystemBackupDef, ServiceConcerns, etc.).
+//! The manager invokes `nickel export` on workspace declarations and parses
+//! the resulting JSON into these structs. The JSON tag conventions mirror the
+//! Nickel discriminated unions: `kind` field plus payload variants.
+//!
+//! Only the shape needed at runtime is materialised here; the full Nickel
+//! schema includes optional fields (e.g. extended `dns_records.extra`) that
+//! we treat as `serde_json::Value` to avoid coupling the binary to every
+//! schema evolution.
+
+pub mod component;
+pub mod group;
+pub mod scope;
+pub mod system;
+
+pub use component::{
+    BackupPolicy, BackupProviderRef, ConcernState, Destination, DestinationKind,
+    DestinationRole, RetentionPolicy, ScheduleSpec, ServiceConcerns,
+};
+pub use group::{BackupGroup, CoordinationKind, CoordinationStrategy, GroupMember};
+pub use scope::{BackupScope, DumpStrategy, DumpStrategyKind, ScopeKind};
+pub use system::{HostSelector, HostSelectorKind, SystemBackupDef, SystemBackupTarget,
+                 SystemBackupTargetKind};

crates/backup-manager/src/policy/scope.rs

@@ -0,0 +1,175 @@
+//! BackupScope and DumpStrategy types — the unit of backup work.
+//!
+//! Each scope expands to one provider invocation with deterministic tags.
+//! The discriminator is `kind`; the rest of the fields are populated only
+//! for variants that use them (Nickel emits `default = []`/`default = {}`).
+
+use std::collections::BTreeMap;
+
+use serde::{Deserialize, Serialize};
+
+/// Discriminator for [`BackupScope`].
+#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
+#[serde(rename_all = "snake_case")]
+pub enum ScopeKind {
+    /// Snapshot a fixed list of paths.
+    ServiceFull,
+    /// Snapshot per domain (one tag set per domain).
+    PerDomain,
+    /// Snapshot per mailbox (selector-driven enumeration).
+    PerMailbox,
+    /// Snapshot a database via a [`DumpStrategy`].
+    Database,
+    /// Snapshot a CSI volume by name.
+    VolumeSnapshot,
+    /// Archive logs from a source (loki / journald / files).
+    LogsArchive,
+    /// Export a subset of a key-value store (etcd / consul).
+    KvExport,
+}
+
+/// Database engine, when scope kind is [`ScopeKind::Database`].
+#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
+#[serde(rename_all = "lowercase")]
+pub enum DbEngine {
+    /// PostgreSQL — supports streaming pg_dump and WAL archiving externally.
+    Postgresql,
+    /// MariaDB.
+    Mariadb,
+    /// MySQL.
+    Mysql,
+    /// Redis (RDB snapshot or AOF).
+    Redis,
+    /// MongoDB (mongodump).
+    Mongodb,
+    /// SurrealDB.
+    Surrealdb,
+    /// etcd (etcdctl snapshot).
+    Etcd,
+    /// SQLite (file-level).
+    Sqlite,
+}
+
+/// Discriminator for [`DumpStrategy`].
+#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
+#[serde(rename_all = "snake_case")]
+pub enum DumpStrategyKind {
+    /// Stream the dump straight to the provider's stdin.
+    StreamToStdin,
+    /// Write the dump to a path, then snapshot the path.
+    DumpToPath,
+    /// Like `DumpToPath` but with a custom dump command.
+    PreDumpThenPath,
+    /// CSI VolumeSnapshot CRD (crash-consistent, no app coordination).
+    CsiVolumeSnapshot,
+    /// Application quiesce hook + snapshot + unquiesce.
+    AppQuiesceThenSnapshot,
+}
+
+/// Strategy for capturing a database scope. See ADR-013 for the consistency
+/// trade-offs across variants.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct DumpStrategy {
+    /// Variant discriminator.
+    pub kind: DumpStrategyKind,
+
+    /// Custom dump command (optional, used by `PreDumpThenPath`).
+    #[serde(default)]
+    pub dump_command: Option<String>,
+
+    /// Path where the dump file lives (`DumpToPath`, `PreDumpThenPath`).
+    #[serde(default)]
+    pub path: Option<String>,
+
+    /// Whether to remove the dump file after the snapshot completes.
+    #[serde(default = "default_true")]
+    pub cleanup: bool,
+
+    /// CSI volume name (`CsiVolumeSnapshot`, `AppQuiesceThenSnapshot`).
+    #[serde(default)]
+    pub volume: Option<String>,
+
+    /// CSI VolumeSnapshotClass.
+    #[serde(default)]
+    pub snapshot_class: Option<String>,
+
+    /// Quiesce command (`AppQuiesceThenSnapshot`).
+    #[serde(default)]
+    pub quiesce_cmd: Option<String>,
+
+    /// Unquiesce command (`AppQuiesceThenSnapshot`).
+    #[serde(default)]
+    pub unquiesce_cmd: Option<String>,
+}
+
+fn default_true() -> bool { true }
+
+/// One unit of backup work. The provider receives this via [`crate::policy::component::BackupPolicy`].
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct BackupScope {
+    /// Discriminator.
+    pub kind: ScopeKind,
+
+    /// Identifier for the scope (used in CLI: `--scope <name>`).
+    pub name: String,
+
+    /// Path list (`ServiceFull`, `LogsArchive`).
+    #[serde(default)]
+    pub paths: Vec<String>,
+
+    /// Exclusion globs.
+    #[serde(default)]
+    pub exclude: Vec<String>,
+
+    /// Domain list (`PerDomain`).
+    #[serde(default)]
+    pub domains: Vec<String>,
+
+    /// Base path for per-domain / per-mailbox scopes.
+    #[serde(default)]
+    pub base_path: String,
+
+    /// Mailbox selector (`PerMailbox`).
+    #[serde(default)]
+    pub selector: Option<String>,
+
+    /// Database engine (`Database`).
+    #[serde(default)]
+    pub engine: Option<DbEngine>,
+
+    /// Dump strategy (`Database`).
+    #[serde(default)]
+    pub dump_strategy: Option<DumpStrategy>,
+
+    /// Volume names (`VolumeSnapshot`).
+    #[serde(default)]
+    pub volumes: Vec<String>,
+
+    /// CSI VolumeSnapshotClass (`VolumeSnapshot`).
+    #[serde(default)]
+    pub snapshot_class: Option<String>,
+
+    /// Log sources (`LogsArchive`).
+    #[serde(default)]
+    pub sources: Vec<String>,
+
+    /// Log archive format (`LogsArchive`).
+    #[serde(default)]
+    pub format: Option<String>,
+
+    /// Rotation duration (`LogsArchive`).
+    #[serde(default)]
+    pub rotation: Option<String>,
+
+    /// KV source kind (`KvExport`).
+    #[serde(default)]
+    pub source: Option<String>,
+
+    /// Tag prefix prepended to determinístico tags.
+    #[serde(default)]
+    pub tag_prefix: String,
+
+    /// Static extra tags (key-value).
+    #[serde(default)]
+    pub tags: BTreeMap<String, String>,
+}

crates/backup-manager/src/policy/system.rs

@@ -0,0 +1,196 @@
+//! SystemBackupDef — backup specifications for artefacts outside the cluster.
+//!
+//! Targets include etcd, k8s certs, host configs, external DNS configurations,
+//! builder environment tools, provisioning state, log archives, SOPS/Age key
+//! bundles, and vault state. Each target is a discriminated variant; the
+//! manager dispatches based on `kind` (see ADR-011 for vault state custody).
+
+use serde::{Deserialize, Serialize};
+
+use super::component::{
+    BackupProviderRef, Destination, RetentionPolicy, ScheduleSpec, TagStrategy, Throttle,
+    VaultCredRef, VaultKeyRef, VerifyPolicyRef, Hooks,
+};
+
+/// Discriminator for [`HostSelector`].
+#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
+#[serde(rename_all = "snake_case")]
+pub enum HostSelectorKind {
+    /// Run on a single control-plane host (any).
+    CpOnly,
+    /// Run on the first control-plane host (deterministic by name sort).
+    CpFirst,
+    /// Run on all control-plane hosts.
+    ControlPlanes,
+    /// Run on all worker hosts.
+    Workers,
+    /// Run on every server in the workspace.
+    AllServers,
+    /// Run on the listed hosts.
+    List,
+}
+
+/// Selects which hosts execute a system backup. Mirrors the Nickel `HostSelector`.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct HostSelector {
+    /// Variant.
+    pub kind: HostSelectorKind,
+    /// Hostname list when `kind = List`.
+    #[serde(default)]
+    pub members: Vec<String>,
+}
+
+/// Discriminator for [`SystemBackupTarget`].
+#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
+#[serde(rename_all = "snake_case")]
+pub enum SystemBackupTargetKind {
+    /// etcd snapshot via etcdctl.
+    Etcd,
+    /// Kubernetes PKI certificates (`/etc/kubernetes/pki`).
+    K8sCerts,
+    /// Cluster resources (Secret/ConfigMap/CRD content).
+    ClusterResources,
+    /// Longhorn engine state.
+    LonghornEngine,
+    /// Host configuration files.
+    HostConfigs,
+    /// External DNS server configuration and zone files.
+    ExternalDns,
+    /// Builder environment tools and caches.
+    BuilderEnv,
+    /// Provisioning state (definitions, state, locks).
+    ProvisioningState,
+    /// Log archive (loki / journald / files).
+    LogsArchive,
+    /// SOPS/Age key bundle (separate custody chain).
+    SopsKeys,
+    /// secretumvault state (with bootstrap-key encryption per ADR-011).
+    VaultState,
+}
+
+/// Discriminated target description.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct SystemBackupTarget {
+    /// Variant.
+    pub kind: SystemBackupTargetKind,
+
+    // ── 'etcd
+    /// etcd endpoints.
+    #[serde(default)]
+    pub endpoints: Vec<String>,
+    /// CA reference.
+    #[serde(default)]
+    pub ca_ref: Option<VaultCredRef>,
+    /// Client cert reference.
+    #[serde(default)]
+    pub cert_ref: Option<VaultCredRef>,
+    /// Client key reference.
+    #[serde(default)]
+    pub key_ref: Option<VaultCredRef>,
+
+    // ── 'k8s_certs / 'host_configs / 'logs_archive (paths)
+    /// Path list.
+    #[serde(default)]
+    pub paths: Vec<String>,
+    /// Exclusion globs.
+    #[serde(default)]
+    pub exclude: Vec<String>,
+
+    // ── 'cluster_resources
+    /// Namespace list.
+    #[serde(default)]
+    pub namespaces: Vec<String>,
+    /// Kind list (`secret`, `certificate`, ...).
+    #[serde(default)]
+    pub kinds: Vec<String>,
+
+    // ── 'longhorn_engine
+    /// Component list (`volumes`, `engines`, `replicas`, `settings`).
+    #[serde(default)]
+    pub components: Vec<String>,
+
+    // ── 'external_dns
+    /// Source kind (`coredns`, `powerdns`, `unbound`, ...).
+    #[serde(default)]
+    pub source_kind: Option<String>,
+    /// Config file paths.
+    #[serde(default)]
+    pub config_paths: Vec<String>,
+    /// Zone file directories.
+    #[serde(default)]
+    pub zones_paths: Vec<String>,
+
+    // ── 'builder_env
+    /// Tool names (informational tags).
+    #[serde(default)]
+    pub tools: Vec<String>,
+    /// Secret names that must accompany the artefact.
+    #[serde(default)]
+    pub secrets: Vec<String>,
+
+    // ── 'provisioning_state
+    /// Definitions directory.
+    #[serde(default)]
+    pub definitions_path: Option<String>,
+    /// State directory.
+    #[serde(default)]
+    pub state_path: Option<String>,
+    /// Lock directory.
+    #[serde(default)]
+    pub lock_path: Option<String>,
+
+    // ── 'logs_archive
+    /// Loki/journald selector.
+    #[serde(default)]
+    pub selector: Option<String>,
+    /// Archive format.
+    #[serde(default)]
+    pub format: Option<String>,
+
+    // ── 'sops_keys / 'vault_state
+    /// Age key file paths.
+    #[serde(default)]
+    pub age_keys: Vec<String>,
+    /// Recipient public keys.
+    #[serde(default)]
+    pub recipients: Vec<String>,
+    /// Vault HTTP endpoint to back up.
+    #[serde(default)]
+    pub vault_endpoint: Option<String>,
+    /// Specific vault paths to capture (omitted = full state).
+    #[serde(default)]
+    pub vault_paths: Vec<String>,
+}
+
+/// Top-level system backup definition.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct SystemBackupDef {
+    /// Identifier (used in CLI: `prvng-backup one-shot backup <name>`).
+    pub name: String,
+    /// Target description.
+    pub target: SystemBackupTarget,
+    /// Host selector.
+    pub host_selector: HostSelector,
+    /// Provider.
+    pub provider: BackupProviderRef,
+    /// Schedule.
+    pub schedule: ScheduleSpec,
+    /// Retention.
+    pub retention: RetentionPolicy,
+    /// Destinations.
+    pub destinations: Vec<Destination>,
+    /// Encryption key. For `VaultState`, ADR-011 requires a bootstrap key
+    /// stored OUTSIDE vault (offline custody).
+    pub encryption: VaultKeyRef,
+    /// Tag strategy.
+    pub tag_strategy: TagStrategy,
+    /// Optional verify reference.
+    #[serde(default)]
+    pub verify: Option<VerifyPolicyRef>,
+    /// Optional hooks.
+    #[serde(default)]
+    pub hooks: Option<Hooks>,
+    /// Optional throttle.
+    #[serde(default)]
+    pub throttle: Option<Throttle>,
+}

crates/buildkit-launcher/Cargo.toml

@@ -0,0 +1,21 @@
+[package]
+authors.workspace = true
+description = "Ephemeral buildkit runner launcher — spawns hcloud VMs, runs buildctl, emits metrics (ADR-039)"
+edition.workspace = true
+name = "buildkit-launcher"
+version.workspace = true
+
+[dependencies]
+anyhow = { workspace = true }
+clap = { workspace = true }
+reqwest = { workspace = true }
+serde = { workspace = true }
+serde_json = { workspace = true }
+tokio = { workspace = true, features = ["rt-multi-thread", "process", "time", "io-util"] }
+tracing = { workspace = true }
+tracing-subscriber = { workspace = true }
+uuid = { workspace = true }
+
+[[bin]]
+name = "buildkit-launcher"
+path = "src/main.rs"

crates/buildkit-launcher/Cargo.workspace.toml

@@ -0,0 +1,19 @@
+[workspace]
+members = ["crates/buildkit-launcher"]
+resolver = "2"
+
+[workspace.package]
+authors  = ["Jesus Perez <jesus@librecloud.online>"]
+edition  = "2021"
+version  = "1.0.11"
+
+[workspace.dependencies]
+anyhow         = "1.0"
+clap           = { version = "4.5", features = ["derive", "env"] }
+reqwest        = { version = "0.13", features = ["json", "rustls"], default-features = false }
+serde          = { version = "1.0", features = ["derive"] }
+serde_json     = "1.0"
+tokio          = { version = "1.49", features = ["rt-multi-thread", "process", "time", "io-util", "macros"] }
+tracing        = "0.1"
+tracing-subscriber = { version = "0.3", features = ["env-filter"] }
+uuid           = { version = "1.20", features = ["v4", "serde"] }

crates/buildkit-launcher/Dockerfile

@@ -0,0 +1,38 @@
+# syntax=docker/dockerfile:1.7
+# Build context: provisioning/ (pass --local context=. from provisioning/ root)
+# Dockerfile path (buildctl): platform/crates/buildkit-launcher/Dockerfile
+# Target architecture: aarch64 (CAX/CCX Woodpecker agents are ARM64)
+# Cache: --export-cache / --import-cache against zot /cache/buildkit-launcher
+
+FROM rust:bookworm AS builder
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    pkg-config libssl-dev ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN rustup target add aarch64-unknown-linux-gnu
+
+WORKDIR /workspace
+
+# Cargo.workspace.toml is a self-contained workspace manifest: it omits all
+# path deps that point outside this build context (ontoref, stratumiops, etc.).
+# It must define every key that the crate's Cargo.toml inherits via .workspace = true.
+COPY platform/crates/buildkit-launcher/Cargo.workspace.toml Cargo.toml
+COPY platform/crates/buildkit-launcher/ crates/buildkit-launcher/
+
+RUN --mount=type=cache,target=/root/.cargo/registry,sharing=locked \
+    --mount=type=cache,target=/root/.cargo/git,sharing=locked \
+    --mount=type=cache,target=/workspace/target,sharing=locked \
+    cargo build --release --target aarch64-unknown-linux-gnu \
+      --package buildkit-launcher && \
+    cp target/aarch64-unknown-linux-gnu/release/buildkit-launcher /buildkit-launcher-bin
+
+FROM debian:bookworm-slim
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    ca-certificates openssh-client \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY --from=builder /buildkit-launcher-bin /buildkit-launcher
+
+ENTRYPOINT ["/buildkit-launcher"]

crates/buildkit-launcher/src/buildctl_runner.rs

@@ -0,0 +1,137 @@
+use anyhow::{Context, Result};
+use std::path::Path;
+use tokio::process::Command;
+use tracing::{debug, info};
+
+pub const OOM_EXIT_CODE: i32 = 137;
+
+pub struct BuildOutput {
+    pub cpu_secs: f64,
+    pub mem_peak_mb: f64,
+    pub duration_secs: f64,
+}
+
+pub async fn rsync_context(
+    ssh_host: &str,
+    ssh_port: u16,
+    context_path: &Path,
+    ssh_key: &Path,
+) -> Result<()> {
+    info!(host = %ssh_host, port = ssh_port, "rsyncing build context");
+
+    let status = Command::new("rsync")
+        .args([
+            "-az",
+            "--delete",
+            "-e",
+            &format!(
+                "ssh -p {} -i {} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null",
+                ssh_port,
+                ssh_key.display()
+            ),
+        ])
+        .arg(format!("{}/", context_path.display()))
+        .arg(format!("root@{}:/build/context/", ssh_host))
+        .status()
+        .await
+        .context("rsync failed to start")?;
+
+    if !status.success() {
+        anyhow::bail!("rsync exited {}", status);
+    }
+    Ok(())
+}
+
+pub async fn run_buildctl(
+    ssh_host: &str,
+    ssh_port: u16,
+    ssh_key: &Path,
+    image_ref: &str,
+    dockerfile: &str,
+    cache_from: Option<&str>,
+    cache_to: Option<&str>,
+) -> Result<BuildOutput> {
+    let mut buildctl_args = vec![
+        "build".to_string(),
+        "--frontend".to_string(),
+        "dockerfile.v0".to_string(),
+        "--opt".to_string(),
+        format!("filename={}", dockerfile),
+        "--local".to_string(),
+        "context=/build/context".to_string(),
+        "--local".to_string(),
+        "dockerfile=/build/context".to_string(),
+        "--output".to_string(),
+        format!("type=image,name={},push=true", image_ref),
+    ];
+
+    if let Some(from) = cache_from {
+        buildctl_args.push("--import-cache".to_string());
+        buildctl_args.push(from.to_string());
+    }
+    if let Some(to) = cache_to {
+        buildctl_args.push("--export-cache".to_string());
+        buildctl_args.push(to.to_string());
+    }
+
+    let remote_cmd = format!("buildctl {}", buildctl_args.join(" "));
+    debug!(cmd = %remote_cmd, "running buildctl on runner");
+
+    let start = std::time::Instant::now();
+
+    let output = Command::new("ssh")
+        .args([
+            "-p",
+            &ssh_port.to_string(),
+            "-i",
+            &ssh_key.to_string_lossy(),
+            "-o",
+            "StrictHostKeyChecking=no",
+            "-o",
+            "UserKnownHostsFile=/dev/null",
+            &format!("root@{}", ssh_host),
+            &remote_cmd,
+        ])
+        .output()
+        .await
+        .context("ssh buildctl failed to start")?;
+
+    let duration_secs = start.elapsed().as_secs_f64();
+
+    if !output.status.success() {
+        let exit_code = output.status.code().unwrap_or(-1);
+        let stderr = String::from_utf8_lossy(&output.stderr);
+        anyhow::bail!(
+            "buildctl exit={}: {}",
+            exit_code,
+            stderr.lines().last().unwrap_or("(no stderr)")
+        );
+    }
+
+    // Parse /proc/self/status peak memory from buildkitd if available in stdout
+    // Fallback: estimate from image size (not accurate; real metrics come from cgroups)
+    let mem_peak_mb = parse_mem_peak_from_output(&String::from_utf8_lossy(&output.stdout))
+        .unwrap_or(0.0);
+
+    // cpu_secs: rough estimate — not from cgroups; orchestrator records real P95 after multiple runs
+    let cpu_secs = duration_secs;
+
+    Ok(BuildOutput { cpu_secs, mem_peak_mb, duration_secs })
+}
+
+fn parse_mem_peak_from_output(stdout: &str) -> Option<f64> {
+    for line in stdout.lines() {
+        if let Some(val) = line.strip_prefix("MEM_PEAK_MB=") {
+            return val.parse().ok();
+        }
+    }
+    None
+}
+
+pub fn is_oom(err: &anyhow::Error) -> bool {
+    let msg = err.to_string();
+    msg.contains(&format!("exit={}", OOM_EXIT_CODE))
+        || msg.contains("exit=137")
+        || msg.contains("OOM")
+        || msg.contains("Killed")
+}

crates/buildkit-launcher/src/main.rs

@@ -0,0 +1,188 @@
+mod buildctl_runner;
+mod orchestrator_client;
+mod retry;
+mod sizing;
+
+use anyhow::{Context, Result};
+use clap::Parser;
+use orchestrator_client::OrchestratorClient;
+use std::path::PathBuf;
+use tracing::{error, info, warn};
+
+#[derive(Parser)]
+#[command(name = "buildkit-launcher", about = "Ephemeral buildkit runner — ADR-039")]
+struct Args {
+    /// Workspace name (used for P95 lookup and runner labelling)
+    #[arg(long, env = "BUILDKIT_WORKSPACE")]
+    workspace: String,
+
+    /// Build context directory (must be reachable from this host)
+    #[arg(long)]
+    context: PathBuf,
+
+    /// Fully-qualified image reference to push after build
+    #[arg(long)]
+    image: String,
+
+    /// Dockerfile path relative to context
+    #[arg(long, default_value = "Dockerfile")]
+    dockerfile: String,
+
+    /// Cache-from reference (optional, e.g. type=registry,ref=zot.example/cache)
+    #[arg(long)]
+    cache_from: Option<String>,
+
+    /// Cache-to reference (optional)
+    #[arg(long)]
+    cache_to: Option<String>,
+
+    /// SSH key to use when connecting to the runner
+    #[arg(long, env = "BUILDKIT_SSH_KEY")]
+    ssh_key: PathBuf,
+
+    /// Language hint for size defaults (rust, go, java, etc.)
+    #[arg(long)]
+    language: Option<String>,
+
+    /// Golden image snapshot ID or name to boot the runner from
+    #[arg(long, env = "BUILDKIT_RUNNER_IMAGE", default_value = "buildkit-runner-latest")]
+    runner_image: String,
+
+    /// Orchestrator base URL
+    #[arg(long, env = "ORCHESTRATOR_URL", default_value = "http://localhost:9011")]
+    orchestrator_url: String,
+}
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    tracing_subscriber::fmt()
+        .with_env_filter(
+            tracing_subscriber::EnvFilter::from_default_env()
+                .add_directive("buildkit_launcher=info".parse()?),
+        )
+        .init();
+
+    let args = Args::parse();
+    let client = OrchestratorClient::new(args.orchestrator_url.clone());
+
+    // Resolve runner size
+    let p95 = client.get_p95(&args.workspace).await.unwrap_or(None);
+    let (p95_cpu, p95_mem) = p95
+        .as_ref()
+        .map(|s| (Some(s.cpu_p95), Some(s.memory_mb_p95)))
+        .unwrap_or((None, None));
+
+    let size = sizing::resolve(
+        &args.context,
+        p95_cpu,
+        p95_mem,
+        args.language.as_deref(),
+    )
+    .context("failed to resolve runner size")?;
+
+    info!(
+        cpu = size.cpu,
+        memory_gb = size.memory_gb,
+        workspace = %args.workspace,
+        "resolved runner size"
+    );
+
+    let result = run_build(&args, &client, size.clone()).await;
+
+    match result {
+        Ok(()) => Ok(()),
+        Err(e) if buildctl_runner::is_oom(&e) && retry::oom_retry_allowed(0) => {
+            warn!(
+                error = %e,
+                limit = retry::MAX_OOM_RETRIES,
+                "OOM detected — retrying at next size tier"
+            );
+
+            let retry_size = retry::next_size_tier(&size)
+                .context("OOM on largest tier — no retry possible")?;
+
+            info!(
+                cpu = retry_size.cpu,
+                memory_gb = retry_size.memory_gb,
+                "retrying build at next tier"
+            );
+
+            run_build(&args, &client, retry_size).await
+        }
+        Err(e) => Err(e),
+    }
+}
+
+async fn run_build(args: &Args, client: &OrchestratorClient, size: sizing::RunnerSize) -> Result<()> {
+    let runner = client
+        .spawn_runner(&size, &args.workspace, &args.runner_image)
+        .await
+        .context("failed to spawn runner")?;
+
+    info!(
+        lease_id = %runner.lease_id,
+        ssh_host = %runner.ssh_host,
+        ssh_port = runner.ssh_port,
+        expires_at = %runner.expires_at,
+        "runner spawned"
+    );
+
+    let destroy = |lease_id: String| {
+        let client = client.clone();
+        tokio::spawn(async move {
+            if let Err(e) = client.destroy_runner(&lease_id).await {
+                error!(lease_id = %lease_id, error = %e, "failed to destroy runner");
+            }
+        })
+    };
+
+    // Rsync build context to runner
+    if let Err(e) = buildctl_runner::rsync_context(
+        &runner.ssh_host,
+        runner.ssh_port,
+        &args.context,
+        &args.ssh_key,
+    )
+    .await
+    {
+        destroy(runner.lease_id.clone());
+        return Err(e.context("rsync build context failed"));
+    }
+
+    // Run buildctl
+    let build_result = buildctl_runner::run_buildctl(
+        &runner.ssh_host,
+        runner.ssh_port,
+        &args.ssh_key,
+        &args.image,
+        &args.dockerfile,
+        args.cache_from.as_deref(),
+        args.cache_to.as_deref(),
+    )
+    .await;
+
+    // Always destroy the runner, even on build failure
+    destroy(runner.lease_id.clone());
+
+    let output = build_result.context("buildctl failed")?;
+
+    info!(
+        duration_secs = output.duration_secs,
+        mem_peak_mb = output.mem_peak_mb,
+        "build complete — recording metrics"
+    );
+
+    if let Err(e) = client
+        .record_metrics(
+            &args.workspace,
+            output.cpu_secs,
+            output.mem_peak_mb,
+            output.duration_secs,
+        )
+        .await
+    {
+        warn!(error = %e, "failed to record build metrics (non-fatal)");
+    }
+
+    Ok(())
+}

crates/buildkit-launcher/src/orchestrator_client.rs

@@ -0,0 +1,158 @@
+use anyhow::{Context, Result};
+use serde::{Deserialize, Serialize};
+
+use crate::sizing::RunnerSize;
+
+#[derive(Debug, Serialize)]
+pub struct SpawnRequest {
+    pub cpu: u32,
+    pub memory_gb: u32,
+    pub disk_gb: u32,
+    pub time_budget_min: u32,
+    pub workspace: String,
+    pub image: String,
+}
+
+#[derive(Debug, Deserialize)]
+pub struct SpawnResponse {
+    pub lease_id: String,
+    pub ssh_host: String,
+    pub ssh_port: u16,
+    pub expires_at: String,
+}
+
+#[derive(Debug, Deserialize)]
+pub struct P95Stats {
+    pub cpu_p95: f64,
+    pub memory_mb_p95: f64,
+    #[allow(dead_code)]
+    pub duration_secs_p95: f64,
+    #[allow(dead_code)]
+    pub sample_count: u64,
+}
+
+#[derive(Debug, Deserialize)]
+struct ApiResponse<T> {
+    success: bool,
+    data: Option<T>,
+    error: Option<String>,
+}
+
+#[derive(Clone)]
+pub struct OrchestratorClient {
+    base_url: String,
+    http: reqwest::Client,
+}
+
+impl OrchestratorClient {
+    pub fn new(base_url: String) -> Self {
+        Self {
+            base_url,
+            http: reqwest::Client::new(),
+        }
+    }
+
+    pub async fn spawn_runner(
+        &self,
+        size: &RunnerSize,
+        workspace: &str,
+        image: &str,
+    ) -> Result<SpawnResponse> {
+        let req = SpawnRequest {
+            cpu: size.cpu,
+            memory_gb: size.memory_gb,
+            disk_gb: size.disk_gb,
+            time_budget_min: size.time_budget_min,
+            workspace: workspace.to_string(),
+            image: image.to_string(),
+        };
+
+        let resp: ApiResponse<SpawnResponse> = self
+            .http
+            .post(format!("{}/api/v1/vm-pool/spawn", self.base_url))
+            .json(&req)
+            .send()
+            .await
+            .context("POST /api/v1/vm-pool/spawn failed")?
+            .error_for_status()
+            .context("orchestrator returned error for spawn")?
+            .json()
+            .await
+            .context("failed to deserialize SpawnResponse")?;
+
+        if !resp.success {
+            anyhow::bail!(
+                "spawn_runner: {}",
+                resp.error.as_deref().unwrap_or("unknown error")
+            );
+        }
+        resp.data.context("spawn_runner: missing data in response")
+    }
+
+    pub async fn destroy_runner(&self, lease_id: &str) -> Result<()> {
+        let resp = self
+            .http
+            .delete(format!("{}/api/v1/vm-pool/{}", self.base_url, lease_id))
+            .send()
+            .await
+            .context("DELETE /api/v1/vm-pool/{lease_id} failed")?;
+
+        if !resp.status().is_success() {
+            anyhow::bail!("destroy_runner HTTP {}", resp.status());
+        }
+        Ok(())
+    }
+
+    pub async fn get_p95(&self, workspace: &str) -> Result<Option<P95Stats>> {
+        let resp = self
+            .http
+            .get(format!(
+                "{}/api/v1/vm-pool/p95/{}",
+                self.base_url, workspace
+            ))
+            .send()
+            .await
+            .context("GET /api/v1/vm-pool/p95/{workspace} failed")?;
+
+        if resp.status() == reqwest::StatusCode::NOT_FOUND {
+            return Ok(None);
+        }
+
+        let body: ApiResponse<P95Stats> = resp
+            .error_for_status()
+            .context("p95 endpoint error")?
+            .json()
+            .await
+            .context("failed to deserialize P95Stats")?;
+
+        Ok(body.data)
+    }
+
+    pub async fn record_metrics(
+        &self,
+        workspace: &str,
+        cpu_secs: f64,
+        mem_peak_mb: f64,
+        duration_secs: f64,
+    ) -> Result<()> {
+        let payload = serde_json::json!({
+            "workspace": workspace,
+            "cpu_secs": cpu_secs,
+            "mem_peak_mb": mem_peak_mb,
+            "duration_secs": duration_secs,
+        });
+
+        let resp = self
+            .http
+            .post(format!("{}/api/v1/vm-pool/metrics", self.base_url))
+            .json(&payload)
+            .send()
+            .await
+            .context("POST /api/v1/vm-pool/metrics failed")?;
+
+        if !resp.status().is_success() {
+            anyhow::bail!("record_metrics HTTP {}", resp.status());
+        }
+        Ok(())
+    }
+}

crates/buildkit-launcher/src/retry.rs

@@ -0,0 +1,31 @@
+use crate::sizing::RunnerSize;
+
+/// Hard bound on OOM auto-retries per build — ADR-039 constraint oom-retry-bounded.
+/// Repeated OOM after one retry indicates misconfiguration; surface to developer.
+pub const MAX_OOM_RETRIES: u8 = 1;
+
+/// Returns whether an OOM retry is permitted given the count of retries already consumed.
+pub fn oom_retry_allowed(retries_used: u8) -> bool {
+    retries_used < MAX_OOM_RETRIES
+}
+
+const SIZE_TIERS: &[(&str, u32, u32)] = &[
+    ("cx22", 2, 4),
+    ("cx32", 4, 8),
+    ("cx42", 8, 16),
+    ("cx52", 16, 32),
+];
+
+pub fn next_size_tier(current: &RunnerSize) -> Option<RunnerSize> {
+    // Find current tier by matching cpu/memory_gb
+    let current_idx = SIZE_TIERS.iter().position(|(_, cpu, mem)| {
+        *cpu >= current.cpu && *mem >= current.memory_gb
+    })?;
+
+    SIZE_TIERS.get(current_idx + 1).map(|(_, cpu, mem)| RunnerSize {
+        cpu: *cpu,
+        memory_gb: *mem,
+        disk_gb: current.disk_gb,
+        time_budget_min: current.time_budget_min,
+    })
+}

crates/buildkit-launcher/src/sizing.rs

@@ -0,0 +1,92 @@
+use anyhow::Result;
+use serde::Deserialize;
+use std::path::Path;
+
+#[derive(Debug, Clone)]
+pub struct RunnerSize {
+    pub cpu: u32,
+    pub memory_gb: u32,
+    pub disk_gb: u32,
+    pub time_budget_min: u32,
+}
+
+#[derive(Deserialize, Default)]
+struct BuildSpec {
+    cpu: Option<u32>,
+    memory_gb: Option<u32>,
+    disk_gb: Option<u32>,
+    time_budget_min: Option<u32>,
+    language: Option<String>,
+}
+
+impl RunnerSize {
+    fn language_default(language: &str) -> Self {
+        match language {
+            "rust" => Self { cpu: 4, memory_gb: 8, disk_gb: 50, time_budget_min: 60 },
+            "go" => Self { cpu: 2, memory_gb: 4, disk_gb: 30, time_budget_min: 30 },
+            "java" | "kotlin" | "scala" => Self { cpu: 4, memory_gb: 8, disk_gb: 40, time_budget_min: 45 },
+            _ => Self { cpu: 2, memory_gb: 4, disk_gb: 30, time_budget_min: 30 },
+        }
+    }
+
+    pub fn from_p95(cpu_p95: f64, memory_mb_p95: f64) -> Self {
+        let cpu = ((cpu_p95 * 1.2).ceil() as u32).max(2);
+        let memory_gb = (((memory_mb_p95 * 1.2) / 1024.0).ceil() as u32).max(4);
+        Self { cpu, memory_gb, disk_gb: 30, time_budget_min: 60 }
+    }
+}
+
+fn parse_build_spec(context_path: &Path) -> Option<BuildSpec> {
+    let spec_path = context_path.join(".build-spec.ncl");
+    if !spec_path.exists() {
+        return None;
+    }
+    // Extract JSON from nickel export — buildkit-launcher shells out to nickel
+    let output = std::process::Command::new("nickel")
+        .args(["export", "--format", "json"])
+        .arg(&spec_path)
+        .output()
+        .ok()?;
+    if !output.status.success() {
+        return None;
+    }
+    serde_json::from_slice(&output.stdout).ok()
+}
+
+pub fn resolve(
+    context_path: &Path,
+    p95_cpu: Option<f64>,
+    p95_mem_mb: Option<f64>,
+    language_hint: Option<&str>,
+) -> Result<RunnerSize> {
+    let spec = parse_build_spec(context_path).unwrap_or_default();
+
+    // Tier 1: explicit declaration in .build-spec.ncl
+    if spec.cpu.is_some() || spec.memory_gb.is_some() {
+        let base = language_hint
+            .or(spec.language.as_deref())
+            .map(RunnerSize::language_default)
+            .unwrap_or_else(|| RunnerSize::language_default("default"));
+        return Ok(RunnerSize {
+            cpu: spec.cpu.unwrap_or(base.cpu),
+            memory_gb: spec.memory_gb.unwrap_or(base.memory_gb),
+            disk_gb: spec.disk_gb.unwrap_or(base.disk_gb),
+            time_budget_min: spec.time_budget_min.unwrap_or(base.time_budget_min),
+        });
+    }
+
+    // Tier 2: P95 historical × 1.2
+    if let (Some(cpu_p95), Some(mem_p95)) = (p95_cpu, p95_mem_mb) {
+        let mut size = RunnerSize::from_p95(cpu_p95, mem_p95);
+        if let Some(budget) = spec.time_budget_min {
+            size.time_budget_min = budget;
+        }
+        return Ok(size);
+    }
+
+    // Tier 3: language defaults
+    let lang = language_hint
+        .or(spec.language.as_deref())
+        .unwrap_or("default");
+    Ok(RunnerSize::language_default(lang))
+}

crates/contract-tests/Cargo.toml

@@ -0,0 +1,29 @@
+[package]
+authors.workspace = true
+edition.workspace = true
+license.workspace = true
+name = "contract-tests"
+repository.workspace = true
+version.workspace = true
+description = "G3 — CLI↔HTTP↔MCP contract tests for the provisioning-core tool Registry"
+publish = false
+
+[lib]
+path = "src/lib.rs"
+
+[dependencies]
+provisioning-core   = { path = "../provisioning-core" }
+provisioning-daemon = { path = "../provisioning-daemon" }
+provisioning-mcp    = { path = "../mcp-server" }
+
+tokio         = { workspace = true, features = ["full"] }
+axum          = { workspace = true }
+serde         = { workspace = true }
+serde_json    = { workspace = true }
+anyhow        = { workspace = true }
+async-trait   = { workspace = true }
+reqwest       = { workspace = true }
+jsonschema    = "0.28"
+
+[dev-dependencies]
+rustls = { version = "0.23", features = ["aws_lc_rs"] }

crates/contract-tests/src/lib.rs

@@ -0,0 +1,257 @@
+//! G3 — CLI↔HTTP↔MCP contract-test harness.
+//!
+//! Provides fixture tools, a three-tier dispatcher, and envelope-normalisation
+//! helpers so the invariant "same tool + same params → same payload through all
+//! three surfaces" can be asserted in an integration test without touching
+//! filesystem, orchestrator, or provider CLIs.
+
+use async_trait::async_trait;
+use provisioning_core::{
+    Environment, Registry, ToolError,
+    protocol::ToolCategory,
+    tool::{Context, Tool},
+};
+use serde_json::{Value, json};
+use std::sync::Arc;
+
+// ── Fixture tools ─────────────────────────────────────────────────────────────
+
+/// Returns the input params verbatim — schema allows any object.
+pub struct EchoTool;
+
+#[async_trait]
+impl Tool for EchoTool {
+    fn name(&self) -> &'static str { "ct_echo" }
+    fn description(&self) -> &'static str { "Echo the input parameters verbatim" }
+    fn schema(&self) -> Value {
+        json!({
+            "type": "object",
+            "properties": {
+                "message": { "type": "string" }
+            },
+            "required": ["message"],
+            "additionalProperties": true
+        })
+    }
+    fn category(&self) -> ToolCategory { ToolCategory::Read }
+
+    async fn invoke(&self, params: Value, _ctx: &Context) -> Result<Value, ToolError> {
+        let message = params.get("message").and_then(|v| v.as_str()).ok_or_else(|| {
+            ToolError::invalid_param("message", "required string field missing")
+        })?;
+        Ok(json!({
+            "items": [{ "message": message }],
+            "total": 1
+        }))
+    }
+}
+
+/// Deterministic three-item listing — same output regardless of params.
+pub struct ListingTool;
+
+#[async_trait]
+impl Tool for ListingTool {
+    fn name(&self) -> &'static str { "ct_listing" }
+    fn description(&self) -> &'static str { "Return a canonical three-item listing" }
+    fn schema(&self) -> Value {
+        json!({ "type": "object", "properties": {}, "additionalProperties": false })
+    }
+    fn category(&self) -> ToolCategory { ToolCategory::Read }
+
+    async fn invoke(&self, _params: Value, _ctx: &Context) -> Result<Value, ToolError> {
+        Ok(json!({
+            "items": [
+                { "id": "a", "value": 1 },
+                { "id": "b", "value": 2 },
+                { "id": "c", "value": 3 }
+            ],
+            "total": 3
+        }))
+    }
+}
+
+/// Always fails with `InvalidParam` — used to validate error-path contract.
+pub struct FailingTool;
+
+#[async_trait]
+impl Tool for FailingTool {
+    fn name(&self) -> &'static str { "ct_fail" }
+    fn description(&self) -> &'static str { "Always fails with InvalidParam" }
+    fn schema(&self) -> Value {
+        json!({ "type": "object", "properties": {}, "additionalProperties": true })
+    }
+    fn category(&self) -> ToolCategory { ToolCategory::Read }
+
+    async fn invoke(&self, _params: Value, _ctx: &Context) -> Result<Value, ToolError> {
+        Err(ToolError::invalid_param("expected", "fixture always rejects"))
+    }
+}
+
+/// Build a Registry populated with the three fixture tools.
+pub fn make_fixture_registry() -> Registry {
+    let mut reg = Registry::new();
+    reg.register(Arc::new(EchoTool)).expect("register echo");
+    reg.register(Arc::new(ListingTool)).expect("register listing");
+    reg.register(Arc::new(FailingTool)).expect("register fail");
+    reg
+}
+
+// ── Output-shape schema (the contract) ────────────────────────────────────────
+
+/// JSON Schema that all `Read`-category list-style tools must satisfy.
+/// Generated once at build time so each contract-test case can validate
+/// without re-parsing.
+pub fn listing_output_schema() -> Value {
+    json!({
+        "type": "object",
+        "required": ["items", "total"],
+        "properties": {
+            "items": { "type": "array" },
+            "total": { "type": "integer", "minimum": 0 }
+        },
+        "additionalProperties": true
+    })
+}
+
+// ── Three-tier dispatcher ─────────────────────────────────────────────────────
+
+pub struct TierOutcome {
+    /// Inner payload after envelope unwrap. `None` on error.
+    pub payload: Option<Value>,
+    /// Error code on failure, `None` on success. Each tier maps ToolError
+    /// to its own code namespace — the normaliser strips that difference.
+    pub error_code: Option<i32>,
+    /// Error message (trimmed of tier-specific prefixes).
+    pub error_message: Option<String>,
+}
+
+impl TierOutcome {
+    pub fn ok(payload: Value) -> Self {
+        Self { payload: Some(payload), error_code: None, error_message: None }
+    }
+    pub fn err(code: i32, message: impl Into<String>) -> Self {
+        Self { payload: None, error_code: Some(code), error_message: Some(message.into()) }
+    }
+}
+
+/// Tier A — direct Registry invocation. The reference tier: what the other
+/// two surfaces must agree with after envelope unwrapping.
+pub async fn tier_registry(
+    registry: &Registry,
+    env: Arc<Environment>,
+    tool: &str,
+    params: Value,
+) -> TierOutcome {
+    let ctx = Context::new(env);
+    match registry.invoke(tool, params, &ctx).await {
+        Ok(v) => TierOutcome::ok(v),
+        Err(e) => {
+            let code = match &e {
+                ToolError::NotFound(_) => -32001,
+                ToolError::InvalidParam { .. } => -32602,
+                ToolError::Unauthorized(_) => -32003,
+                _ => -32000,
+            };
+            TierOutcome::err(code, e.to_string())
+        }
+    }
+}
+
+/// Tier B — HTTP daemon. The axum router is reused directly on a TcpListener
+/// bound to 127.0.0.1:0 for deterministic port isolation.
+pub async fn tier_http(base_url: &str, tool: &str, params: Value) -> TierOutcome {
+    let client = reqwest::Client::new();
+    let url = format!("{}/api/v1/tools/{}", base_url, tool);
+    let body = json!({ "params": params });
+
+    let resp = match client.post(&url).json(&body).send().await {
+        Ok(r) => r,
+        Err(e) => return TierOutcome::err(-32000, format!("http transport: {e}")),
+    };
+
+    let status = resp.status();
+    let body: Value = match resp.json().await {
+        Ok(v) => v,
+        Err(e) => return TierOutcome::err(-32000, format!("http body: {e}")),
+    };
+
+    if status.is_success() {
+        match body.get("result").cloned() {
+            Some(v) => TierOutcome::ok(v),
+            None => TierOutcome::err(-32000, "missing result field in http response"),
+        }
+    } else {
+        let code = body
+            .get("code")
+            .and_then(|v| v.as_i64())
+            .map(|c| c as i32)
+            .unwrap_or(-32000);
+        let msg = body
+            .get("error")
+            .and_then(|v| v.as_str())
+            .unwrap_or("<missing error>")
+            .to_string();
+        TierOutcome::err(code, msg)
+    }
+}
+
+/// Tier C — MCP surface via McpServer::handle_request.
+/// MCP wraps success payloads in `{content: [{type:"text", text: "<json>"}]}`;
+/// the unwrap parses the text field back into JSON.
+pub async fn tier_mcp(server: &provisioning_mcp_server::registry_server::McpServer, tool: &str, params: Value) -> TierOutcome {
+    let rpc = json!({
+        "jsonrpc": "2.0",
+        "id": 1,
+        "method": "tools/call",
+        "params": { "name": tool, "arguments": params }
+    });
+
+    let resp = server.handle_request(rpc).await;
+
+    if let Some(err) = resp.get("error") {
+        let code = err.get("code").and_then(|v| v.as_i64()).map(|c| c as i32).unwrap_or(-32000);
+        let msg  = err.get("message").and_then(|v| v.as_str()).unwrap_or("<missing>").to_string();
+        return TierOutcome::err(code, msg);
+    }
+
+    let content = match resp.get("result").and_then(|r| r.get("content")) {
+        Some(c) => c,
+        None => return TierOutcome::err(-32000, "mcp response missing result.content"),
+    };
+
+    let text = content
+        .get(0)
+        .and_then(|c| c.get("text"))
+        .and_then(|t| t.as_str());
+    let text = match text {
+        Some(t) => t,
+        None => return TierOutcome::err(-32000, "mcp content[0].text missing"),
+    };
+
+    match serde_json::from_str::<Value>(text) {
+        Ok(v) => TierOutcome::ok(v),
+        Err(e) => TierOutcome::err(-32000, format!("mcp text parse: {e}")),
+    }
+}
+
+// ── Payload normalisation ─────────────────────────────────────────────────────
+
+/// Strip fields that are allowed to diverge between tiers: trace ids, timestamps,
+/// uuids. The contract is on semantic payload, not byte equality.
+pub fn normalise(v: &Value) -> Value {
+    match v {
+        Value::Object(map) => {
+            let mut out = serde_json::Map::new();
+            for (k, val) in map {
+                if matches!(k.as_str(), "trace_id" | "span_id" | "timestamp" | "generated_at" | "request_id")
+                {
+                    continue;
+                }
+                out.insert(k.clone(), normalise(val));
+            }
+            Value::Object(out)
+        }
+        Value::Array(arr) => Value::Array(arr.iter().map(normalise).collect()),
+        _ => v.clone(),
+    }
+}

crates/contract-tests/tests/g3_contract.rs

@@ -0,0 +1,194 @@
+//! G3 — CLI↔HTTP↔MCP contract test.
+//!
+//! For each fixture tool, invoke via:
+//!   Tier A — Registry directly (reference)
+//!   Tier B — HTTP daemon (axum::serve on 127.0.0.1:0)
+//!   Tier C — MCP server (handle_request in-process)
+//!
+//! Assert:
+//!   1. All tiers that succeed produce equal normalised payloads
+//!   2. The payload validates against the listing output schema
+//!   3. All tiers that fail produce the same error code
+//!
+//! The normaliser strips volatile fields (trace_id, timestamp, …) so the
+//! contract is on semantic equivalence, not byte-for-byte equality.
+
+use contract_tests::{
+    listing_output_schema, make_fixture_registry, normalise,
+    tier_http, tier_mcp, tier_registry,
+};
+use jsonschema::Validator;
+use provisioning_mcp_server::registry_server::McpServer;
+use provisioning_core::Environment;
+use provisioning_daemon::{AppState, http::router};
+use serde_json::{Value, json};
+use std::sync::{Arc, OnceLock};
+
+// ── Test harness ──────────────────────────────────────────────────────────────
+
+static CRYPTO: OnceLock<()> = OnceLock::new();
+
+fn init_crypto() {
+    CRYPTO.get_or_init(|| {
+        let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
+    });
+}
+
+struct Harness {
+    env: Arc<Environment>,
+    registry: Arc<provisioning_core::Registry>,
+    mcp: McpServer,
+    http_base: String,
+    _daemon_task: tokio::task::JoinHandle<()>,
+}
+
+async fn boot() -> Harness {
+    init_crypto();
+
+    // Shared fixture registry for all tiers.
+    let registry = Arc::new(make_fixture_registry());
+    let env = Arc::new(Environment::default());
+
+    // Tier B — bind daemon on ephemeral port.
+    let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("bind");
+    let addr = listener.local_addr().expect("local_addr");
+    let http_base = format!("http://{addr}");
+
+    // AppState owns its own Arc<Registry>; we build it from a fresh fixture
+    // registry so the daemon and the reference tier see identical tool sets.
+    let daemon_registry = make_fixture_registry();
+    let app = AppState::new(daemon_registry, Environment::default());
+    let app_router = router(app);
+    let task = tokio::spawn(async move {
+        axum::serve(listener, app_router).await.expect("daemon serve");
+    });
+
+    // Wait for the daemon to accept connections (fast — local TCP).
+    for _ in 0..40 {
+        if reqwest::get(format!("{}/health", http_base)).await.is_ok() {
+            break;
+        }
+        tokio::time::sleep(std::time::Duration::from_millis(25)).await;
+    }
+
+    // Tier C — in-process MCP server with its own fixture registry.
+    let mcp = McpServer::new(make_fixture_registry(), Environment::default(), "g3-test", "0");
+
+    Harness { env, registry, mcp, http_base, _daemon_task: task }
+}
+
+fn validate_listing(v: &Value) {
+    let schema = listing_output_schema();
+    let validator = Validator::new(&schema).expect("compile schema");
+    validator
+        .validate(v)
+        .unwrap_or_else(|errors| panic!("listing schema validation failed: {errors}"));
+}
+
+// ── Success contract ──────────────────────────────────────────────────────────
+
+#[tokio::test]
+async fn listing_tool_agrees_across_three_tiers() {
+    let h = boot().await;
+    let params = json!({});
+
+    let a = tier_registry(&h.registry, h.env.clone(), "ct_listing", params.clone()).await;
+    let b = tier_http(&h.http_base, "ct_listing", params.clone()).await;
+    let c = tier_mcp(&h.mcp, "ct_listing", params.clone()).await;
+
+    let a_payload = a.payload.expect("tier A registry must succeed");
+    let b_payload = b.payload.expect("tier B http must succeed");
+    let c_payload = c.payload.expect("tier C mcp must succeed");
+
+    validate_listing(&a_payload);
+    validate_listing(&b_payload);
+    validate_listing(&c_payload);
+
+    assert_eq!(normalise(&a_payload), normalise(&b_payload), "A↔B diverge");
+    assert_eq!(normalise(&a_payload), normalise(&c_payload), "A↔C diverge");
+    assert_eq!(normalise(&b_payload), normalise(&c_payload), "B↔C diverge");
+}
+
+#[tokio::test]
+async fn echo_tool_agrees_across_three_tiers() {
+    let h = boot().await;
+    let params = json!({ "message": "contract" });
+
+    let a = tier_registry(&h.registry, h.env.clone(), "ct_echo", params.clone()).await;
+    let b = tier_http(&h.http_base, "ct_echo", params.clone()).await;
+    let c = tier_mcp(&h.mcp, "ct_echo", params.clone()).await;
+
+    let a_payload = a.payload.expect("A must succeed");
+    let b_payload = b.payload.expect("B must succeed");
+    let c_payload = c.payload.expect("C must succeed");
+
+    validate_listing(&a_payload);
+    validate_listing(&b_payload);
+    validate_listing(&c_payload);
+
+    assert_eq!(normalise(&a_payload), normalise(&b_payload));
+    assert_eq!(normalise(&a_payload), normalise(&c_payload));
+
+    // Exact content check — echo should round-trip the message
+    assert_eq!(a_payload["items"][0]["message"], "contract");
+    assert_eq!(b_payload["items"][0]["message"], "contract");
+    assert_eq!(c_payload["items"][0]["message"], "contract");
+}
+
+// ── Failure contract ──────────────────────────────────────────────────────────
+
+#[tokio::test]
+async fn invalid_param_error_code_agrees_across_tiers() {
+    let h = boot().await;
+    let params = json!({}); // echo requires "message" → InvalidParam
+
+    let a = tier_registry(&h.registry, h.env.clone(), "ct_echo", params.clone()).await;
+    let b = tier_http(&h.http_base, "ct_echo", params.clone()).await;
+    let c = tier_mcp(&h.mcp, "ct_echo", params.clone()).await;
+
+    // All three must classify as InvalidParam → -32602
+    assert_eq!(a.error_code, Some(-32602), "A: {:?}", a.error_message);
+    assert_eq!(b.error_code, Some(-32602), "B: {:?}", b.error_message);
+    assert_eq!(c.error_code, Some(-32602), "C: {:?}", c.error_message);
+}
+
+#[tokio::test]
+async fn failing_tool_error_code_agrees_across_tiers() {
+    let h = boot().await;
+    let params = json!({});
+
+    let a = tier_registry(&h.registry, h.env.clone(), "ct_fail", params.clone()).await;
+    let b = tier_http(&h.http_base, "ct_fail", params.clone()).await;
+    let c = tier_mcp(&h.mcp, "ct_fail", params.clone()).await;
+
+    assert_eq!(a.error_code, Some(-32602));
+    assert_eq!(b.error_code, Some(-32602));
+    assert_eq!(c.error_code, Some(-32602));
+}
+
+// ── tools/list contract ───────────────────────────────────────────────────────
+
+#[tokio::test]
+async fn tools_list_count_agrees_across_surfaces() {
+    let h = boot().await;
+
+    // Registry baseline
+    let reg_count = h.registry.len();
+
+    // HTTP
+    let http_body: Value = reqwest::get(format!("{}/api/v1/tools", h.http_base))
+        .await
+        .expect("http list")
+        .json()
+        .await
+        .expect("parse");
+    let http_count = http_body["tools"].as_array().expect("tools array").len();
+
+    // MCP
+    let rpc = json!({"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}});
+    let mcp_resp = h.mcp.handle_request(rpc).await;
+    let mcp_count = mcp_resp["result"]["tools"].as_array().expect("mcp tools").len();
+
+    assert_eq!(reg_count, http_count, "HTTP count diverges from registry");
+    assert_eq!(reg_count, mcp_count, "MCP count diverges from registry");
+}

crates/control-center-ui/src/api/orchestrator_client.rs

@@ -224,6 +224,19 @@ impl OrchestratorClient {
         }
     }
 
+    /// Submit a unified component lifecycle operation.
+    pub async fn deploy_component(&self, operation: &str, workflow: &ComponentWorkflow) -> Result<String, OrchestratorError> {
+        let path = format!("/api/v1/workflows/component/{operation}");
+        let request = self.build_request("POST", &path)?;
+        let response = self.execute_json_request::<String, _>(request, workflow).await?;
+
+        if response.success {
+            Ok(response.data.unwrap_or_default())
+        } else {
+            Err(OrchestratorError::Api(response.error.unwrap_or_default()))
+        }
+    }
+
     /// Create cluster workflow
     pub async fn create_cluster_workflow(&self, workflow: &ClusterWorkflow) -> Result<String, OrchestratorError> {
         let request = self.build_request("POST", "/workflows/cluster/create")?;

crates/control-center-ui/src/api/orchestrator_types.rs

@@ -84,6 +84,30 @@ pub struct ClusterWorkflow {
     pub wait: bool,
 }
 
+/// Unified component lifecycle workflow request.
+/// Submitted to `/api/v1/workflows/component/{op}`.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ComponentWorkflow {
+    pub workspace: String,
+    pub infra: String,
+    pub component: String,
+    pub server: String,
+    #[serde(default)]
+    pub namespace: Option<String>,
+    #[serde(default = "default_ssh_user")]
+    pub ssh_user: String,
+    #[serde(default)]
+    pub ssh_key_path: Option<String>,
+    pub settings: String,
+    #[serde(default)]
+    pub check_mode: bool,
+    pub provisioning: String,
+}
+
+fn default_ssh_user() -> String {
+    "root".to_string()
+}
+
 /// Batch operation request
 #[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct BatchOperationRequest {

crates/control-center/Cargo.toml

@@ -18,6 +18,11 @@ tokio = { workspace = true }
 # Web server and API
 axum = { workspace = true }
 hyper = { workspace = true }
+
+# Ontoref API catalog
+ontoref-ontology = { workspace = true }
+ontoref-derive   = { workspace = true }
+inventory        = { workspace = true }
 tower = { workspace = true }
 tower-http = { workspace = true }
 
@@ -40,7 +45,7 @@ clap = { workspace = true }
 config = { workspace = true }
 
 # Centralized observability (logging, metrics, health, tracing)
-observability = { workspace = true, features = ["logging", "metrics-prometheus", "health"] }
+platform-observability = { workspace = true, features = ["logging", "metrics-prometheus", "health"] }
 
 # Error handling
 anyhow = { workspace = true }
@@ -58,7 +63,7 @@ validator = { workspace = true }
 reqwest = { workspace = true }
 
 # HTTP service clients (machines, init, AI) - enables remote service calls
-service-clients = { workspace = true }
+platform-clients = { workspace = true }
 
 # Platform configuration management
 platform-config = { workspace = true }

crates/control-center/src/api_catalog.rs

@@ -0,0 +1,12 @@
+use std::sync::Arc;
+
+use axum::{extract::State, response::IntoResponse, Json};
+use control_center::AppState;
+use ontoref_ontology::api::ApiRouteEntry;
+use serde_json::json;
+
+pub async fn api_catalog(State(_state): State<Arc<AppState>>) -> impl IntoResponse {
+    let mut routes: Vec<&'static ApiRouteEntry> = inventory::iter::<ApiRouteEntry>().collect();
+    routes.sort_by(|a, b| a.path.cmp(b.path).then(a.method.cmp(b.method)));
+    Json(json!({ "service": "control-center", "routes": routes }))
+}

crates/control-center/src/error.rs.old

@@ -1,391 +0,0 @@
-use axum::{
-    http::StatusCode,
-    response::{IntoResponse, Response},
-    Json,
-};
-use thiserror::Error;
-use tracing::error;
-
-/// Result type alias for the control center
-pub type Result<T> = std::result::Result<T, ControlCenterError>;
-
-/// Main error type for the control center service
-#[derive(Error, Debug)]
-pub enum ControlCenterError {
-    /// Database-related errors
-    #[error("Database error: {0}")]
-    Database(String),
-
-    /// Authentication errors
-    #[error("Authentication error: {0}")]
-    Authentication(String),
-
-    /// Authorization errors
-    #[error("Authorization error: {0}")]
-    Authorization(String),
-
-    /// JWT token errors
-    #[error("Token error: {0}")]
-    Token(String),
-
-    /// Validation errors
-    #[error("Validation error: {0}")]
-    Validation(String),
-
-    /// User not found
-    #[error("User not found: {0}")]
-    UserNotFound(String),
-
-    /// Role not found
-    #[error("Role not found: {0}")]
-    RoleNotFound(String),
-
-    /// Permission not found
-    #[error("Permission not found: {0}")]
-    PermissionNotFound(String),
-
-    /// Session not found
-    #[error("Session not found: {0}")]
-    SessionNotFound(String),
-
-    /// Session expired
-    #[error("Session expired: {0}")]
-    SessionExpired(String),
-
-    /// Resource not found
-    #[error("Not found: {0}")]
-    NotFound(String),
-
-    /// Unauthorized access
-    #[error("Unauthorized: {0}")]
-    Unauthorized(String),
-
-    /// Forbidden access
-    #[error("Forbidden: {0}")]
-    Forbidden(String),
-
-    /// Conflict (e.g., duplicate resources)
-    #[error("Conflict: {0}")]
-    Conflict(String),
-
-    /// Bad request
-    #[error("Bad request: {0}")]
-    BadRequest(String),
-
-    /// Internal server error
-    #[error("Internal server error: {0}")]
-    Internal(String),
-
-    /// Configuration errors
-    #[error("Configuration error: {0}")]
-    Configuration(String),
-
-    /// External service errors
-    #[error("External service error: {0}")]
-    ExternalService(String),
-
-    /// Rate limit exceeded
-    #[error("Rate limit exceeded: {0}")]
-    RateLimit(String),
-
-    /// WebSocket errors
-    #[error("WebSocket error: {0}")]
-    WebSocket(String),
-
-    /// Serialization errors
-    #[error("Serialization error: {0}")]
-    Serialization(String),
-
-    /// Policy evaluation error
-    #[error("Policy evaluation error: {0}")]
-    PolicyEvaluation(String),
-
-    /// Policy parsing error
-    #[error("Policy parsing error: {0}")]
-    PolicyParsing(String),
-
-    /// Cedar policy error
-    #[error("Cedar policy error: {0}")]
-    Cedar(String),
-
-    /// Compliance check failed
-    #[error("Compliance check failed: {0}")]
-    Compliance(String),
-
-    /// Anomaly detection error
-    #[error("Anomaly detection error: {0}")]
-    Anomaly(String),
-
-    /// Analysis failed error
-    #[error("Analysis failed: {0}")]
-    AnalysisFailed(String),
-
-    /// Generic anyhow errors
-    #[error(transparent)]
-    Other(#[from] anyhow::Error),
-}
-
-impl ControlCenterError {
-    /// Get the HTTP status code for this error
-    pub fn status_code(&self) -> StatusCode {
-        match self {
-            Self::Database(_) => StatusCode::INTERNAL_SERVER_ERROR,
-            Self::Authentication(_) => StatusCode::UNAUTHORIZED,
-            Self::Authorization(_) => StatusCode::FORBIDDEN,
-            Self::Token(_) => StatusCode::UNAUTHORIZED,
-            Self::Validation(_) => StatusCode::BAD_REQUEST,
-            Self::UserNotFound(_) => StatusCode::NOT_FOUND,
-            Self::RoleNotFound(_) => StatusCode::NOT_FOUND,
-            Self::PermissionNotFound(_) => StatusCode::NOT_FOUND,
-            Self::SessionNotFound(_) => StatusCode::NOT_FOUND,
-            Self::SessionExpired(_) => StatusCode::UNAUTHORIZED,
-            Self::NotFound(_) => StatusCode::NOT_FOUND,
-            Self::Unauthorized(_) => StatusCode::UNAUTHORIZED,
-            Self::Forbidden(_) => StatusCode::FORBIDDEN,
-            Self::Conflict(_) => StatusCode::CONFLICT,
-            Self::BadRequest(_) => StatusCode::BAD_REQUEST,
-            Self::Internal(_) => StatusCode::INTERNAL_SERVER_ERROR,
-            Self::Configuration(_) => StatusCode::INTERNAL_SERVER_ERROR,
-            Self::ExternalService(_) => StatusCode::BAD_GATEWAY,
-            Self::RateLimit(_) => StatusCode::TOO_MANY_REQUESTS,
-            Self::WebSocket(_) => StatusCode::BAD_REQUEST,
-            Self::Serialization(_) => StatusCode::INTERNAL_SERVER_ERROR,
-            Self::PolicyEvaluation(_) => StatusCode::INTERNAL_SERVER_ERROR,
-            Self::PolicyParsing(_) => StatusCode::BAD_REQUEST,
-            Self::Cedar(_) => StatusCode::INTERNAL_SERVER_ERROR,
-            Self::Compliance(_) => StatusCode::FORBIDDEN,
-            Self::Anomaly(_) => StatusCode::INTERNAL_SERVER_ERROR,
-            Self::AnalysisFailed(_) => StatusCode::BAD_REQUEST,
-            Self::Other(_) => StatusCode::INTERNAL_SERVER_ERROR,
-        }
-    }
-
-    /// Get the error code string for API responses
-    pub fn error_code(&self) -> &'static str {
-        match self {
-            Self::Database(_) => "DATABASE_ERROR",
-            Self::Authentication(_) => "AUTHENTICATION_ERROR",
-            Self::Authorization(_) => "AUTHORIZATION_ERROR",
-            Self::Token(_) => "TOKEN_ERROR",
-            Self::Validation(_) => "VALIDATION_ERROR",
-            Self::UserNotFound(_) => "USER_NOT_FOUND",
-            Self::RoleNotFound(_) => "ROLE_NOT_FOUND",
-            Self::PermissionNotFound(_) => "PERMISSION_NOT_FOUND",
-            Self::SessionNotFound(_) => "SESSION_NOT_FOUND",
-            Self::SessionExpired(_) => "SESSION_EXPIRED",
-            Self::NotFound(_) => "NOT_FOUND",
-            Self::Unauthorized(_) => "UNAUTHORIZED",
-            Self::Forbidden(_) => "FORBIDDEN",
-            Self::Conflict(_) => "CONFLICT",
-            Self::BadRequest(_) => "BAD_REQUEST",
-            Self::Internal(_) => "INTERNAL_ERROR",
-            Self::Configuration(_) => "CONFIGURATION_ERROR",
-            Self::ExternalService(_) => "EXTERNAL_SERVICE_ERROR",
-            Self::RateLimit(_) => "RATE_LIMIT_EXCEEDED",
-            Self::WebSocket(_) => "WEBSOCKET_ERROR",
-            Self::Serialization(_) => "SERIALIZATION_ERROR",
-            Self::PolicyEvaluation(_) => "POLICY_EVALUATION_ERROR",
-            Self::PolicyParsing(_) => "POLICY_PARSING_ERROR",
-            Self::Cedar(_) => "CEDAR_ERROR",
-            Self::Compliance(_) => "COMPLIANCE_ERROR",
-            Self::Anomaly(_) => "ANOMALY_ERROR",
-            Self::AnalysisFailed(_) => "ANALYSIS_FAILED",
-            Self::Other(_) => "INTERNAL_ERROR",
-        }
-    }
-
-    /// Check if this error should be logged at ERROR level
-    pub fn should_log_error(&self) -> bool {
-        match self {
-            Self::Database(_) => true,
-            Self::Internal(_) => true,
-            Self::Configuration(_) => true,
-            Self::Serialization(_) => true,
-            Self::PolicyEvaluation(_) => true,
-            Self::Cedar(_) => true,
-            Self::Anomaly(_) => true,
-            Self::AnalysisFailed(_) => true,
-            Self::Other(_) => true,
-            _ => false, // Client errors are logged at WARN level
-        }
-    }
-}
-
-/// Error response structure for API
-#[derive(serde::Serialize)]
-pub struct ErrorResponse {
-    pub error: String,
-    pub error_code: String,
-    pub message: String,
-    #[serde(skip_serializing_if = "Option::is_none")]
-    pub details: Option<serde_json::Value>,
-    pub timestamp: chrono::DateTime<chrono::Utc>,
-}
-
-impl ErrorResponse {
-    pub fn new(error: &ControlCenterError) -> Self {
-        Self {
-            error: error.error_code().to_string(),
-            error_code: error.error_code().to_string(),
-            message: error.to_string(),
-            details: None,
-            timestamp: chrono::Utc::now(),
-        }
-    }
-
-    pub fn with_details(mut self, details: serde_json::Value) -> Self {
-        self.details = Some(details);
-        self
-    }
-}
-
-impl IntoResponse for ControlCenterError {
-    fn into_response(self) -> Response {
-        let status_code = self.status_code();
-
-        // Log the error appropriately
-        if self.should_log_error() {
-            error!("Error occurred: {} (status: {})", self, status_code);
-        } else {
-            tracing::warn!("Client error: {} (status: {})", self, status_code);
-        }
-
-        let error_response = ErrorResponse::new(&self);
-
-        (status_code, Json(error_response)).into_response()
-    }
-}
-
-// Implement conversions for common error types
-
-impl From<surrealdb::Error> for ControlCenterError {
-    fn from(error: surrealdb::Error) -> Self {
-        Self::Database(error.to_string())
-    }
-}
-
-impl From<jsonwebtoken::errors::Error> for ControlCenterError {
-    fn from(error: jsonwebtoken::errors::Error) -> Self {
-        Self::Token(error.to_string())
-    }
-}
-
-impl From<validator::ValidationErrors> for ControlCenterError {
-    fn from(errors: validator::ValidationErrors) -> Self {
-        let details = errors
-            .field_errors()
-            .iter()
-            .map(|(field, errors)| {
-                let messages: Vec<String> = errors
-                    .iter()
-                    .filter_map(|e| e.message.as_ref().map(|m| m.to_string()))
-                    .collect();
-                (field.to_string(), messages)
-            })
-            .collect::<std::collections::HashMap<_, _>>();
-
-        Self::Validation(format!("Validation failed: {:?}", details))
-    }
-}
-
-impl From<serde_json::Error> for ControlCenterError {
-    fn from(error: serde_json::Error) -> Self {
-        Self::Serialization(error.to_string())
-    }
-}
-
-impl From<std::io::Error> for ControlCenterError {
-    fn from(error: std::io::Error) -> Self {
-        Self::Internal(format!("IO error: {}", error))
-    }
-}
-
-impl From<toml::de::Error> for ControlCenterError {
-    fn from(error: toml::de::Error) -> Self {
-        Self::Configuration(format!("TOML parsing error: {}", error))
-    }
-}
-
-impl From<argon2::Error> for ControlCenterError {
-    fn from(error: argon2::Error) -> Self {
-        Self::Authentication(format!("Password hashing error: {}", error))
-    }
-}
-
-impl From<reqwest::Error> for ControlCenterError {
-    fn from(error: reqwest::Error) -> Self {
-        Self::ExternalService(error.to_string())
-    }
-}
-
-impl From<cedar_policy::ParseError> for ControlCenterError {
-    fn from(err: cedar_policy::ParseError) -> Self {
-        ControlCenterError::Cedar(err.to_string())
-    }
-}
-
-impl From<cedar_policy::PolicySetError> for ControlCenterError {
-    fn from(err: cedar_policy::PolicySetError) -> Self {
-        ControlCenterError::Cedar(err.to_string())
-    }
-}
-
-impl From<crate::kms::kms_service_client::KmsClientError> for ControlCenterError {
-    fn from(error: crate::kms::kms_service_client::KmsClientError) -> Self {
-        Self::ExternalService(format!("KMS error: {}", error))
-    }
-}
-
-impl From<crate::kms::error::KmsError> for ControlCenterError {
-    fn from(error: crate::kms::error::KmsError) -> Self {
-        Self::Internal(format!("KMS error: {}", error))
-    }
-}
-
-/// Helper macro for creating specific error types
-#[macro_export]
-macro_rules! control_center_error {
-    ($variant:ident, $msg:expr) => {
-        $crate::error::ControlCenterError::$variant($msg.to_string())
-    };
-    ($variant:ident, $fmt:expr, $($arg:tt)*) => {
-        $crate::error::ControlCenterError::$variant(format!($fmt, $($arg)*))
-    };
-}
-
-/// Helper functions for creating common errors
-pub fn user_not_found(identifier: &str) -> ControlCenterError {
-    ControlCenterError::UserNotFound(identifier.to_string())
-}
-
-pub fn session_not_found(session_id: &str) -> ControlCenterError {
-    ControlCenterError::SessionNotFound(session_id.to_string())
-}
-
-pub fn session_expired(session_id: &str) -> ControlCenterError {
-    ControlCenterError::SessionExpired(session_id.to_string())
-}
-
-pub fn unauthorized(message: &str) -> ControlCenterError {
-    ControlCenterError::Unauthorized(message.to_string())
-}
-
-pub fn forbidden(message: &str) -> ControlCenterError {
-    ControlCenterError::Forbidden(message.to_string())
-}
-
-pub fn bad_request(message: &str) -> ControlCenterError {
-    ControlCenterError::BadRequest(message.to_string())
-}
-
-pub fn conflict(message: &str) -> ControlCenterError {
-    ControlCenterError::Conflict(message.to_string())
-}
-
-pub fn internal_error(message: &str) -> ControlCenterError {
-    ControlCenterError::Internal(message.to_string())
-}
-
-pub fn not_found(message: &str) -> ControlCenterError {
-    ControlCenterError::NotFound(message.to_string())
-}

crates/control-center/src/handlers/auth.rs

@@ -4,6 +4,7 @@ use axum::{
     extract::{Request, State},
     response::Json,
 };
+use ontoref_derive::onto_api;
 use serde::{Deserialize, Serialize};
 use tracing::info;
 
@@ -14,6 +15,15 @@ use crate::services::AuthService;
 use crate::AppState;
 
 /// Login endpoint
+#[onto_api(
+    method = "POST",
+    path = "/auth/login",
+    description = "Authenticate and obtain JWT tokens",
+    auth = "none",
+    actors = "developer, agent",
+    tags = "auth",
+    feature = ""
+)]
 pub async fn login(
     State(app_state): State<Arc<AppState>>,
     Json(request): Json<LoginRequest>,
@@ -32,6 +42,15 @@ pub async fn login(
 }
 
 /// Refresh token endpoint
+#[onto_api(
+    method = "POST",
+    path = "/auth/refresh",
+    description = "Refresh JWT access token",
+    auth = "none",
+    actors = "developer, agent",
+    tags = "auth",
+    feature = ""
+)]
 pub async fn refresh_token(
     State(app_state): State<Arc<AppState>>,
     Json(request): Json<RefreshTokenRequest>,
@@ -42,6 +61,15 @@ pub async fn refresh_token(
 }
 
 /// Logout endpoint
+#[onto_api(
+    method = "POST",
+    path = "/auth/logout",
+    description = "Invalidate current session and logout",
+    auth = "bearer",
+    actors = "developer, agent",
+    tags = "auth",
+    feature = ""
+)]
 pub async fn logout(
     State(app_state): State<Arc<AppState>>,
     Json(logout_request): Json<LogoutRequest>,
@@ -126,6 +154,15 @@ pub async fn invalidate_all_sessions(
 }
 
 /// Health check endpoint (no auth required)
+#[onto_api(
+    method = "GET",
+    path = "/health",
+    description = "Service health check",
+    auth = "none",
+    actors = "developer, agent, ci",
+    tags = "health",
+    feature = ""
+)]
 pub async fn health_check() -> Json<ApiResponse<HealthCheckResponse>> {
     Json(ApiResponse::success(HealthCheckResponse {
         status: "healthy".to_string(),

crates/control-center/src/handlers/iac_deployment.rs

@@ -7,6 +7,7 @@ use axum::{
     http::StatusCode,
     Json,
 };
+use ontoref_derive::onto_api;
 use serde::{Deserialize, Serialize};
 
 // Re-export service types to avoid duplication
@@ -32,6 +33,15 @@ pub struct DeploymentQuery {
 }
 
 /// List deployment plans
+#[onto_api(
+    method = "GET",
+    path = "/deployments",
+    description = "List all deployment plans",
+    auth = "bearer",
+    actors = "developer, agent",
+    tags = "deployments",
+    feature = ""
+)]
 pub async fn list_deployments(
     State(state): State<Arc<AppState>>,
     Query(params): Query<DeploymentQuery>,
@@ -53,6 +63,15 @@ pub async fn list_deployments(
 }
 
 /// Get deployment plan
+#[onto_api(
+    method = "GET",
+    path = "/deployments/{id}",
+    description = "Get a deployment plan by ID",
+    auth = "bearer",
+    actors = "developer, agent",
+    tags = "deployments",
+    feature = ""
+)]
 pub async fn get_deployment(
     State(state): State<Arc<AppState>>,
     Path(id): Path<String>,
@@ -65,6 +84,15 @@ pub async fn get_deployment(
 }
 
 /// Create deployment plan
+#[onto_api(
+    method = "POST",
+    path = "/deployments",
+    description = "Create a new deployment plan",
+    auth = "bearer",
+    actors = "developer, agent",
+    tags = "deployments",
+    feature = ""
+)]
 pub async fn create_deployment(
     State(state): State<Arc<AppState>>,
     Json(plan): Json<DeploymentPlan>,
@@ -74,6 +102,15 @@ pub async fn create_deployment(
 }
 
 /// Update deployment plan
+#[onto_api(
+    method = "PUT",
+    path = "/deployments/{id}",
+    description = "Update a deployment plan",
+    auth = "bearer",
+    actors = "developer, agent",
+    tags = "deployments",
+    feature = ""
+)]
 pub async fn update_deployment(
     State(state): State<Arc<AppState>>,
     Path(id): Path<String>,
@@ -95,6 +132,15 @@ pub struct SubmissionResult {
 }
 
 /// Submit deployment plan to orchestrator
+#[onto_api(
+    method = "POST",
+    path = "/deployments/{id}/submit",
+    description = "Submit a deployment plan to the orchestrator",
+    auth = "bearer",
+    actors = "developer, agent",
+    tags = "deployments",
+    feature = ""
+)]
 pub async fn submit_deployment(
     State(state): State<Arc<AppState>>,
     Path(id): Path<String>,
@@ -171,6 +217,15 @@ pub struct DeploymentStatus {
 }
 
 /// Get deployment execution status
+#[onto_api(
+    method = "GET",
+    path = "/deployments/{id}/status",
+    description = "Get deployment execution status",
+    auth = "bearer",
+    actors = "developer, agent",
+    tags = "deployments",
+    feature = ""
+)]
 pub async fn get_deployment_status(
     State(state): State<Arc<AppState>>,
     Path(id): Path<String>,

crates/control-center/src/handlers/iac_detection.rs

@@ -10,6 +10,7 @@ use axum::{
     http::StatusCode,
     Json,
 };
+use ontoref_derive::onto_api;
 use serde::{Deserialize, Serialize};
 
 // Re-export service types
@@ -45,6 +46,15 @@ pub struct PagedResponse<T> {
 }
 
 /// List all detection results
+#[onto_api(
+    method = "GET",
+    path = "/detections",
+    description = "List all infrastructure detections",
+    auth = "bearer",
+    actors = "developer, agent",
+    tags = "detections",
+    feature = ""
+)]
 pub async fn list_detections(
     State(state): State<Arc<AppState>>,
     Query(params): Query<DetectionQuery>,
@@ -79,6 +89,15 @@ pub async fn list_detections(
 }
 
 /// Get single detection by ID
+#[onto_api(
+    method = "GET",
+    path = "/detections/{id}",
+    description = "Get infrastructure detection by ID",
+    auth = "bearer",
+    actors = "developer, agent",
+    tags = "detections",
+    feature = ""
+)]
 pub async fn get_detection(
     State(state): State<Arc<AppState>>,
     Path(id): Path<String>,
@@ -104,6 +123,15 @@ pub struct AnalyzeProjectRequest {
     pub organization: Option<String>,
 }
 
+#[onto_api(
+    method = "POST",
+    path = "/detections/analyze",
+    description = "Analyze a project for infrastructure technologies",
+    auth = "bearer",
+    actors = "developer, agent",
+    tags = "detections",
+    feature = ""
+)]
 pub async fn analyze_project(
     State(state): State<Arc<AppState>>,
     Json(req): Json<AnalyzeProjectRequest>,

crates/control-center/src/handlers/iac_rules.rs

@@ -7,6 +7,7 @@ use axum::{
     http::StatusCode,
     Json,
 };
+use ontoref_derive::onto_api;
 use serde::{Deserialize, Serialize};
 
 // Re-export service types to avoid duplication
@@ -32,6 +33,15 @@ pub struct RulesQuery {
 }
 
 /// List all inference rules
+#[onto_api(
+    method = "GET",
+    path = "/rules",
+    description = "List all inference rules",
+    auth = "bearer",
+    actors = "developer",
+    tags = "rules",
+    feature = ""
+)]
 pub async fn list_rules(
     State(state): State<Arc<AppState>>,
     Query(params): Query<RulesQuery>,
@@ -53,6 +63,15 @@ pub async fn list_rules(
 }
 
 /// List rules for specific organization
+#[onto_api(
+    method = "GET",
+    path = "/rules/org/{org}",
+    description = "List inference rules for a specific organization",
+    auth = "bearer",
+    actors = "developer",
+    tags = "rules",
+    feature = ""
+)]
 pub async fn list_org_rules(
     State(state): State<Arc<AppState>>,
     Path(org): Path<String>,
@@ -75,6 +94,15 @@ pub async fn list_org_rules(
 }
 
 /// Get single rule
+#[onto_api(
+    method = "GET",
+    path = "/rules/{id}",
+    description = "Get an inference rule by ID",
+    auth = "bearer",
+    actors = "developer",
+    tags = "rules",
+    feature = ""
+)]
 pub async fn get_rule(
     State(state): State<Arc<AppState>>,
     Path(id): Path<String>,
@@ -83,6 +111,15 @@ pub async fn get_rule(
 }
 
 /// Create new rule
+#[onto_api(
+    method = "POST",
+    path = "/rules",
+    description = "Create a new inference rule",
+    auth = "bearer",
+    actors = "developer",
+    tags = "rules",
+    feature = ""
+)]
 pub async fn create_rule(
     State(state): State<Arc<AppState>>,
     Json(rule): Json<InferenceRule>,
@@ -92,6 +129,15 @@ pub async fn create_rule(
 }
 
 /// Update existing rule
+#[onto_api(
+    method = "PUT",
+    path = "/rules/{id}",
+    description = "Update an inference rule",
+    auth = "bearer",
+    actors = "developer",
+    tags = "rules",
+    feature = ""
+)]
 pub async fn update_rule(
     State(state): State<Arc<AppState>>,
     Path(id): Path<String>,
@@ -102,6 +148,15 @@ pub async fn update_rule(
 }
 
 /// Delete rule
+#[onto_api(
+    method = "DELETE",
+    path = "/rules/{id}",
+    description = "Delete an inference rule",
+    auth = "bearer",
+    actors = "developer",
+    tags = "rules",
+    feature = ""
+)]
 pub async fn delete_rule(
     State(state): State<Arc<AppState>>,
     Path(id): Path<String>,
@@ -124,6 +179,15 @@ pub struct TestRuleResult {
     pub confidence_score: f32,
 }
 
+#[onto_api(
+    method = "POST",
+    path = "/rules/{id}/test",
+    description = "Test an inference rule against a project",
+    auth = "bearer",
+    actors = "developer",
+    tags = "rules",
+    feature = ""
+)]
 pub async fn test_rule(
     State(state): State<Arc<AppState>>,
     Path(id): Path<String>,

crates/control-center/src/handlers/permission.rs

@@ -4,6 +4,7 @@ use axum::{
     extract::{Query, Request, State},
     response::Json,
 };
+use ontoref_derive::onto_api;
 use serde::Deserialize;
 
 use crate::error::{auth, http, ControlCenterError, Result};
@@ -13,6 +14,15 @@ use crate::models::PermissionResponse;
 use crate::AppState;
 
 /// List permissions
+#[onto_api(
+    method = "GET",
+    path = "/permissions",
+    description = "List all permissions",
+    auth = "bearer",
+    actors = "developer",
+    tags = "permissions",
+    feature = ""
+)]
 pub async fn list_permissions(
     State(app_state): State<Arc<AppState>>,
     Query(params): Query<ListPermissionsParams>,

crates/control-center/src/handlers/secrets.rs

@@ -6,6 +6,7 @@ use axum::{
     response::IntoResponse,
     Json,
 };
+use ontoref_derive::onto_api;
 use serde::{Deserialize, Serialize};
 use tracing::{error, info};
 
@@ -181,6 +182,15 @@ fn default_limit() -> usize {
 }
 
 /// Handler: Create a new secret
+#[onto_api(
+    method = "POST",
+    path = "/secrets",
+    description = "Create a new secret",
+    auth = "admin",
+    actors = "developer",
+    tags = "secrets",
+    feature = ""
+)]
 pub async fn create_secret(
     State(app_state): State<Arc<AppState>>,
     Extension(security_ctx): Extension<SecurityContext>,
@@ -269,6 +279,15 @@ pub async fn create_secret(
 }
 
 /// Handler: Get a secret value (decrypted)
+#[onto_api(
+    method = "GET",
+    path = "/secrets/{path}",
+    description = "Get a secret value by path",
+    auth = "admin",
+    actors = "developer",
+    tags = "secrets",
+    feature = ""
+)]
 pub async fn get_secret(
     State(app_state): State<Arc<AppState>>,
     Extension(security_ctx): Extension<SecurityContext>,
@@ -356,6 +375,15 @@ pub struct GetSecretQuery {
 }
 
 /// Handler: List secrets
+#[onto_api(
+    method = "GET",
+    path = "/secrets",
+    description = "List all secrets",
+    auth = "admin",
+    actors = "developer",
+    tags = "secrets",
+    feature = ""
+)]
 pub async fn list_secrets(
     State(app_state): State<Arc<AppState>>,
     Extension(security_ctx): Extension<SecurityContext>,
@@ -392,6 +420,15 @@ pub async fn list_secrets(
 }
 
 /// Handler: Update a secret (creates new version)
+#[onto_api(
+    method = "PUT",
+    path = "/secrets/{path}",
+    description = "Update a secret, creating a new version",
+    auth = "admin",
+    actors = "developer",
+    tags = "secrets",
+    feature = ""
+)]
 pub async fn update_secret(
     State(app_state): State<Arc<AppState>>,
     Extension(security_ctx): Extension<SecurityContext>,
@@ -497,6 +534,15 @@ pub async fn update_secret(
 }
 
 /// Handler: Delete a secret
+#[onto_api(
+    method = "DELETE",
+    path = "/secrets/{path}",
+    description = "Delete a secret",
+    auth = "admin",
+    actors = "developer",
+    tags = "secrets",
+    feature = ""
+)]
 pub async fn delete_secret(
     State(app_state): State<Arc<AppState>>,
     Extension(security_ctx): Extension<SecurityContext>,
@@ -579,6 +625,15 @@ pub async fn delete_secret(
 }
 
 /// Handler: Get secret history/versions
+#[onto_api(
+    method = "GET",
+    path = "/secrets/{path}/history",
+    description = "Get version history for a secret",
+    auth = "admin",
+    actors = "developer",
+    tags = "secrets",
+    feature = ""
+)]
 pub async fn get_secret_history(
     State(app_state): State<Arc<AppState>>,
     Extension(security_ctx): Extension<SecurityContext>,
@@ -606,6 +661,15 @@ pub async fn get_secret_history(
 }
 
 /// Handler: Restore a specific version of a secret
+#[onto_api(
+    method = "POST",
+    path = "/secrets/{path}/restore/{version}",
+    description = "Restore a secret to a specific version",
+    auth = "admin",
+    actors = "developer",
+    tags = "secrets",
+    feature = ""
+)]
 pub async fn restore_secret_version(
     State(app_state): State<Arc<AppState>>,
     Extension(security_ctx): Extension<SecurityContext>,
@@ -741,6 +805,15 @@ pub struct RotationStatusResponse {
 }
 
 /// Handler: Force rotate a secret
+#[onto_api(
+    method = "POST",
+    path = "/secrets/{path}/rotate",
+    description = "Force rotate a secret",
+    auth = "admin",
+    actors = "developer",
+    tags = "secrets",
+    feature = ""
+)]
 pub async fn force_rotate_secret(
     State(app_state): State<Arc<AppState>>,
     Extension(security_ctx): Extension<SecurityContext>,
@@ -845,6 +918,15 @@ pub async fn force_rotate_secret(
 }
 
 /// Handler: Get rotation status for a secret
+#[onto_api(
+    method = "GET",
+    path = "/secrets/{path}/rotation-status",
+    description = "Get rotation status for a secret",
+    auth = "admin",
+    actors = "developer",
+    tags = "secrets",
+    feature = ""
+)]
 pub async fn get_rotation_status(
     State(app_state): State<Arc<AppState>>,
     Extension(security_ctx): Extension<SecurityContext>,
@@ -922,6 +1004,15 @@ pub struct GrantResponse {
 }
 
 /// Handler: Create a secret grant (sharing)
+#[onto_api(
+    method = "POST",
+    path = "/secrets/{path}/grant",
+    description = "Create a grant to share a secret",
+    auth = "admin",
+    actors = "developer",
+    tags = "secrets",
+    feature = ""
+)]
 pub async fn create_grant(
     State(app_state): State<Arc<AppState>>,
     Extension(security_ctx): Extension<SecurityContext>,
@@ -998,6 +1089,15 @@ pub struct RevokeGrantRequest {
 }
 
 /// Handler: Revoke a secret grant
+#[onto_api(
+    method = "POST",
+    path = "/secrets/grant/{grant_id}/revoke",
+    description = "Revoke a secret grant",
+    auth = "admin",
+    actors = "developer",
+    tags = "secrets",
+    feature = ""
+)]
 pub async fn revoke_grant(
     State(app_state): State<Arc<AppState>>,
     Extension(security_ctx): Extension<SecurityContext>,
@@ -1035,6 +1135,15 @@ pub async fn revoke_grant(
 // ============== PHASE 3.4: MONITORING HANDLERS ==============
 
 /// Handler: Get dashboard metrics
+#[onto_api(
+    method = "GET",
+    path = "/secrets/monitoring/dashboard",
+    description = "Get secrets monitoring dashboard metrics",
+    auth = "admin",
+    actors = "developer",
+    tags = "secrets",
+    feature = ""
+)]
 pub async fn get_dashboard_metrics(
     State(app_state): State<Arc<AppState>>,
     Extension(_security_ctx): Extension<SecurityContext>,
@@ -1055,6 +1164,15 @@ pub async fn get_dashboard_metrics(
 }
 
 /// Handler: Get alert summary
+#[onto_api(
+    method = "GET",
+    path = "/secrets/monitoring/alerts",
+    description = "Get secrets monitoring alert summary",
+    auth = "admin",
+    actors = "developer",
+    tags = "secrets",
+    feature = ""
+)]
 pub async fn get_alert_summary(
     State(app_state): State<Arc<AppState>>,
     Extension(_security_ctx): Extension<SecurityContext>,
@@ -1072,6 +1190,15 @@ pub async fn get_alert_summary(
 }
 
 /// Handler: Get expiring secrets
+#[onto_api(
+    method = "GET",
+    path = "/secrets/monitoring/expiring",
+    description = "Get list of expiring secrets",
+    auth = "admin",
+    actors = "developer",
+    tags = "secrets",
+    feature = ""
+)]
 pub async fn get_expiring_secrets(
     State(app_state): State<Arc<AppState>>,
     Extension(_security_ctx): Extension<SecurityContext>,

crates/control-center/src/lib.rs

@@ -23,6 +23,7 @@ pub mod error;
 pub mod handlers;
 pub mod middleware;
 pub mod models;
+pub mod ncl_config;
 pub mod services;
 pub mod simple_config;
 pub mod storage;

crates/control-center/src/main.rs

@@ -11,6 +11,8 @@ use axum::{
     routing::{get, post},
     Router,
 };
+
+mod api_catalog;
 use clap::Parser;
 use control_center::handlers::{
     auth::*,
@@ -94,12 +96,22 @@ struct Cli {
     /// Generate default configuration file
     #[arg(long)]
     generate_config: bool,
+
+    /// Print all #[onto_api] registered routes as JSON and exit.
+    /// Pipe to api-catalog-control-center.json: `just export-api-catalog`
+    #[arg(long)]
+    dump_api_catalog: bool,
 }
 
 #[tokio::main]
 async fn main() -> Result<()> {
     let cli = Cli::parse();
 
+    if cli.dump_api_catalog {
+        println!("{}", ontoref_ontology::api::dump_catalog_json());
+        return Ok(());
+    }
+
     // Generate default config if requested
     if cli.generate_config {
         let config_path = cli.config.unwrap_or_else(|| PathBuf::from("config.toml"));
@@ -114,7 +126,7 @@ async fn main() -> Result<()> {
 
     // Check if control-center is enabled in deployment-mode.ncl
     if let Ok(deployment) = platform_config::load_deployment_mode() {
-        if let Ok(enabled) = deployment.is_service_enabled("control-center") {
+        if let Ok(enabled) = deployment.is_service_enabled("control_center") {
             if !enabled {
                 warn!("⚠ Control Center is DISABLED in deployment-mode.ncl");
                 std::process::exit(1);
@@ -123,25 +135,28 @@ async fn main() -> Result<()> {
         }
     }
 
-    // Try to load control-center.ncl
+    // Load configuration from NCL using the same pattern as orchestrator and vault-service:
-    if let Ok(config) = platform_config::load_service_config_from_ncl("control-center") {
+    // ControlCenterNclConfig implements ConfigLoader which reads the NCL via
-        info!("✓ Loaded control-center configuration from NCL");
+    // PROVISIONING_CONFIG_DIR, exports to JSON, and deserializes into typed structs.
-        tracing::debug!("Config: {:?}", config);
+    use control_center::ncl_config::ControlCenterNclConfig;
+    use platform_config::format::ConfigLoader as _;
+    let ncl = ControlCenterNclConfig::load().map_err(|e| {
+        control_center::ControlCenterError::Infrastructure(
+            control_center::error::infrastructure::InfrastructureError::Configuration(format!(
+                "Failed to load control-center NCL config: {}",
+                e
+            )),
+        )
+    })?;
+    let mut config = ncl.to_runtime_config()?;
+
+    // Apply explicit CLI overrides on top of NCL values
+    if let Some(path) = &cli.config {
+        // If a TOML config file was explicitly given, merge it on top of NCL
+        let toml_config = Config::load_from_file(path)?;
+        config = toml_config;
     }
 
-    // Resolve config file path using new resolver
-    let resolver = platform_config::ConfigResolver::new()
-        .with_cli_config(cli.config.clone())
-        .with_cli_config_dir(cli.config_dir.clone())
-        .with_cli_mode(cli.mode.clone());
-
-    // Load configuration
-    let mut config = if let Some(path) = resolver.resolve("control-center") {
-        Config::load_from_file(path)?
-    } else {
-        Config::load()?
-    };
-
     // Apply CLI overrides
     if let Some(port) = cli.port {
         config.server.port = port;
@@ -207,6 +222,7 @@ async fn create_router(app_state: Arc<AppState>) -> Result<Router> {
         .route("/health", get(health_check))
         .route("/auth/login", post(login))
         .route("/auth/refresh", post(refresh_token))
+        .route("/api/catalog", get(api_catalog::api_catalog))
         .layer(auth_rate_limit);
 
     // Protected routes (authentication required)
@@ -239,44 +255,44 @@ async fn create_router(app_state: Arc<AppState>) -> Result<Router> {
         // .route("/permissions/actions", get(get_actions))
         // Detection routes (Infrastructure-from-Code)
         .route("/detections", get(list_detections))
-        .route("/detections/:id", get(get_detection))
+        .route("/detections/{id}", get(get_detection))
         .route("/detections/analyze", post(analyze_project))
         // Rules routes (Inference rules)
         .route("/rules", get(list_rules).post(create_rule))
-        .route("/rules/org/:org", get(list_org_rules))
+        .route("/rules/org/{org}", get(list_org_rules))
         .route(
-            "/rules/:id",
+            "/rules/{id}",
             get(get_rule).put(update_rule).delete(delete_rule),
         )
-        .route("/rules/:id/test", post(test_rule))
+        .route("/rules/{id}/test", post(test_rule))
         // Deployment routes
         .route(
             "/deployments",
             get(list_deployments).post(create_deployment),
         )
         .route(
-            "/deployments/:id",
+            "/deployments/{id}",
             get(get_deployment).put(update_deployment),
         )
-        .route("/deployments/:id/submit", post(submit_deployment))
+        .route("/deployments/{id}/submit", post(submit_deployment))
-        .route("/deployments/:id/status", get(get_deployment_status))
+        .route("/deployments/{id}/status", get(get_deployment_status))
         // Secrets routes (Phase 1.5 - Now active with SecretsService state initialization)
         .route("/secrets", post(create_secret).get(list_secrets))
         .route(
-            "/secrets/:path",
+            "/secrets/{path}",
             get(get_secret).put(update_secret).delete(delete_secret),
         )
-        .route("/secrets/:path/history", get(get_secret_history))
+        .route("/secrets/{path}/history", get(get_secret_history))
         .route(
-            "/secrets/:path/restore/:version",
+            "/secrets/{path}/restore/{version}",
             post(restore_secret_version),
         )
         // Secrets Phase 3.1: Rotation routes
-        .route("/secrets/:path/rotate", post(force_rotate_secret))
+        .route("/secrets/{path}/rotate", post(force_rotate_secret))
-        .route("/secrets/:path/rotation-status", get(get_rotation_status))
+        .route("/secrets/{path}/rotation-status", get(get_rotation_status))
         // Secrets Phase 3.2: Sharing routes
-        .route("/secrets/:path/grant", post(create_grant))
+        .route("/secrets/{path}/grant", post(create_grant))
-        .route("/secrets/grant/:grant_id/revoke", post(revoke_grant))
+        .route("/secrets/grant/{grant_id}/revoke", post(revoke_grant))
         // Secrets Phase 3.4: Monitoring routes
         .route("/secrets/monitoring/dashboard", get(get_dashboard_metrics))
         .route("/secrets/monitoring/alerts", get(get_alert_summary))

crates/control-center/src/models/permission.rs

@@ -1,13 +1,14 @@
 use chrono::{DateTime, Utc};
 use serde::{Deserialize, Serialize};
-use surrealdb::sql::Thing;
+use surrealdb::types::{RecordId, SurrealValue};
 use uuid::Uuid;
 use validator::Validate;
 
 /// Permission model for fine-grained access control
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct Permission {
-    pub id: Option<Thing>,
+    pub id: Option<RecordId>,
     pub permission_id: Uuid,
     pub name: String,
     pub resource: String,

crates/control-center/src/models/role.rs

@@ -1,13 +1,14 @@
 use chrono::{DateTime, Utc};
 use serde::{Deserialize, Serialize};
-use surrealdb::sql::Thing;
+use surrealdb::types::{RecordId, SurrealValue};
 use uuid::Uuid;
 use validator::Validate;
 
 /// Role model for permission-based access control
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct Role {
-    pub id: Option<Thing>,
+    pub id: Option<RecordId>,
     pub role_id: Uuid,
     pub name: String,
     pub description: Option<String>,

crates/control-center/src/models/session.rs

@@ -1,12 +1,13 @@
 use chrono::{DateTime, Utc};
 use serde::{Deserialize, Serialize};
-use surrealdb::sql::Thing;
+use surrealdb::types::{RecordId, SurrealValue};
 use uuid::Uuid;
 
 /// Session model for managing user sessions and refresh tokens
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct Session {
-    pub id: Option<Thing>,
+    pub id: Option<RecordId>,
     pub session_id: Uuid,
     pub user_id: Uuid,
     pub refresh_token: String,
@@ -18,7 +19,8 @@ pub struct Session {
 }
 
 /// Client information for session tracking
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct ClientInfo {
     pub user_agent: Option<String>,
     pub ip_address: Option<String>,

crates/control-center/src/models/user.rs

@@ -1,13 +1,14 @@
 use chrono::{DateTime, Utc};
 use serde::{Deserialize, Serialize};
-use surrealdb::sql::Thing;
+use surrealdb::types::{RecordId, SurrealValue};
 use uuid::Uuid;
 use validator::Validate;
 
 /// User model for SurrealDB storage
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct User {
-    pub id: Option<Thing>,
+    pub id: Option<RecordId>,
     pub user_id: Uuid,
     pub email: String,
     pub username: String,

crates/control-center/src/ncl_config.rs

@@ -0,0 +1,404 @@
+use platform_config::format::ConfigLoader;
+use serde::{Deserialize, Serialize};
+
+use crate::error::{infrastructure, ControlCenterError, Result};
+use crate::simple_config::{
+    Config, CorsConfig, DatabaseConfig, JwtConfig, LoggingConfig, RateLimitConfig, SecurityConfig,
+    ServerConfig,
+};
+
+/// Root wrapper matching the NCL export key `control_center`.
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub struct ControlCenterNclConfig {
+    #[serde(default)]
+    pub control_center: NclSettings,
+}
+
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub struct NclSettings {
+    #[serde(default)]
+    pub server: NclServer,
+    #[serde(default)]
+    pub database: NclDatabase,
+    #[serde(default)]
+    pub security: NclSecurity,
+    #[serde(default)]
+    pub logging: NclLogging,
+    #[serde(default)]
+    pub users: NclUsers,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct NclServer {
+    #[serde(default = "default_host")]
+    pub host: String,
+    #[serde(default = "default_port")]
+    pub port: u16,
+    pub workers: Option<usize>,
+    pub keep_alive: Option<u64>,
+    pub max_connections: Option<usize>,
+}
+
+impl Default for NclServer {
+    fn default() -> Self {
+        Self {
+            host: default_host(),
+            port: default_port(),
+            workers: Some(2),
+            keep_alive: Some(75),
+            max_connections: Some(100),
+        }
+    }
+}
+
+fn default_host() -> String {
+    "0.0.0.0".to_string()
+}
+fn default_port() -> u16 {
+    9012
+}
+
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub struct NclDatabase {
+    /// Internal storage backend of the SurrealDB server (e.g. "rocksdb").
+    /// Does NOT determine how the Rust client connects — always WebSocket for Docker.
+    #[serde(default)]
+    pub backend: Option<String>,
+    #[serde(default)]
+    pub mode: Option<String>,
+    /// RocksDB data path (server-side configuration, not used by Rust client)
+    #[serde(default)]
+    pub path: Option<String>,
+    /// WebSocket connection string for remote SurrealDB (Docker).
+    /// Falls back to "127.0.0.1:8000" if unset.
+    #[serde(default)]
+    pub connection_string: Option<String>,
+    #[serde(default)]
+    pub namespace: Option<String>,
+    /// SurrealDB authentication credentials (required when server runs with --user/--pass)
+    #[serde(default)]
+    pub credentials: Option<NclDbCredentials>,
+}
+
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub struct NclDbCredentials {
+    pub username: Option<String>,
+    pub password: Option<String>,
+}
+
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub struct NclSecurity {
+    #[serde(default)]
+    pub jwt: NclJwt,
+    #[serde(default)]
+    pub rate_limiting: NclRateLimiting,
+    #[serde(default)]
+    pub cors: NclCors,
+    #[serde(default)]
+    pub session: NclSession,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct NclJwt {
+    #[serde(default = "default_issuer")]
+    pub issuer: String,
+    #[serde(default = "default_audience")]
+    pub audience: String,
+    /// Access token lifetime in seconds
+    #[serde(default = "default_expiration")]
+    pub expiration: i64,
+    /// Refresh token lifetime in seconds
+    #[serde(default = "default_refresh_expiration")]
+    pub refresh_expiration: i64,
+}
+
+impl Default for NclJwt {
+    fn default() -> Self {
+        Self {
+            issuer: default_issuer(),
+            audience: default_audience(),
+            expiration: default_expiration(),
+            refresh_expiration: default_refresh_expiration(),
+        }
+    }
+}
+
+fn default_issuer() -> String {
+    "control-center".to_string()
+}
+fn default_audience() -> String {
+    "provisioning".to_string()
+}
+fn default_expiration() -> i64 {
+    3600
+}
+fn default_refresh_expiration() -> i64 {
+    86400
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct NclRateLimiting {
+    #[serde(default)]
+    pub enabled: bool,
+    /// NCL schema types this field as String ("1000"), so we use Value to handle both.
+    #[serde(default = "default_max_requests_json")]
+    pub max_requests: serde_json::Value,
+    #[serde(default = "default_window_secs")]
+    pub window_seconds: u64,
+}
+
+impl Default for NclRateLimiting {
+    fn default() -> Self {
+        Self {
+            enabled: false,
+            max_requests: serde_json::Value::Number(100.into()),
+            window_seconds: 60,
+        }
+    }
+}
+
+impl NclRateLimiting {
+    fn max_requests_u32(&self) -> u32 {
+        match &self.max_requests {
+            serde_json::Value::Number(n) => n.as_u64().unwrap_or(100) as u32,
+            serde_json::Value::String(s) => s.parse().unwrap_or(100),
+            _ => 100,
+        }
+    }
+}
+
+fn default_max_requests_json() -> serde_json::Value {
+    serde_json::Value::Number(100.into())
+}
+fn default_window_secs() -> u64 {
+    60
+}
+
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub struct NclCors {
+    #[serde(default)]
+    pub enabled: bool,
+    #[serde(default)]
+    pub allow_credentials: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct NclSession {
+    /// Idle timeout in seconds — drives session_cleanup_interval_minutes
+    #[serde(default = "default_idle_timeout")]
+    pub idle_timeout: u64,
+}
+
+impl Default for NclSession {
+    fn default() -> Self {
+        Self {
+            idle_timeout: default_idle_timeout(),
+        }
+    }
+}
+
+fn default_idle_timeout() -> u64 {
+    3600
+}
+
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub struct NclUsers {
+    #[serde(default)]
+    pub sessions: NclUserSessions,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct NclUserSessions {
+    pub max_active: Option<usize>,
+    #[serde(default = "default_idle_timeout")]
+    pub idle_timeout: u64,
+}
+
+impl Default for NclUserSessions {
+    fn default() -> Self {
+        Self {
+            max_active: Some(5),
+            idle_timeout: 3600,
+        }
+    }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct NclLogging {
+    #[serde(default = "default_log_level")]
+    pub level: String,
+    #[serde(default = "default_log_format")]
+    pub format: String,
+}
+
+impl Default for NclLogging {
+    fn default() -> Self {
+        Self {
+            level: default_log_level(),
+            format: default_log_format(),
+        }
+    }
+}
+
+fn default_log_level() -> String {
+    "info".to_string()
+}
+fn default_log_format() -> String {
+    "json".to_string()
+}
+
+impl ConfigLoader for ControlCenterNclConfig {
+    fn service_name() -> &'static str {
+        "control-center"
+    }
+
+    fn load_from_hierarchy(
+    ) -> std::result::Result<Self, Box<dyn std::error::Error + Send + Sync>> {
+        let service = Self::service_name();
+        if let Some(path) = platform_config::resolve_config_path(service) {
+            return Self::from_path(&path);
+        }
+        Ok(Self::default())
+    }
+
+    fn from_path<P: AsRef<std::path::Path>>(
+        path: P,
+    ) -> std::result::Result<Self, Box<dyn std::error::Error + Send + Sync>> {
+        let path = path.as_ref();
+        let json_value = platform_config::format::load_config(path).map_err(|e| {
+            Box::new(e) as Box<dyn std::error::Error + Send + Sync>
+        })?;
+        serde_json::from_value(json_value).map_err(|e| {
+            Box::new(std::io::Error::new(
+                std::io::ErrorKind::InvalidData,
+                format!(
+                    "Failed to deserialize control-center NCL config from {:?}: {}",
+                    path, e
+                ),
+            )) as Box<dyn std::error::Error + Send + Sync>
+        })
+    }
+}
+
+impl ControlCenterNclConfig {
+    /// Convert the NCL platform config into the runtime `Config` used by services.
+    ///
+    /// JWT RSA keys are generated fresh on each startup (NCL uses HS256 which
+    /// is incompatible with the service's RS256 implementation). For solo mode,
+    /// sessions do not persist across restarts, so ephemeral keys are acceptable.
+    pub fn to_runtime_config(&self) -> Result<Config> {
+        let cc = &self.control_center;
+
+        let key_pair =
+            crate::services::jwt::generate_rsa_key_pair().map_err(|e| {
+                ControlCenterError::Infrastructure(
+                    infrastructure::InfrastructureError::Configuration(format!(
+                        "Failed to generate RSA key pair: {}",
+                        e
+                    )),
+                )
+            })?;
+
+        let db_url = cc
+            .database
+            .connection_string
+            .as_deref()
+            .unwrap_or("127.0.0.1:8000")
+            .to_string();
+
+        let namespace = cc
+            .database
+            .namespace
+            .as_deref()
+            .unwrap_or("control_center")
+            .to_string();
+
+        // Credentials: NCL → env vars SURREALDB_USER / SURREALDB_PASS → None
+        let db_username = cc
+            .database
+            .credentials
+            .as_ref()
+            .and_then(|c| c.username.clone())
+            .or_else(|| std::env::var("SURREALDB_USER").ok());
+        let db_password = cc
+            .database
+            .credentials
+            .as_ref()
+            .and_then(|c| c.password.clone())
+            .or_else(|| std::env::var("SURREALDB_PASS").ok());
+
+        Ok(Config {
+            server: ServerConfig {
+                host: cc.server.host.clone(),
+                port: cc.server.port,
+                workers: cc.server.workers,
+                keep_alive: cc.server.keep_alive,
+                max_connections: cc.server.max_connections,
+            },
+            database: DatabaseConfig {
+                url: db_url,
+                namespace,
+                database: "main".to_string(),
+                username: db_username,
+                password: db_password,
+            },
+            jwt: JwtConfig {
+                issuer: cc.security.jwt.issuer.clone(),
+                audience: cc.security.jwt.audience.clone(),
+                access_token_expiration_hours: (cc.security.jwt.expiration / 3600).max(1),
+                refresh_token_expiration_hours: (cc.security.jwt.refresh_expiration / 3600).max(1),
+                private_key_pem: key_pair.private_key_pem,
+                public_key_pem: key_pair.public_key_pem,
+            },
+            rate_limiting: RateLimitConfig {
+                max_requests: cc.security.rate_limiting.max_requests_u32(),
+                window_seconds: cc.security.rate_limiting.window_seconds,
+                per_ip: true,
+                global: false,
+            },
+            cors: CorsConfig {
+                allowed_origins: vec!["*".to_string()],
+                allowed_methods: vec![
+                    "GET".to_string(),
+                    "POST".to_string(),
+                    "PUT".to_string(),
+                    "DELETE".to_string(),
+                    "PATCH".to_string(),
+                    "OPTIONS".to_string(),
+                ],
+                allowed_headers: vec![
+                    "content-type".to_string(),
+                    "authorization".to_string(),
+                    "accept".to_string(),
+                    "x-requested-with".to_string(),
+                    "x-session-id".to_string(),
+                ],
+                expose_headers: vec![
+                    "x-total-count".to_string(),
+                    "x-rate-limit-remaining".to_string(),
+                    "x-rate-limit-limit".to_string(),
+                    "x-rate-limit-reset".to_string(),
+                ],
+                max_age: 86400,
+                allow_credentials: cc.security.cors.allow_credentials,
+            },
+            security: SecurityConfig {
+                session_cleanup_interval_minutes: (cc.security.session.idle_timeout / 60).max(1),
+                max_sessions_per_user: cc.users.sessions.max_active,
+                password_min_length: 8,
+                password_require_special_chars: false,
+                password_require_numbers: false,
+                password_require_uppercase: false,
+                failed_login_lockout_attempts: Some(5),
+                failed_login_lockout_duration_minutes: Some(15),
+            },
+            logging: LoggingConfig {
+                level: cc.logging.level.clone(),
+                format: cc.logging.format.clone(),
+                file_path: None,
+                max_file_size: Some("100MB".to_string()),
+                max_files: Some(10),
+            },
+        })
+    }
+}

crates/control-center/src/services/database.rs

@@ -1,6 +1,7 @@
 use anyhow::Context;
-use surrealdb::engine::local::Mem;
+use surrealdb::engine::remote::ws::{Client, Ws};
 use surrealdb::opt::auth::Root;
+use surrealdb::types::SurrealValue;
 use surrealdb::Surreal;
 use tracing::{info, warn};
 
@@ -9,7 +10,7 @@ use crate::error::{auth, ControlCenterError, Result};
 /// Database service for SurrealDB operations
 #[derive(Clone)]
 pub struct DatabaseService {
-    pub db: Surreal<surrealdb::engine::local::Db>,
+    pub db: Surreal<Client>,
 }
 
 // Use the configuration from simple_config
@@ -18,15 +19,18 @@ use crate::simple_config::DatabaseConfig;
 impl DatabaseService {
     /// Create a new database service and connect
     pub async fn new(config: DatabaseConfig) -> Result<Self> {
-        info!("Connecting to SurrealDB (in-memory) at {}", config.url);
+        info!("Connecting to SurrealDB at ws://{}", config.url);
 
-        let db = Surreal::new::<Mem>(())
+        let db = Surreal::new::<Ws>(&*config.url)
             .await
             .context("Failed to connect to SurrealDB")?;
 
-        // Sign in with root credentials if provided
+        // Sign in only when credentials are explicitly configured
         if let (Some(username), Some(password)) = (&config.username, &config.password) {
-            db.signin(Root { username, password })
+            db.signin(Root {
+                username: username.clone(),
+                password: password.clone(),
+            })
                 .await
                 .context("Failed to sign in to SurrealDB")?;
         }
@@ -414,7 +418,8 @@ impl DatabaseService {
 }
 
 /// Database statistics
-#[derive(Debug, serde::Deserialize, serde::Serialize)]
+#[derive(Debug, serde::Deserialize, serde::Serialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct DatabaseStatistics {
     pub users_count: i64,
     pub roles_count: i64,

crates/control-center/src/services/iac_deployment.rs

@@ -6,13 +6,15 @@ use std::sync::Arc;
 
 use anyhow::Context;
 use serde::{Deserialize, Serialize};
+use surrealdb::types::SurrealValue;
 use uuid::Uuid;
 
 use super::DatabaseService;
 use crate::error::{http, ControlCenterError, Result};
 
 /// Deployment task status
-#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Default)]
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Default, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 #[serde(rename_all = "lowercase")]
 pub enum TaskStatus {
     #[default]
@@ -34,7 +36,8 @@ impl std::fmt::Display for TaskStatus {
 }
 
 /// Deployment task
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct DeploymentTask {
     pub id: String,
     pub task_type: String,
@@ -47,7 +50,8 @@ pub struct DeploymentTask {
 }
 
 /// Deployment plan
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct DeploymentPlan {
     pub id: String,
     pub name: String,
@@ -59,7 +63,8 @@ pub struct DeploymentPlan {
 }
 
 /// Deployment execution status
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct DeploymentExecution {
     pub id: String,
     pub plan_id: String,

crates/control-center/src/services/iac_detection.rs

@@ -4,6 +4,8 @@
 
 use std::sync::Arc;
 
+use surrealdb::types::SurrealValue;
+
 use anyhow::Context;
 use serde::{Deserialize, Serialize};
 use uuid::Uuid;
@@ -12,7 +14,8 @@ use super::DatabaseService;
 use crate::error::{http, ControlCenterError, Result};
 
 /// Detected technology information
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct DetectedTechnology {
     pub technology: String,
     pub confidence: f32,
@@ -21,7 +24,8 @@ pub struct DetectedTechnology {
 }
 
 /// Detection result with analysis metadata
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct DetectionResult {
     pub id: String,
     pub project_path: String,

crates/control-center/src/services/iac_rules.rs

@@ -4,6 +4,8 @@
 
 use std::sync::Arc;
 
+use surrealdb::types::SurrealValue;
+
 use anyhow::Context;
 use serde::{Deserialize, Serialize};
 use uuid::Uuid;
@@ -12,7 +14,8 @@ use super::DatabaseService;
 use crate::error::{http, ControlCenterError, Result};
 
 /// Single inference in a rule
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct RuleInference {
     pub technology: String,
     pub reason: String,
@@ -21,7 +24,8 @@ pub struct RuleInference {
 }
 
 /// Inference rule for infrastructure completion
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct InferenceRule {
     pub id: String,
     pub name: String,

crates/control-center/src/services/secrets.rs

@@ -3,6 +3,7 @@ use std::sync::Arc;
 use base64::Engine;
 use chrono::Utc;
 use serde::{Deserialize, Serialize};
+use surrealdb::types::SurrealValue;
 use tracing::{debug, error, info};
 
 use crate::audit::AuditLogger;
@@ -14,7 +15,8 @@ use crate::kms::kms_service_client::KmsServiceClient;
 use crate::storage::surrealdb_storage::SurrealDbStorage;
 
 /// Database connection information
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct DatabaseConnection {
     pub host: String,
     pub port: u16,
@@ -32,7 +34,8 @@ pub enum SecretLifecycle {
 }
 
 /// Secret type classification
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub enum SecretType {
     /// Database credentials
     Database {
@@ -58,7 +61,8 @@ pub enum SecretType {
 }
 
 /// Vault secret metadata stored in SurrealDB
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct VaultSecret {
     // Existing fields
     pub id: String,

crates/control-center/src/simple_config.rs

@@ -126,7 +126,7 @@ impl Default for ServerConfig {
     fn default() -> Self {
         Self {
             host: "0.0.0.0".to_string(),
-            port: 9080,
+            port: 9012,
             workers: None,
             keep_alive: Some(75),
             max_connections: Some(1000),

crates/control-center/src/storage/mod.rs

@@ -11,6 +11,7 @@ use std::sync::Arc;
 use async_trait::async_trait;
 pub use database::{Database, DatabaseConfig};
 use serde::{Deserialize, Serialize};
+use surrealdb::types::SurrealValue;
 // TODO: Re-enable when policies module is fixed
 // use crate::policies::{PolicyMetadata, PolicyVersion};
 // use crate::policies::versioning::RollbackResult;
@@ -145,7 +146,8 @@ impl Default for PolicySearchQuery {
 }
 
 /// Policy evaluation event for audit trail
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct PolicyEvaluationEvent {
     pub id: String,
     pub policy_id: String,
@@ -174,7 +176,8 @@ pub struct PolicyMetrics {
 }
 
 /// Compliance check result
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct ComplianceCheckResult {
     pub id: String,
     pub framework: String,

crates/control-center/src/storage/surrealdb_storage.rs

@@ -17,7 +17,8 @@ use crate::error::{auth, policy, ControlCenterError, Result};
 use crate::services::secrets::SecretType;
 use crate::simple_config::Config;
 
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct PolicyMetadata {
     pub id: String,
     pub name: String,
@@ -32,7 +33,8 @@ pub struct PolicyMetadata {
     pub enabled: bool,
 }
 
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 pub struct PolicyVersion {
     pub version_id: String,
     pub policy_id: String,
@@ -54,12 +56,14 @@ pub struct RollbackResult {
 use async_trait::async_trait;
 use surrealdb::engine::local::Mem;
 use surrealdb::engine::remote::ws::{Client, Ws};
-use surrealdb::{RecordId, Surreal};
+use surrealdb::types::{RecordId, SurrealValue};
+use surrealdb::Surreal;
 use tracing::{debug, info};
 use uuid::Uuid;
 
 /// SurrealDB record for policies
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 struct PolicyRecord {
     pub id: RecordId,
     pub policy_id: String,
@@ -70,21 +74,24 @@ struct PolicyRecord {
 }
 
 /// SurrealDB record for policy versions
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 struct PolicyVersionRecord {
     pub id: RecordId,
     pub version: PolicyVersion,
 }
 
 /// SurrealDB record for policy evaluations
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 struct PolicyEvaluationRecord {
     pub id: RecordId,
     pub evaluation: PolicyEvaluationEvent,
 }
 
 /// SurrealDB record for compliance checks
-#[derive(Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize, SurrealValue)]
+#[surreal(crate = "surrealdb::types")]
 struct ComplianceCheckRecord {
     pub id: RecordId,
     pub result: ComplianceCheckResult,
@@ -110,8 +117,11 @@ impl SurrealDbPolicyStorage<Client> {
         if let (Some(username), Some(password)) =
             (&config.database.username, &config.database.password)
         {
-            db.signin(surrealdb::opt::auth::Root { username, password })
+            db.signin(surrealdb::opt::auth::Root {
-                .await?;
+                username: username.clone(),
+                password: password.clone(),
+            })
+            .await?;
         }
 
         // Use namespace and database
@@ -285,7 +295,7 @@ where
 {
     /// Generate record ID for table
     fn generate_record_id(&self, table: &str) -> RecordId {
-        RecordId::from_table_key(table, Uuid::new_v4().to_string())
+        RecordId::new(table, Uuid::new_v4().to_string())
     }
 }
 
@@ -701,7 +711,7 @@ where
     pub async fn create_secret(&self, secret: &VaultSecret) -> Result<()> {
         let _: Option<VaultSecret> = self
             .db
-            .create(("vault_secrets", &secret.id))
+            .create(("vault_secrets", secret.id.as_str()))
             .content(secret.clone())
             .await?;
 
@@ -866,7 +876,7 @@ where
         // Update current version in vault_secrets
         let _: Option<VaultSecret> = self
             .db
-            .update(("vault_secrets", &secret.id))
+            .update(("vault_secrets", secret.id.as_str()))
             .content(secret.clone())
             .await?;
 

crates/daemon/Cargo.toml

@@ -1,48 +0,0 @@
-[package]
-authors.workspace = true
-edition.workspace = true
-license.workspace = true
-name = "daemon"
-repository.workspace = true
-version.workspace = true
-
-[[bin]]
-name = "provisioning-daemon"
-path = "src/main.rs"
-
-[dependencies]
-# Core daemon library from prov-ecosystem
-daemon-cli = { workspace = true }
-
-# Async runtime and networking
-axum = { workspace = true }
-tokio = { workspace = true }
-tower = { workspace = true }
-tower-http = { workspace = true }
-
-# Serialization
-serde = { workspace = true, features = ["derive"] }
-serde_json = { workspace = true }
-toml = { workspace = true }
-
-# Platform configuration
-platform-config = { workspace = true }
-
-# Centralized observability (logging, metrics, health, tracing)
-observability = { workspace = true, features = ["logging", "metrics-prometheus", "health"] }
-
-# Error handling
-anyhow = { workspace = true }
-thiserror = { workspace = true }
-
-# Logging
-tracing = { workspace = true }
-tracing-subscriber = { workspace = true }
-
-# CLI
-clap = { workspace = true, features = ["derive"] }
-
-# Utilities
-chrono = { workspace = true }
-dirs = { workspace = true }
-uuid = { workspace = true }

crates/data/tasks/00b7d705-2ac2-4532-9ce9-d161dd320011.json

@@ -0,0 +1,23 @@
+{
+  "id": "00b7d705-2ac2-4532-9ce9-d161dd320011",
+  "name": "component_install_democratic_csi",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_00b7d705-2ac2-4532-9ce9-d161dd320011.tar.gz.b64' > /tmp/democratic_csi.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/democratic_csi.tar.gz 'root@libre-wuji-cp-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-cp-0' 'rm -rf /tmp/democratic_csi && mkdir -p /tmp/democratic_csi && tar xzf /tmp/democratic_csi.tar.gz -C /tmp/democratic_csi && cd /tmp/democratic_csi && sudo bash install-democratic_csi.sh install ; rc=$?; rm -f /tmp/democratic_csi.tar.gz && rm -rf /tmp/democratic_csi; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_00b7d705-2ac2-4532-9ce9-d161dd320011.tar.gz.b64' /tmp/democratic_csi.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T03:15:03.134943Z",
+  "started_at": "2026-04-21T03:15:03.377192Z",
+  "completed_at": "2026-04-21T03:15:30.771379Z",
+  "output": "=== democratic-csi: v0.14.6 ===\n=== democratic-csi: deploying nfs-client driver (server=10.0.8.20) ===\nnamespace/democratic-csi unchanged\n\"democratic-csi\" has been added to your repositories\nHang tight while we grab the latest from your chart repositories...\n...Successfully got an update from the \"democratic-csi\" chart repository\nUpdate Complete. ⎈Happy Helming!⎈\nserviceaccount/democratic-csi-controller-sa created\nserviceaccount/democratic-csi-node-sa created\nsecret/democratic-csi-driver-config created\nconfigmap/democratic-csi created\nstorageclass.storage.k8s.io/nfs-shared created\nclusterrole.rbac.authorization.k8s.io/democratic-csi-controller-cr created\nclusterrole.rbac.authorization.k8s.io/democratic-csi-node-cr created\nclusterrolebinding.rbac.authorization.k8s.io/democratic-csi-controller-rb created\nclusterrolebinding.rbac.authorization.k8s.io/democratic-csi-node-rb created\ndaemonset.apps/democratic-csi-node created\ndeployment.apps/democratic-csi-controller created\ncsidriver.storage.k8s.io/org.democratic-csi.nfs-client created\n=== democratic-csi: waiting for controller ===\nWaiting for deployment \"democratic-csi-controller\" rollout to finish: 0 of 1 updated replicas are available...\ndeployment \"democratic-csi-controller\" successfully rolled out\n=== democratic-csi: ready (nfs=10.0.8.20/var/lib/data/shared, class=nfs-shared) ===\n",
+  "error": null,
+  "tags": {
+    "type": "component",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "component": "democratic_csi",
+    "operation": "install",
+    "server": "libre-wuji-cp-0"
+  }
+}

crates/data/tasks/059e4edd-7f6b-4766-85d8-379c9fc92694.json

@@ -0,0 +1,23 @@
+{
+  "id": "059e4edd-7f6b-4766-85d8-379c9fc92694",
+  "name": "component_install_os",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_059e4edd-7f6b-4766-85d8-379c9fc92694.tar.gz.b64' > /tmp/os.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/os.tar.gz 'root@libre-wuji-cp-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-cp-0' 'rm -rf /tmp/os && mkdir -p /tmp/os && tar xzf /tmp/os.tar.gz -C /tmp/os && cd /tmp/os && sudo bash install-os.sh install ; rc=$?; rm -f /tmp/os.tar.gz && rm -rf /tmp/os; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_059e4edd-7f6b-4766-85d8-379c9fc92694.tar.gz.b64' /tmp/os.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Failed",
+  "created_at": "2026-04-21T01:37:11.770013Z",
+  "started_at": "2026-04-21T01:37:11.879953Z",
+  "completed_at": "2026-04-21T01:37:12.257125Z",
+  "output": null,
+  "error": "Command execution failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\n@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\nIT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!\nSomeone could be eavesdropping on you right now (man-in-the-middle attack)!\nIt is also possible that a host key has just been changed.\nThe fingerprint for the ED25519 key sent by the remote host is\nSHA256:n47ih/YVibQVEAIYpId7Q5YYswXe0aBwPp/FrJrtESY.\nPlease contact your system administrator.\nAdd correct host key in /Users/jesusperezlorenzo/.ssh/known_hosts to get rid of this message.\nOffending ECDSA key in /Users/jesusperezlorenzo/.ssh/known_hosts:228\nHost key for libre-wuji-cp-0 has changed and you have requested strict checking.\nHost key verification failed.\nscp: Connection closed\n",
+  "tags": {
+    "type": "component",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "component": "os",
+    "server": "libre-wuji-cp-0",
+    "operation": "install"
+  }
+}

crates/data/tasks/0b72338e-1d92-4370-8ac3-f0c725940c82.json

@@ -0,0 +1,23 @@
+{
+  "id": "0b72338e-1d92-4370-8ac3-f0c725940c82",
+  "name": "component_install_democratic_csi",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_0b72338e-1d92-4370-8ac3-f0c725940c82.tar.gz.b64' > /tmp/democratic_csi.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/democratic_csi.tar.gz 'root@libre-wuji-cp-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-cp-0' 'rm -rf /tmp/democratic_csi && mkdir -p /tmp/democratic_csi && tar xzf /tmp/democratic_csi.tar.gz -C /tmp/democratic_csi && cd /tmp/democratic_csi && sudo bash install-democratic_csi.sh install ; rc=$?; rm -f /tmp/democratic_csi.tar.gz && rm -rf /tmp/democratic_csi; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_0b72338e-1d92-4370-8ac3-f0c725940c82.tar.gz.b64' /tmp/democratic_csi.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T02:44:06.586118Z",
+  "started_at": "2026-04-21T02:44:06.727424Z",
+  "completed_at": "2026-04-21T02:44:10.649693Z",
+  "output": "=== democratic-csi: v0.14.6 ===\n=== democratic-csi: installing helm ===\nDownloading https://get.helm.sh/helm-v3.20.2-linux-arm64.tar.gz\nVerifying checksum... Done.\nPreparing to install helm into /usr/local/bin\nhelm installed into /usr/local/bin/helm\n=== democratic-csi: nfs_server empty — deferred (namespace only) ===\nnamespace/democratic-csi created\n    Set nfs_server to the private IP of the external_nfs node and re-run.\n",
+  "error": null,
+  "tags": {
+    "component": "democratic_csi",
+    "server": "libre-wuji-cp-0",
+    "operation": "install",
+    "type": "component",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji"
+  }
+}

crates/data/tasks/0c6642ff-9cfc-4123-9c2d-b185029f02bf.json

@@ -0,0 +1,17 @@
+{
+  "id": "0c6642ff-9cfc-4123-9c2d-b185029f02bf",
+  "name": "execute_servers_script_libre-wuji-cp-0",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < /tmp/orchestrator_script_0c6642ff-9cfc-4123-9c2d-b185029f02bf.tar.gz.b64 | gunzip | tar -xOf - script.sh | bash +x"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T00:47:28.933303Z",
+  "started_at": "2026-04-21T00:47:29.292291Z",
+  "completed_at": "2026-04-21T00:47:56.758101Z",
+  "output": "✓ Prerequisites: HCLOUD_TOKEN set, hcloud CLI available\n\n=== Managing SSH Keys ===\n✓ SSH public key found: /Users/jesusperezlorenzo/.ssh/htz_ops.pub\nChecking if SSH key 'htz_ops' exists in Hetzner...\n✓ SSH key htz_ops already exists with ID: 106168627\n\n=== SSH Key Management Complete ===\nSSH_KEY_ID: 106168627\nSSH_KEY_NAME: htz_ops\nEnvironment exported to: /tmp/provisioning-wuji-20260421-014729/.env\n\n=== Managing Network ===\n✓ Network config validated: 10.0.8.0/22 with subnet 10.0.8.0/24 in zone eu-central\nChecking if network 'wuwei' exists...\n✓ Network 'wuwei' already exists with ID: 11875940\n  ✓ Subnet 10.0.8.0/24 already present\n\n=== Network Management Complete ===\nNETWORK_ID: 11875940\nNETWORK_NAME: wuwei\nEnvironment exported to: /tmp/provisioning-wuji-20260421-014729/.env\n\u001b[0;34m[INFO]\u001b[0m Step 1/3: Checking server 'libre-wuji-cp-0'...\n\u001b[0;34m[INFO]\u001b[0m Creating server 'libre-wuji-cp-0'...\n\u001b[0;32m[✓]\u001b[0m 127585130 libre-wuji-cp-0 createdIPv4: \n\u001b[0;32m[✓]\u001b[0m Server created: ID=127585130\n\u001b[0;34m[INFO]\u001b[0m Waiting for server 'libre-wuji-cp-0' to reach running state...\n\u001b[0;32m[✓]\u001b[0m Server 'libre-wuji-cp-0' is running\nServer 127585130 attached to Network 11875940\n\u001b[0;32m[✓]\u001b[0m Attached to network wuwei with IP 10.0.8.20\nResource protection enabled for Server 127585130\n\u001b[0;32m[✓]\u001b[0m Protection enabled: delete rebuild\n\u001b[0;34m[INFO]\u001b[0m Step 2/2: Attaching server to firewall 'librecloud-fw'...\nFirewall 10819279 applied to resource\n\u001b[0;32m[✓]\u001b[0m Server attached to firewall 'librecloud-fw'\n\u001b[0;32m[✓]\u001b[0m ✓ Server provisioning complete\n\u001b[0;34m[INFO]\u001b[0m Summary:\n\u001b[0;34m[INFO]\u001b[0m   • Server:   libre-wuji-cp-0 (ID: 127585130)\n\u001b[0;34m[INFO]\u001b[0m   • Firewall: librecloud-fw\n\u001b[0;34m[INFO]\u001b[0m   • Status:   Ready for deployment\n",
+  "error": null,
+  "tags": {}
+}

crates/data/tasks/0db48326-6bd1-4015-a0b9-5146e842c318.json

@@ -0,0 +1,23 @@
+{
+  "id": "0db48326-6bd1-4015-a0b9-5146e842c318",
+  "name": "component_install_kubernetes",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_0db48326-6bd1-4015-a0b9-5146e842c318.tar.gz.b64' > /tmp/kubernetes.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/kubernetes.tar.gz 'root@libre-wuji-wrk-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-wrk-0' 'rm -rf /tmp/kubernetes && mkdir -p /tmp/kubernetes && tar xzf /tmp/kubernetes.tar.gz -C /tmp/kubernetes && cd /tmp/kubernetes && sudo bash install-kubernetes.sh install ; rc=$?; rm -f /tmp/kubernetes.tar.gz && rm -rf /tmp/kubernetes; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_0db48326-6bd1-4015-a0b9-5146e842c318.tar.gz.b64' /tmp/kubernetes.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T04:42:26.575558Z",
+  "started_at": "2026-04-21T04:42:26.914686Z",
+  "completed_at": "2026-04-21T04:42:39.104765Z",
+  "output": "",
+  "error": null,
+  "tags": {
+    "server": "libre-wuji-wrk-0",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "type": "component",
+    "operation": "install",
+    "component": "kubernetes"
+  }
+}

crates/data/tasks/1055cc82-4c97-45ab-9095-871d5a5bf254.json

@@ -0,0 +1,23 @@
+{
+  "id": "1055cc82-4c97-45ab-9095-871d5a5bf254",
+  "name": "component_install_cilium",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_1055cc82-4c97-45ab-9095-871d5a5bf254.tar.gz.b64' > /tmp/cilium.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/cilium.tar.gz 'root@libre-wuji-cp-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-cp-0' 'rm -rf /tmp/cilium && mkdir -p /tmp/cilium && tar xzf /tmp/cilium.tar.gz -C /tmp/cilium && cd /tmp/cilium && sudo bash install-cilium.sh install ; rc=$?; rm -f /tmp/cilium.tar.gz && rm -rf /tmp/cilium; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_1055cc82-4c97-45ab-9095-871d5a5bf254.tar.gz.b64' /tmp/cilium.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Failed",
+  "created_at": "2026-04-20T11:14:59.966046Z",
+  "started_at": "2026-04-20T11:15:00.300158Z",
+  "completed_at": "2026-04-20T11:15:06.758918Z",
+  "output": null,
+  "error": "Command execution failed: \nError: Unable to install Cilium: Kubernetes cluster unreachable: Get \"https://10.0.8.20:6443/version\": dial tcp 10.0.8.20:6443: connect: connection refused\n",
+  "tags": {
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "component": "cilium",
+    "server": "libre-wuji-cp-0",
+    "operation": "install",
+    "type": "component"
+  }
+}

crates/data/tasks/12e0f651-c7ea-403d-b82f-84be04f899bd.json

@@ -0,0 +1,23 @@
+{
+  "id": "12e0f651-c7ea-403d-b82f-84be04f899bd",
+  "name": "component_install_resolv",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_12e0f651-c7ea-403d-b82f-84be04f899bd.tar.gz.b64' > /tmp/resolv.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/resolv.tar.gz 'root@libre-wuji-wrk-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-wrk-0' 'rm -rf /tmp/resolv && mkdir -p /tmp/resolv && tar xzf /tmp/resolv.tar.gz -C /tmp/resolv && cd /tmp/resolv && sudo bash install-resolv.sh install ; rc=$?; rm -f /tmp/resolv.tar.gz && rm -rf /tmp/resolv; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_12e0f651-c7ea-403d-b82f-84be04f899bd.tar.gz.b64' /tmp/resolv.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T02:14:25.735155Z",
+  "started_at": "2026-04-21T02:14:25.900745Z",
+  "completed_at": "2026-04-21T02:14:28.703065Z",
+  "output": "",
+  "error": null,
+  "tags": {
+    "type": "component",
+    "server": "libre-wuji-wrk-0",
+    "component": "resolv",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "operation": "install"
+  }
+}

crates/data/tasks/147f51f7-cf97-4952-84da-4e32c8473e8d.json

@@ -0,0 +1,23 @@
+{
+  "id": "147f51f7-cf97-4952-84da-4e32c8473e8d",
+  "name": "component_install_hetzner_csi",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_147f51f7-cf97-4952-84da-4e32c8473e8d.tar.gz.b64' > /tmp/hetzner_csi.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/hetzner_csi.tar.gz 'root@libre-wuji-cp-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-cp-0' 'rm -rf /tmp/hetzner_csi && mkdir -p /tmp/hetzner_csi && tar xzf /tmp/hetzner_csi.tar.gz -C /tmp/hetzner_csi && cd /tmp/hetzner_csi && sudo bash install-hetzner_csi.sh install ; rc=$?; rm -f /tmp/hetzner_csi.tar.gz && rm -rf /tmp/hetzner_csi; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_147f51f7-cf97-4952-84da-4e32c8473e8d.tar.gz.b64' /tmp/hetzner_csi.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T02:26:38.339315Z",
+  "started_at": "2026-04-21T02:26:38.692203Z",
+  "completed_at": "2026-04-21T02:26:52.261401Z",
+  "output": "=== hetzner-csi: checking prerequisites ===\n=== hetzner-csi: deploying v2.9.0 ===\nserviceaccount/hcloud-csi-controller created\nclusterrole.rbac.authorization.k8s.io/hcloud-csi-controller created\nclusterrolebinding.rbac.authorization.k8s.io/hcloud-csi-controller created\nservice/hcloud-csi-controller-metrics created\nservice/hcloud-csi-node-metrics created\ndaemonset.apps/hcloud-csi-node created\ndeployment.apps/hcloud-csi-controller created\ncsidriver.storage.k8s.io/csi.hetzner.cloud created\n=== hetzner-csi: creating StorageClass hcloud-volumes (reclaimPolicy=Retain) ===\nstorageclass.storage.k8s.io/hcloud-volumes created\nstorageclass.storage.k8s.io/hcloud-volumes annotated\n=== hetzner-csi: waiting for controller pod ===\nWaiting for deployment \"hcloud-csi-controller\" rollout to finish: 0 of 1 updated replicas are available...\ndeployment \"hcloud-csi-controller\" successfully rolled out\n=== hetzner-csi: ready (StorageClass=hcloud-volumes, policy=Retain, fs=ext4) ===\n",
+  "error": null,
+  "tags": {
+    "server": "libre-wuji-cp-0",
+    "operation": "install",
+    "type": "component",
+    "component": "hetzner_csi",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji"
+  }
+}

crates/data/tasks/17ebdbae-5a51-4bc0-881b-24c168c70396.json

@@ -0,0 +1,23 @@
+{
+  "id": "17ebdbae-5a51-4bc0-881b-24c168c70396",
+  "name": "component_install_kubernetes",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_17ebdbae-5a51-4bc0-881b-24c168c70396.tar.gz.b64' > /tmp/kubernetes.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/kubernetes.tar.gz 'root@libre-wuji-wrk-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-wrk-0' 'rm -rf /tmp/kubernetes && mkdir -p /tmp/kubernetes && tar xzf /tmp/kubernetes.tar.gz -C /tmp/kubernetes && cd /tmp/kubernetes && sudo bash install-kubernetes.sh install ; rc=$?; rm -f /tmp/kubernetes.tar.gz && rm -rf /tmp/kubernetes; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_17ebdbae-5a51-4bc0-881b-24c168c70396.tar.gz.b64' /tmp/kubernetes.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T05:10:57.138024Z",
+  "started_at": "2026-04-21T05:10:57.371587Z",
+  "completed_at": "2026-04-21T05:11:09.302706Z",
+  "output": "",
+  "error": null,
+  "tags": {
+    "operation": "install",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "type": "component",
+    "component": "kubernetes",
+    "server": "libre-wuji-wrk-0"
+  }
+}

crates/data/tasks/18c5c3f1-82c7-460c-a7c4-463e1c17017f.json

@@ -0,0 +1,23 @@
+{
+  "id": "18c5c3f1-82c7-460c-a7c4-463e1c17017f",
+  "name": "component_install_containerd",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_18c5c3f1-82c7-460c-a7c4-463e1c17017f.tar.gz.b64' > /tmp/containerd.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/containerd.tar.gz 'root@libre-wuji-cp-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-cp-0' 'rm -rf /tmp/containerd && mkdir -p /tmp/containerd && tar xzf /tmp/containerd.tar.gz -C /tmp/containerd && cd /tmp/containerd && sudo bash install-containerd.sh install ; rc=$?; rm -f /tmp/containerd.tar.gz && rm -rf /tmp/containerd; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_18c5c3f1-82c7-460c-a7c4-463e1c17017f.tar.gz.b64' /tmp/containerd.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Failed",
+  "created_at": "2026-04-20T10:28:47.849117Z",
+  "started_at": "2026-04-20T10:28:48.291233Z",
+  "completed_at": "2026-04-20T10:28:55.043273Z",
+  "output": null,
+  "error": "Command execution failed: tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'\ntar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'\ntar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'\ntar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'\ntar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'\ntar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'\ntar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'\ninstall-containerd.sh: line 42: _common_service_stop: command not found\ninstall-containerd.sh: line 93: _common_service_start: command not found\n",
+  "tags": {
+    "type": "component",
+    "server": "libre-wuji-cp-0",
+    "component": "containerd",
+    "operation": "install",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji"
+  }
+}

crates/data/tasks/1de586af-a397-47c3-bff4-88e77d138496.json

@@ -0,0 +1,23 @@
+{
+  "id": "1de586af-a397-47c3-bff4-88e77d138496",
+  "name": "component_install_kubernetes",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_1de586af-a397-47c3-bff4-88e77d138496.tar.gz.b64' > /tmp/kubernetes.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/kubernetes.tar.gz 'root@libre-wuji-cp-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-cp-0' 'rm -rf /tmp/kubernetes && mkdir -p /tmp/kubernetes && tar xzf /tmp/kubernetes.tar.gz -C /tmp/kubernetes && cd /tmp/kubernetes && sudo bash install-kubernetes.sh install ; rc=$?; rm -f /tmp/kubernetes.tar.gz && rm -rf /tmp/kubernetes; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_1de586af-a397-47c3-bff4-88e77d138496.tar.gz.b64' /tmp/kubernetes.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Failed",
+  "created_at": "2026-04-20T11:01:10.252268Z",
+  "started_at": "2026-04-20T11:01:10.486569Z",
+  "completed_at": "2026-04-20T11:01:13.593520Z",
+  "output": null,
+  "error": "Command execution failed: resources/kubeadm-config.yaml not found\n",
+  "tags": {
+    "component": "kubernetes",
+    "server": "libre-wuji-cp-0",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "operation": "install",
+    "type": "component"
+  }
+}

crates/data/tasks/21a7ed02-b4f0-4f09-98cc-8406a98cd0de.json

@@ -0,0 +1,23 @@
+{
+  "id": "21a7ed02-b4f0-4f09-98cc-8406a98cd0de",
+  "name": "component_install_external_nfs",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_21a7ed02-b4f0-4f09-98cc-8406a98cd0de.tar.gz.b64' > /tmp/external_nfs.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/external_nfs.tar.gz 'root@libre-wuji-cp-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-cp-0' 'rm -rf /tmp/external_nfs && mkdir -p /tmp/external_nfs && tar xzf /tmp/external_nfs.tar.gz -C /tmp/external_nfs && cd /tmp/external_nfs && sudo bash install-external_nfs.sh install ; rc=$?; rm -f /tmp/external_nfs.tar.gz && rm -rf /tmp/external_nfs; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_21a7ed02-b4f0-4f09-98cc-8406a98cd0de.tar.gz.b64' /tmp/external_nfs.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Failed",
+  "created_at": "2026-04-21T02:43:42.534304Z",
+  "started_at": "2026-04-21T02:43:42.637603Z",
+  "completed_at": "2026-04-21T02:43:45.247118Z",
+  "output": null,
+  "error": "Command execution failed: bash: install-external_nfs.sh: No such file or directory\n",
+  "tags": {
+    "server": "libre-wuji-cp-0",
+    "type": "component",
+    "operation": "install",
+    "component": "external_nfs",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji"
+  }
+}

crates/data/tasks/21d299bc-3605-467b-bc88-54694ce92eca.json

@@ -0,0 +1,23 @@
+{
+  "id": "21d299bc-3605-467b-bc88-54694ce92eca",
+  "name": "component_install_os",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_21d299bc-3605-467b-bc88-54694ce92eca.tar.gz.b64' > /tmp/os.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/os.tar.gz 'root@libre-wuji-wrk-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-wrk-0' 'rm -rf /tmp/os && mkdir -p /tmp/os && tar xzf /tmp/os.tar.gz -C /tmp/os && cd /tmp/os && sudo bash install-os.sh install ; rc=$?; rm -f /tmp/os.tar.gz && rm -rf /tmp/os; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_21d299bc-3605-467b-bc88-54694ce92eca.tar.gz.b64' /tmp/os.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T01:55:04.220548Z",
+  "started_at": "2026-04-21T01:55:04.481214Z",
+  "completed_at": "2026-04-21T01:55:11.473780Z",
+  "output": "APT::Get::Update::SourceListWarnings::NonFreeFirmware \"false\";\nHit:1 http://mirror.hetzner.com/debian/packages bookworm InRelease\nHit:2 http://mirror.hetzner.com/debian/packages bookworm-backports InRelease\nHit:3 http://mirror.hetzner.com/debian/packages bookworm-updates InRelease\nHit:4 http://mirror.hetzner.com/debian/security bookworm-security InRelease\nHit:5 http://deb.debian.org/debian bookworm InRelease\nHit:6 http://security.debian.org/debian-security bookworm-security InRelease\nHit:7 http://deb.debian.org/debian bookworm-updates InRelease\nReading package lists...\nReading package lists...\nBuilding dependency tree...\nReading state information...\nCalculating upgrade...\nThe following packages have been kept back:\n  linux-image-arm64\n0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.\nReading package lists...\nBuilding dependency tree...\nReading state information...\n0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.\n",
+  "error": null,
+  "tags": {
+    "component": "os",
+    "server": "libre-wuji-wrk-0",
+    "type": "component",
+    "operation": "install",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji"
+  }
+}

crates/data/tasks/24c4352d-ab7c-4756-95a4-8b9c23712d71.json

@@ -0,0 +1,17 @@
+{
+  "id": "24c4352d-ab7c-4756-95a4-8b9c23712d71",
+  "name": "execute_servers_script_libre-wuji-wrk-0",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < /tmp/orchestrator_script_24c4352d-ab7c-4756-95a4-8b9c23712d71.tar.gz.b64 | gunzip | tar -xOf - script.sh | bash +x"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T00:42:51.560222Z",
+  "started_at": "2026-04-21T00:42:51.712428Z",
+  "completed_at": "2026-04-21T00:43:34.296729Z",
+  "output": "✓ Prerequisites: HCLOUD_TOKEN set, hcloud CLI available\n\n=== Managing SSH Keys ===\n✓ SSH public key found: /Users/jesusperezlorenzo/.ssh/htz_ops.pub\nChecking if SSH key 'htz_ops' exists in Hetzner...\n✓ SSH key htz_ops already exists with ID: 106168627\n\n=== SSH Key Management Complete ===\nSSH_KEY_ID: 106168627\nSSH_KEY_NAME: htz_ops\nEnvironment exported to: /tmp/provisioning-wuji-20260421-014251/.env\n\n=== Managing Network ===\n✓ Network config validated: 10.0.8.0/22 with subnet 10.0.8.0/24 in zone eu-central\nChecking if network 'wuwei' exists...\n✓ Network 'wuwei' already exists with ID: 11875940\n  ✓ Subnet 10.0.8.0/24 already present\n\n=== Network Management Complete ===\nNETWORK_ID: 11875940\nNETWORK_NAME: wuwei\nEnvironment exported to: /tmp/provisioning-wuji-20260421-014251/.env\n\u001b[0;34m[INFO]\u001b[0m Step 1/3: Checking server 'libre-wuji-wrk-0'...\n\u001b[0;34m[INFO]\u001b[0m Creating server 'libre-wuji-wrk-0'...\n\u001b[0;32m[✓]\u001b[0m 127584497 libre-wuji-wrk-0 createdIPv4: \n\u001b[0;32m[✓]\u001b[0m Server created: ID=127584497\n\u001b[0;34m[INFO]\u001b[0m Waiting for server 'libre-wuji-wrk-0' to reach running state...\n\u001b[0;32m[✓]\u001b[0m Server 'libre-wuji-wrk-0' is running\nServer 127584497 attached to Network 11875940\n\u001b[0;32m[✓]\u001b[0m Attached to network wuwei with IP 10.0.8.24\nResource protection enabled for Server 127584497\n\u001b[0;32m[✓]\u001b[0m Protection enabled: delete rebuild\n\u001b[0;34m[INFO]\u001b[0m Step 2/2: Attaching server to firewall 'librecloud-fw'...\nFirewall 10819279 applied to resource\n\u001b[0;32m[✓]\u001b[0m Server attached to firewall 'librecloud-fw'\n\u001b[0;34m[INFO]\u001b[0m Step 3/3: Assigning floating IP 'librecloud-fip-mail' to 'libre-wuji-wrk-0'...\nFloating IP 127388996 assigned to Server 127584497\n\u001b[0;32m[✓]\u001b[0m Assigned floating IP 'librecloud-fip-mail' (id=127388996) → libre-wuji-wrk-0 (id=127584497)\n\u001b[0;32m[✓]\u001b[0m ✓ Server provisioning complete\n\u001b[0;34m[INFO]\u001b[0m Summary:\n\u001b[0;34m[INFO]\u001b[0m   • Server:   libre-wuji-wrk-0 (ID: 127584497)\n\u001b[0;34m[INFO]\u001b[0m   • Firewall: librecloud-fw\n\u001b[0;34m[INFO]\u001b[0m   • Float IP: librecloud-fip-mail (id=127388996)\n\u001b[0;34m[INFO]\u001b[0m   • Status:   Ready for deployment\n",
+  "error": null,
+  "tags": {}
+}

crates/data/tasks/29a4a214-c961-4b14-8256-afaf1541ca81.json

@@ -0,0 +1,23 @@
+{
+  "id": "29a4a214-c961-4b14-8256-afaf1541ca81",
+  "name": "component_install_hetzner_csi",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_29a4a214-c961-4b14-8256-afaf1541ca81.tar.gz.b64' > /tmp/hetzner_csi.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/hetzner_csi.tar.gz 'root@libre-wuji-cp-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-cp-0' 'rm -rf /tmp/hetzner_csi && mkdir -p /tmp/hetzner_csi && tar xzf /tmp/hetzner_csi.tar.gz -C /tmp/hetzner_csi && cd /tmp/hetzner_csi && sudo bash install-hetzner_csi.sh install ; rc=$?; rm -f /tmp/hetzner_csi.tar.gz && rm -rf /tmp/hetzner_csi; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_29a4a214-c961-4b14-8256-afaf1541ca81.tar.gz.b64' /tmp/hetzner_csi.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Failed",
+  "created_at": "2026-04-21T02:16:01.781982Z",
+  "started_at": "2026-04-21T02:16:02.265091Z",
+  "completed_at": "2026-04-21T02:16:05.102310Z",
+  "output": null,
+  "error": "Command execution failed: ERROR: Secret 'hcloud' not found in kube-system.\n       Create it first: kubectl -n kube-system create secret generic hcloud --from-literal=token=<HCLOUD_TOKEN>\n",
+  "tags": {
+    "operation": "install",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "type": "component",
+    "component": "hetzner_csi",
+    "server": "libre-wuji-cp-0"
+  }
+}

crates/data/tasks/2af968fa-1f9e-44b7-85d6-ea913de2142e.json

@@ -0,0 +1,23 @@
+{
+  "id": "2af968fa-1f9e-44b7-85d6-ea913de2142e",
+  "name": "component_install_crun",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_2af968fa-1f9e-44b7-85d6-ea913de2142e.tar.gz.b64' > /tmp/crun.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/crun.tar.gz 'root@libre-wuji-cp-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-cp-0' 'rm -rf /tmp/crun && mkdir -p /tmp/crun && tar xzf /tmp/crun.tar.gz -C /tmp/crun && cd /tmp/crun && sudo bash install-crun.sh install ; rc=$?; rm -f /tmp/crun.tar.gz && rm -rf /tmp/crun; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_2af968fa-1f9e-44b7-85d6-ea913de2142e.tar.gz.b64' /tmp/crun.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T01:44:12.211853Z",
+  "started_at": "2026-04-21T01:44:12.491759Z",
+  "completed_at": "2026-04-21T01:44:15.870915Z",
+  "output": "",
+  "error": null,
+  "tags": {
+    "operation": "install",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "server": "libre-wuji-cp-0",
+    "component": "crun",
+    "type": "component"
+  }
+}

crates/data/tasks/3bc38fc5-9d51-4c44-a761-7c06948606f6.json

@@ -0,0 +1,23 @@
+{
+  "id": "3bc38fc5-9d51-4c44-a761-7c06948606f6",
+  "name": "component_install_kubernetes",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_3bc38fc5-9d51-4c44-a761-7c06948606f6.tar.gz.b64' > /tmp/kubernetes.tar.gz && scp  /tmp/kubernetes.tar.gz 'root@libre-wuji-wkr-0:/tmp/' && ssh  'root@libre-wuji-wkr-0' 'rm -rf /tmp/kubernetes && mkdir -p /tmp/kubernetes && tar xzf /tmp/kubernetes.tar.gz -C /tmp/kubernetes && cd /tmp/kubernetes && sudo bash install-kubernetes.sh install ; rc=$?; rm -f /tmp/kubernetes.tar.gz && rm -rf /tmp/kubernetes; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_3bc38fc5-9d51-4c44-a761-7c06948606f6.tar.gz.b64' /tmp/kubernetes.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Failed",
+  "created_at": "2026-04-21T03:12:56.664977Z",
+  "started_at": "2026-04-21T03:12:56.887653Z",
+  "completed_at": "2026-04-21T03:12:56.927657Z",
+  "output": null,
+  "error": "Command execution failed: ssh: Could not resolve hostname libre-wuji-wkr-0: nodename nor servname provided, or not known\nscp: Connection closed\n",
+  "tags": {
+    "operation": "install",
+    "type": "component",
+    "server": "libre-wuji-wkr-0",
+    "component": "kubernetes",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji"
+  }
+}

crates/data/tasks/3c4b73ce-4ec0-4c63-b8a0-f04408f82c67.json

@@ -0,0 +1,23 @@
+{
+  "id": "3c4b73ce-4ec0-4c63-b8a0-f04408f82c67",
+  "name": "component_install_hetzner_csi",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_3c4b73ce-4ec0-4c63-b8a0-f04408f82c67.tar.gz.b64' > /tmp/hetzner_csi.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/hetzner_csi.tar.gz 'root@libre-wuji-cp-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-cp-0' 'rm -rf /tmp/hetzner_csi && mkdir -p /tmp/hetzner_csi && tar xzf /tmp/hetzner_csi.tar.gz -C /tmp/hetzner_csi && cd /tmp/hetzner_csi && sudo bash install-hetzner_csi.sh install ; rc=$?; rm -f /tmp/hetzner_csi.tar.gz && rm -rf /tmp/hetzner_csi; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_3c4b73ce-4ec0-4c63-b8a0-f04408f82c67.tar.gz.b64' /tmp/hetzner_csi.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Failed",
+  "created_at": "2026-04-21T02:23:24.029018Z",
+  "started_at": "2026-04-21T02:23:24.447791Z",
+  "completed_at": "2026-04-21T02:23:27.237129Z",
+  "output": null,
+  "error": "Command execution failed: ERROR: Secret 'hcloud' not found in kube-system.\n       Create it first: kubectl -n kube-system create secret generic hcloud --from-literal=token=<HCLOUD_TOKEN>\n",
+  "tags": {
+    "component": "hetzner_csi",
+    "type": "component",
+    "server": "libre-wuji-cp-0",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "operation": "install"
+  }
+}

crates/data/tasks/3c7369a3-b249-43e3-8559-f7de35dc7fe6.json

@@ -0,0 +1,23 @@
+{
+  "id": "3c7369a3-b249-43e3-8559-f7de35dc7fe6",
+  "name": "component_install_youki",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_3c7369a3-b249-43e3-8559-f7de35dc7fe6.tar.gz.b64' > /tmp/youki.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/youki.tar.gz 'root@libre-wuji-wrk-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-wrk-0' 'rm -rf /tmp/youki && mkdir -p /tmp/youki && tar xzf /tmp/youki.tar.gz -C /tmp/youki && cd /tmp/youki && sudo bash install-youki.sh install ; rc=$?; rm -f /tmp/youki.tar.gz && rm -rf /tmp/youki; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_3c7369a3-b249-43e3-8559-f7de35dc7fe6.tar.gz.b64' /tmp/youki.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-20T10:14:12.601474Z",
+  "started_at": "2026-04-20T10:14:12.933804Z",
+  "completed_at": "2026-04-20T10:14:16.513877Z",
+  "output": "",
+  "error": null,
+  "tags": {
+    "component": "youki",
+    "server": "libre-wuji-wrk-0",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "operation": "install",
+    "type": "component"
+  }
+}

crates/data/tasks/3cb42c41-866d-428a-887f-341fb816a1b2.json

@@ -0,0 +1,23 @@
+{
+  "id": "3cb42c41-866d-428a-887f-341fb816a1b2",
+  "name": "component_install_crun",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_3cb42c41-866d-428a-887f-341fb816a1b2.tar.gz.b64' > /tmp/crun.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/crun.tar.gz 'root@libre-wuji-wrk-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-wrk-0' 'rm -rf /tmp/crun && mkdir -p /tmp/crun && tar xzf /tmp/crun.tar.gz -C /tmp/crun && cd /tmp/crun && sudo bash install-crun.sh install ; rc=$?; rm -f /tmp/crun.tar.gz && rm -rf /tmp/crun; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_3cb42c41-866d-428a-887f-341fb816a1b2.tar.gz.b64' /tmp/crun.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-20T10:13:45.058247Z",
+  "started_at": "2026-04-20T10:13:45.327751Z",
+  "completed_at": "2026-04-20T10:13:48.708291Z",
+  "output": "",
+  "error": null,
+  "tags": {
+    "component": "crun",
+    "type": "component",
+    "operation": "install",
+    "server": "libre-wuji-wrk-0",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji"
+  }
+}

crates/data/tasks/3ed38c0f-9920-4dc8-b746-0b783cbaab0e.json

@@ -0,0 +1,23 @@
+{
+  "id": "3ed38c0f-9920-4dc8-b746-0b783cbaab0e",
+  "name": "component_install_kubernetes",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_3ed38c0f-9920-4dc8-b746-0b783cbaab0e.tar.gz.b64' > /tmp/kubernetes.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/kubernetes.tar.gz 'root@libre-wuji-wrk-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-wrk-0' 'rm -rf /tmp/kubernetes && mkdir -p /tmp/kubernetes && tar xzf /tmp/kubernetes.tar.gz -C /tmp/kubernetes && cd /tmp/kubernetes && sudo bash install-kubernetes.sh install ; rc=$?; rm -f /tmp/kubernetes.tar.gz && rm -rf /tmp/kubernetes; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_3ed38c0f-9920-4dc8-b746-0b783cbaab0e.tar.gz.b64' /tmp/kubernetes.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T03:28:27.448853Z",
+  "started_at": "2026-04-21T03:28:27.459746Z",
+  "completed_at": "2026-04-21T03:28:32.081653Z",
+  "output": "No k8s_join.sh found\n",
+  "error": null,
+  "tags": {
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "server": "libre-wuji-wrk-0",
+    "type": "component",
+    "component": "kubernetes",
+    "operation": "install"
+  }
+}

crates/data/tasks/3f27a39a-303d-4ef3-a1fb-fdff3b69d530.json

@@ -0,0 +1,23 @@
+{
+  "id": "3f27a39a-303d-4ef3-a1fb-fdff3b69d530",
+  "name": "component_install_kubernetes",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_3f27a39a-303d-4ef3-a1fb-fdff3b69d530.tar.gz.b64' > /tmp/kubernetes.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/kubernetes.tar.gz 'root@libre-wuji-wrk-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-wrk-0' 'rm -rf /tmp/kubernetes && mkdir -p /tmp/kubernetes && tar xzf /tmp/kubernetes.tar.gz -C /tmp/kubernetes && cd /tmp/kubernetes && sudo bash install-kubernetes.sh install ; rc=$?; rm -f /tmp/kubernetes.tar.gz && rm -rf /tmp/kubernetes; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_3f27a39a-303d-4ef3-a1fb-fdff3b69d530.tar.gz.b64' /tmp/kubernetes.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T03:54:35.730627Z",
+  "started_at": "2026-04-21T03:54:35.923566Z",
+  "completed_at": "2026-04-21T03:54:42.449188Z",
+  "output": "",
+  "error": null,
+  "tags": {
+    "component": "kubernetes",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "operation": "install",
+    "server": "libre-wuji-wrk-0",
+    "type": "component"
+  }
+}

crates/data/tasks/43a62145-7402-4a4b-8aa4-76befe884ebd.json

@@ -0,0 +1,23 @@
+{
+  "id": "43a62145-7402-4a4b-8aa4-76befe884ebd",
+  "name": "component_reinstall_kubernetes",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_43a62145-7402-4a4b-8aa4-76befe884ebd.tar.gz.b64' > /tmp/kubernetes.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/kubernetes.tar.gz 'root@libre-wuji-cp-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-cp-0' 'rm -rf /tmp/kubernetes && mkdir -p /tmp/kubernetes && tar xzf /tmp/kubernetes.tar.gz -C /tmp/kubernetes && cd /tmp/kubernetes && sudo bash install-kubernetes.sh reinstall ; rc=$?; rm -f /tmp/kubernetes.tar.gz && rm -rf /tmp/kubernetes; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_43a62145-7402-4a4b-8aa4-76befe884ebd.tar.gz.b64' /tmp/kubernetes.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T03:04:23.522971Z",
+  "started_at": "2026-04-21T03:04:23.925109Z",
+  "completed_at": "2026-04-21T03:04:27.724227Z",
+  "output": "Warning: probe patch failed\n2026_04_21_030427 | apiserver probes patched: startup=300s liveness=120s readiness=15s\n/etc/kubernetes/admin.conf not found\n",
+  "error": null,
+  "tags": {
+    "component": "kubernetes",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "type": "component",
+    "server": "libre-wuji-cp-0",
+    "operation": "reinstall"
+  }
+}

crates/data/tasks/44a67063-a05a-4126-9aac-5bb52fc850f9.json

@@ -0,0 +1,23 @@
+{
+  "id": "44a67063-a05a-4126-9aac-5bb52fc850f9",
+  "name": "component_install_kubernetes",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_44a67063-a05a-4126-9aac-5bb52fc850f9.tar.gz.b64' > /tmp/kubernetes.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/kubernetes.tar.gz 'root@libre-wuji-wrk-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-wrk-0' 'rm -rf /tmp/kubernetes && mkdir -p /tmp/kubernetes && tar xzf /tmp/kubernetes.tar.gz -C /tmp/kubernetes && cd /tmp/kubernetes && sudo bash install-kubernetes.sh install ; rc=$?; rm -f /tmp/kubernetes.tar.gz && rm -rf /tmp/kubernetes; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_44a67063-a05a-4126-9aac-5bb52fc850f9.tar.gz.b64' /tmp/kubernetes.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T03:17:07.185465Z",
+  "started_at": "2026-04-21T03:17:07.354693Z",
+  "completed_at": "2026-04-21T03:17:16.032753Z",
+  "output": "No k8s_join.sh found\n",
+  "error": null,
+  "tags": {
+    "server": "libre-wuji-wrk-0",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "operation": "install",
+    "type": "component",
+    "component": "kubernetes"
+  }
+}

crates/data/tasks/44cf7c11-770c-4f78-a810-06d4ae105a8e.json

@@ -0,0 +1,23 @@
+{
+  "id": "44cf7c11-770c-4f78-a810-06d4ae105a8e",
+  "name": "component_install_democratic_csi",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_44cf7c11-770c-4f78-a810-06d4ae105a8e.tar.gz.b64' > /tmp/democratic_csi.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/democratic_csi.tar.gz 'root@libre-wuji-cp-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-cp-0' 'rm -rf /tmp/democratic_csi && mkdir -p /tmp/democratic_csi && tar xzf /tmp/democratic_csi.tar.gz -C /tmp/democratic_csi && cd /tmp/democratic_csi && sudo bash install-democratic_csi.sh install ; rc=$?; rm -f /tmp/democratic_csi.tar.gz && rm -rf /tmp/democratic_csi; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_44cf7c11-770c-4f78-a810-06d4ae105a8e.tar.gz.b64' /tmp/democratic_csi.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T03:13:05.041471Z",
+  "started_at": "2026-04-21T03:13:05.419506Z",
+  "completed_at": "2026-04-21T03:13:08.431367Z",
+  "output": "=== democratic-csi: v0.14.6 ===\n=== democratic-csi: nfs_server empty — deferred (namespace only) ===\nnamespace/democratic-csi unchanged\n    Set nfs_server to the private IP of the external_nfs node and re-run.\n",
+  "error": null,
+  "tags": {
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "component": "democratic_csi",
+    "operation": "install",
+    "type": "component",
+    "server": "libre-wuji-cp-0"
+  }
+}

crates/data/tasks/49b985ef-68a6-4f56-ab2d-20a523411fe8.json

@@ -0,0 +1,23 @@
+{
+  "id": "49b985ef-68a6-4f56-ab2d-20a523411fe8",
+  "name": "component_install_kubernetes",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_49b985ef-68a6-4f56-ab2d-20a523411fe8.tar.gz.b64' > /tmp/kubernetes.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/kubernetes.tar.gz 'root@libre-wuji-wrk-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-wrk-0' 'rm -rf /tmp/kubernetes && mkdir -p /tmp/kubernetes && tar xzf /tmp/kubernetes.tar.gz -C /tmp/kubernetes && cd /tmp/kubernetes && sudo bash install-kubernetes.sh install ; rc=$?; rm -f /tmp/kubernetes.tar.gz && rm -rf /tmp/kubernetes; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_49b985ef-68a6-4f56-ab2d-20a523411fe8.tar.gz.b64' /tmp/kubernetes.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T02:39:51.849617Z",
+  "started_at": "2026-04-21T02:39:52.264364Z",
+  "completed_at": "2026-04-21T02:39:55.544779Z",
+  "output": "No k8s_join.sh found\n",
+  "error": null,
+  "tags": {
+    "component": "kubernetes",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "type": "component",
+    "server": "libre-wuji-wrk-0",
+    "operation": "install"
+  }
+}

crates/data/tasks/4a072d77-e98e-4776-a7cc-2d64c511162b.json

@@ -0,0 +1,23 @@
+{
+  "id": "4a072d77-e98e-4776-a7cc-2d64c511162b",
+  "name": "component_install_fip",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_4a072d77-e98e-4776-a7cc-2d64c511162b.tar.gz.b64' > /tmp/fip.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/fip.tar.gz 'root@libre-wuji-wrk-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-wrk-0' 'rm -rf /tmp/fip && mkdir -p /tmp/fip && tar xzf /tmp/fip.tar.gz -C /tmp/fip && cd /tmp/fip && sudo bash install-fip.sh install ; rc=$?; rm -f /tmp/fip.tar.gz && rm -rf /tmp/fip; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_4a072d77-e98e-4776-a7cc-2d64c511162b.tar.gz.b64' /tmp/fip.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-20T10:10:19.412561Z",
+  "started_at": "2026-04-20T10:10:19.538669Z",
+  "completed_at": "2026-04-20T10:10:23.400232Z",
+  "output": "Written /etc/network/interfaces.d/60-floating-ip\nFloating IP 49.12.115.133 active on eth0\n",
+  "error": null,
+  "tags": {
+    "server": "libre-wuji-wrk-0",
+    "component": "fip",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "operation": "install",
+    "type": "component"
+  }
+}

crates/data/tasks/5006edf3-c986-443f-9610-5b5cb9224269.json

@@ -0,0 +1,17 @@
+{
+  "id": "5006edf3-c986-443f-9610-5b5cb9224269",
+  "name": "execute_servers_script_libre-wuji-cp-0",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < /tmp/orchestrator_script_5006edf3-c986-443f-9610-5b5cb9224269.tar.gz.b64 | gunzip | tar -xOf - script.sh | bash +x"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T00:42:51.560218Z",
+  "started_at": "2026-04-21T00:42:51.712434Z",
+  "completed_at": "2026-04-21T00:43:20.037945Z",
+  "output": "✓ Prerequisites: HCLOUD_TOKEN set, hcloud CLI available\n\n=== Managing SSH Keys ===\n✓ SSH public key found: /Users/jesusperezlorenzo/.ssh/htz_ops.pub\nChecking if SSH key 'htz_ops' exists in Hetzner...\n✓ SSH key htz_ops already exists with ID: 106168627\n\n=== SSH Key Management Complete ===\nSSH_KEY_ID: 106168627\nSSH_KEY_NAME: htz_ops\nEnvironment exported to: /tmp/provisioning-wuji-20260421-014251/.env\n\n=== Managing Network ===\n✓ Network config validated: 10.0.8.0/22 with subnet 10.0.8.0/24 in zone eu-central\nChecking if network 'wuwei' exists...\n✓ Network 'wuwei' already exists with ID: 11875940\n  ✓ Subnet 10.0.8.0/24 already present\n\n=== Network Management Complete ===\nNETWORK_ID: 11875940\nNETWORK_NAME: wuwei\nEnvironment exported to: /tmp/provisioning-wuji-20260421-014251/.env\n\u001b[0;34m[INFO]\u001b[0m Step 1/3: Checking server 'libre-wuji-cp-0'...\n\u001b[0;34m[INFO]\u001b[0m Creating server 'libre-wuji-cp-0'...\n\u001b[0;32m[✓]\u001b[0m 127584498 libre-wuji-cp-0 createdIPv4: \n\u001b[0;32m[✓]\u001b[0m Server created: ID=127584498\n\u001b[0;34m[INFO]\u001b[0m Waiting for server 'libre-wuji-cp-0' to reach running state...\n\u001b[0;32m[✓]\u001b[0m Server 'libre-wuji-cp-0' is running\nServer 127584498 attached to Network 11875940\n\u001b[0;32m[✓]\u001b[0m Attached to network wuwei with IP 10.0.8.20\nResource protection enabled for Server 127584498\n\u001b[0;32m[✓]\u001b[0m Protection enabled: delete rebuild\n\u001b[0;34m[INFO]\u001b[0m Step 2/2: Attaching server to firewall 'librecloud-fw'...\nFirewall 10819279 applied to resource\n\u001b[0;32m[✓]\u001b[0m Server attached to firewall 'librecloud-fw'\n\u001b[0;32m[✓]\u001b[0m ✓ Server provisioning complete\n\u001b[0;34m[INFO]\u001b[0m Summary:\n\u001b[0;34m[INFO]\u001b[0m   • Server:   libre-wuji-cp-0 (ID: 127584498)\n\u001b[0;34m[INFO]\u001b[0m   • Firewall: librecloud-fw\n\u001b[0;34m[INFO]\u001b[0m   • Status:   Ready for deployment\n",
+  "error": null,
+  "tags": {}
+}

crates/data/tasks/50f27d6d-65a0-4400-88e6-06ffa9bf5ecc.json

@@ -0,0 +1,23 @@
+{
+  "id": "50f27d6d-65a0-4400-88e6-06ffa9bf5ecc",
+  "name": "component_install_kubernetes",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_50f27d6d-65a0-4400-88e6-06ffa9bf5ecc.tar.gz.b64' > /tmp/kubernetes.tar.gz && scp -i '/Users/jesusperezlorenzo/.ssh/htz_ops' /tmp/kubernetes.tar.gz 'root@libre-wuji-wrk-0:/tmp/' && ssh -i '/Users/jesusperezlorenzo/.ssh/htz_ops' 'root@libre-wuji-wrk-0' 'rm -rf /tmp/kubernetes && mkdir -p /tmp/kubernetes && tar xzf /tmp/kubernetes.tar.gz -C /tmp/kubernetes && cd /tmp/kubernetes && sudo bash install-kubernetes.sh install ; rc=$?; rm -f /tmp/kubernetes.tar.gz && rm -rf /tmp/kubernetes; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_50f27d6d-65a0-4400-88e6-06ffa9bf5ecc.tar.gz.b64' /tmp/kubernetes.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T03:52:48.460149Z",
+  "started_at": "2026-04-21T03:52:48.513324Z",
+  "completed_at": "2026-04-21T03:52:59.984673Z",
+  "output": "",
+  "error": null,
+  "tags": {
+    "type": "component",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "operation": "install",
+    "component": "kubernetes",
+    "server": "libre-wuji-wrk-0"
+  }
+}

crates/data/tasks/52c80f88-1fa5-4c3f-8e15-b645d0b7abdd.json

@@ -0,0 +1,17 @@
+{
+  "id": "52c80f88-1fa5-4c3f-8e15-b645d0b7abdd",
+  "name": "execute_servers_script_libre-wuji-cp-0",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < /tmp/orchestrator_script_52c80f88-1fa5-4c3f-8e15-b645d0b7abdd.tar.gz.b64 | gunzip | tar -xOf - script.sh | bash +x"
+  ],
+  "dependencies": [],
+  "status": "Completed",
+  "created_at": "2026-04-21T01:07:42.324101Z",
+  "started_at": "2026-04-21T01:07:42.517244Z",
+  "completed_at": "2026-04-21T01:08:10.615549Z",
+  "output": "✓ Prerequisites: HCLOUD_TOKEN set, hcloud CLI available\n\n=== Managing SSH Keys ===\n✓ SSH public key found: /Users/jesusperezlorenzo/.ssh/htz_ops.pub\nChecking if SSH key 'htz_ops' exists in Hetzner...\n✓ SSH key htz_ops already exists with ID: 106168627\n\n=== SSH Key Management Complete ===\nSSH_KEY_ID: 106168627\nSSH_KEY_NAME: htz_ops\nEnvironment exported to: /tmp/provisioning-wuji-20260421-020742/.env\n\n=== Managing Network ===\n✓ Network config validated: 10.0.8.0/22 with subnet 10.0.8.0/24 in zone eu-central\nChecking if network 'wuwei' exists...\n✓ Network 'wuwei' already exists with ID: 11875940\n  ✓ Subnet 10.0.8.0/24 already present\n\n=== Network Management Complete ===\nNETWORK_ID: 11875940\nNETWORK_NAME: wuwei\nEnvironment exported to: /tmp/provisioning-wuji-20260421-020742/.env\n\u001b[0;34m[INFO]\u001b[0m Step 1/3: Checking server 'libre-wuji-cp-0'...\n\u001b[0;34m[INFO]\u001b[0m Creating server 'libre-wuji-cp-0'...\n\u001b[0;32m[✓]\u001b[0m 127586862 libre-wuji-cp-0 createdIPv4: \n\u001b[0;32m[✓]\u001b[0m Server created: ID=127586862\n\u001b[0;34m[INFO]\u001b[0m Waiting for server 'libre-wuji-cp-0' to reach running state...\n\u001b[0;32m[✓]\u001b[0m Server 'libre-wuji-cp-0' is running\nServer 127586862 attached to Network 11875940\n\u001b[0;32m[✓]\u001b[0m Attached to network wuwei with IP 10.0.8.20\nResource protection enabled for Server 127586862\n\u001b[0;32m[✓]\u001b[0m Protection enabled: delete rebuild\n\u001b[0;34m[INFO]\u001b[0m Step 2/2: Attaching server to firewall 'librecloud-fw'...\nFirewall 10819279 applied to resource\n\u001b[0;32m[✓]\u001b[0m Server attached to firewall 'librecloud-fw'\n\u001b[0;32m[✓]\u001b[0m ✓ Server provisioning complete\n\u001b[0;34m[INFO]\u001b[0m Summary:\n\u001b[0;34m[INFO]\u001b[0m   • Server:   libre-wuji-cp-0 (ID: 127586862)\n\u001b[0;34m[INFO]\u001b[0m   • Firewall: librecloud-fw\n\u001b[0;34m[INFO]\u001b[0m   • Status:   Ready for deployment\n",
+  "error": null,
+  "tags": {}
+}

crates/data/tasks/5406f205-24f1-467c-90cc-7e917b7e3293.json

@@ -0,0 +1,23 @@
+{
+  "id": "5406f205-24f1-467c-90cc-7e917b7e3293",
+  "name": "component_install_containerd",
+  "command": "bash",
+  "args": [
+    "-c",
+    "base64 -d < '/tmp/orchestrator_comp_5406f205-24f1-467c-90cc-7e917b7e3293.tar.gz.b64' > /tmp/containerd.tar.gz && scp  /tmp/containerd.tar.gz 'root@libre-wuji-wk-0:/tmp/' && ssh  'root@libre-wuji-wk-0' 'rm -rf /tmp/containerd && mkdir -p /tmp/containerd && tar xzf /tmp/containerd.tar.gz -C /tmp/containerd && cd /tmp/containerd && sudo bash install-containerd.sh install ; rc=$?; rm -f /tmp/containerd.tar.gz && rm -rf /tmp/containerd; exit $rc' ; rc=$?; rm -f '/tmp/orchestrator_comp_5406f205-24f1-467c-90cc-7e917b7e3293.tar.gz.b64' /tmp/containerd.tar.gz; exit $rc"
+  ],
+  "dependencies": [],
+  "status": "Failed",
+  "created_at": "2026-04-20T10:29:18.910354Z",
+  "started_at": "2026-04-20T10:29:19.413672Z",
+  "completed_at": "2026-04-20T10:29:19.627169Z",
+  "output": null,
+  "error": "Command execution failed: ssh: Could not resolve hostname libre-wuji-wk-0: nodename nor servname provided, or not known\nscp: Connection closed\n",
+  "tags": {
+    "component": "containerd",
+    "workspace": "/Users/Akasha/project-provisioning/workspaces/libre-wuji",
+    "type": "component",
+    "server": "libre-wuji-wk-0",
+    "operation": "install"
+  }
+}