provisioning/provisioning-platform
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

platform: clean-start baseline — Rust workspace: binaries/daemon (constellation materialization)

Browse source
This commit is contained in:
Jesús Pérez 2026-07-09 21:59:23 +01:00
parent df4d09d115
commit 584dd0e079
Signed by: jesus
GPG key ID: 9F243E355E0BC939
4198 changed files with 396792 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.cargo/config.toml Normal file
View file

@ -0,0 +1,2 @@
[build]
target-dir = "/Volumes/Devel/provisioning/platform/target"

57
.dockerignore Normal file
View file

@ -0,0 +1,57 @@
# Rust build artifacts
**/target/
**/*.o
**/*.so
**/*.a
**/*.rlib
# Cargo lock files (we copy them explicitly)
# Cargo.lock
# IDE files
.idea/
.vscode/
*.swp
*.swo
*~
# macOS
.DS_Store
**/.DS_Store
# Logs
*.log
**/*.log
# Node modules (for control-center-ui)
**/node_modules/
**/dist/
**/.cache/
# Test files
**/tests/fixtures/
**/tmp/
**/temp/
# Git
.git/
.gitignore
# Documentation
docs/
*.md
!README.md
# Scripts (not needed in container)
scripts/
# Data directories
data/
**/data/
# Other
.env
.env.*
*.key
*.pem
*.crt

326
.env.example Normal file
View file

@ -0,0 +1,326 @@
# Provisioning Platform Environment Configuration
# Copy this file to .env and customize for your deployment
#==============================================================================
# NICKEL CONFIGURATION (Schema and Import Resolution)
#==============================================================================
# Nickel import path for configuration schema resolution
# Enables proper module resolution in provisioning/schemas and workspaces
NICKEL_IMPORT_PATH=/provisioning:/.
#==============================================================================
# PLATFORM MODE
#==============================================================================
# Deployment mode: solo, multi-user, cicd, enterprise
PROVISIONING_MODE=solo
# Platform metadata
PLATFORM_NAME=provisioning
PLATFORM_VERSION=3.0.0
PLATFORM_ENVIRONMENT=development
#==============================================================================
# NETWORK CONFIGURATION
#==============================================================================
# Docker network subnet
NETWORK_SUBNET=172.20.0.0/16
NETWORK_GATEWAY=172.20.0.1
# External access
EXTERNAL_DOMAIN=provisioning.local
ENABLE_TLS=false
TLS_CERT_PATH=/etc/ssl/certs/provisioning.crt
TLS_KEY_PATH=/etc/ssl/private/provisioning.key
#==============================================================================
# ORCHESTRATOR SERVICE
#==============================================================================
ORCHESTRATOR_ENABLED=true
ORCHESTRATOR_HOST=0.0.0.0
ORCHESTRATOR_PORT=8080
ORCHESTRATOR_WORKERS=4
ORCHESTRATOR_LOG_LEVEL=info
ORCHESTRATOR_DATA_DIR=/data
ORCHESTRATOR_STORAGE_TYPE=filesystem
ORCHESTRATOR_MAX_CONCURRENT_TASKS=5
ORCHESTRATOR_RETRY_ATTEMPTS=3
# CPU and memory limits
ORCHESTRATOR_CPU_LIMIT=2000m
ORCHESTRATOR_MEMORY_LIMIT=2048M
#==============================================================================
# CONTROL CENTER SERVICE
#==============================================================================
CONTROL_CENTER_ENABLED=true
CONTROL_CENTER_HOST=0.0.0.0
CONTROL_CENTER_PORT=8081
CONTROL_CENTER_LOG_LEVEL=info
CONTROL_CENTER_DATABASE_TYPE=rocksdb
CONTROL_CENTER_SESSION_TIMEOUT=3600
# JWT Configuration
CONTROL_CENTER_JWT_SECRET=CHANGE_ME_RANDOM_SECRET_HERE
CONTROL_CENTER_ACCESS_TOKEN_EXPIRATION=3600
CONTROL_CENTER_REFRESH_TOKEN_EXPIRATION=86400
# CPU and memory limits
CONTROL_CENTER_CPU_LIMIT=1000m
CONTROL_CENTER_MEMORY_LIMIT=1024M
#==============================================================================
# COREDNS SERVICE
#==============================================================================
COREDNS_ENABLED=true
COREDNS_PORT=53
COREDNS_TCP_PORT=53
COREDNS_ZONES_DIR=/zones
COREDNS_LOG_LEVEL=info
# CPU and memory limits
COREDNS_CPU_LIMIT=500m
COREDNS_MEMORY_LIMIT=512M
#==============================================================================
# GITEA SERVICE (Multi-user mode and above)
#==============================================================================
GITEA_ENABLED=false
GITEA_HTTP_PORT=3000
GITEA_SSH_PORT=222
GITEA_DOMAIN=localhost
GITEA_ROOT_URL=http://localhost:3000/
GITEA_DB_TYPE=sqlite3
GITEA_SECRET_KEY=CHANGE_ME_GITEA_SECRET_KEY
# Admin user (created on first run)
GITEA_ADMIN_USERNAME=provisioning
GITEA_ADMIN_PASSWORD=CHANGE_ME_ADMIN_PASSWORD
GITEA_ADMIN_EMAIL=admin@provisioning.local
# CPU and memory limits
GITEA_CPU_LIMIT=1000m
GITEA_MEMORY_LIMIT=1024M
#==============================================================================
# OCI REGISTRY SERVICE
#==============================================================================
OCI_REGISTRY_ENABLED=true
OCI_REGISTRY_TYPE=zot
OCI_REGISTRY_PORT=5000
OCI_REGISTRY_NAMESPACE=provisioning-extensions
OCI_REGISTRY_LOG_LEVEL=info
# Authentication (disabled for solo mode)
OCI_REGISTRY_AUTH_ENABLED=false
OCI_REGISTRY_AUTH_HTPASSWD_PATH=/etc/registry/htpasswd
# Storage
OCI_REGISTRY_STORAGE_ROOT=/var/lib/registry
OCI_REGISTRY_DEDUPE_ENABLED=true
# CPU and memory limits
OCI_REGISTRY_CPU_LIMIT=1000m
OCI_REGISTRY_MEMORY_LIMIT=1024M
#==============================================================================
# EXTENSION REGISTRY SERVICE
#==============================================================================
EXTENSION_REGISTRY_ENABLED=true
EXTENSION_REGISTRY_HOST=0.0.0.0
EXTENSION_REGISTRY_PORT=8082
EXTENSION_REGISTRY_LOG_LEVEL=info
EXTENSION_REGISTRY_DATA_DIR=/app/data
# OCI integration
EXTENSION_REGISTRY_OCI_URL=http://oci-registry:5000
EXTENSION_REGISTRY_NAMESPACE=provisioning-extensions
# CPU and memory limits
EXTENSION_REGISTRY_CPU_LIMIT=500m
EXTENSION_REGISTRY_MEMORY_LIMIT=512M
#==============================================================================
# PROVISIONING API SERVER SERVICE
#==============================================================================
API_SERVER_ENABLED=false
API_SERVER_HOST=0.0.0.0
API_SERVER_PORT=8083
API_SERVER_LOG_LEVEL=info
# JWT Configuration
API_SERVER_JWT_SECRET=CHANGE_ME_API_SERVER_JWT_SECRET
API_SERVER_TOKEN_EXPIRATION=3600
# Integration
API_SERVER_ORCHESTRATOR_URL=http://orchestrator:8080
API_SERVER_CONTROL_CENTER_URL=http://control-center:8081
# CPU and memory limits
API_SERVER_CPU_LIMIT=1000m
API_SERVER_MEMORY_LIMIT=1024M
#==============================================================================
# MCP SERVER SERVICE (Optional)
#==============================================================================
MCP_SERVER_ENABLED=false
MCP_SERVER_HOST=0.0.0.0
MCP_SERVER_PORT=8084
MCP_SERVER_PROTOCOL=http
MCP_SERVER_LOG_LEVEL=info
# Capabilities
MCP_SERVER_TOOLS_ENABLED=true
MCP_SERVER_PROMPTS_ENABLED=true
MCP_SERVER_RESOURCES_ENABLED=true
# CPU and memory limits
MCP_SERVER_CPU_LIMIT=500m
MCP_SERVER_MEMORY_LIMIT=512M
#==============================================================================
# DATABASE SERVICE (PostgreSQL for enterprise mode)
#==============================================================================
POSTGRES_ENABLED=false
POSTGRES_HOST=postgres
POSTGRES_PORT=5432
POSTGRES_DB=provisioning
POSTGRES_USER=provisioning
POSTGRES_PASSWORD=CHANGE_ME_POSTGRES_PASSWORD
# CPU and memory limits
POSTGRES_CPU_LIMIT=2000m
POSTGRES_MEMORY_LIMIT=2048M
#==============================================================================
# COSMIAN KMS SERVICE (Enterprise mode)
#==============================================================================
KMS_ENABLED=false
KMS_SERVER=http://kms:9998
KMS_AUTH_METHOD=certificate
KMS_CERT_PATH=/etc/kms/client.crt
KMS_KEY_PATH=/etc/kms/client.key
# CPU and memory limits
KMS_CPU_LIMIT=1000m
KMS_MEMORY_LIMIT=1024M
#==============================================================================
# HARBOR REGISTRY (Enterprise mode alternative to Zot)
#==============================================================================
HARBOR_ENABLED=false
HARBOR_ADMIN_PASSWORD=CHANGE_ME_HARBOR_ADMIN_PASSWORD
HARBOR_DATABASE_PASSWORD=CHANGE_ME_HARBOR_DB_PASSWORD
HARBOR_CORE_SECRET=CHANGE_ME_HARBOR_CORE_SECRET
HARBOR_JOBSERVICE_SECRET=CHANGE_ME_HARBOR_JOBSERVICE_SECRET
# CPU and memory limits
HARBOR_CORE_CPU_LIMIT=2000m
HARBOR_CORE_MEMORY_LIMIT=2048M
#==============================================================================
# MONITORING STACK (Prometheus, Grafana)
#==============================================================================
MONITORING_ENABLED=false
PROMETHEUS_PORT=9090
PROMETHEUS_RETENTION_TIME=15d
GRAFANA_PORT=3001
GRAFANA_ADMIN_PASSWORD=CHANGE_ME_GRAFANA_PASSWORD
# CPU and memory limits
PROMETHEUS_CPU_LIMIT=2000m
PROMETHEUS_MEMORY_LIMIT=2048M
GRAFANA_CPU_LIMIT=500m
GRAFANA_MEMORY_LIMIT=512M
#==============================================================================
# LOGGING STACK (Loki, Promtail)
#==============================================================================
LOGGING_ENABLED=false
LOKI_PORT=3100
LOKI_RETENTION_PERIOD=168h
# CPU and memory limits
LOKI_CPU_LIMIT=1000m
LOKI_MEMORY_LIMIT=1024M
#==============================================================================
# ELASTICSEARCH + KIBANA (Enterprise audit logs)
#==============================================================================
ELASTICSEARCH_ENABLED=false
ELASTICSEARCH_PORT=9200
ELASTICSEARCH_CLUSTER_NAME=provisioning-logs
ELASTICSEARCH_HEAP_SIZE=1g
KIBANA_PORT=5601
# CPU and memory limits
ELASTICSEARCH_CPU_LIMIT=2000m
ELASTICSEARCH_MEMORY_LIMIT=2048M
KIBANA_CPU_LIMIT=1000m
KIBANA_MEMORY_LIMIT=1024M
#==============================================================================
# NGINX REVERSE PROXY
#==============================================================================
NGINX_ENABLED=false
NGINX_HTTP_PORT=80
NGINX_HTTPS_PORT=443
NGINX_WORKER_PROCESSES=4
NGINX_WORKER_CONNECTIONS=1024
# Rate limiting
NGINX_RATE_LIMIT_ENABLED=true
NGINX_RATE_LIMIT_REQUESTS=100
NGINX_RATE_LIMIT_PERIOD=1m
# CPU and memory limits
NGINX_CPU_LIMIT=500m
NGINX_MEMORY_LIMIT=256M
#==============================================================================
# BACKUP CONFIGURATION
#==============================================================================
BACKUP_ENABLED=false
BACKUP_SCHEDULE=0 2 * * *
BACKUP_RETENTION_DAYS=7
BACKUP_STORAGE_PATH=/backup
#==============================================================================
# SECURITY CONFIGURATION
#==============================================================================
# Enable security scanning
SECURITY_SCAN_ENABLED=false
# Secrets encryption
SECRETS_ENCRYPTION_ENABLED=false
SECRETS_KEY_PATH=/etc/provisioning/secrets.key
# Network policies
NETWORK_POLICIES_ENABLED=false
#==============================================================================
# RESOURCE LIMITS DEFAULTS
#==============================================================================
DEFAULT_CPU_LIMIT=1000m
DEFAULT_MEMORY_LIMIT=1024M
DEFAULT_RESTART_POLICY=unless-stopped
#==============================================================================
# HEALTHCHECK CONFIGURATION
#==============================================================================
HEALTHCHECK_INTERVAL=30s
HEALTHCHECK_TIMEOUT=10s
HEALTHCHECK_RETRIES=3
HEALTHCHECK_START_PERIOD=30s
#==============================================================================
# LOGGING CONFIGURATION
#==============================================================================
LOG_DRIVER=json-file
LOG_MAX_SIZE=10m
LOG_MAX_FILE=3
#==============================================================================
# USER AND PERMISSION CONFIGURATION
#==============================================================================
USER_UID=1000
USER_GID=1000

124
.gitignore vendored Normal file
View file

@ -0,0 +1,124 @@
.internal.git/
.p
.claude
.opencode
AGENTS.md
.vscode
.shellcheckrc
.coder
.migration
.zed
ai_demo.nu
CLAUDE.md
.cache
.coder
.wrks
rollback_instruction*
ROOT
OLD
# Generated by Cargo
# will have compiled files and executables
debug/
target/
# Encryption keys and related files (CRITICAL - NEVER COMMIT)
.k.backup
*.key.backup
config.*.toml
config.*back
# where book is written
_book
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
Cargo.lock
# These are backup files generated by rustfmt
**/*.rs.bk
# MSVC Windows builds of rustc generate these, which store debugging information
*.pdb
node_modules/
**/output.css
**/input.css
# Environment files
.env
.env.local
.env.production
.env.development
.env.staging
# Keep example files
!.env.example
# Configuration files (may contain sensitive data)
config.prod.toml
config.production.toml
config.local.toml
config.*.local.toml
# Keep example configuration files
!config.toml
!config.dev.toml
!config.example.toml
# Log files
logs/
*.log
# TLS certificates and keys
certs/
*.pem
*.crt
*.key
*.p12
*.pfx
# Database files
*.db
*.sqlite
*.sqlite3
# Backup files
*.bak
*.backup
*.tmp
*~
# Encryption and security related files
*.encrypted
*.enc
secrets/
private/
security/
# Configuration backups that may contain secrets
config.*.backup
config.backup.*
# OS generated files
.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
ehthumbs.db
Thumbs.db
# Documentation build output
book-output/
# Generated setup report
SETUP_COMPLETE.md
# ML model caches (fastembed, huggingface, etc.)
**/.fastembed_cache/
**/.cache/huggingface/
# Orchestrator runtime queue/state (rkvs) — never source
**/wrks/
.coder/
.claude/

9
.gitmodules vendored Normal file
View file

@ -0,0 +1,9 @@
[submodule "secretumvault"]
path = secretumvault
url = ssh://git@repo.jesusperez.pro:32225/jesus/secretumvault.git
[submodule "stratumiops"]
path = stratumiops
url = ssh://git@repo.jesusperez.pro:32225/jesus/stratumiops.git
[submodule "syntaxis"]
path = syntaxis
url = ssh://git@repo.jesusperez.pro:32225/jesus/syntaxis.git

107
.markdownlint-cli2.jsonc Normal file
View file

@ -0,0 +1,107 @@
// Markdownlint-cli2 Configuration
// Documentation quality enforcement aligned with CLAUDE.md guidelines
// See: https://github.com/igorshubovych/markdownlint-cli2
{
"config": {
"default": true,
// Headings - enforce proper hierarchy
"MD001": false, // heading-increment (relaxed - allow flexibility)
"MD026": { "punctuation": ".,;:!?" }, // heading-punctuation
// Lists - enforce consistency
"MD004": { "style": "consistent" }, // ul-style (consistent list markers)
"MD005": false, // inconsistent-indentation (relaxed)
"MD007": { "indent": 2 }, // ul-indent
"MD029": false, // ol-prefix (allow flexible list numbering)
"MD030": { "ul_single": 1, "ol_single": 1, "ul_multi": 1, "ol_multi": 1 },
// Code blocks - fenced only
"MD046": { "style": "fenced" }, // code-block-style
// CRITICAL: MD040 only checks opening fences, NOT closing fences
// It does NOT catch malformed closing fences with language specifiers (e.g., ```plaintext)
// CommonMark spec requires closing fences to be ``` only (no language)
// Use: nu ../scripts/check-malformed-fences.nu (manual validation)
"MD040": true, // fenced-code-language (code blocks need language on OPENING fence)
// Formatting - strict whitespace
"MD009": true, // no-hard-tabs
"MD010": true, // hard-tabs
"MD011": true, // reversed-link-syntax
"MD018": true, // no-missing-space-atx
"MD019": true, // no-multiple-space-atx
"MD020": true, // no-missing-space-closed-atx
"MD021": true, // no-multiple-space-closed-atx
"MD023": true, // heading-starts-line
"MD027": true, // no-multiple-spaces-blockquote
"MD037": true, // no-space-in-emphasis
"MD039": true, // no-space-in-links
// Trailing content
"MD012": false, // no-multiple-blanks (relaxed - allow formatting space)
"MD024": false, // no-duplicate-heading (too strict for docs)
"MD028": false, // no-blanks-blockquote (relaxed)
"MD047": true, // single-trailing-newline
// Links and references
"MD034": true, // no-bare-urls (links must be formatted)
"MD042": true, // no-empty-links
// HTML - allow for documentation formatting and images
"MD033": { "allowed_elements": ["br", "hr", "details", "summary", "p", "img"] },
// Line length - relaxed for technical documentation
// Headers can be longer to accommodate descriptive technical titles
// Code blocks excluded - example JSON/code should not be reformatted
"MD013": {
"line_length": 150,
"heading_line_length": 350, // Allow longer headers for technical docs
"code_blocks": false, // Don't check line length in code blocks (examples, JSON, etc.)
"tables": true,
"headers": true,
"strict": false,
"stern": false
},
// Images
"MD045": true, // image-alt-text
// Tables - enforce proper formatting
"MD060": true, // table-column-style (proper spacing: | ---- | not |------|)
// Disable rules that conflict with relaxed style
"MD003": false, // consistent-indentation
"MD041": false, // first-line-heading
"MD025": false, // single-h1 / multiple-top-level-headings
"MD022": false, // blanks-around-headings (flexible spacing)
"MD032": false, // blanks-around-lists (flexible spacing)
"MD035": false, // hr-style (consistent)
"MD036": false, // no-emphasis-as-heading
"MD044": false // proper-names
},
// Documentation patterns
"globs": [
"**/*.md",
"!node_modules/**",
"!target/**",
"!.git/**",
"!build/**",
"!dist/**"
],
// Ignore build artifacts, external content, and operational directories
"ignores": [
"node_modules/**",
"target/**",
".git/**",
"build/**",
"dist/**",
".coder/**",
".claude/**",
".wrks/**",
".vale/**"
]
}

147
.pre-commit-config.yaml Normal file
View file

@ -0,0 +1,147 @@
# Pre-commit Framework Configuration
# Generated by dev-system/ci
# Configures git pre-commit hooks for Rust + Markdown projects
repos:
# ============================================================================
# Rust Hooks (ACTIVE)
# ============================================================================
- repo: local
hooks:
- id: rust-fmt
name: Rust formatting (cargo +nightly fmt)
entry: bash -c 'cargo +nightly fmt --all -- --check'
language: system
types: [rust]
pass_filenames: false
stages: [pre-commit]
- id: rust-clippy
name: Rust linting (cargo clippy)
entry: bash -c 'cargo clippy --all-targets -- -D warnings'
language: system
types: [rust]
pass_filenames: false
stages: [pre-commit]
# NOTE: Disabled - cargo test blocks git push. Tests should run in CI/CD.
# - id: rust-test
# name: Rust tests
# entry: bash -c 'cargo test --workspace'
# language: system
# types: [rust]
# pass_filenames: false
# stages: [pre-push]
# NOTE: Disabled - cargo deny blocks git push. Should run in CI/CD.
# - id: cargo-deny
# name: Cargo deny (licenses & advisories)
# entry: bash -c 'cargo deny check licenses advisories'
# language: system
# pass_filenames: false
# stages: [pre-push]
# ============================================================================
# SOLID Architecture Boundary Enforcement
# ============================================================================
- repo: local
hooks:
- id: solid-boundary-check
name: SOLID Architecture Boundaries
entry: .pre-commit-hooks/solid-boundary-check.sh
language: script
pass_filenames: false
stages: [pre-commit]
# ============================================================================
# Nushell Hooks (ACTIVE)
# ============================================================================
- repo: local
hooks:
- id: nushell-check
name: Nushell IDE Check (nu --ide-check)
entry: bash -c 'for f in $(git diff --cached --name-only | grep "\.nu$"); do nu --ide-check "$f" || exit 1; done'
language: system
files: \.nu$
pass_filenames: false
stages: [pre-commit]
# ============================================================================
# Bash Hooks (DISABLED)
# ============================================================================
# NOTE: Disabled - shellcheck-py v0.9.1.1 tag doesn't exist in upstream repo
# Re-enable when upstream releases a compatible version
# - repo: https://github.com/shellcheck-py/shellcheck-py
# rev: v0.9.1.1
# hooks:
# - id: shellcheck
# name: Bash linting (shellcheck)
# args: ['--severity=warning']
# stages: [pre-commit]
# ============================================================================
# Nickel Hooks (ACTIVE)
# ============================================================================
- repo: local
hooks:
- id: nickel-typecheck
name: Nickel type checking
entry: >-
bash -c 'export NICKEL_IMPORT_PATH="../:."; for f in $(git diff --cached --name-only --diff-filter=ACM | grep "\.ncl$"); do
echo "Checking: $f"; nickel typecheck "$f" || exit 1; done'
language: system
types: [file]
files: \.ncl$
pass_filenames: false
stages: [pre-commit]
# ============================================================================
# Markdown Hooks (ACTIVE)
# ============================================================================
- repo: local
hooks:
- id: markdownlint
name: Markdown linting (markdownlint-cli2)
entry: markdownlint-cli2
language: system
types: [markdown]
stages: [pre-commit]
# CRITICAL: markdownlint-cli2 MD040 only checks opening fences for language.
# It does NOT catch malformed closing fences (e.g., ```plaintext) - CommonMark violation.
# This hook is ESSENTIAL to prevent malformed closing fences from entering the repo.
# See: .markdownlint-cli2.jsonc line 22-24 for details.
- id: check-malformed-fences
name: Check malformed closing fences (CommonMark)
entry: bash -c 'nu ../scripts/check-malformed-fences.nu $(git diff --cached --name-only --diff-filter=ACM | grep "\.md$" | grep -v ".coder/" | grep -v ".claude/" | grep -v "old_config/" | tr "\n" " ")'
language: system
types: [markdown]
pass_filenames: false
stages: [pre-commit]
exclude: ^\.coder/|^\.claude/|^old_config/
# ============================================================================
# General Pre-commit Hooks
# ============================================================================
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: check-added-large-files
args: ['--maxkb=1000']
- id: check-case-conflict
- id: check-merge-conflict
- id: check-toml
- id: check-yaml
args: ['--unsafe']
exclude: ^\.woodpecker/
- id: end-of-file-fixer
- id: trailing-whitespace
exclude: \.md$
- id: mixed-line-ending

27
.pre-commit-hooks/solid-boundary-check.sh Executable file
View file

@ -0,0 +1,27 @@
#!/usr/bin/env bash
set -euo pipefail
VIOLATIONS=$(git diff --cached --name-only --diff-filter=ACM |
grep -E "\.(nu|rs)$" |
grep -v "templates/" |
grep -v "extensions/providers/" |
grep -v "orchestrator/" |
xargs grep -lE "^\^hcloud|^\^aws |^\^doctl|hcloud server" 2>/dev/null |
grep -v "^$") || true
if [ -n "$VIOLATIONS" ]; then
echo "SOLID VIOLATION: Provider API calls outside orchestrator:"
echo "$VIOLATIONS"
exit 1
fi
SSH_VIOLATIONS=$(git diff --cached --name-only --diff-filter=ACM |
grep -E "\.(rs)$" |
grep -E "control-center|vault-service" |
xargs grep -lE "ssh2?::|russh::" 2>/dev/null) || true
if [ -n "$SSH_VIOLATIONS" ]; then
echo "SOLID VIOLATION: SSH code outside orchestrator:"
echo "$SSH_VIOLATIONS"
exit 1
fi

18
.provisioning-state.json Normal file
View file

@ -0,0 +1,18 @@
{
"cluster": "librecloud",
"timestamp": "2026-02-15 22:17:38",
"version": "1.0.4",
"state": {
"ssh_keys": {
"htz_ops": {
"id": 106168627,
"created": true,
"fingerprint": "unknown"
}
},
"network": {},
"firewall": {},
"volumes": {},
"servers": {}
}
}

371
.typedialog/README.md Normal file
View file

@ -0,0 +1,371 @@
# TypeDialog Integration
TypeDialog enables interactive form-based configuration from Nickel schemas.
## Status
- **TypeDialog Binary**: Not yet installed (planned: `typedialog` command)
- **TypeDialog Forms**: Created and ready (setup wizard, auth login, MFA enrollment)
- **Bash Wrappers**: Implemented to handle TTY input properly
- **ForminQuire**: DEPRECATED - Archived to `.coder/archive/forminquire/`
## Directory Structure
```toml
.typedialog/
└── provisioning/platform/
├── README.md # This file
├── forms/ # Form definitions (to be generated)
│ ├── orchestrator.form.toml
│ ├── control-center.form.toml
│ └── ...
├── templates/ # Jinja2 templates for schema rendering
│ └── service-form.template.j2
├── schemas/ # Symlink to Nickel schemas
│ └── platform/schemas/ → ../../../schemas/platform/
└── constraints/ # Validation constraints
└── constraints.toml # Shared validation rules
```
## How TypeDialog Would Work
### 1. Form Generation from Schemas
```toml
# Auto-generate form from Nickel schema
typedialog generate-form --schema orchestrator.ncl
--output forms/orchestrator.form.toml
```
### 2. Interactive Configuration
```toml
# Run interactive form
typedialog run-form --form forms/orchestrator.form.toml
--output orchestrator-configured.ncl
```
### 3. Validation
```toml
# Validate user input against schema
typedialog validate --form forms/orchestrator.form.toml
--data user-config.ncl
```
## Current Status: TypeDialog Forms Ready
TypeDialog forms have been created and are ready to use:
**Form Locations**:
- Setup wizard: `provisioning/.typedialog/core/forms/setup-wizard.toml`
- Auth login: `provisioning/.typedialog/core/forms/auth-login.toml`
- MFA enrollment: `provisioning/.typedialog/core/forms/mfa-enroll.toml`
**Bash Wrappers** (TTY-safe, handle input properly):
- `provisioning/core/shlib/setup-wizard-tty.sh`
- `provisioning/core/shlib/auth-login-tty.sh`
- `provisioning/core/shlib/mfa-enroll-tty.sh`
**Usage Pattern**:
1. Bash wrapper calls TypeDialog (handles TTY input)
2. TypeDialog generates Nickel config file
3. Nushell scripts read the generated config (no input issues)
**Example**:
```toml
# Run TypeDialog setup wizard
./provisioning/core/shlib/setup-wizard-tty.sh
# Nushell reads the generated config
let config = (open provisioning/.typedialog/core/generated/setup-wizard-result.json | from json)
```
**Note**: ForminQuire (Jinja2-based forms) has been archived to `provisioning/.coder/archive/forminquire/` and is no longer in use.
## Integration Plan (When TypeDialog Available)
### Step 1: Install TypeDialog
```toml
cargo install --path /Users/Akasha/Development/typedialog
typedialog --version
```
### Step 2: Generate Forms from Schemas
```toml
# Batch generate all forms
for schema in provisioning/schemas/platform/*.ncl; do
service=$(basename $schema .ncl)
typedialog generate-form
--schema $schema
--output provisioning/platform/.typedialog/forms/${service}.form.toml
done
```
### Step 3: Create Setup Wizard
```toml
# Unified setup workflow
provisioning setup-platform
--mode solo|multiuser|enterprise
--provider docker|kubernetes
--interactive # Uses TypeDialog forms
```
### Step 4: Update Platform Setup Script
```toml
# provisioning/platform/scripts/setup-platform-config.sh
if command -v typedialog &> /dev/null; then
# TypeDialog is installed - use bash wrapper for proper TTY handling
./provisioning/core/shlib/setup-wizard-tty.sh
# Read generated JSON config
# Nushell scripts can now read the config without input issues
else
# Fallback to basic prompts
echo "TypeDialog not available. Using basic interactive prompts..."
# Nushell wizard with basic input prompts
nu -c "use provisioning/core/nulib/lib_provisioning/setup/wizard.nu *; run-setup-wizard"
fi
```
## Form Definition Example
```toml
# provisioning/platform/.typedialog/forms/orchestrator.form.toml
[metadata]
name = "Orchestrator Configuration"
description = "Configure the Orchestrator service"
version = "1.0.0"
schema = "orchestrator.ncl"
[fields.mode]
type = "enum"
label = "Deployment Mode"
description = "Select deployment mode: solo, multiuser, or enterprise"
options = ["solo", "multiuser", "enterprise"]
default = "solo"
required = true
[fields.server.port]
type = "number"
label = "Server Port"
description = "HTTP server port (1-65535)"
min = 1
max = 65535
default = 8080
required = true
[fields.database.host]
type = "string"
label = "Database Host"
description = "PostgreSQL host"
default = "localhost"
required = true
[fields.logging.level]
type = "enum"
label = "Logging Level"
options = ["debug", "info", "warning", "error"]
default = "info"
required = false
```
## Validation Constraints
```toml
# provisioning/platform/.typedialog/constraints/constraints.toml
[orchestrator]
mode = ["solo", "multiuser", "enterprise"]
port = "range(1, 65535)"
database_pool_size = "range(1, 100)"
memory = "pattern(^\\d+[MG]B$)"
[control-center]
port = "range(1, 65535)"
replicas = "range(1, 10)"
[nginx]
worker_processes = "range(1, 32)"
worker_connections = "range(1, 65536)"
```
## Workflow: Setup to Deployment
```toml
1. User runs setup command
↓
2. TypeDialog displays form
↓
3. User fills form with validation
↓
4. Form data → Nickel config
↓
5. Nickel config → TOML (via ConfigLoader)
↓
6. Service reads TOML config
↓
7. Service starts with configured values
```
## Benefits of TypeDialog Integration
- ✅ **Type-safe forms** - Generated from Nickel schemas
- ✅ **Real-time validation** - Enforce constraints as user types
- ✅ **Progressive disclosure** - Show advanced options only when needed
- ✅ **Consistent UX** - Same forms across platforms (CLI, Web, TUI)
- ✅ **Auto-generated** - Forms stay in sync with schemas automatically
- ✅ **TTY handling** - Bash wrappers solve Nushell input stack issues
- ✅ **Graceful fallback** - Falls back to basic prompts if TypeDialog unavailable
## Testing TypeDialog Forms
```toml
# Validate form structure
typedialog check-form provisioning/platform/.typedialog/forms/orchestrator.form.toml
# Run form with test data
typedialog run-form
--form provisioning/platform/.typedialog/forms/orchestrator.form.toml
--test-mode # Automated validation
# Generate sample output
typedialog generate-sample
--form provisioning/platform/.typedialog/forms/orchestrator.form.toml
--output /tmp/orchestrator-sample.ncl
```
## Migration Path
### Phase A: Legacy (DEPRECATED)
```toml
FormInquire (Jinja2) → Nushell processing → TOML config
Status: ARCHIVED to .coder/archive/forminquire/
```
### Phase B: Current Implementation
```toml
Bash wrapper → TypeDialog (TTY input) → Nickel config → JSON export → Nushell reads JSON
Status: IMPLEMENTED with forms ready
```
### Phase C: TypeDialog Binary Available (Future)
```toml
TypeDialog binary installed → Full nickel-roundtrip workflow → Auto-sync with schemas
Status: PLANNED - awaiting TypeDialog binary release
```
### Phase D: Unified (Future)
```toml
ConfigLoader discovers config → Service reads → TypeDialog updates UI
```
## Integration with Infrastructure Schemas
TypeDialog forms work seamlessly with infrastructure schemas:
### Infrastructure Configuration Workflow
**1. Define Infrastructure Schemas** (completed)
- Location: `provisioning/schemas/infrastructure/`
- 6 schemas: docker-compose, kubernetes, nginx, prometheus, systemd, oci-registry
- All validated with `nickel typecheck`
**2. Generate Infrastructure Configs** (completed)
- Script: `provisioning/platform/scripts/generate-infrastructure-configs.nu`
- Supports: solo, multiuser, enterprise, cicd modes
- Formats: YAML, JSON, conf, service
**3. Validate Generated Configs** (completed)
- Script: `provisioning/platform/scripts/validate-infrastructure.nu`
- Tools: docker-compose config, kubectl apply --dry-run, nginx -t, promtool check
- Examples: `examples-solo-deployment.ncl`, `examples-enterprise-deployment.ncl`
**4. Interactive Setup with Forms** (TypeDialog ready)
- Script: `provisioning/platform/scripts/setup-with-forms.sh`
- Bash wrappers: `provisioning/core/shlib/*-tty.sh` (handle TTY input)
- Forms ready: setup-wizard, auth-login, mfa-enroll
- Fallback: Basic Nushell prompts if TypeDialog unavailable
### Current Status: Full Infrastructure Support
| Component | Status | Details |
| ----------- | -------- | --------- |
| **Schemas** | ✅ Complete | 6 infrastructure schemas (1,577 lines) |
| **Examples** | ✅ Complete | 2 deployment examples (solo, enterprise) |
| **Generation Script** | ✅ Complete | Auto-generates configs for all modes |
| **Validation Script** | ✅ Complete | Validates Docker, K8s, Nginx, Prometheus |
| **Setup Wizard** | ✅ Complete | TypeDialog forms + bash wrappers ready |
| **TypeDialog Integration** | ⏳ Pending | Structure ready, awaiting binary |
### Validated Examples
**Solo Deployment** (`examples-solo-deployment.ncl`):
- ✅ Type-checks without errors
- ✅ Exports to 198 lines of JSON
- ✅ 5 Docker Compose services
- ✅ Resource limits: 1.0-4.0 CPU, 256M-1024M RAM
- ✅ Prometheus: 4 scrape jobs
- ✅ Registry backend: Zot (filesystem)
**Enterprise Deployment** (`examples-enterprise-deployment.ncl`):
- ✅ Type-checks without errors
- ✅ Exports to 313 lines of JSON
- ✅ 6 Docker Compose services with HA
- ✅ Resource limits: 2.0-4.0 CPU, 512M-4096M RAM
- ✅ Prometheus: 7 scrape jobs with remote storage
- ✅ Registry backend: Harbor (S3 distributed)
### Test Infrastructure Generation
```toml
# Export solo infrastructure
nickel export --format json provisioning/schemas/infrastructure/examples-solo-deployment.ncl > /tmp/solo.json
# Validate JSON
jq . /tmp/solo.json
# Check Docker Compose services
jq '.docker_compose_services | keys' /tmp/solo.json
# Compare resource allocation (solo vs enterprise)
jq '.docker_compose_services.orchestrator.deploy.resources.limits' /tmp/solo.json
jq '.docker_compose_services.orchestrator.deploy.resources.limits' /tmp/enterprise.json
```
## Next Steps
1. **Infrastructure Setup** (available now):
- Generate infrastructure configs with automation scripts
- Validate with format-specific tools
- Use interactive setup wizard for configuration
2. **When TypeDialog binary becomes available**:
- Install TypeDialog binary
- Forms already created and ready to use
- Bash wrappers handle TTY input (no Nushell stack issues)
- Full nickel-roundtrip workflow will be enabled
3. **Production Deployment**:
- Use validated infrastructure configs
- Deploy with ConfigLoader + infrastructure schemas
- Monitor via Prometheus (auto-generated from schemas)
---
**Version**: 1.2.0 (TypeDialog Forms + Bash Wrappers)
**Status**: TypeDialog forms ready with bash wrappers; Awaiting TypeDialog Binary
**Last Updated**: 2025-01-09
**ForminQuire Status**: DEPRECATED - Archived to .coder/archive/forminquire/
**Fallback**: Basic Nushell prompts if TypeDialog unavailable
**Tested**: Infrastructure examples (solo + enterprise) validated

63
.typedialog/provisioning/platform/constraints/constraints.toml Normal file
View file

@ -0,0 +1,63 @@
# TypeDialog Validation Constraints
# Defines validation rules for form fields generated from Nickel schemas
[orchestrator]
cpus = "pattern(^[0-9]+(\\.[0-9]+)?$)"
db_pool_size = "range(1, 100)"
log_level = ["debug", "info", "warning", "error"]
memory = "pattern(^[0-9]+[MG]B$)"
mode = ["solo", "multiuser", "enterprise", "cicd"]
port = "range(1, 65535)"
replicas = "range(1, 10)"
[control-center]
log_level = ["debug", "info", "warning", "error"]
port = "range(1, 65535)"
replicas = "range(1, 10)"
[vault-service]
cpus = "pattern(^[0-9]+(\\.[0-9]+)?$)"
memory = "pattern(^[0-9]+[MG]B$)"
port = "range(1, 65535)"
[rag]
max_concurrent_requests = "range(1, 100)"
port = "range(1, 65535)"
timeout_seconds = "range(1, 3600)"
[extension-registry]
port = "range(1, 65535)"
storage_path = "pattern(^/[a-zA-Z0-9/_-]+$)"
[mcp-server]
max_connections = "range(1, 1000)"
port = "range(1, 65535)"
[provisioning-daemon]
max_workers = "range(1, 100)"
port = "range(1, 65535)"
[ai-service]
max_retries = "range(0, 10)"
model_timeout_seconds = "range(1, 3600)"
port = "range(1, 65535)"
[nginx]
client_max_body_size = "pattern(^[0-9]+[MG]B$)"
worker_connections = "range(1, 65536)"
worker_processes = "range(1, 32)"
[prometheus]
evaluation_interval = "pattern(^[0-9]+[smh]$)"
retention = "pattern(^[0-9]+[dhw]$)"
scrape_interval = "pattern(^[0-9]+[smh]$)"
[kubernetes]
cpu = "pattern(^[0-9]+m$|^[0-9]+(\\.[0-9]+)?$)"
memory = "pattern(^[0-9]+Mi$|^[0-9]+Gi$)"
replicas = "range(1, 100)"
[docker-compose]
cpus = "pattern(^[0-9]+(\\.[0-9]+)?$)"
memory = "pattern(^[0-9]+[MG]B$)"
port = "range(1, 65535)"

1
.typedialog/provisioning/platform/schemas/schemas Symbolic link
View file

@ -0,0 +1 @@
/Users/Akasha/project-provisioning/provisioning/schemas

77
.typedialog/provisioning/platform/templates/service-form.template.j2 Normal file
View file

@ -0,0 +1,77 @@
{# Jinja2 template for service configuration form #}
{# This template is used as a reference for schema-to-form transformation #}
{# When TypeDialog is available, forms will be auto-generated from Nickel schemas #}
# {{ service_name }} Configuration Form
# Mode: {{ deployment_mode }}
# Auto-generated from schema: {{ schema_path }}
## Service Settings
### Server Configuration
- **Server Port** (1-65535)
Value: {{ server.port | default("8080") }}
Description: HTTP server port
- **TLS Enabled** (true/false)
Value: {{ server.tls.enabled | default("false") }}
Description: Enable HTTPS/TLS
{% if server.tls.enabled %}
- **TLS Certificate Path**
Value: {{ server.tls.cert_path | default("") }}
- **TLS Key Path**
Value: {{ server.tls.key_path | default("") }}
{% endif %}
### Database Configuration
- **Database Host**
Value: {{ database.host | default("localhost") }}
- **Database Port** (1-65535)
Value: {{ database.port | default("5432") }}
- **Database Name**
Value: {{ database.name | default("provisioning") }}
- **Connection Pool Size** (1-100)
Value: {{ database.pool_size | default("10") }}
### Deployment Configuration
- **Deployment Mode**
Options: solo, multiuser, enterprise, cicd
Value: {{ mode | default("solo") }}
- **Number of Replicas** (1-10)
Value: {{ replicas | default("1") }}
- **CPU Limit**
Value: {{ deploy.resources.limits.cpus | default("1.0") }}
Format: e.g., "1.0", "2.5", "4.0"
- **Memory Limit**
Value: {{ deploy.resources.limits.memory | default("1024M") }}
Format: e.g., "512M", "1024M", "2G"
### Logging Configuration
- **Log Level**
Options: debug, info, warning, error
Value: {{ logging.level | default("info") }}
- **Log Format**
Options: json, text
Value: {{ logging.format | default("json") }}
### Monitoring Configuration
- **Enable Metrics**
Value: {{ monitoring.enabled | default("true") }}
- **Metrics Port** (1-65535)
Value: {{ monitoring.metrics_port | default("9090") }}
{% if monitoring.enabled %}
- **Scrape Interval**
Value: {{ monitoring.scrape_interval | default("15s") }}
Format: e.g., "15s", "1m", "5m"
{% endif %}

138
CHANGELOG.md Normal file
View file

@ -0,0 +1,138 @@
# Platform Changelog
## [Unreleased]
### Added
#### ops control plane (`crates/ops-keeper`, `crates/ops-controller`)
New two-component ops control plane implementing secure, auditable operation dispatch.
**ops-keeper** — policy gate before any operation reaches the orchestrator:
- Glob-based `PolicyDef` matching on `op_type`, `image_patterns`, `target_patterns`
- `Signer` issues compact Ed25519 JWTs (`OpsClaims`) on policy approval
- NATS JetStream pull consumer on `ops.pending.*`
- `AuditEvent` emission to `ops.audit.*` on every terminal decision
**ops-controller** — durable executor with at-least-once delivery:
- Verifies keeper-signed JWT before dispatching to orchestrator HTTP API
- SurrealDB idempotency store; `reconcile_pending` reconciles stale ops on startup
- Structured `AckResult` (Ack / Nak / Term) per JetStream semantics
- Emits audit events to `ops.audit.*` on every terminal outcome
**ADR**: ADR-038 (ops control plane design)
#### audit-mirror (`crates/audit-mirror`)
Sidecar that consumes `ops.audit.*` and creates a signed git commit per event in a local Radicle repository, then announces the repo. JTI deduplication prevents double-commits on consumer redelivery.
**ADR**: ADR-038
#### contract-tests (`crates/contract-tests`)
G3 contract test suite verifying semantic equivalence across three tiers:
- **Tier A**: direct registry invocation (reference baseline)
- **Tier B**: axum HTTP daemon on ephemeral port
- **Tier C**: in-process MCP `handle_request`
Normaliser strips volatile fields; JSON schema validates every tier output against `listing_output_schema`. Regression guard for CLI↔HTTP↔MCP drift.
#### control-center: NATS→WebSocket bridge (`src/handlers/nats_bridge.rs`)
Eliminates long-polling between control-center UI and orchestrator. Durable JetStream consumer `cc-task-status-bridge` subscribes to `TASKS` stream and re-broadcasts each `TaskStatusPayload` to all connected WebSocket clients via `WebSocketManager`.
#### API catalog endpoints (`crates/ai-service`, `crates/control-center`)
`GET /api-catalog` added to ai-service and control-center. Each handler uses `inventory::iter::<ApiRouteEntry>()` to emit a sorted JSON catalog of all registered routes — self-describing API surface without a separate OpenAPI toolchain.
#### SOLID boundary pre-commit check (`.pre-commit-hooks/solid-boundary-check.sh`)
Shell hook enforced in `.pre-commit-config.yaml`. Fails the commit if a crate imports across declared SOLID layer boundaries, keeping dependency direction consistent before code reaches CI.
### Removed / Lifted
- **`crates/backup-manager`** — lifted to `cloudatasave` (LibreCloud/cloudDataSave). **ADR-041**.
- **`crates/buildkit-launcher`** — lifted to `lian-build`. **ADR-040**.
- **`crates/daemon`** — `Cargo.toml` removed; binary consolidated into `provisioning-daemon`.
- **`prov-ecosystem/crates/nu-daemon`** / **`daemon-cli`** — excluded from workspace; `rustls=0.23.28` pin conflicts with surrealdb@3 (`^0.23.36`). Build standalone with `cargo build -p nu-daemon` until `nu-command` relaxes the pin.
---
## [3.6.0] — 2026-04-17
### Added
#### ncl-sync daemon (`crates/ncl-sync/`)
New Rust daemon that eliminates `nickel export` latency from CLI commands by pre-compiling NCL files and maintaining a shared cache.
- **File watching**: `notify` watcher on workspace NCL directories; re-exports automatically on file change
- **Warm-up**: on `prvng platform start`, scans all NCL files in the workspace and exports stale entries concurrently (configurable concurrency)
- **Shared cache**: `~/.cache/provisioning/config-cache/` — same directory and key strategy as `nu_plugin_nickel`
- **Content-addressed keys**: `SHA256(file_content + sorted_import_paths + format)` — zero coordination overhead between daemon and plugin
- **Sync requests**: Nu processes write `.sync-<pid>.json` sidecars after mutations; daemon drains them within 500 ms
- **CLI subcommands**: `daemon`, `warm`, `invalidate`, `key`, `stats`
- **Config-driven** via `platform-config`: `platform/config/ncl-sync.ncl` — idle timeout, concurrency, poll interval, extra import paths
- **No platform dependencies**: intentionally avoids NATS, SurrealDB, and orchestrator to prevent bootstrap circularity
#### Platform lifecycle integration
- `prvng platform start` now launches `ncl-sync daemon` (via `ncl-sync-start` in `service-manager.nu`)
- `prvng platform stop` stops ncl-sync (via PID file)
- `prvng platform status` shows ncl-sync running state
#### Nu cache layer
- `lib_provisioning/config/cache/core.nu`: implemented `cache-lookup`, `cache-write`, `write-sync-request` (previously no-ops)
- `lib_provisioning/config/cache/nickel.nu`: implemented `lookup-nickel-cache`, `derive-ncl-cache-key`, `request-ncl-sync`
#### nickel_processor wrappers
New `lib_provisioning/utils/nickel_processor.nu` functions:
- `ncl-eval`: drop-in for `^nickel export ... | from json` — checks plugin cache, propagates error on failure
- `ncl-eval-soft`: soft-failure variant with configurable fallback value
#### nu_plugin_nickel updates
- `nickel-eval` and `nickel-export` now accept `--import-path LIST` flag (`-I`)
- New command `nickel-cache-key`: prints the cache key for a file (parity testing)
- Cache key derivation updated to `SHA256(file_content + sorted_import_paths + format)` — aligned with ncl-sync
- Cache key now includes import paths: same file with different import paths produces different cache entries
#### Hot-path migration (C1)
Replaced `^nickel export ... | from json` with `ncl-eval`/`ncl-eval-soft` in the four highest-frequency call sites:
- `main_provisioning/dispatcher.nu` — commands-registry load
- `main_provisioning/components.nu` — workspace settings export
- `main_provisioning/workflow.nu` — workflow NCL exports
- `main_provisioning/extensions.nu` — per-extension metadata
#### Extended migration (C2 + C3)
Migrated remaining high-value call sites (~55 additional sites across 30+ files) including `config/export.nu`, `taskservs/discover.nu`, `servers/create.nu`, `ontoref-queries.nu`, `service-manager.nu`, `dag.nu`, and others.
#### ADRs
- `adrs/adr-022-ncl-sync-daemon.ncl`: daemon design decisions, key strategy rationale, constraint enforcement
- `adrs/adr-023-ncl-export-wrapper.ncl`: `ncl-eval`/`ncl-eval-soft` wrapper design, migration strategy
#### Tests
- `tests/cache/test_key_parity.nu`: validates that ncl-sync and nu_plugin_nickel produce identical keys for the same `(file, import_paths, format)` triple
### Performance
| Command | Before | After (warm cache) |
|---------|--------|--------------------|
| `prvng component list` | 3–7 s | ~1.5 s |
| `prvng workflow list` | 3–5 s | ~1.5 s |
| `prvng deploy` | 15–30 s | ~3–5 s |
| Multi-export commands (ontoref) | 12–30 s | ~1.5 s |
Nu module parse startup (~1.2 s) is unaffected — separate concern.
---
## [3.5.0] — 2025-10-07
Initial platform version with orchestrator, control center, installer, MCP server, vault service, and extension registry.

489
Cargo.toml Normal file
View file

@ -0,0 +1,489 @@
[workspace]
members = [
"crates/platform-config",
"crates/platform-nats",
"crates/platform-db",
"crates/service-clients",
"crates/ai-service",
"crates/catalog-registry",
"crates/orchestrator",
"crates/control-center",
"crates/control-center-ui",
"crates/vault-service",
"crates/mcp-server",
# lifted: "crates/backup-manager" → cloudatasave (LibreCloud/cloudDataSave) — adr-041
# archived: "crates/detector" → archive/detector (no dependents, stale since Jan 2026)
"prov-ecosystem/crates/machines",
"prov-ecosystem/crates/encrypt",
"prov-ecosystem/crates/backup",
"prov-ecosystem/crates/observability",
"crates/ontoref-compute",
"crates/ontoref-dns",
"crates/ncl-sync",
"crates/prvng-cli",
"crates/provisioning-core",
"crates/provisioning-tool",
"crates/provisioning-daemon",
"crates/provisioning-desktop",
"crates/contract-tests",
"crates/extension-manager",
"crates/ops-keeper",
"crates/audit-mirror",
"crates/ops-controller",
# lifted: "crates/buildkit-launcher" → lian-build — adr-040
]
exclude = [
# archived: syntaxis/ → archive/syntaxis (Jan 2026, all refs commented out)
# archived: stratumiops/ → archive/stratumiops (Jan 2026, workspace uses canonical ../../../Development/stratumiops)
"prov-ecosystem/crates/syntaxis-integration",
"prov-ecosystem/crates/audit",
"prov-ecosystem/crates/valida",
"prov-ecosystem/crates/runtime",
"prov-ecosystem/crates/gitops",
# nu-daemon + daemon-cli are excluded: nu-command@0.110.0 (via nushell feature) pins
# rustls=0.23.28, hard conflict with surrealdb@3 (requires ^0.23.36). Not resolvable
# until nu-command relaxes its rustls pin. Build standalone: cargo build -p nu-daemon
"crates/nu-daemon",
"prov-ecosystem/crates/daemon-cli",
]
resolver = "2"
[workspace.package]
authors = ["Jesus Perez <jesus@librecloud.online>"]
edition = "2021"
license = "MIT"
repository = "https://github.com/jesusperezlorenzo/provisioning"
version = "1.0.11"
[workspace.dependencies]
# ============================================================================
# SHARED ASYNC RUNTIME AND CORE LIBRARIES
# ============================================================================
async-trait = "0.1"
futures = "0.3"
tokio = { version = "1.49", features = ["full"] }
tokio-util = "0.7"
# ============================================================================
# SERIALIZATION AND DATA HANDLING
# ============================================================================
chrono = { version = "0.4", features = ["serde"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
toml = "0.9"
uuid = { version = "1.20", features = ["v4", "serde"] }
# ============================================================================
# ERROR HANDLING
# ============================================================================
anyhow = "1.0"
thiserror = "2.0"
# ============================================================================
# LOGGING AND TRACING
# ============================================================================
log = "0.4"
tracing = "0.1"
tracing-appender = "0.2"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
# ============================================================================
# WEB SERVER AND NETWORKING
# ============================================================================
axum = { version = "0.8", features = ["ws", "macros"] }
hyper = "1.8"
reqwest = { version = "0.13", features = ["json", "rustls"], default-features = false }
tower = { version = "0.5", features = ["full"] }
tower-http = { version = "0.6", features = [
"cors",
"trace",
"fs",
"compression-gzip",
"timeout",
] }
# ============================================================================
# CLI AND CONFIGURATION
# ============================================================================
clap = { version = "4.5", features = ["derive", "env"] }
config = "0.15"
# ============================================================================
# DATABASE AND STORAGE
# ============================================================================
sqlx = { version = "0.8", features = ["runtime-tokio-rustls", "sqlite", "chrono", "uuid"] }
# kv-surrealkv: core relational/graph (orchestrator state, control-center)
# kv-rocksdb: hot data (embeddings cache, audit logs) — via platform-db embedded-rocksdb feature
# rustls excluded: nu-command@0.110.0 pins rustls=0.23.28, SurrealDB 3 requires ^0.23.36 (conflict)
# TLS for remote connections is handled at the proxy layer (nginx/Caddy) in production.
surrealdb = { version = "3", features = ["kv-mem", "kv-surrealkv", "protocol-ws"], default-features = false }
# ============================================================================
# MESSAGING (NATS)
# ============================================================================
async-nats = "0.46"
# ============================================================================
# SECURITY AND CRYPTOGRAPHY
# ============================================================================
aes-gcm = "0.10"
argon2 = "0.5"
base64 = "0.22"
git2 = { version = "0.20", default-features = false, features = ["https", "ssh"] }
hmac = "0.12"
jsonwebtoken = { version = "10.3", default-features = false, features = ["aws_lc_rs"] }
rand = { version = "0.9", features = ["std_rng", "os_rng"] }
ring = "0.17"
sha2 = "0.10"
# AWS SDK for KMS
aws-config = "1"
aws-credential-types = "1"
aws-sdk-kms = "1"
# ============================================================================
# VALIDATION AND REGEX
# ============================================================================
regex = "1.12"
validator = { version = "0.20", features = ["derive"] }
globset = "0.4"
# ============================================================================
# GRAPH ALGORITHMS AND UTILITIES
# ============================================================================
petgraph = "0.8"
# ============================================================================
# CONCURRENT DATA STRUCTURES
# ============================================================================
dashmap = "6"
# ============================================================================
# ADDITIONAL SHARED DEPENDENCIES
# ============================================================================
# System utilities
dirs = "6.0"
# Filesystem operations
notify = "8.2"
walkdir = "2.5"
# Statistics and templates
statistics = "0.4"
tera = "1.20"
# Additional cryptography
hkdf = "0.12"
rsa = "0.9.10"
zeroize = { version = "1.8", features = ["derive"] }
# Additional security
constant_time_eq = "0.4"
subtle = "2.6"
# Caching and storage
redis = { version = "1.0", features = ["tokio-comp", "connection-manager"] }
# Tower services
tower-service = "0.3"
tower_governor = "0.8"
# Scheduling
cron = "0.15"
tokio-cron-scheduler = "0.15"
# Policy engine
cedar-policy = "4.8"
# URL handling
url = "2.5"
# Icons and UI
icondata = "0.7"
leptos_icons = "0.7"
# Image processing
image = { version = "0.25", default-features = false, features = ["png"] }
qrcode = "0.14"
# Authentication
hex = "0.4"
lazy_static = "1.5"
totp-rs = { version = "5.7", features = ["qr"] }
webauthn-rs = "0.5"
webauthn-rs-proto = "0.5"
# Additional serialization
serde-wasm-bindgen = "0.6"
# Gloo utilities (for WASM)
gloo-net = { version = "0.6", features = ["http", "websocket"] }
gloo-storage = "0.3"
gloo-timers = "0.3"
gloo-utils = { version = "0.2", features = ["serde"] }
# Plotting and canvas
plotters = "0.3"
plotters-canvas = "0.3"
# WASM utilities
console_error_panic_hook = "0.1"
js-sys = "0.3"
tracing-wasm = "0.2"
wasm-bindgen-futures = "0.4"
# Random number generation
getrandom = { version = "0.4" }
# ============================================================================
# TUI (Terminal User Interface)
# ============================================================================
crossterm = "0.29"
ratatui = { version = "0.30", features = ["all-widgets", "serde"] }
# ============================================================================
# WASM AND FRONTEND DEPENDENCIES (for control-center-ui)
# ============================================================================
leptos = { version = "0.8", features = ["csr"] }
leptos_meta = { version = "0.8", features = ["default"] }
leptos_router = { version = "0.8" }
wasm-bindgen = "0.2"
# ============================================================================
# DEVELOPMENT AND TESTING DEPENDENCIES
# ============================================================================
assert_matches = "1.5"
criterion = { version = "0.8", features = ["html_reports"] }
mockito = "1"
tempfile = "3.24"
tokio-test = "0.4"
# Additional caching and binary discovery
lru = "0.16"
parking_lot = "0.12"
which = "8"
yaml-rust = "0.4"
humantime-serde = "1.1"
# Metrics
prometheus = "0.14"
approx = "0.5"
# Utilities
xxhash-rust = { version = "0.8", features = ["xxh3"] }
# ============================================================================
# RAG AND TEXT PROCESSING
# ============================================================================
tokenizers = "0.22"
# ============================================================================
# STRATUM ECOSYSTEM DEPENDENCIES (for RAG embeddings & LLM)
# ============================================================================
moka = { version = "0.12", features = ["future"] }
sled = "0.34"
fastembed = "5.11"
lancedb = "0.26"
arrow = "=56"
# ============================================================================
# INTERNAL WORKSPACE CRATES (Local path dependencies)
# ============================================================================
platform-config = { path = "./crates/platform-config" }
platform-nats = { path = "./crates/platform-nats" }
platform-db = { path = "./crates/platform-db" }
platform-clients = { path = "./crates/service-clients" }
platform-rag = { path = "./crates/rag" }
provisioning-mcp = { path = "./crates/mcp-server" }
ai-service = { path = "./crates/ai-service" }
# ============================================================================
# PROV-ECOSYSTEM (Now members of workspace)
# ============================================================================
daemon-cli = { path = "./prov-ecosystem/crates/daemon-cli" }
platform-machines = { path = "./prov-ecosystem/crates/machines" }
platform-encrypt = { path = "./prov-ecosystem/crates/encrypt" }
platform-backup = { path = "./prov-ecosystem/crates/backup" }
platform-observability = { path = "./prov-ecosystem/crates/observability" }
init-servs = { path = "./prov-ecosystem/crates/init-servs" }
# ============================================================================
# ONTOREF PROTOCOL ADOPTION (API catalog surface — Phase 4)
# ============================================================================
ontoref-ontology = { path = "../../ontoref/code/crates/ontoref-ontology", features = ["derive"] }
ontoref-derive = { path = "../../ontoref/code/crates/ontoref-derive" }
inventory = "0.3"
# ============================================================================
# ONTOREF FOUNDATION SIBLINGS (Task D.2 — provisioning identity shift)
# ============================================================================
# ontoref-dns: provider-agnostic DNS lifecycle. Replaces internal CoreDNS client.
ontoref-dns = { path = "./crates/ontoref-dns", default-features = false, features = ["coredns"] }
# ontoref-compute: provider-agnostic compute lifecycle. Used by policy layer
# (BatchWorkflowEngine, InfraStateService) for fleet operations against Hetzner.
# HcloudClient remains in provisioning-core for volume/raw-JSON operations that
# do not fit the ComputeProvider lifecycle surface (see adrs/adr-042).
ontoref-compute = { path = "./crates/ontoref-compute", default-features = false, features = ["hetzner"] }
# Stratum ecosystem — sourced from canonical stratumiops repo (SurrealDB v3 throughout)
stratum-embeddings = { path = "../../stratumiops/code/crates/stratum-embeddings", features = ["openai-provider", "ollama-provider", "fastembed-provider", "memory-cache"] }
stratum-llm = { path = "../../stratumiops/code/crates/stratum-llm", features = ["anthropic", "openai", "ollama"] }
stratum-graph = { path = "../../stratumiops/code/crates/stratum-graph" }
stratum-state = { path = "../../stratumiops/code/crates/stratum-state" }
# ============================================================================
# SECRETUMVAULT (Enterprise Secrets Management - canonical source)
# ============================================================================
secretumvault = { path = "../../secretumvault", features = ["surrealdb-storage", "filesystem", "server", "cedar"] }
# ============================================================================
# WASM/WEB-SPECIFIC DEPENDENCIES
# ============================================================================
web-sys = { version = "0.3", features = [
"console",
"Window",
"Document",
"Element",
"HtmlElement",
"HtmlCanvasElement",
"CanvasRenderingContext2d",
"EventTarget",
"Event",
"DragEvent",
"DataTransfer",
"HtmlInputElement",
"HtmlSelectElement",
"HtmlTextAreaElement",
"HtmlButtonElement",
"HtmlDivElement",
"Storage",
"Location",
"History",
"Navigator",
"ServiceWorkerRegistration",
"ServiceWorker",
"NotificationPermission",
"Notification",
"Headers",
"Request",
"RequestInit",
"RequestMode",
"Response",
"AbortController",
"AbortSignal",
"WebSocket",
"MessageEvent",
"CloseEvent",
"ErrorEvent",
"Blob",
"Url",
"FileReader",
"File",
"HtmlAnchorElement",
"MouseEvent",
"TouchEvent",
"KeyboardEvent",
"ResizeObserver",
"ResizeObserverEntry",
"IntersectionObserver",
"IntersectionObserverEntry",
"MediaQueryList",
"MediaQueryListEvent",
"CredentialsContainer",
"PublicKeyCredential",
"PublicKeyCredentialCreationOptions",
"PublicKeyCredentialRequestOptions",
"AuthenticatorResponse",
"AuthenticatorAttestationResponse",
"AuthenticatorAssertionResponse",
"Crypto",
"SubtleCrypto",
"CryptoKey",
] }
# ============================================================================
# ADDITIONAL MISSING DEPENDENCIES (Not in original workspace)
# ============================================================================
ed25519-dalek = "2.2"
http-body-util = "0.1"
# ============================================================================
# BYTES MANIPULATION
# ============================================================================
bytes = "1.11"
# ============================================================================
# HTTP AND PROTOCOL UTILITIES
# ============================================================================
http = "1"
# ============================================================================
# CONTAINER MANAGEMENT AND SSH
# ============================================================================
bollard = "0.20"
russh = "0.57"
russh-keys = "0.49"
# ============================================================================
# SECRETS MANAGEMENT
# ============================================================================
age = "0.11"
rusty_vault = "0.2.1"
# ============================================================================
# ADDITIONAL DATA FORMAT SERIALIZATION
# ============================================================================
serde_yaml = "0.9"
# ============================================================================
# PATH AND SHELL UTILITIES
# ============================================================================
shellexpand = "3.1"
# ============================================================================
# DESKTOP APP (Tauri 2 — provisioning-desktop)
# ============================================================================
tauri = { version = "2", features = ["tray-icon"] }
tauri-build = { version = "2", default-features = false }
tauri-plugin-notification = "2"
tauri-plugin-shell = "2"
tauri-plugin-dialog = "2"
keyring = "3"
[workspace.metadata]
description = "Provisioning Platform - Rust workspace for cloud infrastructure automation tools"
# Profile configurations shared across all workspace members
[profile.dev]
codegen-units = 256
debug = true
debug-assertions = true
incremental = true
lto = false
opt-level = 0
overflow-checks = true
panic = 'unwind'
[profile.release]
codegen-units = 1
lto = true
opt-level = 3
panic = "abort"
strip = "debuginfo"
# Fast release profile for development
[profile.dev-release]
debug = true
inherits = "release"
lto = "thin"
opt-level = 2
# Profile for benchmarks
[profile.bench]
debug = true
inherits = "release"

78
config/external-services.ncl Normal file
View file

@ -0,0 +1,78 @@
# External Infrastructure Services Configuration
# Defines the external services (databases, registries, CI/CD, etc.) that the platform integrates with
# These services are NOT managed by provisioning, only monitored for health/status
#
# Schema validation: Loaded from provisioning/schemas/platform/external-services.ncl
let schema = import "schemas/platform/external-services.ncl" in
[
# SecretumVault - Secrets management and encryption
({
name = "svault_server-vault",
srvc = "vault",
desc = "SecretumVault server for secrets management and encryption",
url = "http://127.0.0.1:8082",
port = 8082,
required = true,
dependencies = [],
binary_path = "~/.local/bin/svault",
startup_command = "svault server --config ~/.config/provisioning/secretumvault-dev.toml",
health_check_timeout = 5,
} | schema.ExternalService),
# SurrealDB - Multi-model database
({
name = "surrealdb-dbs",
srvc = "dbs",
desc = "SurrealDB multi-model database for data storage and queries",
url = "http://127.0.0.1:8000",
port = 8000,
required = true,
dependencies = [],
} | schema.ExternalService),
# PostgreSQL - Database for Forgejo and Woodpecker
({
name = "postgresql-db",
srvc = "postgres",
desc = "PostgreSQL database for Forgejo and Woodpecker services",
url = "postgresql://127.0.0.1:5432",
port = 5432,
required = false,
dependencies = [],
} | schema.ExternalService),
# Forgejo - Git server
({
name = "forgejo-git",
srvc = "git",
desc = "Forgejo Git server for version control and collaboration",
url = "http://127.0.0.1:3000",
port = 3000,
required = false,
dependencies = ["postgresql-db"],
} | schema.ExternalService),
# Zot - OCI container registry
({
name = "zot-register",
srvc = "register",
desc = "Zot OCI-compliant container registry for container images",
url = "http://127.0.0.1:5001",
port = 5001,
required = false,
dependencies = [],
} | schema.ExternalService),
# Woodpecker - CI/CD pipeline engine
({
name = "woodpecker-cdci",
srvc = "cdci",
desc = "Woodpecker CI/CD pipeline engine for automation and testing",
url = "http://127.0.0.1:8180",
port = 8180,
required = false,
dependencies = ["forgejo-git", "postgresql-db"],
} | schema.ExternalService),
]

23
config/ncl-sync.ncl Normal file
View file

@ -0,0 +1,23 @@
let schema = import "schemas/platform/ncl-sync.ncl" in
{
ncl_sync | schema.NclSyncConfig = {
idle_timeout_secs = 600,
sync_poll_interval_ms = 500,
warm_concurrency = 4,
# ontoref root (installed) — resolves imports like `reflection/schemas/*.ncl`
# and `ontology/defaults/*.ncl` used by workspace .ontology/ and reflection/ files.
# Run `ontoref` without args to see the current Root.
extra_import_paths = [
"~/Library/Application Support/ontoref",
],
# NATS event-driven cache invalidation (opt-in).
# Requires: platform-nats running, and a publisher emitting
# provisioning.workspace.ncl.{changed,removed} events.
nats = {
enabled = true,
url = "",
},
},
}

74
crates/ai-service/Cargo.toml Normal file
View file

@ -0,0 +1,74 @@
[package]
authors.workspace = true
description = "HTTP service for AI capabilities including RAG, MCP tool invocation, and knowledge graph operations"
edition.workspace = true
name = "ai-service"
version.workspace = true
[[bin]]
name = "provisioning-ai-service"
path = "src/main.rs"
[dependencies]
# Workspace dependencies
async-trait = { workspace = true }
futures = { workspace = true }
tokio = { workspace = true, features = ["full"] }
# Web server and API
axum = { workspace = true }
tower = { workspace = true, features = ["full"] }
tower-http = { workspace = true, features = ["cors", "trace"] }
# Ontoref API catalog
ontoref-ontology = { workspace = true }
ontoref-derive = { workspace = true }
inventory = { workspace = true }
# Serialization
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true }
toml = { workspace = true }
# Platform configuration
platform-config = { workspace = true }
# Centralized observability (logging, metrics, health, tracing)
platform-observability = { workspace = true, features = ["logging", "metrics-prometheus", "health"] }
# Error handling
anyhow = { workspace = true }
thiserror = { workspace = true }
# Logging
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
# UUID and time
chrono = { workspace = true, features = ["serde"] }
uuid = { workspace = true, features = ["v4", "serde"] }
# CLI
clap = { workspace = true, features = ["derive"] }
# RAG crate for AI capabilities
platform-rag = { workspace = true }
# MCP server tools for real implementations
provisioning-mcp = { workspace = true }
# Graph operations for DAG
petgraph = { workspace = true }
# Stratum ecosystem - embeddings and LLM abstraction (optional - requires external setup)
stratum-embeddings = { workspace = true }
stratum-llm = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }
tokio-test = { workspace = true }
# Library target
[lib]
name = "ai_service"
path = "src/lib.rs"

12
crates/ai-service/src/api_catalog.rs Normal file
View file

@ -0,0 +1,12 @@
use axum::{extract::State, response::IntoResponse, Json};
use ontoref_ontology::api::ApiRouteEntry;
use serde_json::json;
use std::sync::Arc;
use crate::service::AiService;
pub async fn api_catalog(State(_state): State<Arc<AiService>>) -> impl IntoResponse {
let mut routes: Vec<&'static ApiRouteEntry> = inventory::iter::<ApiRouteEntry>().collect();
routes.sort_by(|a, b| a.path.cmp(b.path).then(a.method.cmp(b.method)));
Json(json!({ "service": "ai-service", "routes": routes }))
}

149
crates/ai-service/src/config.rs Normal file
View file

@ -0,0 +1,149 @@
use std::collections::HashMap;
use std::path::Path;
use anyhow::Context;
use platform_config::ConfigLoader;
use serde::{Deserialize, Serialize};
/// AI Service configuration
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct AiServiceConfig {
pub ai_service: AiServiceSettings,
}
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct AiServiceSettings {
pub server: ServerConfig,
pub rag: RagConfig,
pub mcp: McpConfig,
pub dag: DagConfig,
#[serde(default)]
pub monitoring: Option<MonitoringConfig>,
#[serde(default)]
pub logging: Option<LoggingConfig>,
#[serde(default)]
pub build: Option<DockerBuildConfig>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ServerConfig {
pub host: String,
pub port: u16,
#[serde(default)]
pub workers: Option<usize>,
}
impl Default for ServerConfig {
fn default() -> Self {
Self {
host: "127.0.0.1".to_string(),
port: 8082,
workers: Some(4),
}
}
}
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct RagConfig {
pub enabled: bool,
pub rag_service_url: Option<String>,
pub timeout: Option<u64>,
}
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct McpConfig {
pub enabled: bool,
pub mcp_service_url: Option<String>,
pub timeout: Option<u64>,
}
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct DagConfig {
pub max_concurrent_tasks: Option<usize>,
pub task_timeout: Option<u64>,
pub retry_attempts: Option<u32>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MonitoringConfig {
pub enabled: bool,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LoggingConfig {
pub level: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DockerBuildConfig {
pub base_image: String,
#[serde(default)]
pub build_args: HashMap<String, String>,
}
impl AiServiceConfig {
pub fn load() -> anyhow::Result<Self> {
let config_json = platform_config::load_service_config_from_ncl("ai-service")
.context("Failed to load ai-service configuration from Nickel")?;
serde_json::from_value(config_json)
.context("Failed to deserialize ai-service configuration")
}
}
impl ConfigLoader for AiServiceConfig {
fn service_name() -> &'static str {
"ai-service"
}
fn load_from_hierarchy() -> std::result::Result<Self, Box<dyn std::error::Error + Send + Sync>>
{
if let Some(path) = platform_config::resolve_config_path(Self::service_name()) {
return Self::from_path(&path);
}
Ok(Self::default())
}
fn apply_env_overrides(
&mut self,
) -> std::result::Result<(), Box<dyn std::error::Error + Send + Sync>> {
if let Ok(host) = std::env::var("AI_SERVICE_SERVER_HOST") {
self.ai_service.server.host = host;
}
if let Ok(port) = std::env::var("AI_SERVICE_SERVER_PORT") {
if let Ok(p) = port.parse() {
self.ai_service.server.port = p;
}
}
Ok(())
}
fn from_path<P: AsRef<Path>>(
path: P,
) -> std::result::Result<Self, Box<dyn std::error::Error + Send + Sync>> {
let path = path.as_ref();
let json_value = platform_config::format::load_config(path)
.map_err(|e| Box::new(e) as Box<dyn std::error::Error + Send + Sync>)?;
serde_json::from_value(json_value).map_err(|e| {
let err_msg = format!("Failed to deserialize ai-service config: {}", e);
Box::new(std::io::Error::new(
std::io::ErrorKind::InvalidData,
err_msg,
)) as Box<dyn std::error::Error + Send + Sync>
})
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_default_config() {
let config = AiServiceConfig::default();
assert_eq!(config.ai_service.server.port, 8082);
assert!(!config.ai_service.rag.enabled);
assert!(!config.ai_service.mcp.enabled);
}
}

108
crates/ai-service/src/dag.rs Normal file
View file

@ -0,0 +1,108 @@
//! Extension DAG (Directed Acyclic Graph) operations for dependency resolution
use std::collections::HashMap;
use petgraph::graph::{DiGraph, NodeIndex};
use serde::{Deserialize, Serialize};
/// Extension dependency graph
pub struct ExtensionDag {
graph: DiGraph<Extension, String>,
nodes: HashMap<String, NodeIndex>,
}
/// Extension metadata
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Extension {
pub name: String,
pub version: String,
pub description: String,
}
impl ExtensionDag {
/// Create a new extension DAG
pub fn new() -> Self {
Self {
graph: DiGraph::new(),
nodes: HashMap::new(),
}
}
/// Add an extension to the DAG
pub fn add_extension(&mut self, name: String, version: String, description: String) {
let extension = Extension {
name: name.clone(),
version,
description,
};
let idx = self.graph.add_node(extension);
self.nodes.insert(name, idx);
}
/// Add a dependency between extensions
pub fn add_dependency(&mut self, from: &str, to: &str) -> Result<(), String> {
let from_idx = self
.nodes
.get(from)
.ok_or_else(|| format!("Extension not found: {}", from))?;
let to_idx = self
.nodes
.get(to)
.ok_or_else(|| format!("Extension not found: {}", to))?;
self.graph
.add_edge(*from_idx, *to_idx, format!("{} depends on {}", from, to));
Ok(())
}
/// Get topological sort (initialization order)
pub fn topological_sort(&self) -> Result<Vec<String>, String> {
match petgraph::algo::toposort(&self.graph, None) {
Ok(order) => Ok(order
.iter()
.map(|idx| self.graph[*idx].name.clone())
.collect()),
Err(_) => Err("Circular dependency detected".to_string()),
}
}
/// Get dependencies for an extension
pub fn get_dependencies(&self, name: &str) -> Result<Vec<String>, String> {
let idx = self
.nodes
.get(name)
.ok_or_else(|| format!("Extension not found: {}", name))?;
let deps = self
.graph
.neighbors(*idx)
.map(|neighbor_idx| self.graph[neighbor_idx].name.clone())
.collect();
Ok(deps)
}
/// Export DAG as nodes and edges
pub fn export(&self) -> (Vec<Extension>, Vec<(String, String)>) {
let nodes: Vec<Extension> = self.graph.node_weights().cloned().collect();
let edges: Vec<(String, String)> = self
.graph
.raw_edges()
.iter()
.map(|edge| {
let from = self.graph[edge.source()].name.clone();
let to = self.graph[edge.target()].name.clone();
(from, to)
})
.collect();
(nodes, edges)
}
}
impl Default for ExtensionDag {
fn default() -> Self {
Self::new()
}
}

213
crates/ai-service/src/handlers.rs Normal file
View file

@ -0,0 +1,213 @@
//! HTTP request handlers for AI service API
use std::sync::Arc;
use axum::{
extract::State,
http::StatusCode,
response::IntoResponse,
routing::{get, post},
Json, Router,
};
use ontoref_derive::onto_api;
use serde_json::json;
use tracing::debug;
use crate::service::{
AiService, AskRequest, AskResponse, BestPractice, DagResponse, McpToolRequest, McpToolResponse,
};
/// Create AI service HTTP routes
pub fn create_routes(state: Arc<AiService>) -> Router {
Router::new()
.route("/api/v1/ai/mcp/tool", post(call_mcp_tool_handler))
.route("/api/v1/ai/ask", post(ask_handler))
.route("/api/v1/ai/dag/extensions", get(get_extension_dag_handler))
.route(
"/api/v1/ai/knowledge/best-practices",
get(get_best_practices_handler),
)
.route("/health", get(health_check_handler))
.route("/api/catalog", get(crate::api_catalog::api_catalog))
.with_state(state)
}
#[onto_api(
method = "POST",
path = "/api/v1/ai/mcp/tool",
description = "Invoke an MCP tool by name with arguments",
auth = "bearer",
actors = "developer, agent",
tags = "ai, mcp",
feature = ""
)]
async fn call_mcp_tool_handler(
State(service): State<Arc<AiService>>,
Json(req): Json<McpToolRequest>,
) -> Result<Json<McpToolResponse>, McpToolError> {
debug!("Calling MCP tool: {}", req.tool_name);
let response = service
.call_mcp_tool(req)
.await
.map_err(McpToolError::from)?;
Ok(Json(response))
}
#[onto_api(
method = "POST",
path = "/api/v1/ai/ask",
description = "Ask a RAG-powered question grounded in Nickel schemas and deployment history",
auth = "bearer",
actors = "developer, agent",
tags = "ai, rag",
feature = ""
)]
async fn ask_handler(
State(service): State<Arc<AiService>>,
Json(req): Json<AskRequest>,
) -> Result<Json<AskResponse>, AskError> {
debug!("Processing RAG question: {}", req.question);
let response = service.ask(req).await.map_err(AskError::from)?;
Ok(Json(response))
}
#[onto_api(
method = "GET",
path = "/api/v1/ai/dag/extensions",
description = "Get the extension dependency DAG used by AI for schema-aware config generation",
auth = "bearer",
actors = "developer, agent",
tags = "ai, dag, extensions",
feature = ""
)]
async fn get_extension_dag_handler(
State(service): State<Arc<AiService>>,
) -> Result<Json<DagResponse>, InternalError> {
debug!("Getting extension DAG");
let dag = service
.get_extension_dag()
.await
.map_err(|e| InternalError(e.to_string()))?;
Ok(Json(dag))
}
#[onto_api(
method = "GET",
path = "/api/v1/ai/knowledge/best-practices",
description = "Get best practices for a given category from the knowledge base",
auth = "bearer",
actors = "developer, agent",
tags = "ai, knowledge",
feature = ""
)]
async fn get_best_practices_handler(
State(service): State<Arc<AiService>>,
axum::extract::Query(params): axum::extract::Query<std::collections::HashMap<String, String>>,
) -> Result<Json<Vec<BestPractice>>, InternalError> {
let category = params
.get("category")
.map(|s| s.as_str())
.unwrap_or("general");
debug!("Getting best practices for category: {}", category);
let practices = service
.get_best_practices(category)
.await
.map_err(|e| InternalError(e.to_string()))?;
Ok(Json(practices))
}
#[onto_api(
method = "GET",
path = "/health",
description = "AI service health check",
auth = "none",
actors = "developer, agent, ci",
tags = "health",
feature = ""
)]
async fn health_check_handler(
State(service): State<Arc<AiService>>,
) -> Result<StatusCode, InternalError> {
service
.health_check()
.await
.map_err(|e| InternalError(e.to_string()))?;
Ok(StatusCode::OK)
}
// Error types for handlers
/// MCP tool error
enum McpToolError {
Internal(String),
}
impl From<anyhow::Error> for McpToolError {
fn from(err: anyhow::Error) -> Self {
McpToolError::Internal(err.to_string())
}
}
impl IntoResponse for McpToolError {
fn into_response(self) -> axum::response::Response {
match self {
McpToolError::Internal(err) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(json!({
"error": format!("MCP tool execution failed: {}", err)
})),
)
.into_response(),
}
}
}
/// Ask/RAG error
enum AskError {
Internal(String),
}
impl From<anyhow::Error> for AskError {
fn from(err: anyhow::Error) -> Self {
AskError::Internal(err.to_string())
}
}
impl IntoResponse for AskError {
fn into_response(self) -> axum::response::Response {
match self {
AskError::Internal(err) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(json!({
"error": format!("Question answering failed: {}", err)
})),
)
.into_response(),
}
}
}
/// Internal server error
struct InternalError(String);
impl IntoResponse for InternalError {
fn into_response(self) -> axum::response::Response {
(
StatusCode::INTERNAL_SERVER_ERROR,
Json(json!({
"error": self.0
})),
)
.into_response()
}
}

206
crates/ai-service/src/knowledge.rs Normal file
View file

@ -0,0 +1,206 @@
//! Knowledge graph for storing and retrieving best practices
use std::collections::HashMap;
use serde::{Deserialize, Serialize};
/// Knowledge graph node for best practices
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct KnowledgeNode {
pub id: String,
pub title: String,
pub description: String,
pub category: String,
pub relevance: u8,
pub tags: Vec<String>,
}
/// Knowledge graph relationship
#[derive(Debug, Clone)]
pub struct Relationship {
from: String,
to: String,
#[allow(dead_code)]
relation_type: String,
}
/// Knowledge graph for best practices and system knowledge
pub struct KnowledgeGraph {
nodes: HashMap<String, KnowledgeNode>,
relationships: Vec<Relationship>,
}
impl KnowledgeGraph {
/// Create a new knowledge graph
pub fn new() -> Self {
let mut graph = Self {
nodes: HashMap::new(),
relationships: Vec::new(),
};
graph.populate_best_practices();
graph
}
/// Populate knowledge graph with best practices
fn populate_best_practices(&mut self) {
let practices = vec![
KnowledgeNode {
id: "bp_001".to_string(),
title: "Use Configuration as Code".to_string(),
description: "Always store infrastructure configuration in version control"
.to_string(),
category: "deployment".to_string(),
relevance: 95,
tags: vec!["config".to_string(), "iac".to_string()],
},
KnowledgeNode {
id: "bp_002".to_string(),
title: "Implement Health Checks".to_string(),
description: "Define health check endpoints for all services".to_string(),
category: "reliability".to_string(),
relevance: 90,
tags: vec!["monitoring".to_string(), "health".to_string()],
},
KnowledgeNode {
id: "bp_003".to_string(),
title: "Monitor Resource Usage".to_string(),
description: "Track CPU, memory, and network metrics continuously".to_string(),
category: "operations".to_string(),
relevance: 85,
tags: vec!["monitoring".to_string(), "metrics".to_string()],
},
KnowledgeNode {
id: "bp_004".to_string(),
title: "Encrypt Data in Transit".to_string(),
description: "Use TLS for all network communications".to_string(),
category: "security".to_string(),
relevance: 100,
tags: vec!["security".to_string(), "encryption".to_string()],
},
KnowledgeNode {
id: "bp_005".to_string(),
title: "Implement Access Control".to_string(),
description: "Use RBAC and principle of least privilege".to_string(),
category: "security".to_string(),
relevance: 95,
tags: vec!["security".to_string(), "access-control".to_string()],
},
KnowledgeNode {
id: "bp_006".to_string(),
title: "Use Container Images".to_string(),
description: "Containerize services for consistency and portability".to_string(),
category: "deployment".to_string(),
relevance: 88,
tags: vec!["containers".to_string(), "docker".to_string()],
},
KnowledgeNode {
id: "bp_007".to_string(),
title: "Implement Automated Testing".to_string(),
description: "Run unit, integration, and e2e tests in CI/CD".to_string(),
category: "quality".to_string(),
relevance: 90,
tags: vec!["testing".to_string(), "ci-cd".to_string()],
},
KnowledgeNode {
id: "bp_008".to_string(),
title: "Use Service Mesh".to_string(),
description: "Implement service-to-service communication control".to_string(),
category: "architecture".to_string(),
relevance: 80,
tags: vec!["architecture".to_string(), "networking".to_string()],
},
];
for practice in practices {
self.nodes.insert(practice.id.clone(), practice);
}
}
/// Search best practices by category
pub fn search_by_category(&self, category: &str) -> Vec<KnowledgeNode> {
self.nodes
.values()
.filter(|node| node.category == category)
.cloned()
.collect()
}
/// Search best practices by tag
pub fn search_by_tag(&self, tag: &str) -> Vec<KnowledgeNode> {
self.nodes
.values()
.filter(|node| node.tags.contains(&tag.to_string()))
.cloned()
.collect()
}
/// Search best practices by relevance threshold
pub fn search_by_relevance(&self, min_relevance: u8) -> Vec<KnowledgeNode> {
let mut results: Vec<_> = self
.nodes
.values()
.filter(|node| node.relevance >= min_relevance)
.cloned()
.collect();
results.sort_by(|a, b| b.relevance.cmp(&a.relevance));
results
}
/// Get all categories
pub fn get_categories(&self) -> Vec<String> {
let mut categories: Vec<String> = self
.nodes
.values()
.map(|node| node.category.clone())
.collect();
categories.sort();
categories.dedup();
categories
}
/// Get all tags
pub fn get_tags(&self) -> Vec<String> {
let mut tags: Vec<String> = self
.nodes
.values()
.flat_map(|node| node.tags.clone())
.collect();
tags.sort();
tags.dedup();
tags
}
/// Add relationship between knowledge nodes
pub fn add_relationship(&mut self, from: String, to: String, relation_type: String) {
self.relationships.push(Relationship {
from,
to,
relation_type,
});
}
/// Get related practices
pub fn get_related(&self, id: &str) -> Vec<KnowledgeNode> {
let related_ids: Vec<String> = self
.relationships
.iter()
.filter(|rel| rel.from == id)
.map(|rel| rel.to.clone())
.collect();
self.nodes
.values()
.filter(|node| related_ids.contains(&node.id))
.cloned()
.collect()
}
}
impl Default for KnowledgeGraph {
fn default() -> Self {
Self::new()
}
}

23
crates/ai-service/src/lib.rs Normal file
View file

@ -0,0 +1,23 @@
//! HTTP service wrapper for AI capabilities including RAG, MCP tool invocation,
//! DAG operations, and knowledge graphs
//!
//! Exposes Claude-based question answering, MCP tool execution, extension
//! dependency graphs, and best practice recommendations via HTTP API.
pub mod api_catalog;
pub mod config;
pub mod dag;
pub mod handlers;
pub mod knowledge;
pub mod mcp;
pub mod service;
pub mod tool_integration;
pub use config::AiServiceConfig;
pub use service::AiService;
/// HTTP API version
pub const API_VERSION: &str = "v1";
/// Default port for AI service
pub const DEFAULT_PORT: u16 = 8083;

86
crates/ai-service/src/main.rs Normal file
View file

@ -0,0 +1,86 @@
//! AI service binary - HTTP wrapper for AI capabilities including RAG, MCP tool
//! invocation, and knowledge graphs
use std::net::SocketAddr;
use std::sync::Arc;
use ai_service::{handlers, AiService, DEFAULT_PORT};
use axum::Router;
use clap::Parser;
#[derive(Parser, Debug)]
#[command(name = "ai-service")]
#[command(about = "HTTP service for AI capabilities including RAG, MCP tool invocation, DAG operations, and knowledge graphs", long_about = None)]
struct Args {
/// Configuration file path (highest priority)
#[arg(short = 'c', long, env = "AI_SERVICE_CONFIG")]
config: Option<std::path::PathBuf>,
/// Configuration directory (searches for ai-service.ncl|toml|json)
#[arg(long, env = "PROVISIONING_CONFIG_DIR")]
config_dir: Option<std::path::PathBuf>,
/// Deployment mode (solo, multiuser, cicd, enterprise)
#[arg(short = 'm', long, env = "AI_SERVICE_MODE")]
mode: Option<String>,
/// Service bind address
#[arg(short = 'H', long, default_value = "127.0.0.1")]
host: String,
/// Service bind port
#[arg(short = 'p', long, default_value_t = DEFAULT_PORT)]
port: u16,
/// Print all #[onto_api] registered routes as JSON and exit.
/// Pipe to api-catalog-ai-service.json: `just export-api-catalog`
#[arg(long)]
dump_api_catalog: bool,
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Parse CLI arguments FIRST (so --help works before any other processing)
let args = Args::parse();
if args.dump_api_catalog {
println!("{}", ontoref_ontology::api::dump_catalog_json());
return Ok(());
}
// Initialize centralized observability (logging, metrics, health checks)
let _guard = observability::init_from_env("ai-service", env!("CARGO_PKG_VERSION"))?;
// Check if ai-service is enabled in deployment-mode.ncl
if let Ok(deployment) = platform_config::load_deployment_mode() {
if let Ok(enabled) = deployment.is_service_enabled("ai_service") {
if !enabled {
tracing::warn!("⚠ AI Service is DISABLED in deployment-mode.ncl");
std::process::exit(1);
}
tracing::info!("✓ AI Service is ENABLED in deployment-mode.ncl");
}
}
// Try to load ai-service.ncl
if let Ok(config) = platform_config::load_service_config_from_ncl("ai-service") {
tracing::info!("✓ Loaded ai-service configuration from NCL");
tracing::debug!("Config: {:?}", config);
}
let addr: SocketAddr = format!("{}:{}", args.host, args.port).parse()?;
// Create service
let service = Arc::new(AiService::new(addr));
tracing::info!("Starting AI service on {}", addr);
// Create router
let app = Router::new()
.merge(handlers::create_routes(service))
.fallback(|| async { (axum::http::StatusCode::NOT_FOUND, "Not found") });
// Start server
let listener = tokio::net::TcpListener::bind(&addr).await?;
axum::serve(listener, app).await?;
Ok(())
}

735
crates/ai-service/src/mcp.rs Normal file
View file

@ -0,0 +1,735 @@
//! MCP (Model Context Protocol) tool registry and execution
//!
//! Provides tool definition, registration, and execution for RAG, Guidance,
//! Settings, and IaC tools.
use provisioning_mcp_server::tools::settings::{DeploymentMode, SettingsTools};
use serde::{Deserialize, Serialize};
use serde_json::{json, Value};
use tokio::sync::Mutex;
/// Tool execution result
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ToolExecution {
pub tool_name: String,
pub result: Value,
pub duration_ms: u64,
}
/// MCP tool registry for provisioning system
pub struct ToolRegistry {
tools: std::collections::HashMap<String, ToolDefinition>,
settings_tools: Mutex<SettingsTools>,
}
/// Tool definition for MCP
#[derive(Debug, Clone)]
pub struct ToolDefinition {
pub name: String,
pub description: String,
pub category: ToolCategory,
pub input_schema: Value,
}
/// Tool categories
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ToolCategory {
Rag,
Guidance,
Settings,
Iac,
}
impl ToolRegistry {
/// Create a new tool registry
pub fn new() -> Self {
let mut registry = Self {
tools: std::collections::HashMap::new(),
settings_tools: Mutex::new(SettingsTools::new()),
};
registry.register_all_tools();
registry
}
/// Register all tool categories (RAG, Guidance, Settings, IaC)
fn register_all_tools(&mut self) {
self.register_rag_tools();
self.register_guidance_tools();
self.register_settings_tools();
self.register_iac_tools();
}
/// Register RAG tools
fn register_rag_tools(&mut self) {
self.tools.insert(
"rag_ask_question".to_string(),
ToolDefinition {
name: "rag_ask_question".to_string(),
description: "Ask a question using RAG (Retrieval-Augmented Generation) with knowledge base search".to_string(),
category: ToolCategory::Rag,
input_schema: json!({
"type": "object",
"properties": {
"question": {"type": "string", "description": "The question to ask"},
"context": {"type": "string", "description": "Optional context for the question"},
"top_k": {"type": "integer", "description": "Number of top results to return", "default": 3}
},
"required": ["question"]
}),
},
);
self.tools.insert(
"rag_semantic_search".to_string(),
ToolDefinition {
name: "rag_semantic_search".to_string(),
description: "Perform semantic search on the knowledge base".to_string(),
category: ToolCategory::Rag,
input_schema: json!({
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"},
"category": {"type": "string", "description": "Optional category filter"},
"top_k": {"type": "integer", "description": "Number of results", "default": 5}
},
"required": ["query"]
}),
},
);
self.tools.insert(
"rag_get_status".to_string(),
ToolDefinition {
name: "rag_get_status".to_string(),
description: "Get the current status of the RAG knowledge base".to_string(),
category: ToolCategory::Rag,
input_schema: json!({
"type": "object",
"properties": {}
}),
},
);
}
/// Register Guidance tools
fn register_guidance_tools(&mut self) {
self.tools.insert(
"guidance_check_system_status".to_string(),
ToolDefinition {
name: "guidance_check_system_status".to_string(),
description: "Check the current system status and configuration".to_string(),
category: ToolCategory::Guidance,
input_schema: json!({
"type": "object",
"properties": {}
}),
},
);
self.tools.insert(
"guidance_suggest_next_action".to_string(),
ToolDefinition {
name: "guidance_suggest_next_action".to_string(),
description: "Get suggestions for the next action based on current system state".to_string(),
category: ToolCategory::Guidance,
input_schema: json!({
"type": "object",
"properties": {
"context": {"type": "string", "description": "Optional context for suggestion"}
}
}),
},
);
self.tools.insert(
"guidance_find_docs".to_string(),
ToolDefinition {
name: "guidance_find_docs".to_string(),
description: "Find relevant documentation and guides".to_string(),
category: ToolCategory::Guidance,
input_schema: json!({
"type": "object",
"properties": {
"query": {"type": "string", "description": "What to search for"},
"context": {"type": "string", "description": "Optional context"}
},
"required": ["query"]
}),
},
);
self.tools.insert(
"guidance_troubleshoot".to_string(),
ToolDefinition {
name: "guidance_troubleshoot".to_string(),
description: "Troubleshoot an issue or error".to_string(),
category: ToolCategory::Guidance,
input_schema: json!({
"type": "object",
"properties": {
"error": {"type": "string", "description": "Error message or description"},
"context": {"type": "string", "description": "Context about the issue"}
},
"required": ["error"]
}),
},
);
self.tools.insert(
"guidance_validate_config".to_string(),
ToolDefinition {
name: "guidance_validate_config".to_string(),
description: "Validate a configuration file or settings".to_string(),
category: ToolCategory::Guidance,
input_schema: json!({
"type": "object",
"properties": {
"config_path": {"type": "string", "description": "Path to configuration file"}
},
"required": ["config_path"]
}),
},
);
}
/// Register Settings tools
fn register_settings_tools(&mut self) {
self.tools.insert(
"installer_get_settings".to_string(),
ToolDefinition {
name: "installer_get_settings".to_string(),
description: "Get installer settings and configuration".to_string(),
category: ToolCategory::Settings,
input_schema: json!({
"type": "object",
"properties": {
"query": {"type": "string", "description": "Optional settings query"}
}
}),
},
);
self.tools.insert(
"installer_complete_config".to_string(),
ToolDefinition {
name: "installer_complete_config".to_string(),
description: "Complete partial configuration with defaults".to_string(),
category: ToolCategory::Settings,
input_schema: json!({
"type": "object",
"properties": {
"config": {"type": "object", "description": "Partial configuration"}
}
}),
},
);
self.tools.insert(
"installer_validate_config".to_string(),
ToolDefinition {
name: "installer_validate_config".to_string(),
description: "Validate configuration against schema".to_string(),
category: ToolCategory::Settings,
input_schema: json!({
"type": "object",
"properties": {
"config": {"type": "object", "description": "Configuration to validate"}
},
"required": ["config"]
}),
},
);
self.tools.insert(
"installer_get_defaults".to_string(),
ToolDefinition {
name: "installer_get_defaults".to_string(),
description: "Get default settings for a deployment mode".to_string(),
category: ToolCategory::Settings,
input_schema: json!({
"type": "object",
"properties": {
"mode": {"type": "string", "description": "Deployment mode"}
},
"required": ["mode"]
}),
},
);
self.tools.insert(
"installer_platform_recommendations".to_string(),
ToolDefinition {
name: "installer_platform_recommendations".to_string(),
description: "Get platform-specific recommendations".to_string(),
category: ToolCategory::Settings,
input_schema: json!({
"type": "object",
"properties": {}
}),
},
);
self.tools.insert(
"installer_service_recommendations".to_string(),
ToolDefinition {
name: "installer_service_recommendations".to_string(),
description: "Get service recommendations for a deployment mode".to_string(),
category: ToolCategory::Settings,
input_schema: json!({
"type": "object",
"properties": {
"mode": {"type": "string", "description": "Deployment mode"}
},
"required": ["mode"]
}),
},
);
self.tools.insert(
"installer_resource_recommendations".to_string(),
ToolDefinition {
name: "installer_resource_recommendations".to_string(),
description: "Get resource recommendations for a deployment mode".to_string(),
category: ToolCategory::Settings,
input_schema: json!({
"type": "object",
"properties": {
"mode": {"type": "string", "description": "Deployment mode"}
},
"required": ["mode"]
}),
},
);
}
/// Register IaC tools
fn register_iac_tools(&mut self) {
self.tools.insert(
"iac_detect_technologies".to_string(),
ToolDefinition {
name: "iac_detect_technologies".to_string(),
description: "Detect technologies used in infrastructure".to_string(),
category: ToolCategory::Iac,
input_schema: json!({
"type": "object",
"properties": {
"path": {"type": "string", "description": "Path to analyze"}
}
}),
},
);
self.tools.insert(
"iac_analyze_completeness".to_string(),
ToolDefinition {
name: "iac_analyze_completeness".to_string(),
description: "Analyze completeness of infrastructure configuration".to_string(),
category: ToolCategory::Iac,
input_schema: json!({
"type": "object",
"properties": {
"config": {"type": "object", "description": "Configuration to analyze"}
}
}),
},
);
self.tools.insert(
"iac_infer_requirements".to_string(),
ToolDefinition {
name: "iac_infer_requirements".to_string(),
description: "Infer infrastructure requirements from description".to_string(),
category: ToolCategory::Iac,
input_schema: json!({
"type": "object",
"properties": {
"description": {"type": "string", "description": "Infrastructure description"}
},
"required": ["description"]
}),
},
);
}
/// Get all tool definitions
pub fn list_tools(&self) -> Vec<ToolDefinition> {
self.tools.values().cloned().collect()
}
/// Get tools by category
pub fn tools_by_category(&self, category: ToolCategory) -> Vec<ToolDefinition> {
self.tools
.values()
.filter(|t| t.category == category)
.cloned()
.collect()
}
/// Check if tool exists
pub fn has_tool(&self, name: &str) -> bool {
self.tools.contains_key(name)
}
/// Execute a tool (async)
pub async fn execute(&self, tool_name: &str, args: &Value) -> Result<Value, String> {
match tool_name {
// RAG tools
"rag_ask_question" => self.rag_ask_question(args).await,
"rag_semantic_search" => self.rag_semantic_search(args).await,
"rag_get_status" => self.rag_get_status(args).await,
// Guidance tools
"guidance_check_system_status" => self.guidance_check_system_status(args).await,
"guidance_suggest_next_action" => self.guidance_suggest_next_action(args).await,
"guidance_find_docs" => self.guidance_find_docs(args).await,
"guidance_troubleshoot" => self.guidance_troubleshoot(args).await,
"guidance_validate_config" => self.guidance_validate_config(args).await,
// Settings tools
"installer_get_settings" => self.installer_get_settings(args).await,
"installer_complete_config" => self.installer_complete_config(args).await,
"installer_validate_config" => self.installer_validate_config(args).await,
"installer_get_defaults" => self.installer_get_defaults(args).await,
"installer_platform_recommendations" => {
self.installer_platform_recommendations(args).await
}
"installer_service_recommendations" => {
self.installer_service_recommendations(args).await
}
"installer_resource_recommendations" => {
self.installer_resource_recommendations(args).await
}
// IaC tools
"iac_detect_technologies" => self.iac_detect_technologies(args).await,
"iac_analyze_completeness" => self.iac_analyze_completeness(args).await,
"iac_infer_requirements" => self.iac_infer_requirements(args).await,
_ => Err(format!("Unknown tool: {}", tool_name)),
}
}
// ========== RAG TOOL IMPLEMENTATIONS ==========
async fn rag_ask_question(&self, args: &Value) -> Result<Value, String> {
let question = args
.get("question")
.and_then(|v| v.as_str())
.ok_or("question parameter required")?;
Ok(json!({
"status": "success",
"tool": "rag_ask_question",
"message": format!("RAG query would be processed for: {}", question)
}))
}
async fn rag_semantic_search(&self, args: &Value) -> Result<Value, String> {
let query = args
.get("query")
.and_then(|v| v.as_str())
.ok_or("query parameter required")?;
Ok(json!({
"status": "success",
"tool": "rag_semantic_search",
"message": format!("Semantic search would be performed for: {}", query),
"results": []
}))
}
async fn rag_get_status(&self, _args: &Value) -> Result<Value, String> {
Ok(json!({
"status": "active",
"tool": "rag_get_status",
"knowledge_base": {
"documents_loaded": true,
"total_documents": 76,
"categories": ["architecture", "deployment", "security", "reliability"]
}
}))
}
// ========== GUIDANCE TOOL IMPLEMENTATIONS ==========
/// Execute a Nushell command and parse JSON output
async fn execute_nu_command(cmd: &str) -> Result<Value, String> {
use tokio::process::Command;
let output = Command::new("nu")
.arg("-c")
.arg(cmd)
.output()
.await
.map_err(|e| format!("Failed to execute Nushell: {}", e))?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(format!("Nushell command failed: {}", stderr));
}
serde_json::from_slice(&output.stdout)
.map_err(|e| format!("Failed to parse JSON output: {}", e))
}
async fn guidance_check_system_status(&self, _args: &Value) -> Result<Value, String> {
Self::execute_nu_command("provisioning status-json").await
}
async fn guidance_suggest_next_action(&self, _args: &Value) -> Result<Value, String> {
Self::execute_nu_command("provisioning next").await
}
async fn guidance_find_docs(&self, args: &Value) -> Result<Value, String> {
let query = args
.get("query")
.and_then(|v| v.as_str())
.ok_or("query parameter required")?;
let cmd = format!("provisioning guide {}", query);
Self::execute_nu_command(&cmd).await
}
async fn guidance_troubleshoot(&self, _args: &Value) -> Result<Value, String> {
Self::execute_nu_command("provisioning health-json").await
}
async fn guidance_validate_config(&self, args: &Value) -> Result<Value, String> {
let config_path = args
.get("config_path")
.and_then(|v| v.as_str())
.ok_or("config_path parameter required")?;
let cmd = format!("validate-system-config {}", config_path);
Self::execute_nu_command(&cmd).await
}
// ========== SETTINGS TOOL IMPLEMENTATIONS ==========
async fn installer_get_settings(&self, args: &Value) -> Result<Value, String> {
let query = args.get("query").and_then(|v| v.as_str());
let mut settings_tools = self.settings_tools.lock().await;
let settings = settings_tools
.get_settings(query)
.await
.map_err(|e| format!("Failed to get settings: {}", e))?;
Ok(json!({
"status": "success",
"tool": "installer_get_settings",
"platforms": settings.get("platforms"),
"modes": settings.get("modes"),
"default_domain": settings.get("default_domain"),
"auto_generate_secrets": settings.get("auto_generate_secrets"),
"available_services": settings.get("available_services")
}))
}
async fn installer_complete_config(&self, args: &Value) -> Result<Value, String> {
let config = args
.get("config")
.cloned()
.ok_or("config parameter required")?;
let mut settings_tools = self.settings_tools.lock().await;
settings_tools
.complete_config(config)
.await
.map_err(|e| format!("Failed to complete config: {}", e))
}
async fn installer_validate_config(&self, args: &Value) -> Result<Value, String> {
let config = args
.get("config")
.cloned()
.ok_or("config parameter required")?;
let settings_tools = self.settings_tools.lock().await;
settings_tools
.validate_config(config)
.map_err(|e| format!("Failed to validate config: {}", e))
}
async fn installer_get_defaults(&self, args: &Value) -> Result<Value, String> {
let mode = args
.get("mode")
.and_then(|v| v.as_str())
.ok_or("mode parameter required")?;
let settings_tools = self.settings_tools.lock().await;
let defaults = settings_tools
.get_mode_defaults(mode)
.map_err(|e| format!("Failed to get defaults: {}", e))?;
Ok(json!({
"status": "success",
"tool": "installer_get_defaults",
"mode": defaults.get("mode"),
"description": defaults.get("description"),
"service_count": defaults.get("service_count"),
"min_cpu_cores": defaults.get("min_cpu_cores"),
"min_memory_gb": defaults.get("min_memory_gb"),
"recommended_services": defaults.get("recommended_services"),
"auto_generate_secrets": defaults.get("auto_generate_secrets"),
"domain": defaults.get("domain")
}))
}
async fn installer_platform_recommendations(&self, _args: &Value) -> Result<Value, String> {
let mut settings_tools = self.settings_tools.lock().await;
let recommendations = settings_tools
.get_platform_recommendations()
.await
.map_err(|e| format!("Failed to get platform recommendations: {}", e))?;
Ok(json!({
"status": "success",
"tool": "installer_platform_recommendations",
"recommendations": recommendations
}))
}
async fn installer_service_recommendations(&self, args: &Value) -> Result<Value, String> {
let mode_str = args
.get("mode")
.and_then(|v| v.as_str())
.ok_or("mode parameter required")?;
let mode = DeploymentMode::from_str(mode_str)
.ok_or(format!("Invalid deployment mode: {}", mode_str))?;
let settings_tools = self.settings_tools.lock().await;
let recommendations = settings_tools.get_service_recommendations(&mode);
Ok(json!({
"status": "success",
"tool": "installer_service_recommendations",
"mode": mode_str,
"recommendations": recommendations
}))
}
async fn installer_resource_recommendations(&self, args: &Value) -> Result<Value, String> {
let mode_str = args
.get("mode")
.and_then(|v| v.as_str())
.ok_or("mode parameter required")?;
let mode = DeploymentMode::from_str(mode_str)
.ok_or(format!("Invalid deployment mode: {}", mode_str))?;
let settings_tools = self.settings_tools.lock().await;
let recommendations = settings_tools.get_resource_recommendations(&mode);
Ok(json!({
"status": "success",
"tool": "installer_resource_recommendations",
"mode": mode_str,
"recommendations": recommendations
}))
}
// ========== IAC TOOL IMPLEMENTATIONS ==========
async fn iac_detect_technologies(&self, args: &Value) -> Result<Value, String> {
let path = args
.get("path")
.and_then(|v| v.as_str())
.ok_or("path parameter required")?;
let path_obj = std::path::Path::new(path);
let mut technologies = Vec::new();
// Check for Kubernetes
if path_obj.join("kustomization.yaml").exists() || path_obj.join("deployment.yaml").exists()
{
technologies.push("kubernetes");
}
// Check for Docker
if path_obj.join("Dockerfile").exists() || path_obj.join("docker-compose.yml").exists() {
technologies.push("docker");
}
// Check for Terraform
if path_obj.join("main.tf").exists() {
technologies.push("terraform");
}
// Check for KCL (legacy)
if path_obj.join("kcl.mod").exists() {
technologies.push("kcl");
}
// Check for Nickel (current IaC)
if path_obj.join("main.ncl").exists() {
technologies.push("nickel");
}
Ok(json!({
"status": "success",
"tool": "iac_detect_technologies",
"path": path,
"technologies": technologies
}))
}
async fn iac_analyze_completeness(&self, args: &Value) -> Result<Value, String> {
let path = args
.get("path")
.and_then(|v| v.as_str())
.ok_or("path parameter required")?;
let path_obj = std::path::Path::new(path);
let mut missing = Vec::new();
// Check for essential infrastructure files
if !path_obj.join("infrastructure.ncl").exists() {
missing.push("infrastructure.ncl");
}
if !path_obj.join("config.toml").exists() {
missing.push("config.toml");
}
if !path_obj.join("README.md").exists() {
missing.push("README.md");
}
let complete = missing.is_empty();
let completeness_score = if missing.is_empty() { 1.0 } else { 0.7 };
Ok(json!({
"status": "success",
"tool": "iac_analyze_completeness",
"complete": complete,
"completeness_score": completeness_score,
"missing_files": missing
}))
}
async fn iac_infer_requirements(&self, args: &Value) -> Result<Value, String> {
let _description = args
.get("description")
.and_then(|v| v.as_str())
.ok_or("description parameter required")?;
// Basic requirements inference (can be enhanced with ML later)
Ok(json!({
"status": "success",
"tool": "iac_infer_requirements",
"requirements": {
"compute": "2 CPU, 4GB RAM (minimum)",
"storage": "20GB",
"network": "Private network recommended",
"high_availability": false
}
}))
}
}
impl Default for ToolRegistry {
fn default() -> Self {
Self::new()
}
}

495
crates/ai-service/src/service.rs Normal file
View file

@ -0,0 +1,495 @@
//! Core AI service implementation with RAG integration
use std::net::SocketAddr;
use std::sync::Arc;
use anyhow::Result;
use serde::{Deserialize, Serialize};
use tokio::sync::RwLock;
use tracing::{debug, info};
/// MCP tool invocation request
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct McpToolRequest {
/// Tool name (e.g., "execute_provisioning_plan")
pub tool_name: String,
/// Tool arguments as JSON
pub args: serde_json::Value,
}
/// MCP tool invocation response
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct McpToolResponse {
/// Tool execution result
pub result: serde_json::Value,
/// Execution time in milliseconds
pub duration_ms: u64,
}
/// RAG-powered question request with optional hybrid tool execution
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AskRequest {
/// User question
pub question: String,
/// Optional context
pub context: Option<String>,
/// Enable automatic tool execution (hybrid mode)
/// When true, the RAG answer may trigger tool calls automatically
/// Default: false (explicit tool calls only)
#[serde(default)]
pub enable_tool_execution: Option<bool>,
/// Maximum number of tools to execute automatically
/// Default: 3
#[serde(default)]
pub max_tool_calls: Option<u32>,
}
/// RAG-powered question response with optional tool execution results
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AskResponse {
/// Answer from AI
pub answer: String,
/// Source documents used
pub sources: Vec<String>,
/// Confidence level (0-100)
pub confidence: u8,
/// Reasoning explanation
pub reasoning: String,
/// Tool executions performed (if hybrid mode enabled)
#[serde(skip_serializing_if = "Option::is_none")]
pub tool_executions: Option<Vec<crate::mcp::ToolExecution>>,
}
/// Extension DAG node
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DagNode {
/// Extension/component name
pub name: String,
/// Dependencies on other nodes
pub dependencies: Vec<String>,
/// Component version
pub version: String,
}
/// Extension DAG response
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DagResponse {
/// DAG nodes (extensions)
pub nodes: Vec<DagNode>,
/// DAG edges (dependencies)
pub edges: Vec<(String, String)>,
}
/// Best practice entry
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BestPractice {
/// Practice title
pub title: String,
/// Practice description
pub description: String,
/// Category (e.g., "deployment", "security")
pub category: String,
/// Relevance score (0-100)
pub relevance: u8,
}
/// Knowledge base document (from RAG ingestion)
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct KnowledgeDocument {
/// Document ID
pub id: String,
/// Document type
#[serde(rename = "type")]
pub doc_type: String,
/// Document title
pub title: Option<String>,
/// Document name (for extensions)
pub name: Option<String>,
/// Full content
pub content: String,
/// Document category
pub category: String,
/// Tags
pub tags: Vec<String>,
/// Relevance/importance
pub relevance: Option<u8>,
/// Dependencies (for extensions)
pub dependencies: Option<Vec<String>>,
}
/// Knowledge base with documents and relationships
#[derive(Debug, Clone)]
pub struct KnowledgeBase {
/// All documents indexed by ID
pub documents: std::collections::HashMap<String, KnowledgeDocument>,
/// Document relationships
pub relationships: Vec<Relationship>,
}
/// Document relationship
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Relationship {
/// Source document ID
pub source_id: String,
/// Target document ID
pub target_id: String,
/// Relationship type
pub relationship_type: String,
/// Strength (0-1)
pub strength: f32,
}
/// Core AI service
pub struct AiService {
/// Server address
addr: SocketAddr,
/// Knowledge base
knowledge_base: Arc<RwLock<KnowledgeBase>>,
/// MCP tool registry
tool_registry: crate::mcp::ToolRegistry,
}
impl AiService {
/// Create new AI service
pub fn new(addr: SocketAddr) -> Self {
Self {
addr,
knowledge_base: Arc::new(RwLock::new(KnowledgeBase {
documents: std::collections::HashMap::new(),
relationships: Vec::new(),
})),
tool_registry: crate::mcp::ToolRegistry::new(),
}
}
/// Load knowledge base from JSON files
pub async fn load_knowledge_base(&self, knowledge_base_dir: &str) -> Result<()> {
info!("Loading knowledge base from: {}", knowledge_base_dir);
// Load best practice documents
let bp_path = format!("{}/best-practices-docs.json", knowledge_base_dir);
let bp_content = std::fs::read_to_string(&bp_path)
.map_err(|e| anyhow::anyhow!("Failed to read best practices: {}", e))?;
let bp_docs: Vec<KnowledgeDocument> = serde_json::from_str(&bp_content)?;
// Load extension documents
let ext_path = format!("{}/extension-docs.json", knowledge_base_dir);
let ext_content = std::fs::read_to_string(&ext_path)
.map_err(|e| anyhow::anyhow!("Failed to read extensions: {}", e))?;
let ext_docs: Vec<KnowledgeDocument> = serde_json::from_str(&ext_content)?;
// Load relationships
let rel_path = format!("{}/relationships.json", knowledge_base_dir);
let rel_content = std::fs::read_to_string(&rel_path)
.map_err(|e| anyhow::anyhow!("Failed to read relationships: {}", e))?;
let relationships: Vec<Relationship> = serde_json::from_str(&rel_content)?;
// Build document index
let mut documents = std::collections::HashMap::new();
for doc in bp_docs.into_iter().chain(ext_docs.into_iter()) {
documents.insert(doc.id.clone(), doc);
}
// Update knowledge base
let mut kb = self.knowledge_base.write().await;
kb.documents = documents;
kb.relationships = relationships;
info!(
"Knowledge base loaded: {} documents, {} relationships",
kb.documents.len(),
kb.relationships.len()
);
Ok(())
}
/// Get service address
pub fn addr(&self) -> SocketAddr {
self.addr
}
/// Search knowledge base by keyword and category
async fn search_knowledge(
&self,
query: &str,
category: Option<&str>,
) -> Vec<KnowledgeDocument> {
let kb = self.knowledge_base.read().await;
let query_lower = query.to_lowercase();
kb.documents
.values()
.filter(|doc| {
let matches_query = doc.content.to_lowercase().contains(&query_lower)
|| doc
.tags
.iter()
.any(|t| t.to_lowercase().contains(&query_lower));
let matches_category = category.map(|c| doc.category == c).unwrap_or(true);
matches_query && matches_category
})
.cloned()
.collect()
}
/// Call an MCP tool via registry
pub async fn call_mcp_tool(&self, req: McpToolRequest) -> Result<McpToolResponse> {
let start = std::time::Instant::now();
debug!("Calling MCP tool: {}", req.tool_name);
let result = self.execute_mcp_tool(&req.tool_name, &req.args).await?;
let duration_ms = start.elapsed().as_millis() as u64;
Ok(McpToolResponse {
result,
duration_ms,
})
}
/// Execute MCP tool with arguments via registry
async fn execute_mcp_tool(
&self,
tool_name: &str,
args: &serde_json::Value,
) -> Result<serde_json::Value> {
// Check if tool exists in registry
if !self.tool_registry.has_tool(tool_name) {
return Err(anyhow::anyhow!("Unknown tool: {}", tool_name));
}
// Execute tool through registry
match self.tool_registry.execute(tool_name, args).await {
Ok(result) => Ok(result),
Err(e) => Err(anyhow::anyhow!("Tool execution failed: {}", e)),
}
}
/// Execute suggested tools and collect results
async fn execute_tools(
&self,
suggestions: &[crate::tool_integration::ToolSuggestion],
max_tools: usize,
) -> Option<(
Vec<crate::mcp::ToolExecution>,
Vec<(String, serde_json::Value)>,
)> {
if suggestions.is_empty() {
return None;
}
debug!(
"Executing {} suggested tools in hybrid mode",
suggestions.len().min(max_tools)
);
let mut executions = Vec::new();
let mut results = Vec::new();
for suggestion in suggestions.iter().take(max_tools) {
match self
.tool_registry
.execute(&suggestion.tool_name, &suggestion.args)
.await
{
Ok(result) => {
debug!("Tool {} executed successfully", suggestion.tool_name);
results.push((suggestion.tool_name.clone(), result.clone()));
executions.push(crate::mcp::ToolExecution {
tool_name: suggestion.tool_name.clone(),
result: result.clone(),
duration_ms: 0,
});
}
Err(e) => {
debug!("Tool {} execution failed: {}", suggestion.tool_name, e);
}
}
}
if executions.is_empty() {
None
} else {
Some((executions, results))
}
}
/// Ask AI a question using RAG with optional hybrid tool execution
pub async fn ask(&self, req: AskRequest) -> Result<AskResponse> {
debug!("Processing RAG question: {}", req.question);
// Search knowledge base for relevant documents
let search_results = self.search_knowledge(&req.question, None).await;
if search_results.is_empty() {
return Ok(AskResponse {
answer: "I couldn't find any relevant information in the knowledge base for this \
question."
.to_string(),
sources: vec![],
confidence: 20,
reasoning: "No matching documents found".to_string(),
tool_executions: None,
});
}
// Sort by relevance (best practices have explicit relevance scores)
let mut results = search_results;
results.sort_by(|a, b| {
let a_rel = a.relevance.unwrap_or(50);
let b_rel = b.relevance.unwrap_or(50);
b_rel.cmp(&a_rel)
});
// Get top 3 most relevant documents
let top_results: Vec<_> = results.iter().take(3).collect();
// Build answer from top results
let mut answer_parts =
vec!["Based on the knowledge base, here's what I found:".to_string()];
for doc in &top_results {
if let Some(title) = &doc.title {
answer_parts.push(format!(
"\n- **{}**: {}",
title,
&doc.content[..std::cmp::min(150, doc.content.len())]
));
} else if let Some(name) = &doc.name {
answer_parts.push(format!(
"\n- **{}**: {}",
name,
&doc.content[..std::cmp::min(150, doc.content.len())]
));
}
}
let mut answer = answer_parts.join("\n");
let sources: Vec<String> = top_results
.iter()
.filter_map(|d| d.title.clone().or_else(|| d.name.clone()))
.collect();
let confidence = (top_results.iter().filter_map(|d| d.relevance).sum::<u8>() as f32
/ top_results.len() as f32) as u8;
// Handle hybrid execution mode (auto-trigger tools if enabled)
let mut tool_executions = None;
if req.enable_tool_execution.unwrap_or(false) {
let max_tools = req.max_tool_calls.unwrap_or(3) as usize;
let tool_suggestions =
crate::tool_integration::analyze_for_tools(&answer, &req.question);
if let Some((executions, results)) =
self.execute_tools(&tool_suggestions, max_tools).await
{
if !results.is_empty() {
answer = crate::tool_integration::enrich_answer_with_results(answer, &results);
tool_executions = Some(executions);
}
}
}
Ok(AskResponse {
answer,
sources,
confidence,
reasoning: format!(
"Retrieved {} relevant documents using keyword search across {} total documents",
top_results.len(),
results.len()
),
tool_executions,
})
}
/// Get extension dependency DAG
pub async fn get_extension_dag(&self) -> Result<DagResponse> {
debug!("Building extension DAG");
let kb = self.knowledge_base.read().await;
// Build nodes from extension documents
let nodes: Vec<DagNode> = kb
.documents
.values()
.filter(|doc| doc.doc_type == "extension_metadata")
.map(|doc| DagNode {
name: doc.name.clone().unwrap_or_else(|| doc.id.clone()),
dependencies: doc.dependencies.clone().unwrap_or_default(),
version: "1.0.0".to_string(),
})
.collect();
// Build edges from dependency relationships
let edges: Vec<(String, String)> = kb
.relationships
.iter()
.filter(|rel| rel.relationship_type == "depends_on")
.map(|rel| {
let source = rel.source_id.strip_prefix("ext_").unwrap_or(&rel.source_id);
let target = rel.target_id.strip_prefix("ext_").unwrap_or(&rel.target_id);
(source.to_string(), target.to_string())
})
.collect();
Ok(DagResponse { nodes, edges })
}
/// Get best practices for a category
pub async fn get_best_practices(&self, category: &str) -> Result<Vec<BestPractice>> {
debug!("Retrieving best practices for category: {}", category);
let kb = self.knowledge_base.read().await;
// Filter documents by category and type
let mut practices: Vec<BestPractice> = kb
.documents
.values()
.filter(|doc| doc.category == category && doc.doc_type == "best_practice")
.map(|doc| BestPractice {
title: doc.title.clone().unwrap_or_else(|| doc.id.clone()),
description: doc.content.clone(),
category: doc.category.clone(),
relevance: doc.relevance.unwrap_or(70),
})
.collect();
// Sort by relevance descending
practices.sort_by(|a, b| b.relevance.cmp(&a.relevance));
Ok(practices)
}
/// Health check endpoint
pub async fn health_check(&self) -> Result<()> {
Ok(())
}
/// Get all available tools
pub fn list_all_tools(&self) -> Vec<crate::mcp::ToolDefinition> {
self.tool_registry.list_tools()
}
/// Get tools by category
pub fn tools_by_category(
&self,
category: crate::mcp::ToolCategory,
) -> Vec<crate::mcp::ToolDefinition> {
self.tool_registry.tools_by_category(category)
}
/// Check if a tool exists
pub fn has_tool(&self, name: &str) -> bool {
self.tool_registry.has_tool(name)
}
}
impl Default for AiService {
fn default() -> Self {
Self::new("127.0.0.1:8083".parse().unwrap())
}
}

203
crates/ai-service/src/tool_integration.rs Normal file
View file

@ -0,0 +1,203 @@
//! Tool integration and hybrid execution mode
//!
//! Analyzes RAG responses to suggest tool executions and enriches answers with
//! tool results.
use serde_json::Value;
/// Tool suggestion from RAG answer analysis
#[derive(Debug, Clone)]
pub struct ToolSuggestion {
pub tool_name: String,
pub confidence: f32,
pub args: Value,
}
/// Analyzes RAG answers to suggest relevant tools
///
/// Uses keyword matching and question pattern detection to identify tools that
/// might be useful.
pub fn analyze_for_tools(_answer: &str, question: &str) -> Vec<ToolSuggestion> {
let mut suggestions = Vec::new();
// System status patterns
if question_matches_any(
question,
&["status", "health", "running", "check", "what's"],
) {
suggestions.push(ToolSuggestion {
tool_name: "guidance_check_system_status".to_string(),
confidence: 0.7,
args: serde_json::json!({}),
});
}
// Configuration validation patterns
if question_matches_any(
question,
&["valid", "config", "configuration", "validate", "verify"],
) {
suggestions.push(ToolSuggestion {
tool_name: "guidance_validate_config".to_string(),
confidence: 0.6,
args: serde_json::json!({
"config_path": "/etc/provisioning/config.toml"
}),
});
}
// Documentation patterns
if question_matches_any(question, &["doc", "help", "guide", "tutorial", "how to"]) {
suggestions.push(ToolSuggestion {
tool_name: "guidance_find_docs".to_string(),
confidence: 0.5,
args: serde_json::json!({
"query": extract_main_topic(question)
}),
});
}
// Troubleshooting patterns
if question_matches_any(
question,
&["error", "fail", "problem", "issue", "fix", "debug"],
) {
suggestions.push(ToolSuggestion {
tool_name: "guidance_troubleshoot".to_string(),
confidence: 0.65,
args: serde_json::json!({
"error": extract_error_description(question)
}),
});
}
// Next action suggestions
if question_matches_any(question, &["next", "then", "after", "what should"]) {
suggestions.push(ToolSuggestion {
tool_name: "guidance_suggest_next_action".to_string(),
confidence: 0.55,
args: serde_json::json!({}),
});
}
// RAG tools based on keywords
if question_matches_any(question, &["search", "find", "look for", "retrieve"]) {
suggestions.push(ToolSuggestion {
tool_name: "rag_semantic_search".to_string(),
confidence: 0.6,
args: serde_json::json!({
"query": question.to_string(),
"top_k": 5
}),
});
}
// Filter out low confidence suggestions and sort by confidence descending
suggestions.sort_by(|a, b| b.confidence.partial_cmp(&a.confidence).unwrap());
suggestions
}
/// Checks if question contains any of the keywords
fn question_matches_any(question: &str, keywords: &[&str]) -> bool {
let lower = question.to_lowercase();
keywords.iter().any(|kw| lower.contains(kw))
}
/// Extracts main topic from question for search
fn extract_main_topic(question: &str) -> String {
// Simple heuristic: take the longest word or meaningful phrase
let words: Vec<&str> = question.split_whitespace().collect();
if words.is_empty() {
"provisioning".to_string()
} else {
words
.iter()
.max_by_key(|w| w.len())
.map(|w| w.to_string())
.unwrap_or_else(|| "provisioning".to_string())
}
}
/// Extracts error description from question
fn extract_error_description(question: &str) -> String {
// Take the full question as error context
question.to_string()
}
/// Enriches RAG answer with tool execution results
///
/// Appends tool execution results to the original answer.
pub fn enrich_answer_with_results(mut answer: String, tool_results: &[(String, Value)]) -> String {
if tool_results.is_empty() {
return answer;
}
answer.push_str("\n\n---\n\n**Tool Results:**\n\n");
for (tool_name, result) in tool_results {
answer.push_str(&format!("**{}:**\n", tool_name));
if let Some(status) = result.get("status") {
answer.push_str(&format!("Status: {}\n", status));
}
// Add tool-specific result formatting
if let Some(msg) = result.get("message") {
answer.push_str(&format!("{}\n", msg));
}
if let Some(suggestion) = result.get("suggestion") {
answer.push_str(&format!("→ {}\n", suggestion));
}
if let Some(diagnosis) = result.get("diagnosis") {
answer.push_str(&format!("{}\n", diagnosis));
}
answer.push('\n');
}
answer
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_system_status_detection() {
let question = "What is the current system status?";
let suggestions = analyze_for_tools("some answer", question);
assert!(suggestions
.iter()
.any(|s| s.tool_name == "guidance_check_system_status"));
}
#[test]
fn test_config_validation_detection() {
let question = "Is my configuration valid?";
let suggestions = analyze_for_tools("some answer", question);
assert!(suggestions
.iter()
.any(|s| s.tool_name == "guidance_validate_config"));
}
#[test]
fn test_doc_search_detection() {
let question = "How do I use the provisioning guide?";
let suggestions = analyze_for_tools("some answer", question);
assert!(suggestions
.iter()
.any(|s| s.tool_name == "guidance_find_docs"));
}
#[test]
fn test_answer_enrichment() {
let original = "RAG answer about provisioning".to_string();
let results = vec![(
"test_tool".to_string(),
serde_json::json!({"status": "success", "message": "Tool ran"}),
)];
let enriched = enrich_answer_with_results(original, &results);
assert!(enriched.contains("Tool Results"));
assert!(enriched.contains("test_tool"));
}
}

479
crates/ai-service/tests/phase4_integration_test.rs Normal file
View file

@ -0,0 +1,479 @@
//! Phase 4 Integration Tests: MCP Tool Integration with RAG
//!
//! Tests for tool registry, explicit tool calls, hybrid mode, and all tool
//! categories.
use std::net::SocketAddr;
use ai_service::mcp::ToolCategory;
use ai_service::service::{AiService, AskRequest, McpToolRequest};
use serde_json::json;
#[tokio::test]
async fn test_tool_registry_initialization() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
// Test that all tool categories are registered
let all_tools = service.list_all_tools();
assert!(!all_tools.is_empty(), "Tool registry should not be empty");
// Verify we have tools from each category
let categories: Vec<_> = all_tools.iter().map(|t| t.category).collect();
assert!(
categories.contains(&ToolCategory::Rag),
"RAG tools should be registered"
);
assert!(
categories.contains(&ToolCategory::Guidance),
"Guidance tools should be registered"
);
assert!(
categories.contains(&ToolCategory::Settings),
"Settings tools should be registered"
);
assert!(
categories.contains(&ToolCategory::Iac),
"IaC tools should be registered"
);
}
#[tokio::test]
async fn test_rag_tool_count() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
let rag_tools = service.tools_by_category(ToolCategory::Rag);
assert_eq!(
rag_tools.len(),
3,
"Should have 3 RAG tools: ask, search, status"
);
let tool_names: Vec<_> = rag_tools.iter().map(|t| t.name.as_str()).collect();
assert!(tool_names.contains(&"rag_ask_question"));
assert!(tool_names.contains(&"rag_semantic_search"));
assert!(tool_names.contains(&"rag_get_status"));
}
#[tokio::test]
async fn test_guidance_tool_count() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
let guidance_tools = service.tools_by_category(ToolCategory::Guidance);
assert_eq!(guidance_tools.len(), 5, "Should have 5 Guidance tools");
let tool_names: Vec<_> = guidance_tools.iter().map(|t| t.name.as_str()).collect();
assert!(tool_names.contains(&"guidance_check_system_status"));
assert!(tool_names.contains(&"guidance_suggest_next_action"));
assert!(tool_names.contains(&"guidance_find_docs"));
assert!(tool_names.contains(&"guidance_troubleshoot"));
assert!(tool_names.contains(&"guidance_validate_config"));
}
#[tokio::test]
async fn test_settings_tool_count() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
let settings_tools = service.tools_by_category(ToolCategory::Settings);
assert_eq!(settings_tools.len(), 7, "Should have 7 Settings tools");
}
#[tokio::test]
async fn test_iac_tool_count() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
let iac_tools = service.tools_by_category(ToolCategory::Iac);
assert_eq!(iac_tools.len(), 3, "Should have 3 IaC tools");
}
#[tokio::test]
async fn test_explicit_tool_call_rag_ask() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
let req = McpToolRequest {
tool_name: "rag_ask_question".to_string(),
args: json!({"question": "What is Nushell?"}),
};
let response = service
.call_mcp_tool(req)
.await
.expect("MCP tool call failed");
assert_eq!(response.result["status"], "success");
assert_eq!(response.result["tool"], "rag_ask_question");
}
#[tokio::test]
#[ignore]
async fn test_explicit_tool_call_guidance_status() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
let req = McpToolRequest {
tool_name: "guidance_check_system_status".to_string(),
args: json!({}),
};
let response = service
.call_mcp_tool(req)
.await
.expect("guidance_check_system_status failed");
assert_eq!(response.result["status"], "healthy");
assert_eq!(response.result["tool"], "guidance_check_system_status");
}
#[tokio::test]
async fn test_explicit_tool_call_settings() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
let req = McpToolRequest {
tool_name: "installer_get_settings".to_string(),
args: json!({}),
};
let response = service
.call_mcp_tool(req)
.await
.expect("MCP tool call failed");
assert_eq!(response.result["status"], "success");
// Verify real SettingsTools data is returned (not empty placeholder)
assert!(
response.result.get("platforms").is_some()
|| response.result.get("modes").is_some()
|| response.result.get("available_services").is_some(),
"Should return real settings data from SettingsTools"
);
}
#[tokio::test]
async fn test_settings_tools_platform_recommendations() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
let req = McpToolRequest {
tool_name: "installer_platform_recommendations".to_string(),
args: json!({}),
};
let response = service
.call_mcp_tool(req)
.await
.expect("MCP tool call failed");
assert_eq!(response.result["status"], "success");
// Should have real recommendations array from SettingsTools platform detection
assert!(response.result.get("recommendations").is_some());
}
#[tokio::test]
async fn test_settings_tools_mode_defaults() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
let req = McpToolRequest {
tool_name: "installer_get_defaults".to_string(),
args: json!({"mode": "solo"}),
};
let response = service
.call_mcp_tool(req)
.await
.expect("MCP tool call failed");
assert_eq!(response.result["status"], "success");
// Verify real mode defaults (resource requirements)
assert!(response.result.get("min_cpu_cores").is_some());
assert!(response.result.get("min_memory_gb").is_some());
}
#[tokio::test]
async fn test_explicit_tool_call_iac() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
let req = McpToolRequest {
tool_name: "iac_detect_technologies".to_string(),
args: json!({"path": "/tmp/infra"}),
};
let response = service
.call_mcp_tool(req)
.await
.expect("MCP tool call failed");
assert_eq!(response.result["status"], "success");
// Verify real technology detection (returns technologies array)
assert!(response.result.get("technologies").is_some());
}
#[tokio::test]
async fn test_iac_detect_technologies_real() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
// Test with provisioning directory that has Nickel files
let req = McpToolRequest {
tool_name: "iac_detect_technologies".to_string(),
args: json!({"path": "../../provisioning"}),
};
let response = service
.call_mcp_tool(req)
.await
.expect("MCP tool call failed");
assert_eq!(response.result["status"], "success");
// Should detect technologies as an array
let techs = response.result.get("technologies");
assert!(techs.is_some(), "Should have technologies array");
assert!(techs.unwrap().is_array());
}
#[tokio::test]
async fn test_iac_analyze_completeness() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
let req = McpToolRequest {
tool_name: "iac_analyze_completeness".to_string(),
args: json!({"path": "/tmp/test-infra"}),
};
let response = service
.call_mcp_tool(req)
.await
.expect("MCP tool call failed");
assert_eq!(response.result["status"], "success");
// Verify real analysis data
assert!(response.result.get("complete").is_some());
assert!(response.result.get("completeness_score").is_some());
assert!(response.result.get("missing_files").is_some());
}
#[tokio::test]
async fn test_unknown_tool_error() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
let req = McpToolRequest {
tool_name: "unknown_tool_xyz".to_string(),
args: json!({}),
};
let result = service.call_mcp_tool(req).await;
assert!(result.is_err(), "Should fail with unknown tool");
}
#[tokio::test]
async fn test_hybrid_mode_disabled() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
// Load knowledge base (required for ask)
service
.load_knowledge_base("../../config/knowledge-base")
.await
.ok();
let req = AskRequest {
question: "What are deployment best practices?".to_string(),
context: None,
enable_tool_execution: Some(false), // Explicitly disabled
max_tool_calls: None,
};
let response = service.ask(req).await.unwrap();
// Should not have tool executions when disabled
assert!(
response.tool_executions.is_none() || response.tool_executions.as_ref().unwrap().is_empty(),
"Tool executions should be empty when disabled"
);
}
#[tokio::test]
async fn test_hybrid_mode_enabled() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
// Load knowledge base
service
.load_knowledge_base("../../config/knowledge-base")
.await
.ok();
let req = AskRequest {
question: "What is the current system status and best practices?".to_string(),
context: None,
enable_tool_execution: Some(true), // Enable hybrid mode
max_tool_calls: Some(3),
};
let response = service.ask(req).await.unwrap();
// Should have RAG answer
assert!(!response.answer.is_empty(), "Should have RAG answer");
// Tool executions may or may not occur depending on tool suggestions
// The important thing is that when enabled, the mechanism works
}
#[tokio::test]
async fn test_max_tool_calls_limit() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
service
.load_knowledge_base("../../config/knowledge-base")
.await
.ok();
let req = AskRequest {
question: "What is system status and what should I do next and how do I find \
documentation?"
.to_string(),
context: None,
enable_tool_execution: Some(true),
max_tool_calls: Some(1), // Limit to 1 tool
};
let response = service.ask(req).await.unwrap();
// Even if multiple tools are suggested, only max_tool_calls should execute
if let Some(executions) = &response.tool_executions {
assert!(
executions.len() <= 1,
"Should respect max_tool_calls limit of 1"
);
}
}
#[tokio::test]
async fn test_tool_definition_schemas() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
let all_tools = service.list_all_tools();
// Verify all tools have proper definitions
for tool in all_tools {
assert!(!tool.name.is_empty(), "Tool name should not be empty");
assert!(
!tool.description.is_empty(),
"Tool description should not be empty"
);
// Verify input schema is valid JSON
assert!(
tool.input_schema.is_object(),
"Input schema should be an object"
);
assert!(
tool.input_schema.get("type").is_some(),
"Input schema should have 'type' field"
);
}
}
#[tokio::test]
async fn test_tool_execution_with_required_args() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
// Tool that requires arguments should work when provided
let req = McpToolRequest {
tool_name: "rag_semantic_search".to_string(),
args: json!({"query": "kubernetes"}),
};
let response = service
.call_mcp_tool(req)
.await
.expect("MCP tool call failed");
assert_eq!(response.result["status"], "success");
}
#[tokio::test]
async fn test_tool_execution_error_handling() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
// Tool that requires arguments should fail when not provided
let req = McpToolRequest {
tool_name: "rag_semantic_search".to_string(),
args: json!({}), // Missing required 'query'
};
let result = service.call_mcp_tool(req).await;
// Should either fail or return an error in the result
match result {
Ok(response) => {
// Even if it doesn't fail, it should indicate an error
assert!(
response.result.get("status").is_some() || response.result.get("error").is_some()
);
}
Err(_) => {
// Expected: missing required parameter
}
}
}
#[tokio::test]
#[ignore] // Requires Nushell to be available
async fn test_guidance_tools_nushell_execution() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
// Test system status tool (requires Nushell)
let req = McpToolRequest {
tool_name: "guidance_check_system_status".to_string(),
args: json!({}),
};
let response = service.call_mcp_tool(req).await;
// Should either succeed with Nushell data or fail with Nushell not found
match response {
Ok(result) => {
// If Nushell is available, should have JSON data
assert!(result.result.is_object());
}
Err(e) => {
// Expected if Nushell not available
let err_msg = e.to_string();
assert!(err_msg.contains("Nushell") || err_msg.contains("command"));
}
}
}
#[tokio::test]
#[ignore] // Requires Nushell to be available
async fn test_guidance_find_docs() {
let addr: SocketAddr = "127.0.0.1:8083".parse().unwrap();
let service = AiService::new(addr);
// Test documentation finding tool (requires Nushell)
let req = McpToolRequest {
tool_name: "guidance_find_docs".to_string(),
args: json!({"query": "deployment"}),
};
let response = service.call_mcp_tool(req).await;
match response {
Ok(result) => {
// If Nushell is available, should have JSON data
assert!(result.result.is_object());
}
Err(e) => {
// Expected if Nushell not available
let err_msg = e.to_string();
assert!(err_msg.contains("Nushell") || err_msg.contains("command"));
}
}
}

30
crates/audit-mirror/Cargo.toml Normal file
View file

@ -0,0 +1,30 @@
[package]
authors.workspace = true
edition.workspace = true
license.workspace = true
name = "audit-mirror"
version.workspace = true
description = "NATS ops.audit.* → Radicle git commit sidecar with jti idempotency (ADR-038)"
[[bin]]
name = "audit-mirror"
path = "src/main.rs"
[dependencies]
async-nats = { workspace = true }
tokio = { workspace = true, features = ["full"] }
serde = { workspace = true }
serde_json = { workspace = true }
thiserror = { workspace = true }
anyhow = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
chrono = { workspace = true }
clap = { workspace = true }
futures = { workspace = true }
git2 = { workspace = true }
bytes = { workspace = true }
reqwest = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }

99
crates/audit-mirror/src/commit_writer.rs Normal file
View file

@ -0,0 +1,99 @@
use git2::{Repository, Signature};
use crate::error::MirrorError;
/// Writes a single commit into the HEAD of `repo` embedding `jti` in the message.
///
/// The commit message format is:
/// audit: jti=<jti>
///
/// <json_payload>
///
/// This is the canonical format that `jti_check::already_committed` searches for.
/// An empty tree is used because the audit repo contains only commit-message blobs;
/// no working-tree files are modified.
pub fn write_audit_commit(
repo: &Repository,
jti: &str,
json_payload: &str,
author_name: &str,
author_email: &str,
) -> Result<git2::Oid, MirrorError> {
let sig = Signature::now(author_name, author_email)
.map_err(|e| MirrorError::Git(e.to_string()))?;
let tree_id = {
let mut index = repo.index().map_err(|e| MirrorError::Git(e.to_string()))?;
// Write tree from current index (may be empty — no working-tree blobs needed)
index.write_tree().map_err(|e| MirrorError::Git(e.to_string()))?
};
let tree = repo
.find_tree(tree_id)
.map_err(|e| MirrorError::Git(e.to_string()))?;
let message = format!("audit: jti={jti}\n\n{json_payload}");
// Parent is HEAD if it exists; first commit for an empty (unborn) branch has no parent.
let parent_commit;
let parents: Vec<&git2::Commit<'_>>;
match repo.head() {
Ok(head_ref) => {
parent_commit = head_ref
.peel_to_commit()
.map_err(|e| MirrorError::Git(e.to_string()))?;
parents = vec![&parent_commit];
}
Err(e) if e.code() == git2::ErrorCode::UnbornBranch => {
parents = vec![];
}
Err(e) => return Err(MirrorError::Git(e.to_string())),
}
let oid = repo
.commit(Some("HEAD"), &sig, &sig, &message, &tree, &parents)
.map_err(|e| MirrorError::Git(e.to_string()))?;
Ok(oid)
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::TempDir;
fn init_repo() -> (TempDir, Repository) {
let dir = TempDir::new().unwrap();
let repo = Repository::init(dir.path()).unwrap();
let mut cfg = repo.config().unwrap();
cfg.set_str("user.name", "test").unwrap();
cfg.set_str("user.email", "test@test.com").unwrap();
(dir, repo)
}
#[test]
fn writes_commit_to_empty_repo() {
let (_dir, repo) = init_repo();
let oid = write_audit_commit(&repo, "jti-001", r#"{"event":"test"}"#, "ci", "ci@ci.local").unwrap();
let commit = repo.find_commit(oid).unwrap();
assert!(commit.message().unwrap().contains("jti=jti-001"));
assert!(commit.message().unwrap().contains(r#"{"event":"test"}"#));
}
#[test]
fn writes_second_commit_with_correct_parent() {
let (_dir, repo) = init_repo();
let first = write_audit_commit(&repo, "jti-001", "{}", "ci", "ci@ci.local").unwrap();
let second = write_audit_commit(&repo, "jti-002", "{}", "ci", "ci@ci.local").unwrap();
let commit = repo.find_commit(second).unwrap();
assert_eq!(commit.parent_id(0).unwrap(), first);
}
#[test]
fn message_contains_jti_marker() {
let (_dir, repo) = init_repo();
let oid = write_audit_commit(&repo, "abc-xyz", r#"{"op":"deploy"}"#, "keeper", "keeper@ops").unwrap();
let msg = repo.find_commit(oid).unwrap().message().unwrap().to_string();
assert!(msg.starts_with("audit: jti=abc-xyz\n\n"));
}
}

19
crates/audit-mirror/src/error.rs Normal file
View file

@ -0,0 +1,19 @@
use thiserror::Error;
#[derive(Debug, Error)]
pub enum MirrorError {
#[error("git error: {0}")]
Git(String),
#[error("NATS error: {0}")]
Nats(String),
#[error("json error: {0}")]
Json(#[from] serde_json::Error),
#[error("io error: {0}")]
Io(#[from] std::io::Error),
}
impl From<git2::Error> for MirrorError {
fn from(e: git2::Error) -> Self {
MirrorError::Git(e.to_string())
}
}

92
crates/audit-mirror/src/jti_check.rs Normal file
View file

@ -0,0 +1,92 @@
use git2::Repository;
use crate::error::MirrorError;
/// Returns true if a commit with the given jti already exists in the HEAD ancestor chain.
/// Searches commit messages for the marker `jti=<value>` added by commit_writer.
///
/// This is the idempotency guard required by ADR-038 constraint audit-mirror-idempotent-on-jti.
pub fn already_committed(repo: &Repository, jti: &str) -> Result<bool, MirrorError> {
let head = match repo.head() {
Ok(r) => r,
Err(e) if e.code() == git2::ErrorCode::UnbornBranch => return Ok(false),
Err(e) => return Err(MirrorError::Git(e.to_string())),
};
let head_commit = head
.peel_to_commit()
.map_err(|e| MirrorError::Git(e.to_string()))?;
let marker = format!("jti={jti}");
let mut revwalk = repo
.revwalk()
.map_err(|e| MirrorError::Git(e.to_string()))?;
revwalk
.push(head_commit.id())
.map_err(|e| MirrorError::Git(e.to_string()))?;
revwalk.set_sorting(git2::Sort::TOPOLOGICAL | git2::Sort::TIME)
.map_err(|e| MirrorError::Git(e.to_string()))?;
for oid in revwalk {
let oid = oid.map_err(|e| MirrorError::Git(e.to_string()))?;
let commit = repo
.find_commit(oid)
.map_err(|e| MirrorError::Git(e.to_string()))?;
if commit.message().unwrap_or("").contains(&marker) {
return Ok(true);
}
}
Ok(false)
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::TempDir;
fn init_bare_repo_with_commit(message: &str) -> (TempDir, Repository) {
let dir = TempDir::new().unwrap();
let repo = Repository::init(dir.path()).unwrap();
{
let mut config = repo.config().unwrap();
config.set_str("user.name", "test").unwrap();
config.set_str("user.email", "test@test.com").unwrap();
}
let sig = git2::Signature::now("test", "test@test.com").unwrap();
{
let tree_id = {
let mut index = repo.index().unwrap();
index.write_tree().unwrap()
};
let tree = repo.find_tree(tree_id).unwrap();
repo.commit(Some("HEAD"), &sig, &sig, message, &tree, &[])
.unwrap();
// tree drops here so repo can be moved below
}
(dir, repo)
}
#[test]
fn detects_existing_jti_in_head_commit() {
let (_dir, repo) = init_bare_repo_with_commit("audit: jti=abc-123\n\npayload");
assert!(already_committed(&repo, "abc-123").unwrap());
}
#[test]
fn returns_false_for_unknown_jti() {
let (_dir, repo) = init_bare_repo_with_commit("audit: jti=abc-123\n\npayload");
assert!(!already_committed(&repo, "unknown-jti").unwrap());
}
#[test]
fn returns_false_for_empty_repo() {
let dir = TempDir::new().unwrap();
let repo = Repository::init(dir.path()).unwrap();
assert!(!already_committed(&repo, "any-jti").unwrap());
}
}

188
crates/audit-mirror/src/main.rs Normal file
View file

@ -0,0 +1,188 @@
mod commit_writer;
mod error;
mod jti_check;
mod radicle_publish;
use std::path::PathBuf;
use async_nats::jetstream::{self, consumer::PullConsumer};
use bytes::Bytes;
use clap::Parser;
use futures::StreamExt;
use git2::Repository;
use tracing::{error, info, warn};
use commit_writer::write_audit_commit;
use jti_check::already_committed;
use radicle_publish::announce_repo;
#[derive(Parser)]
#[command(name = "audit-mirror", about = "NATS ops.audit.* → Radicle git commit sidecar (ADR-038)")]
struct Cli {
/// NATS server URL
#[arg(long, default_value = "nats://127.0.0.1:4222")]
nats_url: String,
/// Workspace name (used to derive stream and consumer names)
#[arg(long)]
workspace: String,
/// Path to the local Radicle storage repo for this workspace's state
/// (e.g. ~/.radicle/storage/<rid>)
#[arg(long)]
state_repo_path: PathBuf,
/// Radicle Repository ID (RID) for gossip announce
#[arg(long)]
rid: String,
/// Author name embedded in git commits
#[arg(long, default_value = "audit-mirror")]
author_name: String,
/// Author email embedded in git commits
#[arg(long, default_value = "audit-mirror@ops.local")]
author_email: String,
/// Radicle local HTTP API base URL
#[arg(long, default_value = "http://127.0.0.1:8776")]
radicle_api_url: String,
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
tracing_subscriber::fmt()
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
.init();
let cli = Cli::parse();
let nats = async_nats::connect(&cli.nats_url).await?;
let js = jetstream::new(nats);
let stream_name = format!("OPS_AUDIT_{}", cli.workspace.to_uppercase());
let consumer_name = format!("{}-audit-mirror", cli.workspace);
// Ensure stream exists (idempotent — ADR-038 requires the stream be pre-provisioned
// by ops-controller, so we only access it here, not create it).
let stream = js.get_stream(&stream_name).await.map_err(|e| {
format!("stream '{stream_name}' not found — ensure ops-controller has provisioned it: {e}")
})?;
let consumer: PullConsumer = stream
.get_or_create_consumer(
&consumer_name,
jetstream::consumer::pull::Config {
durable_name: Some(consumer_name.clone()),
ack_policy: jetstream::consumer::AckPolicy::Explicit,
filter_subject: format!("ops.audit.{}.>", cli.workspace),
..Default::default()
},
)
.await?;
info!(
stream = stream_name,
consumer = consumer_name,
repo = ?cli.state_repo_path,
"audit-mirror ready"
);
let repo = Repository::open(&cli.state_repo_path).map_err(|e| {
format!("cannot open state repo at {:?}: {e}", cli.state_repo_path)
})?;
let mut messages = consumer.messages().await?;
while let Some(msg_result) = messages.next().await {
let msg = match msg_result {
Ok(m) => m,
Err(e) => {
error!("NATS message error: {e}");
continue;
}
};
let jti = match extract_jti(&msg.payload) {
Some(j) => j,
None => {
warn!(subject = %msg.subject, "no jti in audit payload; acking and skipping");
let _ = msg.ack().await;
continue;
}
};
match already_committed(&repo, &jti) {
Ok(true) => {
info!(jti, "jti already committed — idempotent skip");
let _ = msg.ack().await;
continue;
}
Ok(false) => {}
Err(e) => {
error!(jti, error = %e, "jti_check failed; leaving in queue for retry");
continue;
}
}
let payload_str = String::from_utf8_lossy(&msg.payload);
match write_audit_commit(
&repo,
&jti,
&payload_str,
&cli.author_name,
&cli.author_email,
) {
Ok(oid) => {
info!(jti, commit = %oid, "audit commit written");
}
Err(e) => {
error!(jti, error = %e, "commit_writer failed; leaving in queue for retry");
continue;
}
}
// Best-effort gossip announce — non-fatal, commit is durable.
if let Err(e) = announce_repo(&cli.rid, &cli.radicle_api_url).await {
warn!(jti, error = %e, "announce_repo failed (non-fatal)");
}
if let Err(e) = msg.ack().await {
warn!(jti, error = %e, "ack failed (JetStream will redeliver; jti_check will deduplicate)");
}
}
warn!("NATS message stream ended; audit-mirror exiting");
Ok(())
}
/// Extracts the `jti` field from a JSON payload.
fn extract_jti(payload: &Bytes) -> Option<String> {
let v: serde_json::Value = serde_json::from_slice(payload).ok()?;
v.get("jti")?.as_str().map(str::to_owned)
}
#[cfg(test)]
mod tests {
use super::*;
use bytes::Bytes;
#[test]
fn extracts_jti_from_payload() {
let payload = Bytes::from(r#"{"jti":"abc-123","event":"deploy"}"#);
assert_eq!(extract_jti(&payload), Some("abc-123".to_string()));
}
#[test]
fn returns_none_for_missing_jti() {
let payload = Bytes::from(r#"{"event":"deploy"}"#);
assert_eq!(extract_jti(&payload), None);
}
#[test]
fn returns_none_for_invalid_json() {
let payload = Bytes::from(b"not-json" as &[u8]);
assert_eq!(extract_jti(&payload), None);
}
}

35
crates/audit-mirror/src/radicle_publish.rs Normal file
View file

@ -0,0 +1,35 @@
use tracing::{info, warn};
/// Signals the local Radicle node to announce updated refs for `rid`.
///
/// Radicle Heartwood exposes an HTTP API at `http://127.0.0.1:8776`. After
/// writing a new audit commit to the Radicle storage repo, we POST to the node
/// sync endpoint so gossip propagates immediately rather than waiting for the
/// node's background scan interval.
///
/// Failure here is non-fatal: the commit is already durable in the local git
/// store and radicled will eventually pick it up via its storage-directory
/// watcher. We log a warning and return Ok(()) so the caller can ack the
/// JetStream message.
pub async fn announce_repo(rid: &str, node_api_url: &str) -> Result<(), crate::error::MirrorError> {
let url = format!("{node_api_url}/v1/repos/{rid}/sync");
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(5))
.build()
.map_err(|e| crate::error::MirrorError::Nats(format!("HTTP client init: {e}")))?;
match client.post(&url).send().await {
Ok(resp) if resp.status().is_success() => {
info!(rid, "Radicle repo sync triggered successfully");
}
Ok(resp) => {
warn!(rid, status = %resp.status(), "Radicle sync returned non-success; node will sync on next scan");
}
Err(e) => {
warn!(rid, error = %e, "Radicle sync request failed (non-fatal; local commit is durable)");
}
}
Ok(())
}

39
crates/backup-manager/Cargo.toml Normal file
View file

@ -0,0 +1,39 @@
[package]
authors.workspace = true
edition.workspace = true
license.workspace = true
name = "backup-manager"
repository.workspace = true
version.workspace = true
description = "Multi-context backup orchestrator: restic-first with kopia opt-in, multi-destination replication, ontoref-driven docs. One-shot, daemon, standalone, or coordinator modes. Distinct from platform-backup (../prov-ecosystem/crates/backup) which is a low-level multi-backend library; backup-manager is the orchestrator binary."
keywords = ["backup", "restic", "kopia", "orchestrator", "ontoref"]
categories = ["command-line-utilities", "development-tools"]
[lib]
name = "backup_manager"
path = "src/lib.rs"
[[bin]]
name = "prvng-backup"
path = "src/main.rs"
[dependencies]
platform-config = { workspace = true }
anyhow = { workspace = true }
async-trait = { workspace = true }
chrono = { workspace = true }
clap = { version = "4", features = ["derive", "env"] }
serde = { workspace = true }
serde_json = { workspace = true }
thiserror = { workspace = true }
tokio = { workspace = true, features = ["full"] }
toml = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
uuid = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }
tokio-test = { workspace = true }

329
crates/backup-manager/src/config.rs Normal file
View file

@ -0,0 +1,329 @@
//! Configuration loading for the backup-manager via [`platform_config::ConfigLoader`].
//!
//! Configuration files are searched along the standard hierarchy
//! (`~/.config/provisioning/platform/config/backup-manager.ncl` on Linux,
//! `~/Library/Application Support/...` on macOS) and merged with environment
//! variable overrides (`BACKUP_*` mapped to JSON paths). Nickel contracts
//! validate the merged configuration BEFORE serde deserialisation, so type
//! errors surface as Nickel error messages, not serde "expected struct field".
//!
//! See ADR-015 (`adrs/adr-015-config-driven-platform-config.ncl`) for the
//! mandate that every platform crate uses this loader.
use std::path::Path;
use platform_config::ConfigLoader;
use serde::{Deserialize, Serialize};
/// Top-level configuration for the backup-manager binary across all four modes.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct BackupManagerConfig {
/// Daemon-mode runtime parameters. Ignored in `one-shot` and `standalone`.
#[serde(default)]
pub daemon: DaemonConfig,
/// Provider configuration (default provider, version pinning).
#[serde(default)]
pub providers: ProvidersConfig,
/// Path (or glob) to the directory holding policy Nickel files.
/// Resolved at runtime by invoking `nickel export` on each match.
#[serde(default)]
pub policies: PoliciesRef,
/// secretumvault client configuration.
#[serde(default)]
pub vault: VaultClientConfig,
/// platform-nats client configuration.
#[serde(default)]
pub nats: NatsClientConfig,
/// Prometheus metrics emission configuration.
#[serde(default)]
pub metrics: MetricsConfig,
}
/// Daemon-mode parameters: priority queue, concurrency, API endpoint.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DaemonConfig {
/// HTTP API listen address (e.g. `127.0.0.1:9099`).
#[serde(default = "default_daemon_http_addr")]
pub http_addr: String,
/// Unix socket path for local CLI communication.
#[serde(default = "default_daemon_unix_socket")]
pub unix_socket: String,
/// Maximum concurrent backup operations across all destinations.
#[serde(default = "default_max_parallel_backups")]
pub max_parallel_backups: usize,
/// Maximum concurrent operations per destination.
#[serde(default = "default_max_parallel_per_destination")]
pub max_parallel_per_destination: usize,
/// Path where the daemon persists its state (queue, last-run timestamps).
#[serde(default = "default_state_path")]
pub state_path: String,
}
impl Default for DaemonConfig {
fn default() -> Self {
Self {
http_addr: default_daemon_http_addr(),
unix_socket: default_daemon_unix_socket(),
max_parallel_backups: default_max_parallel_backups(),
max_parallel_per_destination: default_max_parallel_per_destination(),
state_path: default_state_path(),
}
}
}
fn default_daemon_http_addr() -> String { "127.0.0.1:9099".to_string() }
fn default_daemon_unix_socket() -> String { "/run/prvng-backup.sock".to_string() }
fn default_max_parallel_backups() -> usize { 4 }
fn default_max_parallel_per_destination() -> usize { 2 }
fn default_state_path() -> String { "/var/lib/prvng-backup/state".to_string() }
/// Provider defaults and version pinning.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ProvidersConfig {
/// Provider used when a `BackupPolicy` does not specify one explicitly.
#[serde(default = "default_provider_name")]
pub default: String,
/// Minimum acceptable restic CLI version. The daemon emits a warning if
/// the installed binary is older. Format: `semver` ("0.16.0").
#[serde(default)]
pub restic_min_version: Option<String>,
/// Minimum acceptable kopia CLI version, when kopia is referenced.
#[serde(default)]
pub kopia_min_version: Option<String>,
}
impl Default for ProvidersConfig {
fn default() -> Self {
Self {
default: default_provider_name(),
restic_min_version: None,
kopia_min_version: None,
}
}
}
fn default_provider_name() -> String { "restic".to_string() }
/// Reference to the directory tree where `BackupPolicy` / `BackupGroup` /
/// `SystemBackupDef` Nickel files are committed.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PoliciesRef {
/// Root directory containing component policies (`infra/<workspace>/components/`).
#[serde(default = "default_components_path")]
pub components_path: String,
/// Path to the BackupGroup declarations file.
#[serde(default = "default_groups_path")]
pub groups_path: String,
/// Path to the SystemBackupDef declarations file.
#[serde(default = "default_system_backups_path")]
pub system_backups_path: String,
/// Nickel `--import-path` value used when exporting policies.
#[serde(default = "default_nickel_import_path")]
pub nickel_import_path: String,
}
impl Default for PoliciesRef {
fn default() -> Self {
Self {
components_path: default_components_path(),
groups_path: default_groups_path(),
system_backups_path: default_system_backups_path(),
nickel_import_path: default_nickel_import_path(),
}
}
}
fn default_components_path() -> String {
"infra/libre-wuji/components".to_string()
}
fn default_groups_path() -> String {
"infra/libre-wuji/backup-groups.ncl".to_string()
}
fn default_system_backups_path() -> String {
"infra/libre-wuji/system-backups.ncl".to_string()
}
fn default_nickel_import_path() -> String { "provisioning".to_string() }
/// secretumvault connection parameters.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct VaultClientConfig {
/// Vault HTTP endpoint (e.g. `https://vault.libre-wuji.svc.cluster.local:8200`).
#[serde(default)]
pub endpoint: Option<String>,
/// Path inside vault where the manager's NKey/JWT lives for auth.
#[serde(default)]
pub auth_path: Option<String>,
/// Audit log target (subject prefix or file path).
#[serde(default)]
pub audit_target: Option<String>,
}
/// platform-nats connection parameters (mirrors `NatsConnectionConfig`).
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct NatsClientConfig {
/// NATS URL (e.g. `nats://nats.ops-system.svc:4222`).
#[serde(default)]
pub url: Option<String>,
/// Path inside vault where the NKey seed is stored.
#[serde(default)]
pub nkey_seed_vault_path: Option<String>,
/// Require signed messages on subscribed subjects.
#[serde(default)]
pub require_signed_messages: bool,
}
/// Prometheus metrics configuration.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MetricsConfig {
/// Metrics emission mode. Daemon mode uses HTTP; one-shot uses textfile.
#[serde(default)]
pub mode: MetricsMode,
/// HTTP listen address when `mode = "http"`.
#[serde(default = "default_metrics_http_addr")]
pub http_addr: String,
/// Textfile path when `mode = "textfile"` (consumed by node_exporter).
#[serde(default = "default_metrics_textfile_path")]
pub textfile_path: String,
}
impl Default for MetricsConfig {
fn default() -> Self {
Self {
mode: MetricsMode::default(),
http_addr: default_metrics_http_addr(),
textfile_path: default_metrics_textfile_path(),
}
}
}
/// Metrics emission mode.
#[derive(Debug, Clone, Copy, Serialize, Deserialize, Default, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum MetricsMode {
/// HTTP endpoint scraped by Prometheus (used by daemon mode).
#[default]
Http,
/// Textfile written to a directory scraped by node_exporter (used by one-shot).
Textfile,
}
fn default_metrics_http_addr() -> String { "127.0.0.1:9100".to_string() }
fn default_metrics_textfile_path() -> String {
"/var/lib/node-exporter/textfile_collector/prvng-backup.prom".to_string()
}
impl ConfigLoader for BackupManagerConfig {
fn service_name() -> &'static str {
"backup-manager"
}
fn collect_env_overrides() -> serde_json::Value {
// Map `BACKUP_*` environment variables to nested JSON paths matching
// the structure of `BackupManagerConfig`. Only set keys are included;
// the merge happens inside `platform_config::format::load_config_with_overrides`.
let mut overrides = serde_json::Map::new();
let mut daemon = serde_json::Map::new();
if let Ok(v) = std::env::var("BACKUP_DAEMON_HTTP_ADDR") {
daemon.insert("http_addr".into(), serde_json::Value::String(v));
}
if let Ok(v) = std::env::var("BACKUP_DAEMON_UNIX_SOCKET") {
daemon.insert("unix_socket".into(), serde_json::Value::String(v));
}
if !daemon.is_empty() {
overrides.insert("daemon".into(), serde_json::Value::Object(daemon));
}
let mut vault = serde_json::Map::new();
if let Ok(v) = std::env::var("BACKUP_VAULT_ENDPOINT") {
vault.insert("endpoint".into(), serde_json::Value::String(v));
}
if !vault.is_empty() {
overrides.insert("vault".into(), serde_json::Value::Object(vault));
}
let mut nats = serde_json::Map::new();
if let Ok(v) = std::env::var("BACKUP_NATS_URL") {
nats.insert("url".into(), serde_json::Value::String(v));
}
if !nats.is_empty() {
overrides.insert("nats".into(), serde_json::Value::Object(nats));
}
let mut providers = serde_json::Map::new();
if let Ok(v) = std::env::var("BACKUP_PROVIDER_DEFAULT") {
providers.insert("default".into(), serde_json::Value::String(v));
}
if !providers.is_empty() {
overrides.insert("providers".into(), serde_json::Value::Object(providers));
}
serde_json::Value::Object(overrides)
}
fn from_path<P: AsRef<Path>>(
path: P,
) -> std::result::Result<Self, Box<dyn std::error::Error + Send + Sync>> {
let path = path.as_ref();
let overrides = Self::collect_env_overrides();
let json_value = platform_config::format::load_config_with_overrides(path, &overrides)
.map_err(|e| -> Box<dyn std::error::Error + Send + Sync> { Box::new(e) })?;
serde_json::from_value(json_value).map_err(|e| {
let err_msg = format!(
"Failed to deserialize backup-manager config from {:?}: {}",
path, e
);
Box::new(std::io::Error::new(std::io::ErrorKind::InvalidData, err_msg))
as Box<dyn std::error::Error + Send + Sync>
})
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn defaults_are_consistent() {
let cfg = BackupManagerConfig::default();
assert_eq!(cfg.daemon.http_addr, "127.0.0.1:9099");
assert_eq!(cfg.providers.default, "restic");
assert_eq!(cfg.metrics.mode, MetricsMode::Http);
assert!(!cfg.nats.require_signed_messages);
}
#[test]
fn service_name_matches_practice_node_artifact_path() {
assert_eq!(BackupManagerConfig::service_name(), "backup-manager");
}
#[test]
fn env_overrides_emit_only_set_keys() {
// Without env vars set, the override map should be empty.
// (The test runs in an isolated process; if BACKUP_* leaks from the
// host shell the assertion is loose: just ensure it's a JSON object.)
let v = BackupManagerConfig::collect_env_overrides();
assert!(v.is_object());
}
}

72
crates/backup-manager/src/error.rs Normal file
View file

@ -0,0 +1,72 @@
//! Error types for the backup-manager crate.
//!
//! All public APIs return [`Result<T>`] which is shorthand for
//! `std::result::Result<T, BackupError>`.
use thiserror::Error;
/// Top-level error type for the backup-manager.
#[derive(Debug, Error)]
pub enum BackupError {
/// Configuration could not be loaded or validated.
#[error("configuration error: {0}")]
Config(String),
/// A policy file could not be deserialised or violated a contract.
#[error("policy error: {0}")]
Policy(String),
/// A backup provider (restic, kopia) failed.
#[error("provider '{provider}' error: {message}")]
Provider {
/// Provider identifier (`restic`, `kopia`, ...).
provider: String,
/// Underlying error message from the provider invocation.
message: String,
},
/// A destination was unreachable or rejected the operation.
#[error("destination '{destination}' error: {message}")]
Destination {
/// Destination identifier (matches `BackupPolicy.destinations[].name`).
destination: String,
/// Underlying error.
message: String,
},
/// The vault (secretumvault) returned an error or was unreachable.
#[error("vault error: {0}")]
Vault(String),
/// NATS publish/subscribe error.
#[error("nats error: {0}")]
Nats(String),
/// A Kubernetes API call failed.
#[error("kubernetes error: {0}")]
Kubernetes(String),
/// Filesystem I/O error.
#[error("io error: {0}")]
Io(#[from] std::io::Error),
/// JSON (de)serialisation error.
#[error("json error: {0}")]
Json(#[from] serde_json::Error),
/// A pre/post hook exited non-zero.
#[error("hook '{hook}' failed: exit {exit_code}")]
HookFailed {
/// Hook identifier (`pre`, `post`, or a named hook).
hook: String,
/// Process exit code.
exit_code: i32,
},
/// Any other anyhow-flavoured failure that does not fit the categories above.
#[error("{0}")]
Other(#[from] anyhow::Error),
}
/// Crate-local result alias.
pub type Result<T> = std::result::Result<T, BackupError>;

60
crates/backup-manager/src/lib.rs Normal file
View file

@ -0,0 +1,60 @@
//! Multi-context backup orchestrator: restic-first with kopia opt-in, multi-destination replication, ontoref-driven docs.
//!
//! `backup-manager` runs in four modes that share a single binary, configuration
//! loader, schema set, and event surface:
//!
//! - **one-shot** — invoked by a scheduler (Kubernetes CronJob, system cron,
//! systemd timer). Loads the policy, executes the requested operation
//! (`backup`, `restore`, `verify`, `prune`, `list`), reports through metrics
//! and NATS events, terminates.
//! - **daemon** — long-running orchestrator. Holds a priority queue, enforces
//! concurrency budgets, reloads policies via [`platform_config`] watcher,
//! exposes a local HTTP/Unix socket API, and emits Prometheus metrics on a
//! dedicated endpoint.
//! - **standalone** — workstation mode. Connects to a destination directly with
//! the operator's age key and runs `mount`, `restore`, or `list` without
//! needing the cluster, the daemon, or even a Nickel runtime (the binary
//! embeds the schema subset required to deserialise policies).
//! - **coordinator** — orchestrates `BackupGroup` consistent cuts across
//! members. Tags every member's snapshot with the same `group_id` and
//! `group_ts` so a single timestamp recovers the whole group.
//!
//! ## Schema integration
//!
//! Policies are declared in Nickel (`provisioning/schemas/lib/backup_policy.ncl`,
//! `backup_group.ncl`, `system_backup.ncl`, `verify_policy.ncl`, `concerns.ncl`).
//! The Nickel layer enforces three invariants at export time:
//!
//! - `EncryptionRequired` — every [`BackupProvider`](policy::BackupProvider)
//! must declare `features.encryption = true`.
//! - `MultiDestinationRequired` — every enabled
//! [`BackupPolicy`](policy::BackupPolicy) must declare ≥2 destinations,
//! at least one with `role = 'primary`.
//! - `NonEmptyScopes` — every enabled `BackupPolicy` must list at least one
//! `BackupScope`.
//!
//! ## Custody model (ADR-011)
//!
//! Four custody boxes, never co-dependent: keys live in `secretumvault`,
//! definitions in git, data in ≥2 destinations, production in the cluster.
//! Standalone restore needs only one custody box plus the bootstrap key.
//!
//! ## ontoref integration
//!
//! This crate is registered as the `backup-manager` Practice node in
//! `.ontology/core.ncl`. `ontoref generate-mdbook` extracts this `//!` and
//! produces architecture documentation automatically. The first sentence above
//! is Jaccard-aligned with the node `description` and validated by
//! `ontoref sync diff --docs`.
#![forbid(unsafe_code)]
#![warn(missing_docs)]
#![warn(rust_2018_idioms)]
pub mod config;
pub mod error;
pub mod modes;
pub mod policy;
pub use config::BackupManagerConfig;
pub use error::{BackupError, Result};

348
crates/backup-manager/src/main.rs Normal file
View file

@ -0,0 +1,348 @@
//! `prvng-backup` — entry point for the backup-manager binary.
//!
//! Subcommands map to the four modes described in the crate-level docs.
//! This first cut implements the `policy` subcommand end-to-end (validate
//! and render Nickel-exported policies); the other modes are stubs whose
//! signatures and CLI surface are wired so the contract is visible to users
//! and downstream code while the implementations land in subsequent commits.
use std::path::{Path, PathBuf};
use std::process::ExitCode;
use clap::{Args, Parser, Subcommand};
use backup_manager::modes::policy_cmds;
/// CLI for the backup-manager.
#[derive(Debug, Parser)]
#[command(
name = "prvng-backup",
version,
about = "Multi-context backup orchestrator (restic/kopia, ontoref-integrated)"
)]
struct Cli {
/// Path to a configuration file. When absent, the standard hierarchy
/// (`~/.config/provisioning/platform/config/backup-manager.ncl`) is used.
#[arg(short, long, env = "BACKUP_CONFIG")]
config: Option<PathBuf>,
/// Subcommand selecting the mode of operation.
#[command(subcommand)]
command: Mode,
}
/// Top-level mode dispatch.
#[derive(Debug, Subcommand)]
enum Mode {
/// Render and validate Nickel policy declarations.
Policy(PolicyArgs),
/// One-shot operation invoked by a scheduler (cron, systemd timer).
OneShot(OneShotArgs),
/// Long-running daemon orchestrator.
Daemon(DaemonArgs),
/// Standalone workstation mode (mount/restore/list without cluster).
Standalone(StandaloneArgs),
/// BackupGroup coordinator.
Coordinator(CoordinatorArgs),
}
// ── policy ──────────────────────────────────────────────────────────────────
#[derive(Debug, Args)]
struct PolicyArgs {
/// Sub-action to perform.
#[command(subcommand)]
action: PolicyAction,
}
#[derive(Debug, Subcommand)]
enum PolicyAction {
/// Validate one or more Nickel policy files (export + deserialise).
Validate {
/// Files or directories to validate.
#[arg(required = true)]
paths: Vec<PathBuf>,
/// Nickel `--import-path` value.
#[arg(long, default_value = "provisioning")]
import_path: String,
},
/// Render a policy file to JSON on stdout.
Render {
/// Policy file to render.
path: PathBuf,
/// Nickel `--import-path` value.
#[arg(long, default_value = "provisioning")]
import_path: String,
},
}
// ── one-shot ────────────────────────────────────────────────────────────────
#[derive(Debug, Args)]
struct OneShotArgs {
/// Operation to perform.
#[command(subcommand)]
op: OneShotOp,
}
#[derive(Debug, Subcommand)]
enum OneShotOp {
/// Run a backup for a component, group, or system target.
Backup {
/// Target identifier.
target: String,
/// Optional scope filter.
#[arg(long)]
scope: Option<String>,
},
/// Restore a snapshot to a directory.
Restore {
/// Target identifier.
target: String,
/// Snapshot ID or `latest`.
#[arg(long)]
snapshot: String,
/// Restore destination directory.
#[arg(long)]
to: PathBuf,
},
/// Verify snapshots (provider-level integrity checks).
Verify {
/// Target identifier.
target: String,
/// Verify level.
#[arg(long, default_value = "quick")]
level: String,
},
/// Apply retention policy and prune old snapshots.
Prune {
/// Target identifier.
target: String,
/// Dry run (do not delete).
#[arg(long)]
dry_run: bool,
},
/// List snapshots.
List {
/// Target identifier.
target: String,
/// Optional scope filter.
#[arg(long)]
scope: Option<String>,
},
}
// ── daemon ──────────────────────────────────────────────────────────────────
#[derive(Debug, Args)]
struct DaemonArgs {
/// Daemon control action.
#[command(subcommand)]
action: DaemonAction,
}
#[derive(Debug, Subcommand)]
enum DaemonAction {
/// Start the daemon (foreground).
Start,
/// Reload policies (sends SIGHUP equivalent through the local API).
Reload,
/// Drain the queue and shut down cleanly.
Drain,
/// Print status as JSON.
Status,
}
// ── standalone ──────────────────────────────────────────────────────────────
#[derive(Debug, Args)]
struct StandaloneArgs {
/// Standalone operation.
#[command(subcommand)]
op: StandaloneOp,
}
#[derive(Debug, Subcommand)]
enum StandaloneOp {
/// Mount a snapshot via FUSE.
Mount {
/// Component name.
component: String,
/// Snapshot ID or `latest`.
#[arg(long)]
snapshot: String,
/// Mountpoint directory.
#[arg(long)]
at: PathBuf,
/// Tag filters (`key=value`).
#[arg(long)]
tag: Vec<String>,
},
/// Restore directly to a directory without going through the daemon.
Restore {
/// Component name.
component: String,
/// Snapshot ID or `latest`.
#[arg(long)]
snapshot: String,
/// Restore destination directory.
#[arg(long)]
to: PathBuf,
},
/// List snapshots (no daemon).
List {
/// Component name.
component: String,
/// Optional scope filter.
#[arg(long)]
scope: Option<String>,
},
}
// ── coordinator ─────────────────────────────────────────────────────────────
#[derive(Debug, Args)]
struct CoordinatorArgs {
/// Coordinator action.
#[command(subcommand)]
action: CoordinatorAction,
}
#[derive(Debug, Subcommand)]
enum CoordinatorAction {
/// Run a BackupGroup.
GroupRun {
/// Group name.
name: String,
},
/// Restore a BackupGroup at a given timestamp.
GroupRestore {
/// Group name.
name: String,
/// RFC3339 timestamp.
#[arg(long)]
at: String,
/// Restore destination directory.
#[arg(long)]
to: PathBuf,
},
}
#[tokio::main]
async fn main() -> ExitCode {
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_env("BACKUP_LOG")
.unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")),
)
.init();
let cli = Cli::parse();
match cli.command {
Mode::Policy(args) => run_policy(args),
Mode::OneShot(_) | Mode::Daemon(_) | Mode::Standalone(_) | Mode::Coordinator(_) => {
eprintln!(
"this mode is wired in the CLI surface but its implementation \
lands in a subsequent commit. The `policy` subcommand is fully \
functional today (validate / render Nickel-exported policies)."
);
ExitCode::from(2)
}
}
}
fn run_policy(args: PolicyArgs) -> ExitCode {
match args.action {
PolicyAction::Validate { paths, import_path } => run_policy_validate(&paths, &import_path),
PolicyAction::Render { path, import_path } => run_policy_render(&path, &import_path),
}
}
fn run_policy_validate(paths: &[PathBuf], import_path: &str) -> ExitCode {
let mut total_failures = 0usize;
for path in paths {
total_failures += validate_path(path, import_path);
}
if total_failures == 0 {
ExitCode::SUCCESS
} else {
eprintln!("{} validation failure(s)", total_failures);
ExitCode::from(1)
}
}
fn validate_path(path: &Path, import_path: &str) -> usize {
if path.is_dir() {
validate_directory(path, import_path)
} else {
let v = policy_cmds::validate_policy(path, import_path);
print_validation(&v);
usize::from(!v.ok)
}
}
fn validate_directory(path: &Path, import_path: &str) -> usize {
let entries = match std::fs::read_dir(path) {
Ok(e) => e,
Err(e) => {
eprintln!("could not read directory {:?}: {}", path, e);
return 1;
}
};
entries
.flatten()
.filter_map(|entry| {
let p = entry.path();
if p.extension().is_some_and(|e| e == "ncl") {
Some(p)
} else {
None
}
})
.map(|p| {
let v = policy_cmds::validate_policy(&p, import_path);
print_validation(&v);
usize::from(!v.ok)
})
.sum()
}
fn run_policy_render(path: &Path, import_path: &str) -> ExitCode {
let value = match policy_cmds::nickel_export_to_json(path, import_path) {
Ok(v) => v,
Err(e) => {
eprintln!("{}", e);
return ExitCode::from(1);
}
};
match serde_json::to_string_pretty(&value) {
Ok(rendered) => {
println!("{}", rendered);
ExitCode::SUCCESS
}
Err(e) => {
eprintln!("failed to render JSON: {}", e);
ExitCode::from(1)
}
}
}
fn print_validation(v: &policy_cmds::PolicyValidation) {
if v.ok {
println!(
"OK {} components={} backup-enabled={}",
v.source, v.component_count, v.enabled_backup_components.len()
);
} else {
println!(
"FAIL {} error={}",
v.source,
v.error.as_deref().unwrap_or("unknown")
);
}
}

7
crates/backup-manager/src/modes/mod.rs Normal file
View file

@ -0,0 +1,7 @@
//! Execution modes — `one-shot`, `daemon`, `standalone`, `coordinator`.
//!
//! Each mode is a distinct subcommand of the [`prvng-backup`](../../bin/main.rs)
//! binary. The mode is selected at process start; the configuration loaded
//! from [`crate::BackupManagerConfig`] is shared across all modes.
pub mod policy_cmds;

170
crates/backup-manager/src/modes/policy_cmds.rs Normal file
View file

@ -0,0 +1,170 @@
//! `policy` subcommand — render and validate `BackupPolicy` declarations.
//!
//! This is the first end-to-end mode wired into the binary. It exercises:
//!
//! - Reading workspace declarations via `nickel export --import-path …`.
//! - Deserialising the resulting JSON into [`crate::policy`] structs.
//! - Reporting structured information about policies, groups, and system
//! backup definitions without touching destinations or vault.
//!
//! Subsequent modes (`backup`, `restore`, `mount`, etc.) build on the same
//! deserialisation pipeline implemented here.
use std::path::Path;
use std::process::Command;
use crate::error::{BackupError, Result};
use crate::policy::BackupPolicy;
/// Render a Nickel file to JSON via the `nickel` CLI.
///
/// Honours the `nickel_import_path` provided in [`crate::config::PoliciesRef`].
pub fn nickel_export_to_json(path: &Path, import_path: &str) -> Result<serde_json::Value> {
let output = Command::new("nickel")
.args(["export", "--format", "json", "--import-path", import_path])
.arg(path)
.output()
.map_err(|e| BackupError::Policy(format!(
"failed to invoke `nickel export` on {:?}: {}", path, e
)))?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(BackupError::Policy(format!(
"nickel export failed for {:?}: {}", path, stderr
)));
}
serde_json::from_slice::<serde_json::Value>(&output.stdout).map_err(BackupError::from)
}
/// Result of validating a single policy file.
#[derive(Debug)]
pub struct PolicyValidation {
/// Path to the source Nickel file.
pub source: String,
/// Whether the file successfully exported and deserialised.
pub ok: bool,
/// Error message if validation failed.
pub error: Option<String>,
/// Component count (top-level keys in the JSON object).
pub component_count: usize,
/// Names of components for which `concerns.backup.kind == "enabled"`.
pub enabled_backup_components: Vec<String>,
}
/// Validate a single Nickel policy file by exporting and deserialising it.
pub fn validate_policy(path: &Path, import_path: &str) -> PolicyValidation {
let source = path.display().to_string();
let json = match nickel_export_to_json(path, import_path) {
Ok(v) => v,
Err(e) => {
return PolicyValidation {
source,
ok: false,
error: Some(e.to_string()),
component_count: 0,
enabled_backup_components: Vec::new(),
};
}
};
let object = match json.as_object() {
Some(map) => map,
None => {
return PolicyValidation {
source,
ok: false,
error: Some("policy file did not export a JSON object".to_string()),
component_count: 0,
enabled_backup_components: Vec::new(),
};
}
};
let mut enabled = Vec::new();
for (component_name, component_value) in object {
// Best-effort: locate concerns.backup.kind inside the component.
// Components produced by `ext.make_X` helpers may nest the policy in
// different ways; we don't fail on missing concerns here, we just
// skip when not found. Strict enforcement is the Nickel layer's job.
if let Some(kind) = component_value
.pointer("/concerns/backup/kind")
.and_then(serde_json::Value::as_str)
{
if kind == "enabled" {
enabled.push(component_name.clone());
}
}
}
PolicyValidation {
source,
ok: true,
error: None,
component_count: object.len(),
enabled_backup_components: enabled,
}
}
/// Try to deserialise a `BackupPolicy` directly from its JSON representation.
/// Used by tests and by future scope-expansion code; not exposed via CLI yet.
pub fn deserialize_backup_policy(value: serde_json::Value) -> Result<BackupPolicy> {
serde_json::from_value(value).map_err(BackupError::from)
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn deserialize_minimal_backup_policy() {
let value = json!({
"provider": { "name": "restic" },
"destinations": [
{
"name": "primary",
"kind": "s3",
"uri": "s3:fsn1.hetzner/test",
"cred_ref": { "path": "creds/test", "kind": "s3" },
"role": "primary"
},
{
"name": "replica",
"kind": "b2",
"uri": "b2:test-replica",
"cred_ref": { "path": "creds/replica", "kind": "b2" },
"role": "replica"
}
],
"encryption": {
"path": "backup-manager/master-key",
"algorithm": "age_x25519"
},
"schedule": { "kind": "cron", "cron_expr": "0 2 * * *" },
"retention": {
"keep_last": 7,
"keep_daily": 7,
"keep_weekly": 4,
"keep_monthly": 6,
"keep_yearly": 0
},
"scopes": [
{
"kind": "service_full",
"name": "main",
"paths": ["/data"]
}
],
"tag_strategy": {
"component_label": "test-component"
}
});
let policy = deserialize_backup_policy(value).expect("deserialise");
assert_eq!(policy.destinations.len(), 2);
assert_eq!(policy.scopes.len(), 1);
assert_eq!(policy.provider.name, "restic");
}
}

324
crates/backup-manager/src/policy/component.rs Normal file
View file

@ -0,0 +1,324 @@
//! Component-level policy: `BackupPolicy`, `Destination`, `RetentionPolicy`,
//! `Schedule`, `BackupProviderRef`, plus the surrounding `ServiceConcerns`
//! umbrella.
use std::collections::BTreeMap;
use serde::{Deserialize, Serialize};
use super::scope::BackupScope;
// ============================================================================
// ServiceConcerns umbrella (ADR-008)
// ============================================================================
/// One of `enabled`, `disabled`, `pending`, `inherited`.
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum ConcernStateKind {
/// Concern is implemented; the impl payload field is populated.
Enabled,
/// Concern is explicitly opted out.
Disabled,
/// Concern is deferred; `backlog_ref` points to the tracking item.
Pending,
/// Concern is inherited from a parent component.
Inherited,
}
/// State of one concern within [`ServiceConcerns`].
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ConcernState {
/// Discriminator.
pub kind: ConcernStateKind,
/// `disabled` reason.
#[serde(default)]
pub reason: Option<String>,
/// `disabled` since (ISO date).
#[serde(default)]
pub since: Option<String>,
/// `pending` backlog reference.
#[serde(default)]
pub backlog_ref: Option<String>,
/// `pending` target iteration.
#[serde(default)]
pub target_iteration: Option<String>,
/// `inherited` parent component.
#[serde(default)]
pub from: Option<String>,
/// `enabled` payload — only one of these is populated based on which
/// concern this state belongs to.
#[serde(default)]
pub backup_impl: Option<BackupPolicy>,
/// `enabled` payload for non-backup concerns kept opaque at this layer
/// (TLS/DNS/certs/observability/security details are not consumed by the
/// backup-manager binary).
#[serde(default, flatten)]
pub other_impl: BTreeMap<String, serde_json::Value>,
}
/// Mandatory umbrella in `ComponentDef.concerns`.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ServiceConcerns {
/// TLS concern state (opaque to the backup binary).
pub tls: ConcernState,
/// DNS concern state.
pub dns: ConcernState,
/// Certificates concern state.
pub certs: ConcernState,
/// Backup concern state — when `kind = Enabled`, `backup_impl` is set.
pub backup: ConcernState,
/// Observability concern state.
pub observability: ConcernState,
/// Security concern state.
pub security: ConcernState,
}
// ============================================================================
// BackupPolicy
// ============================================================================
/// Reference to a [`BackupProvider`] declared under
/// `provisioning/catalog/providers/backup/<name>/`.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BackupProviderRef {
/// Provider directory name (e.g. `"restic"`, `"kopia"`).
pub name: String,
/// Pinned CLI version. The daemon emits a warning if the installed
/// binary is older.
#[serde(default)]
pub version: Option<String>,
}
/// Schedule discriminated union (cron / interval / NATS-event).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ScheduleSpec {
/// `cron`, `interval`, or `on_event`.
pub kind: String,
/// Cron expression when `kind = "cron"`.
#[serde(default)]
pub cron_expr: Option<String>,
/// Jitter seconds applied to cron firing.
#[serde(default)]
pub jitter_sec: Option<u64>,
/// Period when `kind = "interval"`.
#[serde(default)]
pub every: Option<String>,
/// Random jitter for interval scheduling.
#[serde(default)]
pub jitter: Option<String>,
/// NATS subject when `kind = "on_event"`.
#[serde(default)]
pub subject: Option<String>,
/// Debounce duration for event-driven schedules.
#[serde(default)]
pub debounce: Option<String>,
}
/// Retention preset. Mirrors restic's `--keep-*` flags.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct RetentionPolicy {
/// `--keep-last`.
#[serde(default)]
pub keep_last: u32,
/// `--keep-daily`.
#[serde(default)]
pub keep_daily: u32,
/// `--keep-weekly`.
#[serde(default)]
pub keep_weekly: u32,
/// `--keep-monthly`.
#[serde(default)]
pub keep_monthly: u32,
/// `--keep-yearly`.
#[serde(default)]
pub keep_yearly: u32,
/// Hard upper bound on snapshot age (deleted regardless of keep_*).
#[serde(default)]
pub prune_after: Option<String>,
}
/// Destination protocol kind.
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum DestinationKind {
/// S3-compatible object storage (Hetzner, MinIO, AWS S3, etc.).
S3,
/// Backblaze B2.
B2,
/// Local filesystem (only valid for archive role).
Local,
/// SFTP target.
Sftp,
/// restic REST server.
RestServer,
}
/// Role of a destination within a policy. The `MultiDestinationRequired`
/// Nickel contract requires at least one `Primary` and a total of ≥2 entries.
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum DestinationRole {
/// First-write target. Backups land here directly.
Primary,
/// Replicated copy of `Primary` for off-site durability.
Replica,
/// Cold archive (typically slower or cheaper storage).
Archive,
}
/// Backup destination specification.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Destination {
/// Stable identifier used in metric labels and tags.
pub name: String,
/// Protocol kind.
pub kind: DestinationKind,
/// Provider URI (e.g. `s3:fsn1.hetzner/libre-wuji-backups`).
pub uri: String,
/// Reference to credentials in `secretumvault`.
pub cred_ref: VaultCredRef,
/// Role within the policy.
pub role: DestinationRole,
/// Region (for cloud providers that support it).
#[serde(default)]
pub region: Option<String>,
}
/// Reference to a credentials entry in vault.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct VaultCredRef {
/// Path under vault (e.g. `backup-manager/destinations/hetzner-primary`).
pub path: String,
/// Type of credential payload at the path.
pub kind: String,
}
/// Reference to an encryption key in vault.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct VaultKeyRef {
/// Path under vault.
pub path: String,
/// Algorithm (`age_x25519`, `aes_gcm_256`, etc.).
#[serde(default)]
pub algorithm: String,
/// HKDF derivation parameters when applicable.
#[serde(default)]
pub derivation: Option<HkdfDerivation>,
}
/// HKDF derivation parameters.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct HkdfDerivation {
/// Method (`none` or `hkdf_sha256`).
pub method: String,
/// HKDF info parameter.
pub info: String,
}
/// Tag generation strategy for snapshots.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TagStrategy {
/// Used as the `component=<value>` tag.
pub component_label: String,
/// Additional static tags as `k=v` strings.
#[serde(default)]
pub extra: Vec<String>,
}
/// Pre/post hooks executed by the manager around the backup run.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct Hooks {
/// Commands run before backup; non-zero exit aborts when `abort_on_failure = true`.
#[serde(default)]
pub pre: Vec<String>,
/// Commands run after backup, regardless of outcome.
#[serde(default)]
pub post: Vec<String>,
/// Hook timeout (e.g. `"5m"`).
#[serde(default = "default_hook_timeout")]
pub timeout: String,
/// Whether a non-zero pre hook aborts the backup.
#[serde(default = "default_abort_on_failure")]
pub abort_on_failure: bool,
}
fn default_hook_timeout() -> String { "5m".to_string() }
fn default_abort_on_failure() -> bool { true }
/// Bandwidth throttle (passed to provider as `--limit-upload`/`--limit-download`).
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct Throttle {
/// Upload limit in KB/s.
#[serde(default)]
pub upload_kbps: Option<u32>,
/// Download limit in KB/s.
#[serde(default)]
pub download_kbps: Option<u32>,
}
/// Reference to a verify policy (drill spec lives separately to keep policies
/// focused).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct VerifyPolicyRef {
/// Verify schedule.
#[serde(default)]
pub schedule: Option<ScheduleSpec>,
/// Level: `quick`, `deep`, `restore_drill`, `full_dr`.
#[serde(default = "default_verify_level")]
pub level: String,
/// Reference to a `DrillSpec` by name.
#[serde(default)]
pub drill_ref: Option<String>,
}
fn default_verify_level() -> String { "quick".to_string() }
/// Component-level backup policy.
///
/// The Nickel contracts `EncryptionRequired`, `MultiDestinationRequired`, and
/// `NonEmptyScopes` are enforced at policy export time, so values reaching
/// this struct already satisfy the invariants.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BackupPolicy {
/// Provider reference.
pub provider: BackupProviderRef,
/// ≥2 destinations, at least one with `role = Primary`.
pub destinations: Vec<Destination>,
/// Encryption key reference.
pub encryption: VaultKeyRef,
/// Schedule.
pub schedule: ScheduleSpec,
/// Retention policy.
pub retention: RetentionPolicy,
/// Scopes (≥1).
pub scopes: Vec<BackupScope>,
/// Tag strategy.
pub tag_strategy: TagStrategy,
/// Optional hooks.
#[serde(default)]
pub hooks: Option<Hooks>,
/// Optional verify reference.
#[serde(default)]
pub verify: Option<VerifyPolicyRef>,
/// Optional throttle.
#[serde(default)]
pub throttle: Option<Throttle>,
/// Optional consistency-group name (this policy participates in a `BackupGroup`).
#[serde(default)]
pub consistency_group: Option<String>,
}

68
crates/backup-manager/src/policy/group.rs Normal file
View file

@ -0,0 +1,68 @@
//! BackupGroup — multi-component consistency points.
use serde::{Deserialize, Serialize};
use super::component::{Destination, RetentionPolicy, ScheduleSpec, TagStrategy, VaultKeyRef,
VerifyPolicyRef};
/// Member of a [`BackupGroup`].
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GroupMember {
/// Component name.
pub component: String,
/// Optional scope filter; omitted means all scopes of the component.
#[serde(default)]
pub scope: Option<String>,
}
/// Coordination strategy discriminator.
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum CoordinationKind {
/// Snapshots fired in parallel; consistency is tag-based (group_id).
BestEffort,
/// Ordered pre-hooks pause writers, snapshot, unquiesce. Bounded downtime.
QuiesceWindow,
/// Atomic at CSI layer via Longhorn VolumeSnapshotGroup.
CsiConsistentGroup,
}
/// Coordination strategy applied during a group run.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CoordinationStrategy {
/// Variant.
pub kind: CoordinationKind,
/// Quiesce sequence (`QuiesceWindow`).
#[serde(default)]
pub quiesce_seq: Vec<String>,
/// Maximum downtime for `QuiesceWindow`.
#[serde(default)]
pub max_downtime: Option<String>,
/// CSI VolumeSnapshotClass for `CsiConsistentGroup`.
#[serde(default)]
pub snapshot_class: Option<String>,
}
/// A group of components/scopes captured atomically (à la Chandy-Lamport).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BackupGroup {
/// Group identifier (used in CLI: `--group <name>`).
pub name: String,
/// Members participating in the consistent cut.
pub members: Vec<GroupMember>,
/// Group schedule (independent of per-member policies).
pub schedule: ScheduleSpec,
/// Coordination strategy.
pub coordination: CoordinationStrategy,
/// Retention shared across all members for the group's snapshots.
pub retention: RetentionPolicy,
/// Destinations (same `MultiDestinationRequired` invariant).
pub destinations: Vec<Destination>,
/// Encryption key.
pub encryption: VaultKeyRef,
/// Tag strategy.
pub tag_strategy: TagStrategy,
/// Optional verify reference.
#[serde(default)]
pub verify: Option<VerifyPolicyRef>,
}

26
crates/backup-manager/src/policy/mod.rs Normal file
View file

@ -0,0 +1,26 @@
//! Policy types deserialised from Nickel-exported JSON.
//!
//! Nickel files under `provisioning/schemas/lib/` define the canonical schema
//! (BackupPolicy, BackupGroup, SystemBackupDef, ServiceConcerns, etc.).
//! The manager invokes `nickel export` on workspace declarations and parses
//! the resulting JSON into these structs. The JSON tag conventions mirror the
//! Nickel discriminated unions: `kind` field plus payload variants.
//!
//! Only the shape needed at runtime is materialised here; the full Nickel
//! schema includes optional fields (e.g. extended `dns_records.extra`) that
//! we treat as `serde_json::Value` to avoid coupling the binary to every
//! schema evolution.
pub mod component;
pub mod group;
pub mod scope;
pub mod system;
pub use component::{
BackupPolicy, BackupProviderRef, ConcernState, Destination, DestinationKind,
DestinationRole, RetentionPolicy, ScheduleSpec, ServiceConcerns,
};
pub use group::{BackupGroup, CoordinationKind, CoordinationStrategy, GroupMember};
pub use scope::{BackupScope, DumpStrategy, DumpStrategyKind, ScopeKind};
pub use system::{HostSelector, HostSelectorKind, SystemBackupDef, SystemBackupTarget,
SystemBackupTargetKind};

175
crates/backup-manager/src/policy/scope.rs Normal file
View file

@ -0,0 +1,175 @@
//! BackupScope and DumpStrategy types — the unit of backup work.
//!
//! Each scope expands to one provider invocation with deterministic tags.
//! The discriminator is `kind`; the rest of the fields are populated only
//! for variants that use them (Nickel emits `default = []`/`default = {}`).
use std::collections::BTreeMap;
use serde::{Deserialize, Serialize};
/// Discriminator for [`BackupScope`].
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum ScopeKind {
/// Snapshot a fixed list of paths.
ServiceFull,
/// Snapshot per domain (one tag set per domain).
PerDomain,
/// Snapshot per mailbox (selector-driven enumeration).
PerMailbox,
/// Snapshot a database via a [`DumpStrategy`].
Database,
/// Snapshot a CSI volume by name.
VolumeSnapshot,
/// Archive logs from a source (loki / journald / files).
LogsArchive,
/// Export a subset of a key-value store (etcd / consul).
KvExport,
}
/// Database engine, when scope kind is [`ScopeKind::Database`].
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum DbEngine {
/// PostgreSQL — supports streaming pg_dump and WAL archiving externally.
Postgresql,
/// MariaDB.
Mariadb,
/// MySQL.
Mysql,
/// Redis (RDB snapshot or AOF).
Redis,
/// MongoDB (mongodump).
Mongodb,
/// SurrealDB.
Surrealdb,
/// etcd (etcdctl snapshot).
Etcd,
/// SQLite (file-level).
Sqlite,
}
/// Discriminator for [`DumpStrategy`].
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum DumpStrategyKind {
/// Stream the dump straight to the provider's stdin.
StreamToStdin,
/// Write the dump to a path, then snapshot the path.
DumpToPath,
/// Like `DumpToPath` but with a custom dump command.
PreDumpThenPath,
/// CSI VolumeSnapshot CRD (crash-consistent, no app coordination).
CsiVolumeSnapshot,
/// Application quiesce hook + snapshot + unquiesce.
AppQuiesceThenSnapshot,
}
/// Strategy for capturing a database scope. See ADR-013 for the consistency
/// trade-offs across variants.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DumpStrategy {
/// Variant discriminator.
pub kind: DumpStrategyKind,
/// Custom dump command (optional, used by `PreDumpThenPath`).
#[serde(default)]
pub dump_command: Option<String>,
/// Path where the dump file lives (`DumpToPath`, `PreDumpThenPath`).
#[serde(default)]
pub path: Option<String>,
/// Whether to remove the dump file after the snapshot completes.
#[serde(default = "default_true")]
pub cleanup: bool,
/// CSI volume name (`CsiVolumeSnapshot`, `AppQuiesceThenSnapshot`).
#[serde(default)]
pub volume: Option<String>,
/// CSI VolumeSnapshotClass.
#[serde(default)]
pub snapshot_class: Option<String>,
/// Quiesce command (`AppQuiesceThenSnapshot`).
#[serde(default)]
pub quiesce_cmd: Option<String>,
/// Unquiesce command (`AppQuiesceThenSnapshot`).
#[serde(default)]
pub unquiesce_cmd: Option<String>,
}
fn default_true() -> bool { true }
/// One unit of backup work. The provider receives this via [`crate::policy::component::BackupPolicy`].
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BackupScope {
/// Discriminator.
pub kind: ScopeKind,
/// Identifier for the scope (used in CLI: `--scope <name>`).
pub name: String,
/// Path list (`ServiceFull`, `LogsArchive`).
#[serde(default)]
pub paths: Vec<String>,
/// Exclusion globs.
#[serde(default)]
pub exclude: Vec<String>,
/// Domain list (`PerDomain`).
#[serde(default)]
pub domains: Vec<String>,
/// Base path for per-domain / per-mailbox scopes.
#[serde(default)]
pub base_path: String,
/// Mailbox selector (`PerMailbox`).
#[serde(default)]
pub selector: Option<String>,
/// Database engine (`Database`).
#[serde(default)]
pub engine: Option<DbEngine>,
/// Dump strategy (`Database`).
#[serde(default)]
pub dump_strategy: Option<DumpStrategy>,
/// Volume names (`VolumeSnapshot`).
#[serde(default)]
pub volumes: Vec<String>,
/// CSI VolumeSnapshotClass (`VolumeSnapshot`).
#[serde(default)]
pub snapshot_class: Option<String>,
/// Log sources (`LogsArchive`).
#[serde(default)]
pub sources: Vec<String>,
/// Log archive format (`LogsArchive`).
#[serde(default)]
pub format: Option<String>,
/// Rotation duration (`LogsArchive`).
#[serde(default)]
pub rotation: Option<String>,
/// KV source kind (`KvExport`).
#[serde(default)]
pub source: Option<String>,
/// Tag prefix prepended to determinístico tags.
#[serde(default)]
pub tag_prefix: String,
/// Static extra tags (key-value).
#[serde(default)]
pub tags: BTreeMap<String, String>,
}

196
crates/backup-manager/src/policy/system.rs Normal file
View file

@ -0,0 +1,196 @@
//! SystemBackupDef — backup specifications for artefacts outside the cluster.
//!
//! Targets include etcd, k8s certs, host configs, external DNS configurations,
//! builder environment tools, provisioning state, log archives, SOPS/Age key
//! bundles, and vault state. Each target is a discriminated variant; the
//! manager dispatches based on `kind` (see ADR-011 for vault state custody).
use serde::{Deserialize, Serialize};
use super::component::{
BackupProviderRef, Destination, RetentionPolicy, ScheduleSpec, TagStrategy, Throttle,
VaultCredRef, VaultKeyRef, VerifyPolicyRef, Hooks,
};
/// Discriminator for [`HostSelector`].
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum HostSelectorKind {
/// Run on a single control-plane host (any).
CpOnly,
/// Run on the first control-plane host (deterministic by name sort).
CpFirst,
/// Run on all control-plane hosts.
ControlPlanes,
/// Run on all worker hosts.
Workers,
/// Run on every server in the workspace.
AllServers,
/// Run on the listed hosts.
List,
}
/// Selects which hosts execute a system backup. Mirrors the Nickel `HostSelector`.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct HostSelector {
/// Variant.
pub kind: HostSelectorKind,
/// Hostname list when `kind = List`.
#[serde(default)]
pub members: Vec<String>,
}
/// Discriminator for [`SystemBackupTarget`].
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum SystemBackupTargetKind {
/// etcd snapshot via etcdctl.
Etcd,
/// Kubernetes PKI certificates (`/etc/kubernetes/pki`).
K8sCerts,
/// Cluster resources (Secret/ConfigMap/CRD content).
ClusterResources,
/// Longhorn engine state.
LonghornEngine,
/// Host configuration files.
HostConfigs,
/// External DNS server configuration and zone files.
ExternalDns,
/// Builder environment tools and caches.
BuilderEnv,
/// Provisioning state (definitions, state, locks).
ProvisioningState,
/// Log archive (loki / journald / files).
LogsArchive,
/// SOPS/Age key bundle (separate custody chain).
SopsKeys,
/// secretumvault state (with bootstrap-key encryption per ADR-011).
VaultState,
}
/// Discriminated target description.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SystemBackupTarget {
/// Variant.
pub kind: SystemBackupTargetKind,
// ── 'etcd
/// etcd endpoints.
#[serde(default)]
pub endpoints: Vec<String>,
/// CA reference.
#[serde(default)]
pub ca_ref: Option<VaultCredRef>,
/// Client cert reference.
#[serde(default)]
pub cert_ref: Option<VaultCredRef>,
/// Client key reference.
#[serde(default)]
pub key_ref: Option<VaultCredRef>,
// ── 'k8s_certs / 'host_configs / 'logs_archive (paths)
/// Path list.
#[serde(default)]
pub paths: Vec<String>,
/// Exclusion globs.
#[serde(default)]
pub exclude: Vec<String>,
// ── 'cluster_resources
/// Namespace list.
#[serde(default)]
pub namespaces: Vec<String>,
/// Kind list (`secret`, `certificate`, ...).
#[serde(default)]
pub kinds: Vec<String>,
// ── 'longhorn_engine
/// Component list (`volumes`, `engines`, `replicas`, `settings`).
#[serde(default)]
pub components: Vec<String>,
// ── 'external_dns
/// Source kind (`coredns`, `powerdns`, `unbound`, ...).
#[serde(default)]
pub source_kind: Option<String>,
/// Config file paths.
#[serde(default)]
pub config_paths: Vec<String>,
/// Zone file directories.
#[serde(default)]
pub zones_paths: Vec<String>,
// ── 'builder_env
/// Tool names (informational tags).
#[serde(default)]
pub tools: Vec<String>,
/// Secret names that must accompany the artefact.
#[serde(default)]
pub secrets: Vec<String>,
// ── 'provisioning_state
/// Definitions directory.
#[serde(default)]
pub definitions_path: Option<String>,
/// State directory.
#[serde(default)]
pub state_path: Option<String>,
/// Lock directory.
#[serde(default)]
pub lock_path: Option<String>,
// ── 'logs_archive
/// Loki/journald selector.
#[serde(default)]
pub selector: Option<String>,
/// Archive format.
#[serde(default)]
pub format: Option<String>,
// ── 'sops_keys / 'vault_state
/// Age key file paths.
#[serde(default)]
pub age_keys: Vec<String>,
/// Recipient public keys.
#[serde(default)]
pub recipients: Vec<String>,
/// Vault HTTP endpoint to back up.
#[serde(default)]
pub vault_endpoint: Option<String>,
/// Specific vault paths to capture (omitted = full state).
#[serde(default)]
pub vault_paths: Vec<String>,
}
/// Top-level system backup definition.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SystemBackupDef {
/// Identifier (used in CLI: `prvng-backup one-shot backup <name>`).
pub name: String,
/// Target description.
pub target: SystemBackupTarget,
/// Host selector.
pub host_selector: HostSelector,
/// Provider.
pub provider: BackupProviderRef,
/// Schedule.
pub schedule: ScheduleSpec,
/// Retention.
pub retention: RetentionPolicy,
/// Destinations.
pub destinations: Vec<Destination>,
/// Encryption key. For `VaultState`, ADR-011 requires a bootstrap key
/// stored OUTSIDE vault (offline custody).
pub encryption: VaultKeyRef,
/// Tag strategy.
pub tag_strategy: TagStrategy,
/// Optional verify reference.
#[serde(default)]
pub verify: Option<VerifyPolicyRef>,
/// Optional hooks.
#[serde(default)]
pub hooks: Option<Hooks>,
/// Optional throttle.
#[serde(default)]
pub throttle: Option<Throttle>,
}

21
crates/buildkit-launcher/Cargo.toml Normal file
View file

@ -0,0 +1,21 @@
[package]
authors.workspace = true
description = "Ephemeral buildkit runner launcher — spawns hcloud VMs, runs buildctl, emits metrics (ADR-039)"
edition.workspace = true
name = "buildkit-launcher"
version.workspace = true
[dependencies]
anyhow = { workspace = true }
clap = { workspace = true }
reqwest = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
tokio = { workspace = true, features = ["rt-multi-thread", "process", "time", "io-util"] }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
uuid = { workspace = true }
[[bin]]
name = "buildkit-launcher"
path = "src/main.rs"

19
crates/buildkit-launcher/Cargo.workspace.toml Normal file
View file

@ -0,0 +1,19 @@
[workspace]
members = ["crates/buildkit-launcher"]
resolver = "2"
[workspace.package]
authors = ["Jesus Perez <jesus@librecloud.online>"]
edition = "2021"
version = "1.0.11"
[workspace.dependencies]
anyhow = "1.0"
clap = { version = "4.5", features = ["derive", "env"] }
reqwest = { version = "0.13", features = ["json", "rustls"], default-features = false }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
tokio = { version = "1.49", features = ["rt-multi-thread", "process", "time", "io-util", "macros"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
uuid = { version = "1.20", features = ["v4", "serde"] }

38
crates/buildkit-launcher/Dockerfile Normal file
View file

@ -0,0 +1,38 @@
# syntax=docker/dockerfile:1.7
# Build context: provisioning/ (pass --local context=. from provisioning/ root)
# Dockerfile path (buildctl): platform/crates/buildkit-launcher/Dockerfile
# Target architecture: aarch64 (CAX/CCX Woodpecker agents are ARM64)
# Cache: --export-cache / --import-cache against zot /cache/buildkit-launcher
FROM rust:bookworm AS builder
RUN apt-get update && apt-get install -y --no-install-recommends \
pkg-config libssl-dev ca-certificates \
&& rm -rf /var/lib/apt/lists/*
RUN rustup target add aarch64-unknown-linux-gnu
WORKDIR /workspace
# Cargo.workspace.toml is a self-contained workspace manifest: it omits all
# path deps that point outside this build context (ontoref, stratumiops, etc.).
# It must define every key that the crate's Cargo.toml inherits via .workspace = true.
COPY platform/crates/buildkit-launcher/Cargo.workspace.toml Cargo.toml
COPY platform/crates/buildkit-launcher/ crates/buildkit-launcher/
RUN --mount=type=cache,target=/root/.cargo/registry,sharing=locked \
--mount=type=cache,target=/root/.cargo/git,sharing=locked \
--mount=type=cache,target=/workspace/target,sharing=locked \
cargo build --release --target aarch64-unknown-linux-gnu \
--package buildkit-launcher && \
cp target/aarch64-unknown-linux-gnu/release/buildkit-launcher /buildkit-launcher-bin
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates openssh-client \
&& rm -rf /var/lib/apt/lists/*
COPY --from=builder /buildkit-launcher-bin /buildkit-launcher
ENTRYPOINT ["/buildkit-launcher"]

137
crates/buildkit-launcher/src/buildctl_runner.rs Normal file
View file

@ -0,0 +1,137 @@
use anyhow::{Context, Result};
use std::path::Path;
use tokio::process::Command;
use tracing::{debug, info};
pub const OOM_EXIT_CODE: i32 = 137;
pub struct BuildOutput {
pub cpu_secs: f64,
pub mem_peak_mb: f64,
pub duration_secs: f64,
}
pub async fn rsync_context(
ssh_host: &str,
ssh_port: u16,
context_path: &Path,
ssh_key: &Path,
) -> Result<()> {
info!(host = %ssh_host, port = ssh_port, "rsyncing build context");
let status = Command::new("rsync")
.args([
"-az",
"--delete",
"-e",
&format!(
"ssh -p {} -i {} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null",
ssh_port,
ssh_key.display()
),
])
.arg(format!("{}/", context_path.display()))
.arg(format!("root@{}:/build/context/", ssh_host))
.status()
.await
.context("rsync failed to start")?;
if !status.success() {
anyhow::bail!("rsync exited {}", status);
}
Ok(())
}
pub async fn run_buildctl(
ssh_host: &str,
ssh_port: u16,
ssh_key: &Path,
image_ref: &str,
dockerfile: &str,
cache_from: Option<&str>,
cache_to: Option<&str>,
) -> Result<BuildOutput> {
let mut buildctl_args = vec![
"build".to_string(),
"--frontend".to_string(),
"dockerfile.v0".to_string(),
"--opt".to_string(),
format!("filename={}", dockerfile),
"--local".to_string(),
"context=/build/context".to_string(),
"--local".to_string(),
"dockerfile=/build/context".to_string(),
"--output".to_string(),
format!("type=image,name={},push=true", image_ref),
];
if let Some(from) = cache_from {
buildctl_args.push("--import-cache".to_string());
buildctl_args.push(from.to_string());
}
if let Some(to) = cache_to {
buildctl_args.push("--export-cache".to_string());
buildctl_args.push(to.to_string());
}
let remote_cmd = format!("buildctl {}", buildctl_args.join(" "));
debug!(cmd = %remote_cmd, "running buildctl on runner");
let start = std::time::Instant::now();
let output = Command::new("ssh")
.args([
"-p",
&ssh_port.to_string(),
"-i",
&ssh_key.to_string_lossy(),
"-o",
"StrictHostKeyChecking=no",
"-o",
"UserKnownHostsFile=/dev/null",
&format!("root@{}", ssh_host),
&remote_cmd,
])
.output()
.await
.context("ssh buildctl failed to start")?;
let duration_secs = start.elapsed().as_secs_f64();
if !output.status.success() {
let exit_code = output.status.code().unwrap_or(-1);
let stderr = String::from_utf8_lossy(&output.stderr);
anyhow::bail!(
"buildctl exit={}: {}",
exit_code,
stderr.lines().last().unwrap_or("(no stderr)")
);
}
// Parse /proc/self/status peak memory from buildkitd if available in stdout
// Fallback: estimate from image size (not accurate; real metrics come from cgroups)
let mem_peak_mb = parse_mem_peak_from_output(&String::from_utf8_lossy(&output.stdout))
.unwrap_or(0.0);
// cpu_secs: rough estimate — not from cgroups; orchestrator records real P95 after multiple runs
let cpu_secs = duration_secs;
Ok(BuildOutput { cpu_secs, mem_peak_mb, duration_secs })
}
fn parse_mem_peak_from_output(stdout: &str) -> Option<f64> {
for line in stdout.lines() {
if let Some(val) = line.strip_prefix("MEM_PEAK_MB=") {
return val.parse().ok();
}
}
None
}
pub fn is_oom(err: &anyhow::Error) -> bool {
let msg = err.to_string();
msg.contains(&format!("exit={}", OOM_EXIT_CODE))
|| msg.contains("exit=137")
|| msg.contains("OOM")
|| msg.contains("Killed")
}

188
crates/buildkit-launcher/src/main.rs Normal file
View file

@ -0,0 +1,188 @@
mod buildctl_runner;
mod orchestrator_client;
mod retry;
mod sizing;
use anyhow::{Context, Result};
use clap::Parser;
use orchestrator_client::OrchestratorClient;
use std::path::PathBuf;
use tracing::{error, info, warn};
#[derive(Parser)]
#[command(name = "buildkit-launcher", about = "Ephemeral buildkit runner — ADR-039")]
struct Args {
/// Workspace name (used for P95 lookup and runner labelling)
#[arg(long, env = "BUILDKIT_WORKSPACE")]
workspace: String,
/// Build context directory (must be reachable from this host)
#[arg(long)]
context: PathBuf,
/// Fully-qualified image reference to push after build
#[arg(long)]
image: String,
/// Dockerfile path relative to context
#[arg(long, default_value = "Dockerfile")]
dockerfile: String,
/// Cache-from reference (optional, e.g. type=registry,ref=zot.example/cache)
#[arg(long)]
cache_from: Option<String>,
/// Cache-to reference (optional)
#[arg(long)]
cache_to: Option<String>,
/// SSH key to use when connecting to the runner
#[arg(long, env = "BUILDKIT_SSH_KEY")]
ssh_key: PathBuf,
/// Language hint for size defaults (rust, go, java, etc.)
#[arg(long)]
language: Option<String>,
/// Golden image snapshot ID or name to boot the runner from
#[arg(long, env = "BUILDKIT_RUNNER_IMAGE", default_value = "buildkit-runner-latest")]
runner_image: String,
/// Orchestrator base URL
#[arg(long, env = "ORCHESTRATOR_URL", default_value = "http://localhost:9011")]
orchestrator_url: String,
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::from_default_env()
.add_directive("buildkit_launcher=info".parse()?),
)
.init();
let args = Args::parse();
let client = OrchestratorClient::new(args.orchestrator_url.clone());
// Resolve runner size
let p95 = client.get_p95(&args.workspace).await.unwrap_or(None);
let (p95_cpu, p95_mem) = p95
.as_ref()
.map(|s| (Some(s.cpu_p95), Some(s.memory_mb_p95)))
.unwrap_or((None, None));
let size = sizing::resolve(
&args.context,
p95_cpu,
p95_mem,
args.language.as_deref(),
)
.context("failed to resolve runner size")?;
info!(
cpu = size.cpu,
memory_gb = size.memory_gb,
workspace = %args.workspace,
"resolved runner size"
);
let result = run_build(&args, &client, size.clone()).await;
match result {
Ok(()) => Ok(()),
Err(e) if buildctl_runner::is_oom(&e) && retry::oom_retry_allowed(0) => {
warn!(
error = %e,
limit = retry::MAX_OOM_RETRIES,
"OOM detected — retrying at next size tier"
);
let retry_size = retry::next_size_tier(&size)
.context("OOM on largest tier — no retry possible")?;
info!(
cpu = retry_size.cpu,
memory_gb = retry_size.memory_gb,
"retrying build at next tier"
);
run_build(&args, &client, retry_size).await
}
Err(e) => Err(e),
}
}
async fn run_build(args: &Args, client: &OrchestratorClient, size: sizing::RunnerSize) -> Result<()> {
let runner = client
.spawn_runner(&size, &args.workspace, &args.runner_image)
.await
.context("failed to spawn runner")?;
info!(
lease_id = %runner.lease_id,
ssh_host = %runner.ssh_host,
ssh_port = runner.ssh_port,
expires_at = %runner.expires_at,
"runner spawned"
);
let destroy = |lease_id: String| {
let client = client.clone();
tokio::spawn(async move {
if let Err(e) = client.destroy_runner(&lease_id).await {
error!(lease_id = %lease_id, error = %e, "failed to destroy runner");
}
})
};
// Rsync build context to runner
if let Err(e) = buildctl_runner::rsync_context(
&runner.ssh_host,
runner.ssh_port,
&args.context,
&args.ssh_key,
)
.await
{
destroy(runner.lease_id.clone());
return Err(e.context("rsync build context failed"));
}
// Run buildctl
let build_result = buildctl_runner::run_buildctl(
&runner.ssh_host,
runner.ssh_port,
&args.ssh_key,
&args.image,
&args.dockerfile,
args.cache_from.as_deref(),
args.cache_to.as_deref(),
)
.await;
// Always destroy the runner, even on build failure
destroy(runner.lease_id.clone());
let output = build_result.context("buildctl failed")?;
info!(
duration_secs = output.duration_secs,
mem_peak_mb = output.mem_peak_mb,
"build complete — recording metrics"
);
if let Err(e) = client
.record_metrics(
&args.workspace,
output.cpu_secs,
output.mem_peak_mb,
output.duration_secs,
)
.await
{
warn!(error = %e, "failed to record build metrics (non-fatal)");
}
Ok(())
}

158
crates/buildkit-launcher/src/orchestrator_client.rs Normal file
View file

@ -0,0 +1,158 @@
use anyhow::{Context, Result};
use serde::{Deserialize, Serialize};
use crate::sizing::RunnerSize;
#[derive(Debug, Serialize)]
pub struct SpawnRequest {
pub cpu: u32,
pub memory_gb: u32,
pub disk_gb: u32,
pub time_budget_min: u32,
pub workspace: String,
pub image: String,
}
#[derive(Debug, Deserialize)]
pub struct SpawnResponse {
pub lease_id: String,
pub ssh_host: String,
pub ssh_port: u16,
pub expires_at: String,
}
#[derive(Debug, Deserialize)]
pub struct P95Stats {
pub cpu_p95: f64,
pub memory_mb_p95: f64,
#[allow(dead_code)]
pub duration_secs_p95: f64,
#[allow(dead_code)]
pub sample_count: u64,
}
#[derive(Debug, Deserialize)]
struct ApiResponse<T> {
success: bool,
data: Option<T>,
error: Option<String>,
}
#[derive(Clone)]
pub struct OrchestratorClient {
base_url: String,
http: reqwest::Client,
}
impl OrchestratorClient {
pub fn new(base_url: String) -> Self {
Self {
base_url,
http: reqwest::Client::new(),
}
}
pub async fn spawn_runner(
&self,
size: &RunnerSize,
workspace: &str,
image: &str,
) -> Result<SpawnResponse> {
let req = SpawnRequest {
cpu: size.cpu,
memory_gb: size.memory_gb,
disk_gb: size.disk_gb,
time_budget_min: size.time_budget_min,
workspace: workspace.to_string(),
image: image.to_string(),
};
let resp: ApiResponse<SpawnResponse> = self
.http
.post(format!("{}/api/v1/vm-pool/spawn", self.base_url))
.json(&req)
.send()
.await
.context("POST /api/v1/vm-pool/spawn failed")?
.error_for_status()
.context("orchestrator returned error for spawn")?
.json()
.await
.context("failed to deserialize SpawnResponse")?;
if !resp.success {
anyhow::bail!(
"spawn_runner: {}",
resp.error.as_deref().unwrap_or("unknown error")
);
}
resp.data.context("spawn_runner: missing data in response")
}
pub async fn destroy_runner(&self, lease_id: &str) -> Result<()> {
let resp = self
.http
.delete(format!("{}/api/v1/vm-pool/{}", self.base_url, lease_id))
.send()
.await
.context("DELETE /api/v1/vm-pool/{lease_id} failed")?;
if !resp.status().is_success() {
anyhow::bail!("destroy_runner HTTP {}", resp.status());
}
Ok(())
}
pub async fn get_p95(&self, workspace: &str) -> Result<Option<P95Stats>> {
let resp = self
.http
.get(format!(
"{}/api/v1/vm-pool/p95/{}",
self.base_url, workspace
))
.send()
.await
.context("GET /api/v1/vm-pool/p95/{workspace} failed")?;
if resp.status() == reqwest::StatusCode::NOT_FOUND {
return Ok(None);
}
let body: ApiResponse<P95Stats> = resp
.error_for_status()
.context("p95 endpoint error")?
.json()
.await
.context("failed to deserialize P95Stats")?;
Ok(body.data)
}
pub async fn record_metrics(
&self,
workspace: &str,
cpu_secs: f64,
mem_peak_mb: f64,
duration_secs: f64,
) -> Result<()> {
let payload = serde_json::json!({
"workspace": workspace,
"cpu_secs": cpu_secs,
"mem_peak_mb": mem_peak_mb,
"duration_secs": duration_secs,
});
let resp = self
.http
.post(format!("{}/api/v1/vm-pool/metrics", self.base_url))
.json(&payload)
.send()
.await
.context("POST /api/v1/vm-pool/metrics failed")?;
if !resp.status().is_success() {
anyhow::bail!("record_metrics HTTP {}", resp.status());
}
Ok(())
}
}

31
crates/buildkit-launcher/src/retry.rs Normal file
View file

@ -0,0 +1,31 @@
use crate::sizing::RunnerSize;
/// Hard bound on OOM auto-retries per build — ADR-039 constraint oom-retry-bounded.
/// Repeated OOM after one retry indicates misconfiguration; surface to developer.
pub const MAX_OOM_RETRIES: u8 = 1;
/// Returns whether an OOM retry is permitted given the count of retries already consumed.
pub fn oom_retry_allowed(retries_used: u8) -> bool {
retries_used < MAX_OOM_RETRIES
}
const SIZE_TIERS: &[(&str, u32, u32)] = &[
("cx22", 2, 4),
("cx32", 4, 8),
("cx42", 8, 16),
("cx52", 16, 32),
];
pub fn next_size_tier(current: &RunnerSize) -> Option<RunnerSize> {
// Find current tier by matching cpu/memory_gb
let current_idx = SIZE_TIERS.iter().position(|(_, cpu, mem)| {
*cpu >= current.cpu && *mem >= current.memory_gb
})?;
SIZE_TIERS.get(current_idx + 1).map(|(_, cpu, mem)| RunnerSize {
cpu: *cpu,
memory_gb: *mem,
disk_gb: current.disk_gb,
time_budget_min: current.time_budget_min,
})
}

92
crates/buildkit-launcher/src/sizing.rs Normal file
View file

@ -0,0 +1,92 @@
use anyhow::Result;
use serde::Deserialize;
use std::path::Path;
#[derive(Debug, Clone)]
pub struct RunnerSize {
pub cpu: u32,
pub memory_gb: u32,
pub disk_gb: u32,
pub time_budget_min: u32,
}
#[derive(Deserialize, Default)]
struct BuildSpec {
cpu: Option<u32>,
memory_gb: Option<u32>,
disk_gb: Option<u32>,
time_budget_min: Option<u32>,
language: Option<String>,
}
impl RunnerSize {
fn language_default(language: &str) -> Self {
match language {
"rust" => Self { cpu: 4, memory_gb: 8, disk_gb: 50, time_budget_min: 60 },
"go" => Self { cpu: 2, memory_gb: 4, disk_gb: 30, time_budget_min: 30 },
"java" | "kotlin" | "scala" => Self { cpu: 4, memory_gb: 8, disk_gb: 40, time_budget_min: 45 },
_ => Self { cpu: 2, memory_gb: 4, disk_gb: 30, time_budget_min: 30 },
}
}
pub fn from_p95(cpu_p95: f64, memory_mb_p95: f64) -> Self {
let cpu = ((cpu_p95 * 1.2).ceil() as u32).max(2);
let memory_gb = (((memory_mb_p95 * 1.2) / 1024.0).ceil() as u32).max(4);
Self { cpu, memory_gb, disk_gb: 30, time_budget_min: 60 }
}
}
fn parse_build_spec(context_path: &Path) -> Option<BuildSpec> {
let spec_path = context_path.join(".build-spec.ncl");
if !spec_path.exists() {
return None;
}
// Extract JSON from nickel export — buildkit-launcher shells out to nickel
let output = std::process::Command::new("nickel")
.args(["export", "--format", "json"])
.arg(&spec_path)
.output()
.ok()?;
if !output.status.success() {
return None;
}
serde_json::from_slice(&output.stdout).ok()
}
pub fn resolve(
context_path: &Path,
p95_cpu: Option<f64>,
p95_mem_mb: Option<f64>,
language_hint: Option<&str>,
) -> Result<RunnerSize> {
let spec = parse_build_spec(context_path).unwrap_or_default();
// Tier 1: explicit declaration in .build-spec.ncl
if spec.cpu.is_some() || spec.memory_gb.is_some() {
let base = language_hint
.or(spec.language.as_deref())
.map(RunnerSize::language_default)
.unwrap_or_else(|| RunnerSize::language_default("default"));
return Ok(RunnerSize {
cpu: spec.cpu.unwrap_or(base.cpu),
memory_gb: spec.memory_gb.unwrap_or(base.memory_gb),
disk_gb: spec.disk_gb.unwrap_or(base.disk_gb),
time_budget_min: spec.time_budget_min.unwrap_or(base.time_budget_min),
});
}
// Tier 2: P95 historical × 1.2
if let (Some(cpu_p95), Some(mem_p95)) = (p95_cpu, p95_mem_mb) {
let mut size = RunnerSize::from_p95(cpu_p95, mem_p95);
if let Some(budget) = spec.time_budget_min {
size.time_budget_min = budget;
}
return Ok(size);
}
// Tier 3: language defaults
let lang = language_hint
.or(spec.language.as_deref())
.unwrap_or("default");
Ok(RunnerSize::language_default(lang))
}

9
crates/catalog-registry/.dockerignore Normal file
View file

@ -0,0 +1,9 @@
target/
.git/
.gitignore
*.md
tests/
Dockerfile
.dockerignore
config.toml
*.log

24
crates/catalog-registry/.gitignore vendored Normal file
View file

@ -0,0 +1,24 @@
# Rust
/target/
**/*.rs.bk
Cargo.lock
# Configuration
config.toml
*.log
# IDE
.vscode/
.idea/
*.swp
*.swo
*~
# OS
.DS_Store
Thumbs.db
# Secrets
*token*.txt
*.key
*.pem

586
crates/catalog-registry/API.md Normal file
View file

@ -0,0 +1,586 @@
# Catalog Registry API Documentation
Version: 1.0.0
Base URL: `http://localhost:8082/api/v1`
## Table of Contents
- [Authentication](#authentication)
- [Extension Endpoints](#extension-endpoints)
- [System Endpoints](#system-endpoints)
- [Error Responses](#error-responses)
- [Data Models](#data-models)
## Authentication
The Catalog Registry API does not require authentication for read operations. Backend authentication (Gitea/OCI) is handled server-side via
configuration.
## Extension Endpoints
### List Extensions
Retrieve a list of available extensions with optional filtering and pagination.
**Endpoint**: `GET /extensions`
**Query Parameters**:
| Parameter | Type | Required | Description |
| ----------- | ------ | ---------- | ------------- |
| `type` | string | No | Filter by extension type: `provider`, `taskserv`, `cluster` |
| `source` | string | No | Filter by source: `gitea`, `oci` |
| `limit` | integer | No | Maximum results (default: 100, max: 1000) |
| `offset` | integer | No | Pagination offset (default: 0) |
**Example Request**:
```bash
curl "http://localhost:8082/api/v1/extensions?type=provider&limit=10"
```
**Example Response** (200 OK):
```bash
[
{
"name": "aws",
"type": "provider",
"version": "1.2.0",
"description": "AWS provider for provisioning infrastructure",
"author": "provisioning-team",
"repository": "https://gitea.example.com/org/aws_prov",
"source": "gitea",
"published_at": "2025-10-06T12:00:00Z",
"download_url": "https://gitea.example.com/org/aws_prov/releases/download/1.2.0/aws_prov.tar.gz",
"checksum": "sha256:abc123...",
"size": 1024000,
"tags": ["cloud", "aws", "infrastructure"]
},
{
"name": "upcloud",
"type": "provider",
"version": "2.1.3",
"description": "UpCloud provider for European cloud infrastructure",
"author": "provisioning-team",
"repository": "https://gitea.example.com/org/upcloud_prov",
"source": "gitea",
"published_at": "2025-10-05T10:30:00Z",
"download_url": "https://gitea.example.com/org/upcloud_prov/releases/download/2.1.3/upcloud_prov.tar.gz",
"size": 890000
}
]
```
---
### Get Extension Metadata
Retrieve detailed metadata for a specific extension.
**Endpoint**: `GET /extensions/{type}/{name}`
**Path Parameters**:
| Parameter | Type | Required | Description |
| ----------- | ------ | ---------- | ------------- |
| `type` | string | Yes | Extension type: `provider`, `taskserv`, `cluster` |
| `name` | string | Yes | Extension name |
**Example Request**:
```bash
curl "http://localhost:8082/api/v1/extensions/provider/aws"
```
**Example Response** (200 OK):
```json
{
"name": "aws",
"type": "provider",
"version": "1.2.0",
"description": "AWS provider for provisioning infrastructure",
"author": "provisioning-team",
"repository": "https://gitea.example.com/org/aws_prov",
"source": "gitea",
"published_at": "2025-10-06T12:00:00Z",
"download_url": "https://gitea.example.com/org/aws_prov/releases/download/1.2.0/aws_prov.tar.gz",
"checksum": "sha256:abc123...",
"size": 1024000,
"tags": ["cloud", "aws", "infrastructure"]
}
```
**Error Response** (404 Not Found):
```json
{
"error": "not_found",
"message": "Extension provider/nonexistent not found"
}
```
---
### List Extension Versions
Get all available versions for a specific extension.
**Endpoint**: `GET /extensions/{type}/{name}/versions`
**Path Parameters**:
| Parameter | Type | Required | Description |
| ----------- | ------ | ---------- | ------------- |
| `type` | string | Yes | Extension type: `provider`, `taskserv`, `cluster` |
| `name` | string | Yes | Extension name |
**Example Request**:
```bash
curl "http://localhost:8082/api/v1/extensions/taskserv/kubernetes/versions"
```
**Example Response** (200 OK):
```bash
[
{
"version": "1.28.0",
"published_at": "2025-10-06T12:00:00Z",
"download_url": "https://gitea.example.com/org/kubernetes_taskserv/releases/download/1.28.0/kubernetes_taskserv.tar.gz",
"checksum": "sha256:def456...",
"size": 2048000
},
{
"version": "1.27.5",
"published_at": "2025-09-15T10:30:00Z",
"download_url": "https://gitea.example.com/org/kubernetes_taskserv/releases/download/1.27.5/kubernetes_taskserv.tar.gz",
"checksum": "sha256:ghi789...",
"size": 1980000
},
{
"version": "1.27.4",
"published_at": "2025-08-20T08:15:00Z",
"download_url": "https://gitea.example.com/org/kubernetes_taskserv/releases/download/1.27.4/kubernetes_taskserv.tar.gz",
"size": 1950000
}
]
```
---
### Download Extension
Download a specific version of an extension.
**Endpoint**: `GET /extensions/{type}/{name}/{version}`
**Path Parameters**:
| Parameter | Type | Required | Description |
| ----------- | ------ | ---------- | ------------- |
| `type` | string | Yes | Extension type: `provider`, `taskserv`, `cluster` |
| `name` | string | Yes | Extension name |
| `version` | string | Yes | Extension version (e.g., `1.2.0`) |
**Example Request**:
```bash
curl -OJ "http://localhost:8082/api/v1/extensions/provider/aws/1.2.0"
```
**Response**:
- **Content-Type**: `application/octet-stream`
- **Body**: Binary data (tarball or archive)
**Error Response** (404 Not Found):
```json
{
"error": "not_found",
"message": "Extension provider/aws version 1.2.0 not found"
}
```
---
### Search Extensions
Search for extensions by name or description.
**Endpoint**: `GET /extensions/search`
**Query Parameters**:
| Parameter | Type | Required | Description |
| ----------- | ------ | ---------- | ------------- |
| `q` | string | Yes | Search query (case-insensitive) |
| `type` | string | No | Filter by extension type |
| `limit` | integer | No | Maximum results (default: 50, max: 100) |
**Example Request**:
```bash
curl "http://localhost:8082/api/v1/extensions/search?q=kubernetes&type=taskserv&limit=5"
```
**Example Response** (200 OK):
```bash
[
{
"name": "kubernetes",
"type": "taskserv",
"version": "1.28.0",
"description": "Kubernetes container orchestration platform",
"author": "provisioning-team",
"source": "gitea",
"published_at": "2025-10-06T12:00:00Z"
},
{
"name": "k3s",
"type": "taskserv",
"version": "1.27.5",
"description": "Lightweight Kubernetes distribution",
"author": "community",
"source": "oci",
"published_at": "2025-09-20T14:30:00Z"
}
]
```
---
## System Endpoints
### Health Check
Check service health and backend status.
**Endpoint**: `GET /health`
**Example Request**:
```bash
curl "http://localhost:8082/api/v1/health"
```
**Example Response** (200 OK):
```json
{
"status": "healthy",
"version": "0.1.0",
"uptime": 3600,
"backends": {
"gitea": {
"enabled": true,
"healthy": true
},
"oci": {
"enabled": true,
"healthy": true,
"error": null
}
}
}
```
**Degraded Status** (200 OK):
```json
{
"status": "degraded",
"version": "0.1.0",
"uptime": 7200,
"backends": {
"gitea": {
"enabled": true,
"healthy": false,
"error": "Connection timeout"
},
"oci": {
"enabled": true,
"healthy": true
}
}
}
```
---
### Metrics
Get Prometheus-formatted metrics.
**Endpoint**: `GET /metrics`
**Example Request**:
```bash
curl "http://localhost:8082/api/v1/metrics"
```
**Example Response** (200 OK):
```bash
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total 1234
# HELP http_request_duration_seconds HTTP request duration
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.005"} 100
http_request_duration_seconds_bucket{le="0.01"} 200
http_request_duration_seconds_bucket{le="0.025"} 300
http_request_duration_seconds_sum 50.5
http_request_duration_seconds_count 1234
# HELP cache_hits_total Total cache hits
# TYPE cache_hits_total counter
cache_hits_total 987
# HELP cache_misses_total Total cache misses
# TYPE cache_misses_total counter
cache_misses_total 247
# HELP extensions_total Total extensions
# TYPE extensions_total gauge
extensions_total 45
```
---
### Cache Statistics
Get cache performance statistics.
**Endpoint**: `GET /cache/stats`
**Example Request**:
```bash
curl "http://localhost:8082/api/v1/cache/stats"
```
**Example Response** (200 OK):
```json
{
"list_entries": 45,
"metadata_entries": 120,
"version_entries": 80,
"total_entries": 245
}
```
---
## Error Responses
All error responses follow this format:
```json
{
"error": "error_type",
"message": "Human-readable error message",
"details": "Optional additional details"
}
```
### HTTP Status Codes
| Status | Description |
| -------- | ------------- |
| 200 OK | Request successful |
| 400 Bad Request | Invalid input (e.g., invalid extension type) |
| 401 Unauthorized | Authentication failed |
| 404 Not Found | Resource not found |
| 429 Too Many Requests | Rate limit exceeded |
| 500 Internal Server Error | Server error |
| 503 Service Unavailable | Service temporarily unavailable |
### Error Types
| Error Type | HTTP Status | Description |
| ------------ | ------------- | ------------- |
| `not_found` | 404 | Extension or resource not found |
| `invalid_type` | 400 | Invalid extension type provided |
| `invalid_version` | 400 | Invalid version format |
| `auth_error` | 401 | Authentication failed |
| `rate_limit` | 429 | Too many requests |
| `config_error` | 500 | Server configuration error |
| `internal_error` | 500 | Internal server error |
---
## Data Models
### Extension
```bash
interface Extension {
name: string; // Extension name
type: ExtensionType; // Extension type
version: string; // Current version (semver)
description: string; // Description
author?: string; // Author/organization
repository?: string; // Source repository URL
source: ExtensionSource; // Backend source
published_at: string; // ISO 8601 timestamp
download_url?: string; // Download URL
checksum?: string; // Checksum (e.g., sha256:...)
size?: number; // Size in bytes
tags?: string[]; // Tags
}
```
### ExtensionType
```bash
type ExtensionType = "provider" | "taskserv" | "cluster";
```
### ExtensionSource
```bash
type ExtensionSource = "gitea" | "oci";
```
### ExtensionVersion
```bash
interface ExtensionVersion {
version: string; // Version string (semver)
published_at: string; // ISO 8601 timestamp
download_url?: string; // Download URL
checksum?: string; // Checksum
size?: number; // Size in bytes
}
```
### HealthResponse
```bash
interface HealthResponse {
status: string; // "healthy" | "degraded"
version: string; // Service version
uptime: number; // Uptime in seconds
backends: BackendHealth; // Backend health status
}
```
### BackendHealth
```bash
interface BackendHealth {
gitea: BackendStatus;
oci: BackendStatus;
}
```
### BackendStatus
```bash
interface BackendStatus {
enabled: boolean; // Backend enabled
healthy: boolean; // Backend healthy
error?: string; // Error message if unhealthy
}
```
---
## Rate Limiting
Currently, the API does not enforce rate limiting. This may be added in future versions.
For high-volume usage, consider:
- Implementing client-side rate limiting
- Using the cache effectively
- Batching requests when possible
---
## Caching Behavior
The service implements LRU caching with TTL:
- **Cache TTL**: Configurable (default: 5 minutes)
- **Cache Capacity**: Configurable (default: 1000 entries)
- **Cache Keys**:
- List: `list:{type}:{source}`
- Metadata: `{type}/{name}`
- Versions: `{type}/{name}/versions`
Cache headers are not currently exposed. Future versions may include:
- `X-Cache-Hit: true/false`
- `X-Cache-TTL: {seconds}`
---
## Versioning
API version is specified in the URL path: `/api/v1/`
Major version changes will be introduced in new paths (e.g., `/api/v2/`).
---
## Examples
### Complete Workflow
```bash
# 1. Search for Kubernetes extensions
curl "http://localhost:8082/api/v1/extensions/search?q=kubernetes"
# 2. Get extension metadata
curl "http://localhost:8082/api/v1/extensions/taskserv/kubernetes"
# 3. List available versions
curl "http://localhost:8082/api/v1/extensions/taskserv/kubernetes/versions"
# 4. Download specific version
curl -OJ "http://localhost:8082/api/v1/extensions/taskserv/kubernetes/1.28.0"
# 5. Verify checksum (if provided)
sha256sum kubernetes_taskserv.tar.gz
```
### Pagination
```bash
# Get first page
curl "http://localhost:8082/api/v1/extensions?limit=10&offset=0"
# Get second page
curl "http://localhost:8082/api/v1/extensions?limit=10&offset=10"
# Get third page
curl "http://localhost:8082/api/v1/extensions?limit=10&offset=20"
```
### Filtering
```bash
# Only providers from Gitea
curl "http://localhost:8082/api/v1/extensions?type=provider&source=gitea"
# Only taskservs from OCI
curl "http://localhost:8082/api/v1/extensions?type=taskserv&source=oci"
# All clusters
curl "http://localhost:8082/api/v1/extensions?type=cluster"
```
---
## Support
For issues and questions, see the main README or project documentation.

91
crates/catalog-registry/Cargo.toml Normal file
View file

@ -0,0 +1,91 @@
[package]
authors.workspace = true
description = "OCI-compliant catalog registry proxy for managing provisioning system extensions and artifacts"
edition.workspace = true
name = "catalog-registry"
version.workspace = true
[[bin]]
name = "provisioning-catalog-registry"
path = "src/main.rs"
[dependencies]
# Workspace dependencies
async-trait = { workspace = true }
futures = { workspace = true }
tokio = { workspace = true, features = ["full"] }
# Web server and API
axum = { workspace = true }
tower = { workspace = true, features = ["full"] }
# Ontoref API catalog
ontoref-ontology = { workspace = true }
ontoref-derive = { workspace = true }
inventory = { workspace = true }
tower-http = { workspace = true, features = ["cors", "trace"] }
# Serialization
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true }
# Platform configuration
platform-config = { workspace = true }
# Centralized observability (logging, metrics, health, tracing)
platform-observability = { workspace = true, features = ["logging", "metrics-prometheus", "health"] }
# Error handling
anyhow = { workspace = true }
thiserror = { workspace = true }
# Logging
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
# UUID and time
chrono = { workspace = true, features = ["serde"] }
uuid = { workspace = true, features = ["v4", "serde"] }
# CLI
clap = { workspace = true, features = ["derive"] }
# HTTP client for OCI registry operations
reqwest = { workspace = true }
# Cryptography for digest validation
hex = { workspace = true }
sha2 = { workspace = true }
# URL parsing
url = { workspace = true }
# Bytes manipulation
bytes = { workspace = true }
# LRU caching
lru = { workspace = true }
# Parking lot for synchronization
parking_lot = { workspace = true }
# Platform NATS bridge (optional)
platform-nats = { workspace = true, optional = true }
# TOML parsing
toml = { workspace = true }
[features]
nats = ["dep:platform-nats"]
default = []
[dev-dependencies]
http-body-util = { workspace = true }
hyper = { workspace = true }
tempfile = { workspace = true }
tokio-test = { workspace = true }
# Library target
[lib]
name = "catalog_registry"
path = "src/lib.rs"

119
crates/catalog-registry/Dockerfile Normal file
View file

@ -0,0 +1,119 @@
# Multi-stage build for catalog-registry
# Generated from Nickel template - DO NOT EDIT DIRECTLY
# Source: provisioning/schemas/platform/templates/docker/Dockerfile.chef.ncl
# ============================================================================
# Stage 1: PLANNER - Generate dependency recipe
# ============================================================================
FROM rust:1.82-trixie AS planner
WORKDIR /workspace
# Install cargo-chef
RUN cargo install cargo-chef --version 0.1.67
# Copy workspace manifests
COPY Cargo.toml Cargo.lock ./
COPY crates ./crates
COPY daemon-cli ./daemon-cli
COPY secretumvault ./secretumvault
COPY prov-ecosystem ./prov-ecosystem
COPY stratumiops ./stratumiops
# Generate recipe.json (dependency graph)
RUN cargo chef prepare --recipe-path recipe.json --bin catalog-registry
# ============================================================================
# Stage 2: CACHER - Build dependencies only
# ============================================================================
FROM rust:1.82-trixie AS cacher
WORKDIR /workspace
# Install build dependencies
RUN apt-get update && apt-get install -y \
pkg-config \
libssl-dev \
&& rm -rf /var/lib/apt/lists/*
# Install cargo-chef
RUN cargo install cargo-chef --version 0.1.67
# sccache disabled
# Copy recipe from planner
COPY --from=planner /workspace/recipe.json recipe.json
# Build dependencies - This layer will be cached
RUN cargo chef cook --release --recipe-path recipe.json
# ============================================================================
# Stage 3: BUILDER - Build source code
# ============================================================================
FROM rust:1.82-trixie AS builder
WORKDIR /workspace
# Install build dependencies
RUN apt-get update && apt-get install -y \
pkg-config \
libssl-dev \
&& rm -rf /var/lib/apt/lists/*
# sccache disabled
# Copy cached dependencies from cacher stage
COPY --from=cacher /workspace/target target
COPY --from=cacher /usr/local/cargo /usr/local/cargo
# Copy source code
COPY Cargo.toml Cargo.lock ./
COPY crates ./crates
COPY daemon-cli ./daemon-cli
COPY secretumvault ./secretumvault
COPY prov-ecosystem ./prov-ecosystem
COPY stratumiops ./stratumiops
# Build release binary with parallelism
ENV CARGO_BUILD_JOBS=4
RUN cargo build --release --package catalog-registry
# ============================================================================
# Stage 4: RUNTIME - Minimal runtime image
# ============================================================================
FROM debian:trixie-slim
# Install runtime dependencies
RUN apt-get update && apt-get install -y \
ca-certificates \
curl \
&& rm -rf /var/lib/apt/lists/*
# Create non-root user
RUN useradd -m -u 1000 provisioning && \
mkdir -p /data /var/log/catalog-registry && \
chown -R provisioning:provisioning /data /var/log/catalog-registry
# Copy binary from builder
COPY --from=builder /workspace/target/release/catalog-registry /usr/local/bin/catalog-registry
RUN chmod +x /usr/local/bin/catalog-registry
# No config file to copy
# Switch to non-root user
USER provisioning
WORKDIR /app
# Expose service port
EXPOSE 9093
# Environment variables
ENV RUST_LOG=info
ENV DATA_DIR=/data
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
CMD curl -f http://localhost:9093/health || exit 1
# Run the binary
CMD ["catalog-registry"]

67
crates/catalog-registry/Makefile Normal file
View file

@ -0,0 +1,67 @@
.PHONY: help build test run clean docker-build docker-run fmt clippy check
help:
@echo "Catalog Registry Service - Make targets"
@echo ""
@echo " build Build release binary"
@echo " build-dev Build debug binary"
@echo " test Run tests"
@echo " run Run service locally"
@echo " clean Clean build artifacts"
@echo " docker-build Build Docker image"
@echo " docker-run Run Docker container"
@echo " fmt Format code"
@echo " clippy Run clippy linter"
@echo " check Check code without building"
@echo ""
build:
cargo build --release
build-dev:
cargo build
test:
cargo test
run:
cargo run -- --config config.toml --port 8082
clean:
cargo clean
rm -f config.toml
docker-build:
docker build -t catalog-registry:latest .
docker-run:
docker-compose up -d
docker-stop:
docker-compose down
fmt:
cargo fmt
clippy:
cargo clippy -- -D warnings
check:
cargo check
audit:
cargo audit
install-deps:
cargo install cargo-audit
config:
@if [ ! -f config.toml ]; then \
cp config.example.toml config.toml; \
echo "Created config.toml from example"; \
echo "Please edit config.toml with your settings"; \
else \
echo "config.toml already exists"; \
fi
all: fmt clippy test build

635
crates/catalog-registry/README.md Normal file
View file

@ -0,0 +1,635 @@
# Catalog Registry Service
A high-performance Rust microservice that provides a unified REST API for extension discovery, versioning,
and download from multiple sources (Gitea releases and OCI registries).
## Features
- **Multi-Backend Support**: Fetch extensions from Gitea releases and OCI registries
- **Unified REST API**: Single API for all extension operations
- **Smart Caching**: LRU cache with TTL to reduce backend API calls
- **Prometheus Metrics**: Built-in metrics for monitoring
- **Health Monitoring**: Health checks for all backends
- **Type-Safe**: Strong typing for extension metadata
- **Async/Await**: High-performance async operations with Tokio
- **Docker Support**: Production-ready containerization
## Architecture
```bash
┌─────────────────────────────────────────────────
────────────┐
│ Catalog Registry API │
│ (axum) │
├─────────────────────────────────────────────────
────────────┤
│ │
│ ┌────────────────┐ ┌────────────────┐
┌──────────────┐ │
│ │ Gitea Client │ │ OCI Client │ │ LRU Cache │ │
│ │ (reqwest) │ │ (reqwest) │ │ (parking) │ │
│ └────────────────┘ └────────────────┘
└──────────────┘ │
│ │ │ │ │
└─────────┼────────────────────┼──────────────────
──┼─────────┘
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Gitea │ │ OCI │ │ Memory │
│ Releases │ │ Registry │ │ │
└──────────┘ └──────────┘ └──────────┘
```
## Installation
### Building from Source
```bash
cd provisioning/platform/catalog-registry
cargo build --release
```
### Docker Build
```bash
docker build -t catalog-registry:latest .
```
### Running with Cargo
```rust
cargo run -- --config config.toml --port 8082
```
### Running with Docker
```bash
docker run -d
-p 8082:8082
-v $(pwd)/config.toml:/app/config.toml:ro
-v $(pwd)/tokens:/app/tokens:ro
catalog-registry:latest
```
## Configuration
Create a `config.toml` file (see `config.example.toml`):
```toml
[server]
host = "0.0.0.0"
port = 8082
workers = 4
enable_cors = true
enable_compression = true
# Gitea backend (optional)
[gitea]
url = "https://gitea.example.com"
organization = "provisioning-extensions"
token_path = "/path/to/gitea-token.txt"
timeout_seconds = 30
verify_ssl = true
# OCI registry backend (optional)
[oci]
registry = "registry.example.com"
namespace = "provisioning"
auth_token_path = "/path/to/oci-token.txt"
timeout_seconds = 30
verify_ssl = true
# Cache configuration
[cache]
capacity = 1000
ttl_seconds = 300
enable_metadata_cache = true
enable_list_cache = true
```
**Note**: At least one backend (Gitea or OCI) must be configured.
## API Endpoints
### Extension Operations
#### List Extensions
```bash
GET /api/v1/extensions
```
Query parameters:
- `type` (optional): Filter by extension type (`provider`, `taskserv`, `cluster`)
- `source` (optional): Filter by source (`gitea`, `oci`)
- `limit` (optional): Maximum results (default: 100)
- `offset` (optional): Pagination offset (default: 0)
Example:
```bash
curl http://localhost:8082/api/v1/extensions?type=provider&limit=10
```
Response:
```bash
[
{
"name": "aws",
"type": "provider",
"version": "1.2.0",
"description": "AWS provider for provisioning",
"author": "provisioning-team",
"repository": "https://gitea.example.com/org/aws_prov",
"source": "gitea",
"published_at": "2025-10-06T12:00:00Z",
"download_url": "https://gitea.example.com/org/aws_prov/releases/download/1.2.0/aws_prov.tar.gz",
"size": 1024000
}
]
```
#### Get Extension
```bash
GET /api/v1/extensions/{type}/{name}
```
Example:
```bash
curl http://localhost:8082/api/v1/extensions/provider/aws
```
#### List Versions
```bash
GET /api/v1/extensions/{type}/{name}/versions
```
Example:
```bash
curl http://localhost:8082/api/v1/extensions/provider/aws/versions
```
Response:
```bash
[
{
"version": "1.2.0",
"published_at": "2025-10-06T12:00:00Z",
"download_url": "https://gitea.example.com/org/aws_prov/releases/download/1.2.0/aws_prov.tar.gz",
"size": 1024000
},
{
"version": "1.1.0",
"published_at": "2025-09-15T10:30:00Z",
"download_url": "https://gitea.example.com/org/aws_prov/releases/download/1.1.0/aws_prov.tar.gz",
"size": 980000
}
]
```
#### Download Extension
```bash
GET /api/v1/extensions/{type}/{name}/{version}
```
Example:
```bash
curl -O http://localhost:8082/api/v1/extensions/provider/aws/1.2.0
```
Returns binary data with `Content-Type: application/octet-stream`.
#### Search Extensions
```bash
GET /api/v1/extensions/search?q={query}
```
Query parameters:
- `q` (required): Search query
- `type` (optional): Filter by extension type
- `limit` (optional): Maximum results (default: 50)
Example:
```bash
curl http://localhost:8082/api/v1/extensions/search?q=kubernetes&type=taskserv
```
### System Endpoints
#### Health Check
```bash
GET /api/v1/health
```
Example:
```bash
curl http://localhost:8082/api/v1/health
```
Response:
```json
{
"status": "healthy",
"version": "0.1.0",
"uptime": 3600,
"backends": {
"gitea": {
"enabled": true,
"healthy": true
},
"oci": {
"enabled": true,
"healthy": true
}
}
}
```
#### Metrics
```bash
GET /api/v1/metrics
```
Returns Prometheus-formatted metrics:
```bash
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total 1234
# HELP cache_hits_total Total cache hits
# TYPE cache_hits_total counter
cache_hits_total 567
# HELP cache_misses_total Total cache misses
# TYPE cache_misses_total counter
cache_misses_total 123
```
#### Cache Statistics
```bash
GET /api/v1/cache/stats
```
Response:
```json
{
"list_entries": 45,
"metadata_entries": 120,
"version_entries": 80,
"total_entries": 245
}
```
## Extension Naming Conventions
### Gitea Repositories
Extensions in Gitea follow specific naming patterns:
- **Providers**: `{name}_prov` (e.g., `aws_prov`, `upcloud_prov`)
- **Task Services**: `{name}_taskserv` (e.g., `kubernetes_taskserv`, `postgres_taskserv`)
- **Clusters**: `{name}_cluster` (e.g., `buildkit_cluster`, `ci_cluster`)
### OCI Artifacts
Extensions in OCI registries follow these patterns:
- **Providers**: `{namespace}/{name}-provider` (e.g., `provisioning/aws-provider`)
- **Task Services**: `{namespace}/{name}-taskserv` (e.g., `provisioning/kubernetes-taskserv`)
- **Clusters**: `{namespace}/{name}-cluster` (e.g., `provisioning/buildkit-cluster`)
## Caching Strategy
The service implements a multi-level LRU cache with TTL:
1. **List Cache**: Caches extension lists (filtered by type/source)
2. **Metadata Cache**: Caches individual extension metadata
3. **Version Cache**: Caches version lists per extension
Cache behavior:
- **Capacity**: Configurable (default: 1000 entries)
- **TTL**: Configurable (default: 5 minutes)
- **Eviction**: LRU (Least Recently Used)
- **Invalidation**: Automatic on TTL expiration
Cache keys:
- List: `list:{type}:{source}`
- Metadata: `{type}/{name}`
- Versions: `{type}/{name}/versions`
## Error Handling
The API uses standard HTTP status codes:
- `200 OK`: Successful operation
- `400 Bad Request`: Invalid input (e.g., invalid extension type)
- `401 Unauthorized`: Authentication failed
- `404 Not Found`: Extension not found
- `429 Too Many Requests`: Rate limit exceeded
- `500 Internal Server Error`: Server error
Error response format:
```json
{
"error": "not_found",
"message": "Extension provider/nonexistent not found"
}
```
## Metrics and Monitoring
### Prometheus Metrics
Available metrics:
- `http_requests_total`: Total HTTP requests
- `http_request_duration_seconds`: Request duration histogram
- `cache_hits_total`: Total cache hits
- `cache_misses_total`: Total cache misses
- `extensions_total`: Total extensions served
### Health Checks
The health endpoint checks:
- Service uptime
- Gitea backend connectivity
- OCI backend connectivity
- Overall service status
## Development
### Project Structure
```bash
catalog-registry/
├── Cargo.toml # Rust dependencies
├── src/
│ ├── main.rs # Entry point
│ ├── lib.rs # Library exports
│ ├── config.rs # Configuration management
│ ├── error.rs # Error types
│ ├── api/
│ │ ├── handlers.rs # HTTP handlers
│ │ └── routes.rs # Route definitions
│ ├── gitea/
│ │ ├── client.rs # Gitea API client
│ │ └── models.rs # Gitea data models
│ ├── oci/
│ │ ├── client.rs # OCI registry client
│ │ └── models.rs # OCI data models
│ ├── cache/
│ │ └── lru_cache.rs # LRU caching
│ └── models/
│ └── extension.rs # Extension models
├── tests/
│ └── integration_test.rs # Integration tests
├── Dockerfile # Docker build
└── README.md # This file
```
### Running Tests
```bash
# Run all tests
cargo test
# Run with output
cargo test -- --nocapture
# Run specific test
cargo test test_health_check
```
### Code Quality
```bash
# Format code
cargo fmt
# Run clippy
cargo clippy
# Check for security vulnerabilities
cargo audit
```
## Deployment
### Systemd Service
Create `/etc/systemd/system/catalog-registry.service`:
```toml
[Unit]
Description=Catalog Registry Service
After=network.target
[Service]
Type=simple
User=registry
WorkingDirectory=/opt/catalog-registry
ExecStart=/usr/local/bin/catalog-registry --config /etc/catalog-registry/config.toml
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target
```
Enable and start:
```bash
sudo systemctl enable catalog-registry
sudo systemctl start catalog-registry
sudo systemctl status catalog-registry
```
### Docker Compose
```bash
version: '3.8'
services:
catalog-registry:
image: catalog-registry:latest
ports:
- "8082:8082"
volumes:
- ./config.toml:/app/config.toml:ro
- ./tokens:/app/tokens:ro
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8082/api/v1/health"]
interval: 30s
timeout: 3s
retries: 3
start_period: 5s
```
### Kubernetes Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: catalog-registry
spec:
replicas: 3
selector:
matchLabels:
app: catalog-registry
template:
metadata:
labels:
app: catalog-registry
spec:
containers:
- name: catalog-registry
image: catalog-registry:latest
ports:
- containerPort: 8082
volumeMounts:
- name: config
mountPath: /app/config.toml
subPath: config.toml
- name: tokens
mountPath: /app/tokens
livenessProbe:
httpGet:
path: /api/v1/health
port: 8082
initialDelaySeconds: 5
periodSeconds: 10
readinessProbe:
httpGet:
path: /api/v1/health
port: 8082
initialDelaySeconds: 5
periodSeconds: 5
volumes:
- name: config
configMap:
name: catalog-registry-config
- name: tokens
secret:
secretName: catalog-registry-tokens
---
apiVersion: v1
kind: Service
metadata:
name: catalog-registry
spec:
selector:
app: catalog-registry
ports:
- port: 8082
targetPort: 8082
type: ClusterIP
```
## Security
### Authentication
- Gitea: Token-based authentication via `token_path`
- OCI: Optional token authentication via `auth_token_path`
### Best Practices
1. **Store tokens securely**: Use file permissions (600) for token files
2. **Enable SSL verification**: Set `verify_ssl = true` in production
3. **Use HTTPS**: Always use HTTPS for Gitea and OCI registries
4. **Limit CORS**: Configure CORS appropriately for production
5. **Rate limiting**: Consider adding rate limiting for public APIs
6. **Network isolation**: Run service in isolated network segments
## Performance
### Benchmarks
Typical performance characteristics:
- **Cached requests**: <5ms response time
- **Uncached requests**: 50-200ms (depends on backend latency)
- **Cache hit ratio**: ~85-95% in typical workloads
- **Throughput**: 1000+ req/s on modern hardware
### Optimization Tips
1. **Increase cache capacity**: For large extension catalogs
2. **Tune TTL**: Balance freshness vs. performance
3. **Use multiple workers**: Scale with CPU cores
4. **Enable compression**: Reduce bandwidth usage
5. **Connection pooling**: Reuse HTTP connections to backends
## Troubleshooting
### Common Issues
#### Service won't start
- Check configuration file syntax
- Verify token files exist and are readable
- Check network connectivity to backends
#### Extensions not found
- Verify backend configuration (URL, organization, namespace)
- Check backend connectivity with health endpoint
- Review logs for authentication errors
#### Slow responses
- Check backend latency
- Increase cache capacity or TTL
- Review Prometheus metrics for bottlenecks
### Logging
Enable debug logging:
```bash
catalog-registry --log-level debug
```
Enable JSON logging for structured logs:
```bash
catalog-registry --json-log
```
## License
Part of the Provisioning Project.
## Contributing
See main project documentation for contribution guidelines.
## Support
For issues and questions, please refer to the main provisioning project documentation.

31
crates/catalog-registry/config.example.toml Normal file
View file

@ -0,0 +1,31 @@
# Catalog Registry Configuration Example
[server]
enable_compression = true
enable_cors = true
host = "0.0.0.0"
port = 8082
workers = 4
# Gitea backend configuration
[gitea]
organization = "provisioning-extensions"
timeout_seconds = 30
token_path = "/path/to/gitea-token.txt"
url = "https://gitea.example.com"
verify_ssl = true
# OCI registry configuration
[oci]
auth_token_path = "/path/to/oci-token.txt"
namespace = "provisioning"
registry = "registry.example.com"
timeout_seconds = 30
verify_ssl = true
# Cache configuration
[cache]
capacity = 1000
enable_list_cache = true
enable_metadata_cache = true
ttl_seconds = 300

29
crates/catalog-registry/docker-compose.yml Normal file
View file

@ -0,0 +1,29 @@
version: '3.8'
services:
catalog-registry:
build:
context: .
dockerfile: Dockerfile
image: catalog-registry:latest
container_name: catalog-registry
ports:
- "8082:8082"
volumes:
- ./config.toml:/app/config.toml:ro
- ./tokens:/app/tokens:ro
environment:
- RUST_LOG=info
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8082/api/v1/health"]
interval: 30s
timeout: 3s
retries: 3
start_period: 5s
networks:
- extension-net
networks:
extension-net:
driver: bridge

85
crates/catalog-registry/scripts/start-service.sh Executable file
View file

@ -0,0 +1,85 @@
#!/usr/bin/env bash
# Start catalog registry service
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_DIR="$(dirname "$SCRIPT_DIR")"
# Default values
CONFIG_FILE="${PROJECT_DIR}/config.toml"
PORT=8082
LOG_LEVEL="info"
JSON_LOG=false
# Parse arguments
while [[ $# -gt 0 ]]; do
case $1 in
--config)
CONFIG_FILE="$2"
shift 2
;;
--port)
PORT="$2"
shift 2
;;
--log-level)
LOG_LEVEL="$2"
shift 2
;;
--json-log)
JSON_LOG=true
shift
;;
--help)
echo "Usage: $0 [OPTIONS]"
echo ""
echo "Options:"
echo " --config FILE Configuration file (default: config.toml)"
echo " --port PORT Port to listen on (default: 8082)"
echo " --log-level LEVEL Log level (trace|debug|info|warn|error)"
echo " --json-log Enable JSON logging"
echo " --help Show this help"
exit 0
;;
*)
echo "Unknown option: $1"
echo "Use --help for usage information"
exit 1
;;
esac
done
# Check if config file exists
if [[ ! -f "$CONFIG_FILE" ]]; then
echo "Error: Configuration file not found: $CONFIG_FILE"
echo "Create config.toml from config.example.toml"
exit 1
fi
# Build if needed
if [[ ! -f "${PROJECT_DIR}/target/release/catalog-registry" ]]; then
echo "Building catalog-registry..."
cd "$PROJECT_DIR"
cargo build --release
fi
# Start service
echo "Starting catalog-registry service..."
echo " Config: $CONFIG_FILE"
echo " Port: $PORT"
echo " Log level: $LOG_LEVEL"
cd "$PROJECT_DIR"
ARGS=(
--config "$CONFIG_FILE"
--port "$PORT"
--log-level "$LOG_LEVEL"
)
if [[ "$JSON_LOG" == "true" ]]; then
ARGS+=(--json-log)
fi
exec ./target/release/catalog-registry "${ARGS[@]}"

443
crates/catalog-registry/src/api/handlers.rs Normal file
View file

@ -0,0 +1,443 @@
use std::sync::Arc;
use std::time::Instant;
use axum::extract::{Path, Query, State};
use axum::http::{header, StatusCode};
use axum::response::{IntoResponse, Response};
use axum::Json;
use tokio::task;
use tracing::{debug, error, info};
use crate::cache::ExtensionCache;
use crate::client::{ClientFactory, DistributionClient, SourceClient};
use crate::error::{RegistryError, Result};
#[cfg(feature = "nats")]
use crate::events::{spawn_cache_invalidator, EventPublisher};
use crate::models::*;
/// Application state
#[derive(Clone)]
pub struct AppState {
pub source_clients: Vec<Arc<dyn SourceClient>>,
pub distribution_clients: Vec<Arc<dyn DistributionClient>>,
pub cache: Arc<ExtensionCache>,
pub start_time: Instant,
#[cfg(feature = "nats")]
pub events: Option<Arc<EventPublisher>>,
}
impl AppState {
/// Create application state.
///
/// `vault_url` enables vault:// token resolution (e.g. `Some("http://127.0.0.1:9094")`).
/// Pass `None` to use filesystem-only token resolution.
pub async fn new(config: crate::config::Config, vault_url: Option<&str>) -> Result<Self> {
let (source_clients, distribution_clients) =
ClientFactory::create_from_config(&config, vault_url).await?;
let cache = Arc::new(ExtensionCache::new(
config.cache.capacity,
config.cache.ttl_seconds,
config.cache.enable_metadata_cache,
config.cache.enable_list_cache,
));
Ok(Self {
source_clients,
distribution_clients,
cache,
start_time: Instant::now(),
#[cfg(feature = "nats")]
events: None,
})
}
/// Attach a connected NatsBridge and spawn the cache invalidator
/// subscriber.
#[cfg(feature = "nats")]
pub fn with_nats(mut self, bridge: Arc<platform_nats::NatsBridge>) -> Self {
spawn_cache_invalidator(Arc::clone(&bridge), Arc::clone(&self.cache));
self.events = Some(Arc::new(EventPublisher::new(bridge)));
self
}
}
/// List all extensions
///
/// Aggregates results from all source and distribution clients in parallel.
/// Uses fallback strategy if clients fail individually.
pub async fn list_extensions(
State(state): State<Arc<AppState>>,
Query(params): Query<ListParams>,
) -> Result<Json<Vec<Extension>>> {
debug!("Listing extensions with params: {:?}", params);
let cache_key = format!("list:{:?}:{:?}", params.extension_type, params.source);
// Check cache
if let Some(cached) = state.cache.get_list(&cache_key) {
debug!("Cache hit for list: {}", cache_key);
return Ok(Json(cached));
}
let mut all_extensions = Vec::new();
// Collect futures for parallel execution
let mut futures = Vec::new();
// Spawn tasks for source clients (Git-based)
for client in &state.source_clients {
let client = Arc::clone(client);
let ext_type = params.extension_type;
futures.push(task::spawn(async move {
client.list_extensions(ext_type).await
}));
}
// Spawn tasks for distribution clients (OCI)
for client in &state.distribution_clients {
let client = Arc::clone(client);
let ext_type = params.extension_type;
futures.push(task::spawn(async move {
client.list_extensions(ext_type).await
}));
}
// Collect results from all parallel tasks
for future in futures {
match future.await {
Ok(Ok(extensions)) => all_extensions.extend(extensions),
Ok(Err(e)) => error!("Client list failed: {}", e),
Err(e) => error!("Task panicked: {}", e),
}
}
// Deduplicate extensions by name (prefer first occurrence)
all_extensions.sort_by(|a, b| {
a.identifier()
.cmp(&b.identifier())
.then_with(|| b.published_at.cmp(&a.published_at))
});
all_extensions.dedup_by(|a, b| a.identifier() == b.identifier());
// Apply pagination
let offset = params.offset.unwrap_or(0);
let limit = params.limit.unwrap_or(100);
let paginated: Vec<Extension> = all_extensions
.into_iter()
.skip(offset)
.take(limit)
.collect();
// Update cache
state.cache.put_list(cache_key, paginated.clone());
Ok(Json(paginated))
}
/// Get specific extension
///
/// Uses fallback strategy: try source clients first, then distribution clients.
/// Returns first successful result.
pub async fn get_extension(
State(state): State<Arc<AppState>>,
Path((ext_type, name)): Path<(String, String)>,
) -> Result<Json<Extension>> {
let extension_type: ExtensionType = ext_type.parse()?;
debug!("Getting extension: {}/{}", extension_type, name);
let cache_key = format!("{}/{}", extension_type, name);
// Check cache
if let Some(cached) = state.cache.get_metadata(&cache_key) {
debug!("Cache hit for metadata: {}", cache_key);
return Ok(Json(cached));
}
// Try source clients first
for client in &state.source_clients {
if let Ok(extension) = client.get_extension(extension_type, &name).await {
state.cache.put_metadata(cache_key, extension.clone());
return Ok(Json(extension));
}
}
// Try distribution clients
for client in &state.distribution_clients {
if let Ok(extension) = client.get_extension(extension_type, &name).await {
state.cache.put_metadata(cache_key, extension.clone());
return Ok(Json(extension));
}
}
Err(RegistryError::NotFound(format!(
"{}/{}",
extension_type, name
)))
}
/// List versions for extension
///
/// Aggregates versions from all clients in parallel.
/// Deduplicates by version string and sorts by published_at descending.
pub async fn list_versions(
State(state): State<Arc<AppState>>,
Path((ext_type, name)): Path<(String, String)>,
) -> Result<Json<Vec<ExtensionVersion>>> {
let extension_type: ExtensionType = ext_type.parse()?;
debug!("Listing versions for: {}/{}", extension_type, name);
let cache_key = format!("{}/{}/versions", extension_type, name);
// Check cache
if let Some(cached) = state.cache.get_versions(&cache_key) {
debug!("Cache hit for versions: {}", cache_key);
let versions: Vec<ExtensionVersion> = cached
.iter()
.map(|v| ExtensionVersion {
version: v.clone(),
published_at: chrono::Utc::now(),
download_url: None,
checksum: None,
size: None,
})
.collect();
return Ok(Json(versions));
}
let mut all_versions = Vec::new();
let mut futures = Vec::new();
// Spawn tasks for source clients
for client in &state.source_clients {
let client = Arc::clone(client);
let name_clone = name.clone();
futures.push(task::spawn(async move {
client.list_versions(extension_type, &name_clone).await
}));
}
// Spawn tasks for distribution clients
for client in &state.distribution_clients {
let client = Arc::clone(client);
let name_clone = name.clone();
futures.push(task::spawn(async move {
client.list_versions(extension_type, &name_clone).await
}));
}
// Collect results from all parallel tasks
for future in futures {
match future.await {
Ok(Ok(versions)) => all_versions.extend(versions),
Ok(Err(e)) => error!("Client list_versions failed: {}", e),
Err(e) => error!("Task panicked: {}", e),
}
}
if all_versions.is_empty() {
return Err(RegistryError::NotFound(format!(
"{}/{}",
extension_type, name
)));
}
// Deduplicate by version string (prefer first occurrence)
all_versions.sort_by_key(|v| std::cmp::Reverse(v.published_at));
all_versions.dedup_by(|a, b| a.version == b.version);
// Cache version strings
let version_strings: Vec<String> = all_versions.iter().map(|v| v.version.clone()).collect();
state.cache.put_versions(cache_key, version_strings);
Ok(Json(all_versions))
}
/// Download extension
///
/// Uses fallback strategy: try source clients first, then distribution clients.
/// Returns first successful download.
pub async fn download_extension(
State(state): State<Arc<AppState>>,
Path((ext_type, name, version)): Path<(String, String, String)>,
) -> Result<Response> {
let extension_type: ExtensionType = ext_type.parse()?;
info!(
"Downloading extension: {}/{} version {}",
extension_type, name, version
);
// Try source clients first
for client in &state.source_clients {
if let Ok(data) = client
.download_extension(extension_type, &name, &version)
.await
{
#[cfg(feature = "nats")]
if let Some(events) = &state.events {
let events = Arc::clone(events);
let (ty, n, v) = (extension_type, name.clone(), version.clone());
tokio::spawn(async move { events.publish_installed(ty, &n, &v).await });
}
return Ok((
StatusCode::OK,
[(header::CONTENT_TYPE, "application/octet-stream")],
data,
)
.into_response());
}
}
// Try distribution clients
for client in &state.distribution_clients {
if let Ok(data) = client
.download_extension(extension_type, &name, &version)
.await
{
#[cfg(feature = "nats")]
if let Some(events) = &state.events {
let events = Arc::clone(events);
let (ty, n, v) = (extension_type, name.clone(), version.clone());
tokio::spawn(async move { events.publish_installed(ty, &n, &v).await });
}
return Ok((
StatusCode::OK,
[(header::CONTENT_TYPE, "application/octet-stream")],
data,
)
.into_response());
}
}
Err(RegistryError::NotFound(format!(
"{}/{} version {}",
extension_type, name, version
)))
}
/// Search extensions
pub async fn search_extensions(
State(state): State<Arc<AppState>>,
Query(params): Query<SearchParams>,
) -> Result<Json<Vec<Extension>>> {
debug!("Searching extensions with query: {:?}", params);
// Get all extensions
let list_params = ListParams {
extension_type: params.extension_type,
source: None,
limit: None,
offset: None,
};
let all_extensions = match list_extensions(State(state), Query(list_params)).await {
Ok(Json(exts)) => exts,
Err(e) => return Err(e),
};
// Filter by search query
let query_lower = params.q.to_lowercase();
let filtered: Vec<Extension> = all_extensions
.into_iter()
.filter(|ext| {
ext.name.to_lowercase().contains(&query_lower)
|| ext.description.to_lowercase().contains(&query_lower)
})
.take(params.limit.unwrap_or(50))
.collect();
Ok(Json(filtered))
}
/// Health check
///
/// Checks health of all configured backends in parallel.
/// Returns "healthy" if all backends are healthy, "degraded" if at least one is
/// healthy, and "unhealthy" if all backends are down.
pub async fn health_check(State(state): State<Arc<AppState>>) -> Json<HealthResponse> {
let mut futures = Vec::new();
// Spawn health check tasks for source clients
for client in &state.source_clients {
let client = Arc::clone(client);
futures.push(task::spawn(async move {
(client.backend_id().to_string(), client.health_check().await)
}));
}
// Spawn health check tasks for distribution clients
for client in &state.distribution_clients {
let client = Arc::clone(client);
futures.push(task::spawn(async move {
(client.backend_id().to_string(), client.health_check().await)
}));
}
// Collect health status from all clients
let mut healthy_count = 0;
let mut unhealthy_count = 0;
for future in futures {
match future.await {
Ok((backend_id, Ok(()))) => {
debug!("Backend {} is healthy", backend_id);
healthy_count += 1;
}
Ok((backend_id, Err(e))) => {
error!("Backend {} health check failed: {}", backend_id, e);
unhealthy_count += 1;
}
Err(e) => {
error!("Health check task panicked: {}", e);
unhealthy_count += 1;
}
}
}
let total = healthy_count + unhealthy_count;
let status = if total == 0 {
"unhealthy".to_string()
} else if unhealthy_count == 0 {
"healthy".to_string()
} else if healthy_count > 0 {
"degraded".to_string()
} else {
"unhealthy".to_string()
};
Json(HealthResponse {
status,
version: env!("CARGO_PKG_VERSION").to_string(),
uptime: state.start_time.elapsed().as_secs(),
backends: BackendHealth {
gitea: BackendStatus {
enabled: !state.source_clients.is_empty(),
healthy: healthy_count > 0,
error: if unhealthy_count > 0 {
Some(format!("{} backends failed", unhealthy_count))
} else {
None
},
},
oci: BackendStatus {
enabled: !state.distribution_clients.is_empty(),
healthy: healthy_count > 0,
error: if unhealthy_count > 0 {
Some(format!("{} backends failed", unhealthy_count))
} else {
None
},
},
},
})
}
/// Metrics endpoint
pub async fn metrics() -> String {
"# Metrics endpoint\n# Metrics collection not implemented\n".to_string()
}
/// Cache stats endpoint
pub async fn cache_stats(State(state): State<Arc<AppState>>) -> Json<crate::cache::CacheStats> {
Json(state.cache.stats())
}

5
crates/catalog-registry/src/api/mod.rs Normal file
View file

@ -0,0 +1,5 @@
pub mod handlers;
pub mod routes;
pub use handlers::AppState;
pub use routes::build_routes;

43
crates/catalog-registry/src/api/routes.rs Normal file
View file

@ -0,0 +1,43 @@
use std::sync::Arc;
use axum::{routing::get, Router};
use tower_http::cors::{Any, CorsLayer};
use tower_http::trace::TraceLayer;
use crate::api::handlers;
/// Build application routes
pub fn build_routes(state: handlers::AppState) -> Router {
let state = Arc::new(state);
// API routes
let api_routes = Router::new()
// Extension operations
.route("/extensions", get(handlers::list_extensions))
.route("/extensions/search", get(handlers::search_extensions))
.route("/extensions/{type}/{name}", get(handlers::get_extension))
.route(
"/extensions/{type}/{name}/versions",
get(handlers::list_versions),
)
.route(
"/extensions/{type}/{name}/{version}",
get(handlers::download_extension),
)
// System endpoints
.route("/health", get(handlers::health_check))
.route("/metrics", get(handlers::metrics))
.route("/cache/stats", get(handlers::cache_stats))
.with_state(state);
// Apply middleware
Router::new()
.nest("/api/v1", api_routes)
.layer(
CorsLayer::new()
.allow_origin(Any)
.allow_methods(Any)
.allow_headers(Any),
)
.layer(TraceLayer::new_for_http())
}

11
crates/catalog-registry/src/api_catalog.rs Normal file
View file

@ -0,0 +1,11 @@
use axum::{extract::State, response::IntoResponse, Json};
use ontoref_ontology::api::ApiRouteEntry;
use serde_json::json;
use crate::handlers::AppState;
pub async fn api_catalog(State(_state): State<AppState>) -> impl IntoResponse {
let mut routes: Vec<&'static ApiRouteEntry> = inventory::iter::<ApiRouteEntry>().collect();
routes.sort_by(|a, b| a.path.cmp(b.path).then(a.method.cmp(b.method)));
Json(json!({ "service": "catalog-registry", "routes": routes }))
}

257
crates/catalog-registry/src/cache/lru_cache.rs vendored Normal file
View file

@ -0,0 +1,257 @@
use std::num::NonZeroUsize;
use std::sync::Arc;
use chrono::{DateTime, Duration, Utc};
use lru::LruCache;
use parking_lot::Mutex;
use crate::models::Extension;
/// Cached item with expiration
#[derive(Debug, Clone)]
struct CachedItem<T> {
value: T,
expires_at: DateTime<Utc>,
}
impl<T> CachedItem<T> {
fn new(value: T, ttl: Duration) -> Self {
Self {
value,
expires_at: Utc::now() + ttl,
}
}
fn is_expired(&self) -> bool {
Utc::now() > self.expires_at
}
}
/// Extension cache with LRU eviction and TTL
pub struct ExtensionCache {
list_cache: Arc<Mutex<LruCache<String, CachedItem<Vec<Extension>>>>>,
metadata_cache: Arc<Mutex<LruCache<String, CachedItem<Extension>>>>,
version_cache: Arc<Mutex<LruCache<String, CachedItem<Vec<String>>>>>,
ttl: Duration,
enable_metadata: bool,
enable_list: bool,
}
impl ExtensionCache {
/// Create new cache with specified capacity and TTL
pub fn new(
capacity: usize,
ttl_seconds: u64,
enable_metadata: bool,
enable_list: bool,
) -> Self {
let capacity = NonZeroUsize::new(capacity).expect("Capacity must be non-zero");
Self {
list_cache: Arc::new(Mutex::new(LruCache::new(capacity))),
metadata_cache: Arc::new(Mutex::new(LruCache::new(capacity))),
version_cache: Arc::new(Mutex::new(LruCache::new(capacity))),
ttl: Duration::seconds(ttl_seconds as i64),
enable_metadata,
enable_list,
}
}
/// Get cached extension list
pub fn get_list(&self, key: &str) -> Option<Vec<Extension>> {
if !self.enable_list {
return None;
}
let mut cache = self.list_cache.lock();
if let Some(item) = cache.get(key) {
if !item.is_expired() {
return Some(item.value.clone());
} else {
cache.pop(key);
}
}
None
}
/// Put extension list in cache
pub fn put_list(&self, key: String, value: Vec<Extension>) {
if !self.enable_list {
return;
}
let mut cache = self.list_cache.lock();
cache.put(key, CachedItem::new(value, self.ttl));
}
/// Get cached extension metadata
pub fn get_metadata(&self, key: &str) -> Option<Extension> {
if !self.enable_metadata {
return None;
}
let mut cache = self.metadata_cache.lock();
if let Some(item) = cache.get(key) {
if !item.is_expired() {
return Some(item.value.clone());
} else {
cache.pop(key);
}
}
None
}
/// Put extension metadata in cache
pub fn put_metadata(&self, key: String, value: Extension) {
if !self.enable_metadata {
return;
}
let mut cache = self.metadata_cache.lock();
cache.put(key, CachedItem::new(value, self.ttl));
}
/// Get cached version list
pub fn get_versions(&self, key: &str) -> Option<Vec<String>> {
let mut cache = self.version_cache.lock();
if let Some(item) = cache.get(key) {
if !item.is_expired() {
return Some(item.value.clone());
} else {
cache.pop(key);
}
}
None
}
/// Put version list in cache
pub fn put_versions(&self, key: String, value: Vec<String>) {
let mut cache = self.version_cache.lock();
cache.put(key, CachedItem::new(value, self.ttl));
}
/// Invalidate specific cache entry
pub fn invalidate(&self, key: &str) {
{
let mut cache = self.list_cache.lock();
cache.pop(key);
}
{
let mut cache = self.metadata_cache.lock();
cache.pop(key);
}
{
let mut cache = self.version_cache.lock();
cache.pop(key);
}
}
/// Invalidate all cache entries
pub fn invalidate_all(&self) {
{
let mut cache = self.list_cache.lock();
cache.clear();
}
{
let mut cache = self.metadata_cache.lock();
cache.clear();
}
{
let mut cache = self.version_cache.lock();
cache.clear();
}
}
/// Get cache statistics
pub fn stats(&self) -> CacheStats {
let list_size = self.list_cache.lock().len();
let metadata_size = self.metadata_cache.lock().len();
let version_size = self.version_cache.lock().len();
CacheStats {
list_entries: list_size,
metadata_entries: metadata_size,
version_entries: version_size,
total_entries: list_size + metadata_size + version_size,
}
}
}
/// Cache statistics
#[derive(Debug, Clone, serde::Serialize)]
pub struct CacheStats {
pub list_entries: usize,
pub metadata_entries: usize,
pub version_entries: usize,
pub total_entries: usize,
}
#[cfg(test)]
mod tests {
use super::*;
use crate::models::{ExtensionSource, ExtensionType};
fn create_test_extension(name: &str) -> Extension {
Extension {
name: name.to_string(),
extension_type: ExtensionType::Provider,
version: "1.0.0".to_string(),
description: "Test extension".to_string(),
author: None,
repository: None,
source: ExtensionSource::Gitea,
published_at: Utc::now(),
download_url: None,
checksum: None,
size: None,
tags: None,
}
}
#[test]
fn test_metadata_cache() {
let cache = ExtensionCache::new(10, 300, true, true);
let ext = create_test_extension("test");
cache.put_metadata("test-key".to_string(), ext.clone());
let cached = cache.get_metadata("test-key");
assert!(cached.is_some());
assert_eq!(cached.unwrap().name, ext.name);
}
#[test]
fn test_list_cache() {
let cache = ExtensionCache::new(10, 300, true, true);
let extensions = vec![
create_test_extension("test1"),
create_test_extension("test2"),
];
cache.put_list("test-list".to_string(), extensions.clone());
let cached = cache.get_list("test-list");
assert!(cached.is_some());
assert_eq!(cached.unwrap().len(), extensions.len());
}
#[test]
fn test_cache_invalidation() {
let cache = ExtensionCache::new(10, 300, true, true);
let ext = create_test_extension("test");
cache.put_metadata("test-key".to_string(), ext);
cache.invalidate("test-key");
assert!(cache.get_metadata("test-key").is_none());
}
#[test]
fn test_disabled_cache() {
let cache = ExtensionCache::new(10, 300, false, false);
let ext = create_test_extension("test");
cache.put_metadata("test-key".to_string(), ext);
assert!(cache.get_metadata("test-key").is_none());
}
}

3
crates/catalog-registry/src/cache/mod.rs vendored Normal file
View file

@ -0,0 +1,3 @@
pub mod lru_cache;
pub use lru_cache::{CacheStats, ExtensionCache};

125
crates/catalog-registry/src/client/factory.rs Normal file
View file

@ -0,0 +1,125 @@
//! Client factory for creating extension clients from configuration
use std::sync::Arc;
use tracing::{info, warn};
use super::traits::{DistributionClient, SourceClient};
use super::vault_resolver::VaultResolver;
use super::{ForgejoClient, GitHubClient};
use crate::config::Config;
use crate::error::{RegistryError, Result};
use crate::gitea::GiteaClient as GiteaClientImpl;
use crate::oci::OciClient as OciClientImpl;
/// Factory for creating extension clients
pub struct ClientFactory;
impl ClientFactory {
/// Create all configured clients from configuration, resolving any vault://
/// token references.
///
/// When `vault_url` is `Some`, any `token_path` starting with `vault://` is
/// resolved via the vault-service HTTP API with a 5-minute in-memory
/// cache. When `vault_url` is `None`, vault:// paths cause an error.
pub async fn create_from_config(
config: &Config,
vault_url: Option<&str>,
) -> Result<(Vec<Arc<dyn SourceClient>>, Vec<Arc<dyn DistributionClient>>)> {
let resolver = vault_url.map(|u| VaultResolver::new(u.to_string()));
let mut source_clients: Vec<Arc<dyn SourceClient>> = Vec::new();
let mut distribution_clients: Vec<Arc<dyn DistributionClient>> = Vec::new();
// Create Gitea clients (source-based)
for gitea_config in &config.sources.gitea {
let token = resolve_token(&gitea_config.token_path, resolver.as_ref()).await?;
let client = GiteaClientImpl::new(gitea_config, token)?;
source_clients.push(Arc::new(client) as Arc<dyn SourceClient>);
info!("Registered Gitea client: {}", gitea_config.url);
}
// Create Forgejo clients (source-based)
for forgejo_config in &config.sources.forgejo {
let token = resolve_token(&forgejo_config.token_path, resolver.as_ref()).await?;
let client = ForgejoClient::new(forgejo_config, token)?;
source_clients.push(Arc::new(client) as Arc<dyn SourceClient>);
info!("Registered Forgejo client: {}", forgejo_config.url);
}
// Create GitHub clients (source-based)
for github_config in &config.sources.github {
// GitHub tokens are optional; a missing file is treated as unauthenticated
// access
let token = resolve_token_optional(&github_config.token_path, resolver.as_ref()).await;
if token.is_none() {
warn!(
"GitHub client '{}' has no token — API rate limits apply",
github_config.organization
);
}
let client = GitHubClient::new(github_config, token)?;
source_clients.push(Arc::new(client) as Arc<dyn SourceClient>);
info!("Registered GitHub client: {}", github_config.organization);
}
// Create OCI clients (distribution-based)
for oci_config in &config.distributions.oci {
let auth_token = match &oci_config.auth_token_path {
Some(path) => Some(resolve_token(path, resolver.as_ref()).await?),
None => None,
};
let client = OciClientImpl::new(oci_config, auth_token)?;
distribution_clients.push(Arc::new(client) as Arc<dyn DistributionClient>);
info!("Registered OCI client: {}", oci_config.registry);
}
if source_clients.is_empty() && distribution_clients.is_empty() {
return Err(RegistryError::Config(
"No backends configured (gitea, forgejo, github, or oci required)".to_string(),
));
}
info!(
"Initialized {} source clients and {} distribution clients",
source_clients.len(),
distribution_clients.len()
);
Ok((source_clients, distribution_clients))
}
}
/// Resolve a token path to its plaintext value.
///
/// Paths prefixed with `vault://` are resolved via VaultResolver (requires
/// resolver to be Some). All other paths are read from the filesystem.
async fn resolve_token(path: &str, resolver: Option<&VaultResolver>) -> Result<String> {
if path.starts_with("vault://") {
let resolver = resolver.ok_or_else(|| {
RegistryError::Config(format!(
"token_path '{}' is a vault:// reference but no vault_url was provided",
path
))
})?;
return resolver
.try_resolve(path)
.await
.expect("try_resolve always returns Some for vault:// prefixed paths")
.map_err(|e| {
RegistryError::Config(format!("Vault resolution failed for '{}': {}", path, e))
});
}
std::fs::read_to_string(path)
.map(|s| s.trim().to_string())
.map_err(|e| RegistryError::Config(format!("Failed to read token from '{}': {}", path, e)))
}
/// Resolve a token path, returning `None` on any failure.
///
/// Used for optional tokens (e.g., GitHub) where unauthenticated access is
/// acceptable.
async fn resolve_token_optional(path: &str, resolver: Option<&VaultResolver>) -> Option<String> {
resolve_token(path, resolver).await.ok()
}

102
crates/catalog-registry/src/client/forgejo.rs Normal file
View file

@ -0,0 +1,102 @@
//! Forgejo client implementation
//!
//! Forgejo API is fully compatible with Gitea API, so this implementation
//! uses composition to wrap GiteaClient and delegate all operations to it.
//! This allows Forgejo instances to work seamlessly with the same interface
//! while maintaining independent backend identification.
use async_trait::async_trait;
use bytes::Bytes;
use super::traits::{BackendType, ExtensionClient, ReleaseInfo, SourceClient};
use crate::error::Result;
use crate::models::{Extension, ExtensionType, ExtensionVersion};
/// Forgejo client (wraps GiteaClient since Forgejo API is Gitea-compatible)
pub struct ForgejoClient {
backend_id: String,
inner: crate::gitea::GiteaClient,
}
impl ForgejoClient {
/// Create new Forgejo client with a pre-resolved token.
///
/// Token resolution (file read or vault:// fetch) is the caller's
/// responsibility.
pub fn new(config: &crate::config::GiteaConfig, token: String) -> Result<Self> {
let backend_id = config
.id
.clone()
.unwrap_or_else(|| format!("forgejo-{}", config.organization));
let inner = crate::gitea::GiteaClient::new(config, token)?;
Ok(Self { backend_id, inner })
}
}
/// ExtensionClient trait implementation for Forgejo
#[async_trait]
impl ExtensionClient for ForgejoClient {
fn backend_id(&self) -> &str {
&self.backend_id
}
fn backend_type(&self) -> BackendType {
BackendType::Forgejo
}
async fn list_extensions(
&self,
extension_type: Option<ExtensionType>,
) -> Result<Vec<Extension>> {
self.inner.list_extensions(extension_type).await
}
async fn get_extension(&self, extension_type: ExtensionType, name: &str) -> Result<Extension> {
self.inner.get_extension(extension_type, name).await
}
async fn list_versions(
&self,
extension_type: ExtensionType,
name: &str,
) -> Result<Vec<ExtensionVersion>> {
self.inner.list_versions(extension_type, name).await
}
async fn download_extension(
&self,
extension_type: ExtensionType,
name: &str,
version: &str,
) -> Result<Bytes> {
self.inner
.download_extension(extension_type, name, version)
.await
}
async fn health_check(&self) -> Result<()> {
self.inner.health_check().await
}
}
/// SourceClient trait implementation for Forgejo
#[async_trait]
impl SourceClient for ForgejoClient {
async fn get_repository_url(
&self,
extension_type: ExtensionType,
name: &str,
) -> Result<String> {
self.inner.get_repository_url(extension_type, name).await
}
async fn list_releases(&self, repo_name: &str) -> Result<Vec<ReleaseInfo>> {
self.inner.list_releases(repo_name).await
}
async fn get_release_notes(&self, repo_name: &str, version: &str) -> Result<String> {
self.inner.get_release_notes(repo_name, version).await
}
}

346
crates/catalog-registry/src/client/github.rs Normal file
View file

@ -0,0 +1,346 @@
//! GitHub client implementation
//!
//! Integrates with GitHub Releases API to fetch provisioning extensions.
//! Extensions are identified as releases within repositories under a GitHub
//! organization.
use std::time::Duration;
use async_trait::async_trait;
use bytes::Bytes;
use reqwest::Client;
use tracing::debug;
use super::traits::{BackendType, ExtensionClient, ReleaseAsset, ReleaseInfo, SourceClient};
use crate::error::{RegistryError, Result};
use crate::models::{Extension, ExtensionSource, ExtensionType, ExtensionVersion};
/// GitHub configuration (reused GiteaConfig for compatibility)
pub struct GitHubClient {
backend_id: String,
organization: String,
token: Option<String>,
client: Client,
}
impl GitHubClient {
/// Create new GitHub client with a pre-resolved token.
///
/// Token resolution (file read or vault:// fetch) is the caller's
/// responsibility. Pass `None` for unauthenticated access (lower rate
/// limits apply).
pub fn new(config: &crate::config::GiteaConfig, token: Option<String>) -> Result<Self> {
let client = Client::builder()
.timeout(Duration::from_secs(config.timeout_seconds))
.danger_accept_invalid_certs(!config.verify_ssl)
.user_agent("catalog-registry/0.1.0")
.build()
.map_err(|e| RegistryError::Config(format!("Failed to create HTTP client: {}", e)))?;
let backend_id = config
.id
.clone()
.unwrap_or_else(|| format!("github-{}", config.organization));
Ok(Self {
backend_id,
organization: config.organization.clone(),
token,
client,
})
}
/// Internal: List releases from GitHub
async fn list_releases_internal(&self, owner: &str, repo: &str) -> Result<Vec<GitHubRelease>> {
let url = format!(
"https://api.github.com/repos/{}/{}/releases?per_page=100",
owner, repo
);
let mut request = self.client.get(&url);
if let Some(ref token) = self.token {
request = request.header("Authorization", format!("token {}", token));
}
let response = request
.send()
.await
.map_err(|e| RegistryError::Config(format!("GitHub API request failed: {}", e)))?;
if !response.status().is_success() {
return Err(RegistryError::Config(format!(
"GitHub API returned: {}",
response.status()
)));
}
response
.json()
.await
.map_err(|e| RegistryError::Config(format!("Failed to parse GitHub response: {}", e)))
}
/// Internal: Format repository name from extension type and name
fn format_repo_name(&self, extension_type: ExtensionType, name: &str) -> String {
let suffix = match extension_type {
ExtensionType::Provider => "_prov",
ExtensionType::Taskserv => "_taskserv",
ExtensionType::Cluster => "_cluster",
};
format!("{}{}", name, suffix)
}
}
/// GitHub release (minimal fields)
#[derive(Debug, Clone, serde::Deserialize)]
pub struct GitHubRelease {
pub tag_name: String,
pub name: String,
pub body: Option<String>,
pub draft: bool,
pub prerelease: bool,
pub created_at: chrono::DateTime<chrono::Utc>,
pub published_at: Option<chrono::DateTime<chrono::Utc>>,
pub assets: Vec<GitHubAsset>,
}
/// GitHub release asset
#[derive(Debug, Clone, serde::Deserialize)]
pub struct GitHubAsset {
pub name: String,
pub browser_download_url: String,
pub size: u64,
}
/// ExtensionClient trait implementation for GitHub
#[async_trait]
impl ExtensionClient for GitHubClient {
fn backend_id(&self) -> &str {
&self.backend_id
}
fn backend_type(&self) -> BackendType {
BackendType::GitHub
}
async fn list_extensions(
&self,
_extension_type: Option<ExtensionType>,
) -> Result<Vec<Extension>> {
debug!(
"Fetching repositories for GitHub organization: {}",
self.organization
);
// Note: This is a simplified implementation that would require
// listing repos from the organization. For full implementation,
// integrate GitHub's organizations API endpoint.
Ok(Vec::new())
}
async fn get_extension(&self, extension_type: ExtensionType, name: &str) -> Result<Extension> {
let repo_name = self.format_repo_name(extension_type, name);
debug!(
"Fetching GitHub extension: {}/{}",
self.organization, repo_name
);
let releases = self
.list_releases_internal(&self.organization, &repo_name)
.await?;
let latest = releases
.iter()
.find(|r| !r.draft && !r.prerelease)
.ok_or_else(|| {
RegistryError::NotFound(format!(
"No releases found for {}/{}",
self.organization, repo_name
))
})?;
let asset = latest.assets.first().ok_or_else(|| {
RegistryError::NotFound(format!(
"No assets found in latest release of {}/{}",
self.organization, repo_name
))
})?;
Ok(Extension {
name: name.to_string(),
extension_type,
version: latest.tag_name.clone(),
description: latest.body.clone().unwrap_or_default(),
author: None,
repository: Some(format!(
"https://github.com/{}/{}",
self.organization, repo_name
)),
source: ExtensionSource::Github,
published_at: latest.published_at.unwrap_or(latest.created_at),
download_url: Some(asset.browser_download_url.clone()),
checksum: None,
size: Some(asset.size),
tags: None,
})
}
async fn list_versions(
&self,
extension_type: ExtensionType,
name: &str,
) -> Result<Vec<ExtensionVersion>> {
let repo_name = self.format_repo_name(extension_type, name);
debug!(
"Fetching GitHub versions for: {}/{}",
self.organization, repo_name
);
let releases = self
.list_releases_internal(&self.organization, &repo_name)
.await?;
Ok(releases
.into_iter()
.filter(|r| !r.draft && !r.prerelease)
.map(|r| ExtensionVersion {
version: r.tag_name,
published_at: r.published_at.unwrap_or(r.created_at),
download_url: r.assets.first().map(|a| a.browser_download_url.clone()),
checksum: None,
size: r.assets.first().map(|a| a.size),
})
.collect())
}
async fn download_extension(
&self,
extension_type: ExtensionType,
name: &str,
version: &str,
) -> Result<Bytes> {
let repo_name = self.format_repo_name(extension_type, name);
debug!(
"Downloading GitHub extension: {}/{} version {}",
self.organization, repo_name, version
);
let releases = self
.list_releases_internal(&self.organization, &repo_name)
.await?;
let release = releases
.iter()
.find(|r| r.tag_name == version)
.ok_or_else(|| {
RegistryError::NotFound(format!(
"Release {} not found for {}/{}",
version, self.organization, repo_name
))
})?;
let asset = release.assets.first().ok_or_else(|| {
RegistryError::NotFound(format!(
"No assets in release {} of {}/{}",
version, self.organization, repo_name
))
})?;
let response = self
.client
.get(&asset.browser_download_url)
.send()
.await
.map_err(|e| RegistryError::Config(format!("Download failed: {}", e)))?;
if !response.status().is_success() {
return Err(RegistryError::Config(format!(
"Download failed: {}",
response.status()
)));
}
response
.bytes()
.await
.map_err(|e| RegistryError::Config(format!("Failed to read response: {}", e)))
}
async fn health_check(&self) -> Result<()> {
let url = "https://api.github.com/zen";
let response = self
.client
.get(url)
.timeout(Duration::from_secs(5))
.send()
.await
.map_err(|e| RegistryError::Config(format!("GitHub health check failed: {}", e)))?;
if response.status().is_success() {
Ok(())
} else {
Err(RegistryError::Config(format!(
"GitHub health check returned: {}",
response.status()
)))
}
}
}
/// SourceClient trait implementation for GitHub
#[async_trait]
impl SourceClient for GitHubClient {
async fn get_repository_url(
&self,
extension_type: ExtensionType,
name: &str,
) -> Result<String> {
let repo_name = self.format_repo_name(extension_type, name);
Ok(format!(
"https://github.com/{}/{}",
self.organization, repo_name
))
}
async fn list_releases(&self, repo_name: &str) -> Result<Vec<ReleaseInfo>> {
let releases = self
.list_releases_internal(&self.organization, repo_name)
.await?;
Ok(releases
.into_iter()
.map(|r| ReleaseInfo {
tag: r.tag_name,
created_at: r.created_at,
published_at: r.published_at,
assets: r
.assets
.into_iter()
.map(|a| ReleaseAsset {
name: a.name,
download_url: a.browser_download_url,
size: Some(a.size),
})
.collect(),
description: r.body.unwrap_or_default(),
})
.collect())
}
async fn get_release_notes(&self, repo_name: &str, version: &str) -> Result<String> {
let releases = self
.list_releases_internal(&self.organization, repo_name)
.await?;
let release = releases
.iter()
.find(|r| r.tag_name == version)
.ok_or_else(|| {
RegistryError::NotFound(format!(
"Release {} not found in {}/{}",
version, self.organization, repo_name
))
})?;
Ok(release.body.clone().unwrap_or_default())
}
}

19
crates/catalog-registry/src/client/mod.rs Normal file
View file

@ -0,0 +1,19 @@
//! Client module with trait hierarchy and implementations
//!
//! Provides abstraction layer for multiple backend types:
//! - Source clients: Gitea, Forgejo, GitHub (Git-based repositories)
//! - Distribution clients: OCI registries (Docker Hub, Harbor, Quay, ghcr.io)
pub mod factory;
pub mod forgejo;
pub mod github;
pub mod traits;
pub mod vault_resolver;
pub use factory::ClientFactory;
pub use forgejo::ForgejoClient;
pub use github::GitHubClient;
pub use traits::{
BackendType, DistributionClient, ExtensionClient, LayerInfo, ManifestInfo, ReleaseAsset,
ReleaseInfo, SourceClient,
};

135
crates/catalog-registry/src/client/traits.rs Normal file
View file

@ -0,0 +1,135 @@
//! Client trait hierarchy for extension sources and distributions
//!
//! Defines three-tier trait system:
//! - `ExtensionClient`: Base trait for all backends
//! - `SourceClient`: Git-based repository sources (Gitea, Forgejo, GitHub)
//! - `DistributionClient`: OCI-compatible registries (Harbor, Docker Hub, etc.)
use async_trait::async_trait;
use bytes::Bytes;
use crate::error::Result;
use crate::models::{Extension, ExtensionType, ExtensionVersion};
/// Backend type identifier
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum BackendType {
Gitea,
Forgejo,
GitHub,
Oci,
}
/// Base trait for all extension clients
///
/// Common operations that all backends must support, regardless of whether
/// they're source (Git) or distribution (OCI) backends.
#[async_trait]
pub trait ExtensionClient: Send + Sync {
/// Get unique backend identifier for this instance
fn backend_id(&self) -> &str;
/// Get backend type
fn backend_type(&self) -> BackendType;
/// List all extensions available from this backend
async fn list_extensions(
&self,
extension_type: Option<ExtensionType>,
) -> Result<Vec<Extension>>;
/// Get specific extension metadata
async fn get_extension(&self, extension_type: ExtensionType, name: &str) -> Result<Extension>;
/// List all versions for an extension
async fn list_versions(
&self,
extension_type: ExtensionType,
name: &str,
) -> Result<Vec<ExtensionVersion>>;
/// Download extension artifact
async fn download_extension(
&self,
extension_type: ExtensionType,
name: &str,
version: &str,
) -> Result<Bytes>;
/// Check backend health
async fn health_check(&self) -> Result<()>;
}
/// Source client trait for Git-based backends
///
/// Extends `ExtensionClient` with operations specific to Git repository sources
/// like Gitea, Forgejo, and GitHub. These backends treat extensions as releases
/// in repositories.
#[async_trait]
pub trait SourceClient: ExtensionClient {
/// Get repository URL for an extension
async fn get_repository_url(&self, extension_type: ExtensionType, name: &str)
-> Result<String>;
/// List all releases for a repository
///
/// Returns release info including tags, assets, and creation times
async fn list_releases(&self, repo_name: &str) -> Result<Vec<ReleaseInfo>>;
/// Get release notes/description for a specific version
async fn get_release_notes(&self, repo_name: &str, version: &str) -> Result<String>;
}
/// Distribution client trait for OCI registries
///
/// Extends `ExtensionClient` with operations specific to OCI registries
/// like Docker Hub, Harbor, Quay, and ghcr.io. These backends treat extensions
/// as container images or artifacts with OCI manifests.
#[async_trait]
pub trait DistributionClient: ExtensionClient {
/// Get manifest for an artifact
async fn get_manifest(&self, repo_name: &str, tag: &str) -> Result<ManifestInfo>;
/// List all catalogs/repositories in the registry
async fn list_catalog(&self) -> Result<Vec<String>>;
/// Get digest for an artifact
async fn get_digest(&self, repo_name: &str, tag: &str) -> Result<String>;
/// Verify artifact integrity
async fn verify_artifact(&self, repo_name: &str, tag: &str, digest: &str) -> Result<bool>;
}
/// Release information from source backends
#[derive(Debug, Clone)]
pub struct ReleaseInfo {
pub tag: String,
pub created_at: chrono::DateTime<chrono::Utc>,
pub published_at: Option<chrono::DateTime<chrono::Utc>>,
pub assets: Vec<ReleaseAsset>,
pub description: String,
}
/// Asset attached to a release
#[derive(Debug, Clone)]
pub struct ReleaseAsset {
pub name: String,
pub download_url: String,
pub size: Option<u64>,
}
/// OCI manifest information
#[derive(Debug, Clone)]
pub struct ManifestInfo {
pub config_digest: String,
pub layers: Vec<LayerInfo>,
pub total_size: u64,
}
/// OCI layer information
#[derive(Debug, Clone)]
pub struct LayerInfo {
pub digest: String,
pub size: u64,
pub media_type: String,
}

115
crates/catalog-registry/src/client/vault_resolver.rs Normal file
View file

@ -0,0 +1,115 @@
use std::collections::HashMap;
use std::sync::Arc;
use std::time::{Duration, Instant};
use parking_lot::Mutex;
use reqwest::Client;
use serde::Deserialize;
use tracing::{info, warn};
use crate::error::{RegistryError, Result};
const TOKEN_TTL: Duration = Duration::from_secs(300); // 5-minute in-memory cache
#[derive(Debug, Deserialize)]
struct SecretResponse {
value: String,
}
struct CachedToken {
value: String,
expires_at: Instant,
}
/// Resolves `vault://path` references to their plaintext values
/// via the vault-service HTTP API.
///
/// Tokens are cached in-memory for `TOKEN_TTL` to avoid repeated requests
/// on every client creation.
pub struct VaultResolver {
vault_url: String,
http: Client,
cache: Arc<Mutex<HashMap<String, CachedToken>>>,
}
impl VaultResolver {
pub fn new(vault_url: String) -> Self {
Self {
vault_url,
http: Client::builder()
.timeout(Duration::from_secs(5))
.build()
.expect("reqwest client"),
cache: Arc::new(Mutex::new(HashMap::new())),
}
}
/// Resolve `vault://path/to/secret` → plaintext value.
///
/// Returns `None` if the path is not a `vault://` reference.
pub async fn try_resolve(&self, token_path: &str) -> Option<Result<String>> {
let secret_path = token_path.strip_prefix("vault://")?;
// Check cache first
{
let cache = self.cache.lock();
if let Some(cached) = cache.get(secret_path) {
if cached.expires_at > Instant::now() {
return Some(Ok(cached.value.clone()));
}
}
}
// Fetch from vault-service
let url = format!("{}/v1/{}", self.vault_url, secret_path);
let resp = match self.http.get(&url).send().await {
Ok(r) => r,
Err(e) => {
return Some(Err(RegistryError::Config(format!(
"vault resolve HTTP error for '{}': {}",
secret_path, e
))))
}
};
if !resp.status().is_success() {
return Some(Err(RegistryError::Config(format!(
"vault resolve failed for '{}': HTTP {}",
secret_path,
resp.status()
))));
}
let body: SecretResponse = match resp.json().await {
Ok(b) => b,
Err(e) => {
return Some(Err(RegistryError::Config(format!(
"vault resolve parse error for '{}': {}",
secret_path, e
))))
}
};
info!(path = %secret_path, "Token resolved from vault");
// Cache the result
{
let mut cache = self.cache.lock();
cache.insert(
secret_path.to_string(),
CachedToken {
value: body.value.clone(),
expires_at: Instant::now() + TOKEN_TTL,
},
);
}
Some(Ok(body.value))
}
/// Expire all cached tokens, forcing re-resolution on next access.
pub fn invalidate_cache(&self) {
self.cache.lock().clear();
warn!("VaultResolver: token cache invalidated");
}
}

511
crates/catalog-registry/src/config.rs Normal file
View file

@ -0,0 +1,511 @@
use std::env;
use std::path::Path;
use platform_config::ConfigLoader;
use serde::{Deserialize, Serialize};
use crate::error::{RegistryError, Result};
/// Main configuration
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct Config {
#[serde(default)]
pub server: ServerConfig,
// Old single-instance format (for backward compatibility)
pub gitea: Option<GiteaConfig>,
pub oci: Option<OciConfig>,
// New multi-instance format
#[serde(default)]
pub sources: SourcesConfig,
#[serde(default)]
pub distributions: DistributionsConfig,
#[serde(default)]
pub cache: CacheConfig,
}
impl Config {
/// Load configuration from Nickel (catalog-registry.ncl) with fallback to
/// hierarchy
pub fn load() -> Result<Self> {
let mut config = Self::load_from_nickel()
.or_else(|_| Self::load_from_hierarchy())
.unwrap_or_default();
Self::migrate_to_multiinstance(&mut config);
Self::apply_env_overrides_internal(&mut config);
Ok(config)
}
/// Load configuration from Nickel catalog-registry.ncl
fn load_from_nickel() -> Result<Self> {
let config_json = platform_config::load_service_config_from_ncl("catalog-registry")
.map_err(|e| RegistryError::Config(format!("Failed to load from Nickel: {}", e)))?;
serde_json::from_value(config_json).map_err(RegistryError::Json)
}
/// Load configuration from hierarchical sources with mode support
///
/// Priority order:
/// 1. REGISTRY_CONFIG environment variable (explicit path)
/// 2. REGISTRY_MODE environment variable (mode-specific file)
/// 3. Hardcoded default
///
/// After loading, applies environment variable overrides and migrates
/// old single-instance configs to multi-instance format.
pub fn load_from_hierarchy() -> Result<Self> {
let mut config = if let Ok(path) = env::var("REGISTRY_CONFIG") {
// Explicit config path provided
load_config(&path)?
} else if let Ok(mode) = env::var("REGISTRY_MODE") {
// Mode-specific config file
let path = format!(
"provisioning/platform/config/catalog-registry.{}.toml",
mode
);
load_config(&path)?
} else {
// Fall back to default
Self::default()
};
// Migrate old single-instance configs to new multi-instance format if needed
Self::migrate_to_multiinstance(&mut config);
Self::apply_env_overrides_internal(&mut config);
Ok(config)
}
/// Migrate old single-instance configs to new multi-instance format
///
/// Detects old-style Option-based configs and automatically moves them
/// to the new Vec-based multi-instance format for seamless backward
/// compatibility.
fn migrate_to_multiinstance(config: &mut Self) {
let mut migrated = false;
// Migrate single Gitea instance to new format
if let Some(gitea) = config.gitea.take() {
if config.sources.gitea.is_empty() {
config.sources.gitea.push(gitea);
migrated = true;
}
}
// Migrate single OCI instance to new format
if let Some(oci) = config.oci.take() {
if config.distributions.oci.is_empty() {
config.distributions.oci.push(oci);
migrated = true;
}
}
if migrated {
tracing::info!("Auto-migrated single-instance config to multi-instance format");
}
}
/// Apply environment variable overrides to configuration (internal helper)
///
/// Overrides take precedence over loaded config values.
/// Pattern: REGISTRY_{SECTION}_{KEY}
fn apply_env_overrides_internal(config: &mut Self) {
if let Ok(val) = env::var("REGISTRY_SERVER_HOST") {
config.server.host = val;
}
if let Ok(val) = env::var("REGISTRY_SERVER_PORT") {
if let Ok(port) = val.parse() {
config.server.port = port;
}
}
if let Ok(val) = env::var("REGISTRY_SERVER_WORKERS") {
if let Ok(workers) = val.parse() {
config.server.workers = workers;
}
}
if let Ok(val) = env::var("REGISTRY_GITEA_URL") {
if let Some(ref mut gitea) = config.gitea {
gitea.url = val;
}
}
if let Ok(val) = env::var("REGISTRY_GITEA_ORG") {
if let Some(ref mut gitea) = config.gitea {
gitea.organization = val;
}
}
if let Ok(val) = env::var("REGISTRY_OCI_REGISTRY") {
if let Some(ref mut oci) = config.oci {
oci.registry = val;
}
}
if let Ok(val) = env::var("REGISTRY_OCI_NAMESPACE") {
if let Some(ref mut oci) = config.oci {
oci.namespace = val;
}
}
if let Ok(val) = env::var("REGISTRY_CACHE_CAPACITY") {
if let Ok(capacity) = val.parse() {
config.cache.capacity = capacity;
}
}
if let Ok(val) = env::var("REGISTRY_CACHE_TTL") {
if let Ok(ttl) = val.parse() {
config.cache.ttl_seconds = ttl;
}
}
}
/// Validate configuration
pub fn validate(&self) -> Result<()> {
// Check if at least one backend is configured (old or new format)
let has_old_backends = self.gitea.is_some() || self.oci.is_some();
let has_new_source_backends = !self.sources.gitea.is_empty()
|| !self.sources.forgejo.is_empty()
|| !self.sources.github.is_empty();
let has_new_dist_backends = !self.distributions.oci.is_empty();
if !has_old_backends && !has_new_source_backends && !has_new_dist_backends {
return Err(RegistryError::Config(
"At least one backend must be configured (gitea, forgejo, github, or oci)"
.to_string(),
));
}
// Validate old single-instance configs if present
if let Some(ref gitea) = self.gitea {
gitea.validate()?;
}
if let Some(ref oci) = self.oci {
oci.validate()?;
}
// Validate new multi-instance configs
for gitea_config in &self.sources.gitea {
gitea_config.validate()?;
}
for forgejo_config in &self.sources.forgejo {
forgejo_config.validate()?;
}
for github_config in &self.sources.github {
github_config.validate()?;
}
for oci_config in &self.distributions.oci {
oci_config.validate()?;
}
self.cache.validate()?;
Ok(())
}
}
impl ConfigLoader for Config {
fn service_name() -> &'static str {
"catalog-registry"
}
fn load_from_hierarchy() -> std::result::Result<Self, Box<dyn std::error::Error + Send + Sync>>
{
let service = Self::service_name();
if let Some(path) = platform_config::resolve_config_path(service) {
return Self::from_path(&path).map_err(|e| {
Box::new(std::io::Error::other(e.to_string()))
as Box<dyn std::error::Error + Send + Sync>
});
}
// Fallback to defaults
Ok(Self::default())
}
fn apply_env_overrides(
&mut self,
) -> std::result::Result<(), Box<dyn std::error::Error + Send + Sync>> {
Self::apply_env_overrides_internal(self);
Ok(())
}
fn from_path<P: AsRef<Path>>(
path: P,
) -> std::result::Result<Self, Box<dyn std::error::Error + Send + Sync>> {
let path = path.as_ref();
let json_value = platform_config::format::load_config(path).map_err(|e| {
let err: Box<dyn std::error::Error + Send + Sync> = Box::new(e);
err
})?;
serde_json::from_value(json_value).map_err(|e| {
let err_msg = format!(
"Failed to deserialize catalog-registry config from {:?}: {}",
path, e
);
Box::new(std::io::Error::new(
std::io::ErrorKind::InvalidData,
err_msg,
)) as Box<dyn std::error::Error + Send + Sync>
})
}
}
/// Server configuration
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ServerConfig {
#[serde(default = "default_host")]
pub host: String,
#[serde(default = "default_port")]
pub port: u16,
#[serde(default = "default_workers")]
pub workers: usize,
#[serde(default)]
pub enable_cors: bool,
#[serde(default)]
pub enable_compression: bool,
}
impl Default for ServerConfig {
fn default() -> Self {
Self {
host: default_host(),
port: default_port(),
workers: default_workers(),
enable_cors: false,
enable_compression: true,
}
}
}
/// Multi-instance source backends configuration
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct SourcesConfig {
#[serde(default)]
pub gitea: Vec<GiteaConfig>,
#[serde(default)]
pub forgejo: Vec<GiteaConfig>,
#[serde(default)]
pub github: Vec<GiteaConfig>,
}
/// Multi-instance distribution backends configuration
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct DistributionsConfig {
#[serde(default)]
pub oci: Vec<OciConfig>,
}
fn default_host() -> String {
"0.0.0.0".to_string()
}
fn default_port() -> u16 {
8082
}
fn default_workers() -> usize {
4
}
/// Gitea configuration
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GiteaConfig {
#[serde(default)]
pub id: Option<String>,
pub url: String,
pub organization: String,
pub token_path: String,
#[serde(default = "default_timeout")]
pub timeout_seconds: u64,
#[serde(default)]
pub verify_ssl: bool,
}
impl GiteaConfig {
fn validate(&self) -> Result<()> {
if self.url.is_empty() {
return Err(RegistryError::Config(
"Gitea URL cannot be empty".to_string(),
));
}
if self.organization.is_empty() {
return Err(RegistryError::Config(
"Gitea organization cannot be empty".to_string(),
));
}
// vault:// references are resolved at runtime; file check only for filesystem
// paths
if !self.token_path.starts_with("vault://") && !Path::new(&self.token_path).exists() {
return Err(RegistryError::Config(format!(
"Gitea token file not found: {}",
self.token_path
)));
}
Ok(())
}
/// Read token from file. Only call for non-vault:// paths.
pub fn read_token(&self) -> Result<String> {
if self.token_path.starts_with("vault://") {
return Err(RegistryError::Config(
"token_path is a vault:// reference — use ClientFactory::create_from_config_async \
to resolve"
.to_string(),
));
}
std::fs::read_to_string(&self.token_path)
.map(|s| s.trim().to_string())
.map_err(|e| RegistryError::Config(format!("Failed to read Gitea token: {}", e)))
}
}
/// OCI registry configuration
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct OciConfig {
#[serde(default)]
pub id: Option<String>,
pub registry: String,
pub namespace: String,
pub auth_token_path: Option<String>,
#[serde(default = "default_timeout")]
pub timeout_seconds: u64,
#[serde(default)]
pub verify_ssl: bool,
}
impl OciConfig {
fn validate(&self) -> Result<()> {
if self.registry.is_empty() {
return Err(RegistryError::Config(
"OCI registry cannot be empty".to_string(),
));
}
if self.namespace.is_empty() {
return Err(RegistryError::Config(
"OCI namespace cannot be empty".to_string(),
));
}
if let Some(ref token_path) = self.auth_token_path {
// vault:// references are resolved at runtime; file check only for filesystem
// paths
if !token_path.starts_with("vault://") && !Path::new(token_path).exists() {
return Err(RegistryError::Config(format!(
"OCI token file not found: {}",
token_path
)));
}
}
Ok(())
}
/// Read auth token from file. Only call for non-vault:// paths.
pub fn read_token(&self) -> Result<Option<String>> {
if let Some(ref path) = self.auth_token_path {
if path.starts_with("vault://") {
return Err(RegistryError::Config(
"auth_token_path is a vault:// reference — use \
ClientFactory::create_from_config_async to resolve"
.to_string(),
));
}
std::fs::read_to_string(path)
.map(|s| Some(s.trim().to_string()))
.map_err(|e| RegistryError::Config(format!("Failed to read OCI token: {}", e)))
} else {
Ok(None)
}
}
}
fn default_timeout() -> u64 {
30
}
/// Cache configuration
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CacheConfig {
#[serde(default = "default_capacity")]
pub capacity: usize,
#[serde(default = "default_ttl")]
pub ttl_seconds: u64,
#[serde(default)]
pub enable_metadata_cache: bool,
#[serde(default)]
pub enable_list_cache: bool,
}
impl Default for CacheConfig {
fn default() -> Self {
Self {
capacity: default_capacity(),
ttl_seconds: default_ttl(),
enable_metadata_cache: true,
enable_list_cache: true,
}
}
}
impl CacheConfig {
fn validate(&self) -> Result<()> {
if self.capacity == 0 {
return Err(RegistryError::Config(
"Cache capacity cannot be zero".to_string(),
));
}
if self.ttl_seconds == 0 {
return Err(RegistryError::Config(
"Cache TTL cannot be zero".to_string(),
));
}
Ok(())
}
}
fn default_capacity() -> usize {
1000
}
fn default_ttl() -> u64 {
300 // 5 minutes
}
/// Load configuration from file
pub fn load_config(path: &str) -> Result<Config> {
let contents = std::fs::read_to_string(path).map_err(RegistryError::Io)?;
let config: Config = toml::from_str(&contents)
.map_err(|e| RegistryError::Config(format!("Failed to parse config: {}", e)))?;
config.validate()?;
Ok(config)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_default_configs() {
let server = ServerConfig::default();
assert_eq!(server.host, "0.0.0.0");
assert_eq!(server.port, 8082);
let cache = CacheConfig::default();
assert_eq!(cache.capacity, 1000);
assert_eq!(cache.ttl_seconds, 300);
}
}

104
crates/catalog-registry/src/error.rs Normal file
View file

@ -0,0 +1,104 @@
use std::fmt;
use axum::http::StatusCode;
use axum::response::{IntoResponse, Response};
/// Registry error types
#[derive(Debug, thiserror::Error)]
pub enum RegistryError {
#[error("Extension not found: {0}")]
NotFound(String),
#[error("Invalid extension type: {0}")]
InvalidType(String),
#[error("Invalid version format: {0}")]
InvalidVersion(String),
#[error("Gitea API error: {0}")]
Gitea(String),
#[error("Forgejo API error: {0}")]
Forgejo(String),
#[error("GitHub API error: {0}")]
Github(String),
#[error("OCI registry error: {0}")]
Oci(String),
#[error("Cache error: {0}")]
Cache(String),
#[error("HTTP request error: {0}")]
Http(#[from] reqwest::Error),
#[error("IO error: {0}")]
Io(#[from] std::io::Error),
#[error("JSON serialization error: {0}")]
Json(#[from] serde_json::Error),
#[error("Configuration error: {0}")]
Config(String),
#[error("Authentication error: {0}")]
Auth(String),
#[error("Rate limit exceeded")]
RateLimit,
#[error("No backends configured")]
NoBackendsConfigured,
#[error("All backends failed: {0}")]
MultiBackendFailure(String),
#[error("Internal server error: {0}")]
Internal(String),
}
/// Result type alias
pub type Result<T> = std::result::Result<T, RegistryError>;
/// API error response
#[derive(Debug, serde::Serialize)]
pub struct ApiErrorResponse {
pub error: String,
pub message: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub details: Option<String>,
}
impl IntoResponse for RegistryError {
fn into_response(self) -> Response {
let (status, error_type) = match &self {
RegistryError::NotFound(_) => (StatusCode::NOT_FOUND, "not_found"),
RegistryError::InvalidType(_) => (StatusCode::BAD_REQUEST, "invalid_type"),
RegistryError::InvalidVersion(_) => (StatusCode::BAD_REQUEST, "invalid_version"),
RegistryError::Auth(_) => (StatusCode::UNAUTHORIZED, "auth_error"),
RegistryError::RateLimit => (StatusCode::TOO_MANY_REQUESTS, "rate_limit"),
RegistryError::Config(_) | RegistryError::NoBackendsConfigured => {
(StatusCode::INTERNAL_SERVER_ERROR, "config_error")
}
RegistryError::MultiBackendFailure(_) => {
(StatusCode::SERVICE_UNAVAILABLE, "backend_failure")
}
_ => (StatusCode::INTERNAL_SERVER_ERROR, "internal_error"),
};
let body = ApiErrorResponse {
error: error_type.to_string(),
message: self.to_string(),
details: None,
};
(status, axum::Json(body)).into_response()
}
}
impl fmt::Display for ApiErrorResponse {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}: {}", self.error, self.message)
}
}

105
crates/catalog-registry/src/events.rs Normal file
View file

@ -0,0 +1,105 @@
use std::sync::Arc;
use futures::StreamExt;
use platform_nats::NatsBridge;
use serde::Serialize;
use tracing::{error, info, warn};
use crate::cache::ExtensionCache;
use crate::models::ExtensionType;
/// Publishes NATS events for extension lifecycle operations.
pub struct EventPublisher {
bridge: Arc<NatsBridge>,
}
#[derive(Debug, Serialize)]
struct ExtensionEvent<'a> {
name: &'a str,
version: &'a str,
#[serde(rename = "type")]
extension_type: ExtensionType,
}
impl EventPublisher {
pub fn new(bridge: Arc<NatsBridge>) -> Self {
Self { bridge }
}
/// Publish `provisioning.extensions.{type}.installed`.
///
/// Fire-and-forget: logs error on failure but never propagates to handler.
pub async fn publish_installed(
&self,
extension_type: ExtensionType,
name: &str,
version: &str,
) {
let subject = format!("extensions.{}.installed", extension_type);
let payload = ExtensionEvent {
name,
version,
extension_type,
};
match self.bridge.publish_json(&subject, &payload).await {
Ok(_) => {
info!(
subject = %subject,
extension = %name,
version = %version,
"Extension installed event published"
);
}
Err(e) => {
error!(
subject = %subject,
extension = %name,
"Failed to publish extension event: {}", e
);
}
}
}
}
/// Spawn a background task that subscribes to workspace deploy-done events
/// and invalidates the extension cache on each notification.
///
/// Subject: `provisioning.workspace.*.deploy.done` (filter on WORKSPACE stream)
pub fn spawn_cache_invalidator(bridge: Arc<NatsBridge>, cache: Arc<ExtensionCache>) {
tokio::spawn(async move {
run_cache_invalidator(bridge, cache).await;
});
}
async fn run_cache_invalidator(bridge: Arc<NatsBridge>, cache: Arc<ExtensionCache>) {
const STREAM: &str = "WORKSPACE";
const CONSUMER: &str = "ext-registry-cache-invalidator";
let mut messages = match bridge.subscribe_pull(STREAM, CONSUMER).await {
Ok(m) => m,
Err(e) => {
warn!("Extension registry cache invalidator: subscribe failed — {e}");
return;
}
};
info!("Extension registry cache invalidator running on stream {STREAM}");
while let Some(msg_result) = messages.next().await {
match msg_result {
Ok(msg) => {
// Subject pattern: provisioning.workspace.{ws_id}.deploy.done
// Filter is applied at JetStream level; any message here triggers invalidation.
let subject = msg.subject.as_str();
info!(subject = %subject, "Workspace deploy detected — invalidating extension cache");
cache.invalidate_all();
if let Err(e) = msg.ack().await {
warn!("Cache invalidator: ack failed — {e}");
}
}
Err(e) => {
error!("Cache invalidator: message error — {e}");
}
}
}
}

510
crates/catalog-registry/src/gitea/client.rs Normal file
View file

@ -0,0 +1,510 @@
use std::time::Duration;
use async_trait::async_trait;
use bytes::Bytes;
use reqwest::Client;
use tracing::debug;
use url::Url;
use crate::client::traits::{
BackendType, ExtensionClient, ReleaseAsset, ReleaseInfo, SourceClient,
};
use crate::config::GiteaConfig;
use crate::error::{RegistryError, Result};
use crate::gitea::models::{GiteaRelease, GiteaRepository};
use crate::models::{Extension, ExtensionSource, ExtensionType, ExtensionVersion};
/// Gitea API client
pub struct GiteaClient {
backend_id: String,
base_url: Url,
organization: String,
token: String,
client: Client,
}
impl GiteaClient {
/// Create new Gitea client with a pre-resolved token.
///
/// Token resolution (file read or vault:// fetch) is the caller's
/// responsibility. Use `ClientFactory::create_from_config_async` for
/// automatic resolution.
pub fn new(config: &GiteaConfig, token: String) -> Result<Self> {
let base_url = Url::parse(&config.url)
.map_err(|e| RegistryError::Config(format!("Invalid Gitea URL: {}", e)))?;
let client = Client::builder()
.timeout(Duration::from_secs(config.timeout_seconds))
.danger_accept_invalid_certs(!config.verify_ssl)
.build()
.map_err(|e| RegistryError::Gitea(format!("Failed to create HTTP client: {}", e)))?;
let backend_id = config
.id
.clone()
.unwrap_or_else(|| format!("gitea-{}", config.organization));
Ok(Self {
backend_id,
base_url,
organization: config.organization.clone(),
token,
client,
})
}
/// List all extensions from organization repositories
pub async fn list_extensions(
&self,
extension_type: Option<ExtensionType>,
) -> Result<Vec<Extension>> {
debug!(
"Fetching repositories for organization: {}",
self.organization
);
let repos = self.list_repositories().await?;
let mut extensions = Vec::new();
for repo in repos {
// Skip archived repositories
if repo.archived {
continue;
}
// Parse extension type from repository name prefix
let Some(ext_type) = self.parse_extension_type(&repo.name) else {
continue;
};
// Filter by type if specified
if let Some(filter_type) = extension_type {
if ext_type != filter_type {
continue;
}
}
// Get latest release for this repository
let Ok(releases) = self.list_releases(&repo.name).await else {
continue;
};
if let Some(latest_release) = releases.first() {
if let Some(extension) = self.release_to_extension(&repo, latest_release, ext_type)
{
extensions.push(extension);
}
}
}
Ok(extensions)
}
/// Get specific extension metadata
pub async fn get_extension(
&self,
extension_type: ExtensionType,
name: &str,
) -> Result<Extension> {
let repo_name = self.format_repo_name(extension_type, name);
debug!("Fetching extension: {}", repo_name);
let repo = self.get_repository(&repo_name).await?;
let releases = self.list_releases(&repo_name).await?;
let latest_release = releases.first().ok_or_else(|| {
RegistryError::NotFound(format!("No releases found for {}", repo_name))
})?;
self.release_to_extension(&repo, latest_release, extension_type)
.ok_or_else(|| {
RegistryError::NotFound(format!("Invalid extension metadata for {}", repo_name))
})
}
/// List all versions for an extension
pub async fn list_versions(
&self,
extension_type: ExtensionType,
name: &str,
) -> Result<Vec<ExtensionVersion>> {
let repo_name = self.format_repo_name(extension_type, name);
debug!("Fetching versions for: {}", repo_name);
let releases = self.list_releases(&repo_name).await?;
Ok(releases
.iter()
.map(|release| ExtensionVersion {
version: release.tag_name.clone(),
published_at: release.published_at.unwrap_or(release.created_at),
download_url: release
.assets
.first()
.map(|a| a.browser_download_url.clone()),
checksum: None,
size: release.assets.first().map(|a| a.size),
})
.collect())
}
/// Download extension asset
pub async fn download_extension(
&self,
extension_type: ExtensionType,
name: &str,
version: &str,
) -> Result<Bytes> {
let repo_name = self.format_repo_name(extension_type, name);
debug!("Downloading extension: {} version {}", repo_name, version);
let release = self.get_release(&repo_name, version).await?;
let asset = release.assets.first().ok_or_else(|| {
RegistryError::NotFound(format!("No assets found for release {}", version))
})?;
self.download_asset(&asset.browser_download_url).await
}
/// List repositories in organization
async fn list_repositories(&self) -> Result<Vec<GiteaRepository>> {
let url = self
.base_url
.join(&format!("api/v1/orgs/{}/repos", self.organization))
.map_err(|e| RegistryError::Gitea(format!("Invalid URL: {}", e)))?;
let response = self
.client
.get(url)
.header("Authorization", format!("token {}", self.token))
.send()
.await
.map_err(|e| RegistryError::Gitea(format!("Request failed: {}", e)))?;
if !response.status().is_success() {
return Err(RegistryError::Gitea(format!(
"Failed to list repositories: {}",
response.status()
)));
}
response
.json()
.await
.map_err(|e| RegistryError::Gitea(format!("Failed to parse response: {}", e)))
}
/// Get specific repository
async fn get_repository(&self, repo_name: &str) -> Result<GiteaRepository> {
let url = self
.base_url
.join(&format!("api/v1/repos/{}/{}", self.organization, repo_name))
.map_err(|e| RegistryError::Gitea(format!("Invalid URL: {}", e)))?;
let response = self
.client
.get(url)
.header("Authorization", format!("token {}", self.token))
.send()
.await
.map_err(|e| RegistryError::Gitea(format!("Request failed: {}", e)))?;
if response.status().as_u16() == 404 {
return Err(RegistryError::NotFound(format!(
"Repository not found: {}",
repo_name
)));
}
if !response.status().is_success() {
return Err(RegistryError::Gitea(format!(
"Failed to get repository: {}",
response.status()
)));
}
response
.json()
.await
.map_err(|e| RegistryError::Gitea(format!("Failed to parse response: {}", e)))
}
/// List releases for repository
async fn list_releases(&self, repo_name: &str) -> Result<Vec<GiteaRelease>> {
let url = self
.base_url
.join(&format!(
"api/v1/repos/{}/{}/releases",
self.organization, repo_name
))
.map_err(|e| RegistryError::Gitea(format!("Invalid URL: {}", e)))?;
let response = self
.client
.get(url)
.header("Authorization", format!("token {}", self.token))
.send()
.await
.map_err(|e| RegistryError::Gitea(format!("Request failed: {}", e)))?;
if !response.status().is_success() {
return Err(RegistryError::Gitea(format!(
"Failed to list releases: {}",
response.status()
)));
}
let mut releases: Vec<GiteaRelease> = response
.json()
.await
.map_err(|e| RegistryError::Gitea(format!("Failed to parse response: {}", e)))?;
// Filter out drafts and prereleases, sort by date
releases.retain(|r| !r.draft && !r.prerelease);
releases.sort_by(|a, b| {
let a_date = a.published_at.unwrap_or(a.created_at);
let b_date = b.published_at.unwrap_or(b.created_at);
b_date.cmp(&a_date)
});
Ok(releases)
}
/// Get specific release
async fn get_release(&self, repo_name: &str, tag: &str) -> Result<GiteaRelease> {
let url = self
.base_url
.join(&format!(
"api/v1/repos/{}/{}/releases/tags/{}",
self.organization, repo_name, tag
))
.map_err(|e| RegistryError::Gitea(format!("Invalid URL: {}", e)))?;
let response = self
.client
.get(url)
.header("Authorization", format!("token {}", self.token))
.send()
.await
.map_err(|e| RegistryError::Gitea(format!("Request failed: {}", e)))?;
if response.status().as_u16() == 404 {
return Err(RegistryError::NotFound(format!(
"Release not found: {}",
tag
)));
}
if !response.status().is_success() {
return Err(RegistryError::Gitea(format!(
"Failed to get release: {}",
response.status()
)));
}
response
.json()
.await
.map_err(|e| RegistryError::Gitea(format!("Failed to parse response: {}", e)))
}
/// Download asset from URL
async fn download_asset(&self, url: &str) -> Result<Bytes> {
let response = self
.client
.get(url)
.header("Authorization", format!("token {}", self.token))
.send()
.await
.map_err(|e| RegistryError::Gitea(format!("Download failed: {}", e)))?;
if !response.status().is_success() {
return Err(RegistryError::Gitea(format!(
"Download failed: {}",
response.status()
)));
}
response
.bytes()
.await
.map_err(|e| RegistryError::Gitea(format!("Failed to read response: {}", e)))
}
/// Parse extension type from repository name
fn parse_extension_type(&self, repo_name: &str) -> Option<ExtensionType> {
if repo_name.ends_with("_prov") {
Some(ExtensionType::Provider)
} else if repo_name.ends_with("_taskserv") {
Some(ExtensionType::Taskserv)
} else if repo_name.ends_with("_cluster") {
Some(ExtensionType::Cluster)
} else {
None
}
}
/// Format repository name from extension type and name
fn format_repo_name(&self, extension_type: ExtensionType, name: &str) -> String {
let suffix = match extension_type {
ExtensionType::Provider => "_prov",
ExtensionType::Taskserv => "_taskserv",
ExtensionType::Cluster => "_cluster",
};
format!("{}{}", name, suffix)
}
/// Convert Gitea release to Extension
fn release_to_extension(
&self,
repo: &GiteaRepository,
release: &GiteaRelease,
extension_type: ExtensionType,
) -> Option<Extension> {
let name = self.extract_extension_name(&repo.name, extension_type)?;
Some(Extension {
name,
extension_type,
version: release.tag_name.clone(),
description: repo.description.clone().unwrap_or_default(),
author: Some(repo.owner.login.clone()),
repository: Some(repo.html_url.clone()),
source: ExtensionSource::Gitea,
published_at: release.published_at.unwrap_or(release.created_at),
download_url: release
.assets
.first()
.map(|a| a.browser_download_url.clone()),
checksum: None,
size: release.assets.first().map(|a| a.size),
tags: None,
})
}
/// Extract extension name from repository name
fn extract_extension_name(
&self,
repo_name: &str,
extension_type: ExtensionType,
) -> Option<String> {
let suffix = match extension_type {
ExtensionType::Provider => "_prov",
ExtensionType::Taskserv => "_taskserv",
ExtensionType::Cluster => "_cluster",
};
repo_name.strip_suffix(suffix).map(|s| s.to_string())
}
/// Check health of Gitea connection
pub async fn health_check(&self) -> Result<()> {
let url = self
.base_url
.join("api/v1/version")
.map_err(|e| RegistryError::Gitea(format!("Invalid URL: {}", e)))?;
let response = self
.client
.get(url)
.timeout(Duration::from_secs(5))
.send()
.await
.map_err(|e| RegistryError::Gitea(format!("Health check failed: {}", e)))?;
if response.status().is_success() {
Ok(())
} else {
Err(RegistryError::Gitea(format!(
"Health check returned: {}",
response.status()
)))
}
}
}
/// ExtensionClient trait implementation for Gitea
#[async_trait]
impl ExtensionClient for GiteaClient {
fn backend_id(&self) -> &str {
&self.backend_id
}
fn backend_type(&self) -> BackendType {
BackendType::Gitea
}
async fn list_extensions(
&self,
extension_type: Option<ExtensionType>,
) -> Result<Vec<Extension>> {
GiteaClient::list_extensions(self, extension_type).await
}
async fn get_extension(&self, extension_type: ExtensionType, name: &str) -> Result<Extension> {
GiteaClient::get_extension(self, extension_type, name).await
}
async fn list_versions(
&self,
extension_type: ExtensionType,
name: &str,
) -> Result<Vec<ExtensionVersion>> {
GiteaClient::list_versions(self, extension_type, name).await
}
async fn download_extension(
&self,
extension_type: ExtensionType,
name: &str,
version: &str,
) -> Result<Bytes> {
GiteaClient::download_extension(self, extension_type, name, version).await
}
async fn health_check(&self) -> Result<()> {
GiteaClient::health_check(self).await
}
}
/// SourceClient trait implementation for Gitea
#[async_trait]
impl SourceClient for GiteaClient {
async fn get_repository_url(
&self,
extension_type: ExtensionType,
name: &str,
) -> Result<String> {
let repo_name = self.format_repo_name(extension_type, name);
let repo = self.get_repository(&repo_name).await?;
Ok(repo.html_url)
}
async fn list_releases(&self, repo_name: &str) -> Result<Vec<ReleaseInfo>> {
let releases = GiteaClient::list_releases(self, repo_name).await?;
Ok(releases
.into_iter()
.map(|r| ReleaseInfo {
tag: r.tag_name,
created_at: r.created_at,
published_at: r.published_at,
assets: r
.assets
.into_iter()
.map(|a| ReleaseAsset {
name: a.name,
download_url: a.browser_download_url,
size: Some(a.size),
})
.collect(),
description: r.body,
})
.collect())
}
async fn get_release_notes(&self, repo_name: &str, version: &str) -> Result<String> {
let release = self.get_release(repo_name, version).await?;
Ok(release.body)
}
}

5
crates/catalog-registry/src/gitea/mod.rs Normal file
View file

@ -0,0 +1,5 @@
pub mod client;
pub mod models;
pub use client::GiteaClient;
pub use models::{GiteaAsset, GiteaRelease, GiteaRepository, GiteaTag, GiteaUser};

66
crates/catalog-registry/src/gitea/models.rs Normal file
View file

@ -0,0 +1,66 @@
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
/// Gitea release information
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GiteaRelease {
pub id: u64,
pub tag_name: String,
pub name: String,
pub body: String,
pub draft: bool,
pub prerelease: bool,
pub created_at: DateTime<Utc>,
pub published_at: Option<DateTime<Utc>>,
pub author: GiteaUser,
pub assets: Vec<GiteaAsset>,
}
/// Gitea user information
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GiteaUser {
pub id: u64,
pub login: String,
pub full_name: Option<String>,
pub email: Option<String>,
}
/// Gitea release asset
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GiteaAsset {
pub id: u64,
pub name: String,
pub size: u64,
pub download_count: u64,
pub created_at: DateTime<Utc>,
pub browser_download_url: String,
}
/// Gitea repository information
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GiteaRepository {
pub id: u64,
pub name: String,
pub full_name: String,
pub description: Option<String>,
pub owner: GiteaUser,
pub html_url: String,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub archived: bool,
}
/// Gitea tag information
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GiteaTag {
pub name: String,
pub message: String,
pub commit: GiteaCommit,
}
/// Gitea commit information
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GiteaCommit {
pub sha: String,
pub url: String,
}

397
crates/catalog-registry/src/handlers.rs Normal file
View file

@ -0,0 +1,397 @@
use std::sync::Arc;
use axum::{
extract::{Path, State},
http::StatusCode,
response::IntoResponse,
routing::{get, head, post, put},
Json, Router,
};
use catalog_registry::service::{
ExtensionMetadata, CatalogRegistry, ImageManifest, PushBlobRequest,
};
use ontoref_derive::onto_api;
use serde_json::json;
/// Application state
#[derive(Clone)]
pub struct AppState {
registry: Arc<CatalogRegistry>,
}
impl AppState {
pub fn new(registry: Arc<CatalogRegistry>) -> Self {
Self { registry }
}
}
/// Error response
#[derive(Debug)]
pub struct RegistryError {
status: StatusCode,
message: String,
}
impl IntoResponse for RegistryError {
fn into_response(self) -> axum::response::Response {
let body = Json(json!({
"errors": [{
"code": "REGISTRY_ERROR",
"message": self.message,
}]
}));
(self.status, body).into_response()
}
}
impl RegistryError {
pub fn not_found(msg: impl Into<String>) -> Self {
Self {
status: StatusCode::NOT_FOUND,
message: msg.into(),
}
}
pub fn internal(msg: impl Into<String>) -> Self {
Self {
status: StatusCode::INTERNAL_SERVER_ERROR,
message: msg.into(),
}
}
#[allow(dead_code)]
pub fn bad_request(msg: impl Into<String>) -> Self {
Self {
status: StatusCode::BAD_REQUEST,
message: msg.into(),
}
}
}
/// Check if blob exists (HEAD /v2/<name>/blobs/<digest>)
#[onto_api(
method = "HEAD",
path = "/v2/{name}/blobs/{digest}",
description = "Check if blob exists (OCI v2)",
auth = "bearer",
actors = "developer, agent",
tags = "oci, blobs",
feature = ""
)]
async fn blob_exists(
Path((_name, digest)): Path<(String, String)>,
State(state): State<AppState>,
) -> Result<StatusCode, RegistryError> {
state
.registry
.blob_exists(&digest)
.await
.map_err(|e: anyhow::Error| RegistryError::internal(e.to_string()))?;
Ok(StatusCode::OK)
}
/// Pull blob (GET /v2/<name>/blobs/<digest>)
#[onto_api(
method = "GET",
path = "/v2/{name}/blobs/{digest}",
description = "Pull blob content (OCI v2)",
auth = "bearer",
actors = "developer, agent",
tags = "oci, blobs",
feature = ""
)]
async fn pull_blob(
Path((_name, digest)): Path<(String, String)>,
State(state): State<AppState>,
) -> Result<Vec<u8>, RegistryError> {
state
.registry
.pull_blob(&digest)
.await
.map_err(|e: anyhow::Error| {
if e.to_string().contains("not found") {
RegistryError::not_found(e.to_string())
} else {
RegistryError::internal(e.to_string())
}
})
}
/// Push blob (POST /v2/<name>/blobs/uploads/)
#[onto_api(
method = "POST",
path = "/v2/{name}/blobs/uploads",
description = "Push blob content (OCI v2)",
auth = "bearer",
actors = "developer",
tags = "oci, blobs",
feature = ""
)]
async fn push_blob(
Path(_name): Path<String>,
State(state): State<AppState>,
Json(req): Json<PushBlobRequest>,
) -> Result<(StatusCode, Json<serde_json::Value>), RegistryError> {
let response = state
.registry
.push_blob(req)
.await
.map_err(|e: anyhow::Error| RegistryError::internal(e.to_string()))?;
Ok((
StatusCode::CREATED,
Json(json!({
"digest": response.digest,
"size": response.size,
})),
))
}
/// Pull manifest (GET /v2/<name>/manifests/<reference>)
#[onto_api(
method = "GET",
path = "/v2/{name}/manifests/{reference}",
description = "Pull extension manifest (OCI v2)",
auth = "bearer",
actors = "developer, agent",
tags = "oci, manifests",
feature = ""
)]
async fn pull_manifest(
Path((name, reference)): Path<(String, String)>,
State(state): State<AppState>,
) -> Result<(StatusCode, Json<serde_json::Value>), RegistryError> {
let response = state
.registry
.get_manifest(&format!("{}:{}", name, reference))
.await
.map_err(|e: anyhow::Error| {
if e.to_string().contains("not found") {
RegistryError::not_found(e.to_string())
} else {
RegistryError::internal(e.to_string())
}
})?;
Ok((
StatusCode::OK,
Json(json!({
"manifest": response.manifest,
"digest": response.digest,
"content_type": response.content_type,
})),
))
}
/// Push manifest (PUT /v2/<name>/manifests/<reference>)
#[onto_api(
method = "PUT",
path = "/v2/{name}/manifests/{reference}",
description = "Push extension manifest (OCI v2)",
auth = "bearer",
actors = "developer",
tags = "oci, manifests",
feature = ""
)]
async fn push_manifest(
Path((name, reference)): Path<(String, String)>,
State(state): State<AppState>,
Json(manifest): Json<ImageManifest>,
) -> Result<(StatusCode, Json<serde_json::Value>), RegistryError> {
let digest = state
.registry
.put_manifest(format!("{}:{}", name, reference), manifest)
.await
.map_err(|e: anyhow::Error| RegistryError::internal(e.to_string()))?;
Ok((
StatusCode::CREATED,
Json(json!({
"digest": digest,
})),
))
}
/// List repositories catalog (GET /v2/_catalog)
#[onto_api(
method = "GET",
path = "/v2/_catalog",
description = "List all extension repositories (OCI v2)",
auth = "bearer",
actors = "developer, agent",
tags = "oci, catalog",
feature = ""
)]
async fn list_catalog(
State(state): State<AppState>,
) -> Result<Json<serde_json::Value>, RegistryError> {
let extensions = state
.registry
.list_extensions()
.await
.map_err(|e| RegistryError::internal(e.to_string()))?;
let repositories: Vec<String> = extensions.iter().map(|e| e.name.clone()).collect();
Ok(Json(json!({
"repositories": repositories,
})))
}
/// List extension tags (GET /v2/<name>/tags/list)
#[onto_api(
method = "GET",
path = "/v2/{name}/tags/list",
description = "List extension tags (OCI v2)",
auth = "bearer",
actors = "developer, agent",
tags = "oci, tags",
feature = ""
)]
async fn list_tags(
Path(name): Path<String>,
State(state): State<AppState>,
) -> Result<Json<serde_json::Value>, RegistryError> {
let extension = state
.registry
.get_extension(&name)
.await
.map_err(|e: anyhow::Error| {
if e.to_string().contains("not found") {
RegistryError::not_found(e.to_string())
} else {
RegistryError::internal(e.to_string())
}
})?;
Ok(Json(json!({
"name": name,
"tags": [extension.version],
})))
}
/// Register extension metadata (POST /extensions)
#[onto_api(
method = "POST",
path = "/extensions",
description = "Register extension metadata",
auth = "bearer",
actors = "developer",
tags = "extensions",
feature = ""
)]
async fn register_extension(
State(state): State<AppState>,
Json(metadata): Json<ExtensionMetadata>,
) -> Result<(StatusCode, Json<serde_json::Value>), RegistryError> {
state
.registry
.register_extension(metadata.clone())
.await
.map_err(|e: anyhow::Error| RegistryError::internal(e.to_string()))?;
Ok((
StatusCode::CREATED,
Json(json!({
"name": metadata.name,
"version": metadata.version,
})),
))
}
/// Get extension metadata (GET /extensions/:name)
#[onto_api(
method = "GET",
path = "/extensions/{name}",
description = "Get extension metadata by name",
auth = "bearer",
actors = "developer, agent",
tags = "extensions",
feature = ""
)]
async fn get_extension(
Path(name): Path<String>,
State(state): State<AppState>,
) -> Result<Json<ExtensionMetadata>, RegistryError> {
state
.registry
.get_extension(&name)
.await
.map_err(|e: anyhow::Error| {
if e.to_string().contains("not found") {
RegistryError::not_found(e.to_string())
} else {
RegistryError::internal(e.to_string())
}
})
.map(Json)
}
/// List all extensions (GET /extensions)
#[onto_api(
method = "GET",
path = "/extensions",
description = "List all registered extensions",
auth = "bearer",
actors = "developer, agent",
tags = "extensions",
feature = ""
)]
async fn list_extensions(
State(state): State<AppState>,
) -> Result<Json<Vec<ExtensionMetadata>>, RegistryError> {
state
.registry
.list_extensions()
.await
.map_err(|e: anyhow::Error| RegistryError::internal(e.to_string()))
.map(Json)
}
/// Health check (GET /health)
#[onto_api(
method = "GET",
path = "/health",
description = "Service health check",
auth = "none",
actors = "developer, agent, ci",
tags = "health",
feature = ""
)]
async fn health(State(state): State<AppState>) -> Result<StatusCode, RegistryError> {
state
.registry
.health_check()
.await
.map_err(|e: anyhow::Error| RegistryError::internal(e.to_string()))?;
Ok(StatusCode::OK)
}
/// Build API router
pub fn routes(state: AppState) -> Router {
Router::new()
// OCI v2 API
.route("/v2/{name}/blobs/{digest}", head(blob_exists))
.route("/v2/{name}/blobs/{digest}", get(pull_blob))
.route("/v2/{name}/blobs/uploads", post(push_blob))
.route("/v2/{name}/manifests/{reference}", get(pull_manifest))
.route("/v2/{name}/manifests/{reference}", put(push_manifest))
.route("/v2/_catalog", get(list_catalog))
.route("/v2/{name}/tags/list", get(list_tags))
// Extensions API
.route("/extensions", post(register_extension))
.route("/extensions", get(list_extensions))
.route("/extensions/{name}", get(get_extension))
// Health
.route("/health", get(health))
.route("/api/v1/health", get(health))
// API catalog
.route(
"/api/catalog",
get(crate::api_catalog::api_catalog),
)
.with_state(state)
}

44
crates/catalog-registry/src/lib.rs Normal file
View file

@ -0,0 +1,44 @@
//! OCI-compliant catalog registry proxy for managing provisioning system
//! extensions
//!
//! Provides HTTP API for pushing, pulling, and managing OCI container images
//! and artifacts representing provisioning extensions, with support for
//! manifest operations, digest validation, and registry federation.
pub mod api;
pub mod cache;
pub mod client;
pub mod config;
pub mod error;
#[cfg(feature = "nats")]
pub mod events;
pub mod gitea;
pub mod models;
pub mod oci;
pub mod registry;
pub mod service;
pub use api::handlers::AppState;
pub use api::routes::build_routes;
pub use config::Config;
pub use service::CatalogRegistry;
/// HTTP API version
pub const API_VERSION: &str = "v2";
/// Default port for catalog registry
pub const DEFAULT_PORT: u16 = 8084;
/// OCI image media types
pub mod media_types {
pub const APPLICATION_VND_DOCKER_DISTRIBUTION_MANIFEST_V2: &str =
"application/vnd.docker.distribution.manifest.v2+json";
pub const APPLICATION_VND_DOCKER_DISTRIBUTION_MANIFEST_LIST_V2: &str =
"application/vnd.docker.distribution.manifest.list.v2+json";
pub const APPLICATION_VND_OCI_IMAGE_MANIFEST_V1: &str =
"application/vnd.oci.image.manifest.v1+json";
pub const APPLICATION_VND_OCI_IMAGE_INDEX_V1: &str = "application/vnd.oci.image.index.v1+json";
pub const APPLICATION_VND_DOCKER_CONTAINER_IMAGE_V1: &str =
"application/vnd.docker.container.image.v1+json";
pub const APPLICATION_OCTET_STREAM: &str = "application/octet-stream";
}

107
crates/catalog-registry/src/main.rs Normal file
View file

@ -0,0 +1,107 @@
use std::net::SocketAddr;
use std::sync::Arc;
use axum::Router;
use clap::Parser;
use catalog_registry::{config::Config, CatalogRegistry, API_VERSION, DEFAULT_PORT};
use tokio::net::TcpListener;
mod api_catalog;
mod handlers;
use handlers::{routes, AppState};
#[derive(Parser)]
#[command(name = "catalog-registry")]
#[command(about = "OCI-compliant catalog registry proxy", long_about = None)]
struct Cli {
/// Configuration file path (highest priority)
#[arg(short = 'c', long, env = "EXTENSION_REGISTRY_CONFIG")]
config: Option<std::path::PathBuf>,
/// Configuration directory (searches for catalog-registry.ncl|toml|json)
#[arg(long, env = "PROVISIONING_CONFIG_DIR")]
config_dir: Option<std::path::PathBuf>,
/// Deployment mode (solo, multiuser, cicd, enterprise)
#[arg(short = 'm', long, env = "EXTENSION_REGISTRY_MODE")]
mode: Option<String>,
/// Host to bind to
#[arg(long, default_value = "127.0.0.1")]
host: String,
/// Port to bind to
#[arg(long, default_value_t = DEFAULT_PORT)]
port: u16,
/// Print all #[onto_api] registered routes as JSON and exit.
/// Pipe to api-catalog-catalog-registry.json: `just export-api-catalog`
#[arg(long)]
dump_api_catalog: bool,
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Parse CLI arguments FIRST (so --help works before any other processing)
let cli = Cli::parse();
if cli.dump_api_catalog {
println!("{}", ontoref_ontology::api::dump_catalog_json());
return Ok(());
}
// Initialize centralized observability (logging, metrics, health checks)
let _guard = observability::init_from_env("catalog-registry", env!("CARGO_PKG_VERSION"))?;
// Check if catalog-registry is enabled in deployment-mode.ncl
if let Ok(deployment) = platform_config::load_deployment_mode() {
if let Ok(enabled) = deployment.is_service_enabled("catalog_registry") {
if !enabled {
tracing::warn!("⚠ Catalog Registry is DISABLED in deployment-mode.ncl");
std::process::exit(1);
}
tracing::info!("✓ Catalog Registry is ENABLED in deployment-mode.ncl");
}
}
// Load configuration from Nickel or hierarchy
let mut config = Config::load()?;
// Apply CLI overrides if provided
if !cli.host.is_empty() && cli.host != "0.0.0.0" {
config.server.host = cli.host.clone();
}
if cli.port != 0 {
config.server.port = cli.port;
}
tracing::info!(
"🔧 Loaded catalog-registry configuration (host: {}, port: {})",
config.server.host,
config.server.port
);
// Create registry service
let addr: SocketAddr = format!("{}:{}", config.server.host, config.server.port).parse()?;
let registry = Arc::new(CatalogRegistry::new(addr));
// Create application state
let state = AppState::new(registry);
// Build router
let app: Router = routes(state);
// Bind to address
let listener = TcpListener::bind(addr).await?;
tracing::info!(
"catalog-registry v{} starting on {} (OCI v2 API)",
API_VERSION,
addr
);
// Run server
axum::serve(listener, app).await?;
Ok(())
}

171
crates/catalog-registry/src/models/extension.rs Normal file
View file

@ -0,0 +1,171 @@
use std::fmt;
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
/// Extension type enumeration
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum ExtensionType {
Provider,
Taskserv,
Cluster,
}
impl fmt::Display for ExtensionType {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
ExtensionType::Provider => write!(f, "provider"),
ExtensionType::Taskserv => write!(f, "taskserv"),
ExtensionType::Cluster => write!(f, "cluster"),
}
}
}
impl std::str::FromStr for ExtensionType {
type Err = crate::error::RegistryError;
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s.to_lowercase().as_str() {
"provider" | "providers" => Ok(ExtensionType::Provider),
"taskserv" | "taskservs" => Ok(ExtensionType::Taskserv),
"cluster" | "clusters" => Ok(ExtensionType::Cluster),
_ => Err(crate::error::RegistryError::InvalidType(s.to_string())),
}
}
}
/// Extension source backend
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum ExtensionSource {
Gitea,
Forgejo,
Github,
Oci,
}
impl fmt::Display for ExtensionSource {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
ExtensionSource::Gitea => write!(f, "gitea"),
ExtensionSource::Forgejo => write!(f, "forgejo"),
ExtensionSource::Github => write!(f, "github"),
ExtensionSource::Oci => write!(f, "oci"),
}
}
}
/// Extension metadata
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Extension {
pub name: String,
#[serde(rename = "type")]
pub extension_type: ExtensionType,
pub version: String,
pub description: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub author: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub repository: Option<String>,
pub source: ExtensionSource,
pub published_at: DateTime<Utc>,
#[serde(skip_serializing_if = "Option::is_none")]
pub download_url: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub checksum: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub size: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tags: Option<Vec<String>>,
}
impl Extension {
/// Create unique cache key for extension
pub fn cache_key(&self) -> String {
format!("{}/{}/{}", self.extension_type, self.name, self.version)
}
/// Get extension identifier (type/name)
pub fn identifier(&self) -> String {
format!("{}/{}", self.extension_type, self.name)
}
}
/// Extension version information
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ExtensionVersion {
pub version: String,
pub published_at: DateTime<Utc>,
#[serde(skip_serializing_if = "Option::is_none")]
pub download_url: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub checksum: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub size: Option<u64>,
}
/// Extension list parameters
#[derive(Debug, Clone, Deserialize)]
pub struct ListParams {
#[serde(rename = "type")]
pub extension_type: Option<ExtensionType>,
pub source: Option<ExtensionSource>,
pub limit: Option<usize>,
pub offset: Option<usize>,
}
impl Default for ListParams {
fn default() -> Self {
Self {
extension_type: None,
source: None,
limit: Some(100),
offset: Some(0),
}
}
}
/// Search parameters
#[derive(Debug, Clone, Deserialize)]
pub struct SearchParams {
pub q: String,
#[serde(rename = "type")]
pub extension_type: Option<ExtensionType>,
pub limit: Option<usize>,
}
impl Default for SearchParams {
fn default() -> Self {
Self {
q: String::new(),
extension_type: None,
limit: Some(50),
}
}
}
/// Health check response
#[derive(Debug, Serialize)]
pub struct HealthResponse {
pub status: String,
pub version: String,
pub uptime: u64,
pub backends: BackendHealth,
}
/// Backend health status
#[derive(Debug, Serialize)]
pub struct BackendHealth {
pub gitea: BackendStatus,
pub oci: BackendStatus,
}
/// Individual backend status
#[derive(Debug, Serialize)]
pub struct BackendStatus {
pub enabled: bool,
pub healthy: bool,
#[serde(skip_serializing_if = "Option::is_none")]
pub error: Option<String>,
}

6
crates/catalog-registry/src/models/mod.rs Normal file
View file

@ -0,0 +1,6 @@
pub mod extension;
pub use extension::{
BackendHealth, BackendStatus, Extension, ExtensionSource, ExtensionType, ExtensionVersion,
HealthResponse, ListParams, SearchParams,
};

491
crates/catalog-registry/src/oci/client.rs Normal file
View file

@ -0,0 +1,491 @@
use std::time::Duration;
use async_trait::async_trait;
use bytes::Bytes;
use chrono::Utc;
use reqwest::Client;
use tracing::debug;
use crate::client::traits::{
BackendType, DistributionClient, ExtensionClient, LayerInfo, ManifestInfo,
};
use crate::config::OciConfig;
use crate::error::{RegistryError, Result};
use crate::models::{Extension, ExtensionSource, ExtensionType, ExtensionVersion};
use crate::oci::models::{OciCatalog, OciManifest, OciTagsList};
/// OCI registry client
pub struct OciClient {
backend_id: String,
registry: String,
namespace: String,
auth_token: Option<String>,
client: Client,
}
impl OciClient {
/// Create new OCI client with a pre-resolved auth token.
///
/// Token resolution (file read or vault:// fetch) is the caller's
/// responsibility. Pass `None` for unauthenticated registry access.
pub fn new(config: &OciConfig, auth_token: Option<String>) -> Result<Self> {
let client = Client::builder()
.timeout(Duration::from_secs(config.timeout_seconds))
.danger_accept_invalid_certs(!config.verify_ssl)
.build()
.map_err(|e| RegistryError::Oci(format!("Failed to create HTTP client: {}", e)))?;
let backend_id = config
.id
.clone()
.unwrap_or_else(|| format!("oci-{}", config.registry));
Ok(Self {
backend_id,
registry: config.registry.clone(),
namespace: config.namespace.clone(),
auth_token,
client,
})
}
/// List all extensions from OCI registry
pub async fn list_extensions(
&self,
extension_type: Option<ExtensionType>,
) -> Result<Vec<Extension>> {
debug!("Fetching artifacts from OCI registry: {}", self.registry);
let catalog = self.list_catalog().await?;
let mut extensions = Vec::new();
for repo_name in catalog.repositories {
// Skip if not in our namespace
if !repo_name.starts_with(&format!("{}/", self.namespace)) {
continue;
}
// Parse extension type and name from repository
let Some((ext_type, _name)) = self.parse_repository_name(&repo_name) else {
continue;
};
// Filter by type if specified
if let Some(filter_type) = extension_type {
if ext_type != filter_type {
continue;
}
}
// Get tags for this repository
let Ok(tags) = self.list_tags(&repo_name).await else {
continue;
};
let Some(latest_tag) = tags.tags.first() else {
continue;
};
if let Ok(manifest) = self.get_manifest(&repo_name, latest_tag).await {
if let Some(extension) =
self.manifest_to_extension(&repo_name, latest_tag, &manifest, ext_type)
{
extensions.push(extension);
}
}
}
Ok(extensions)
}
/// Get specific extension metadata
pub async fn get_extension(
&self,
extension_type: ExtensionType,
name: &str,
) -> Result<Extension> {
let repo_name = self.format_repository_name(extension_type, name);
debug!("Fetching extension: {}", repo_name);
let tags = self.list_tags(&repo_name).await?;
let latest_tag = tags
.tags
.first()
.ok_or_else(|| RegistryError::NotFound(format!("No tags found for {}", repo_name)))?;
let manifest = self.get_manifest(&repo_name, latest_tag).await?;
self.manifest_to_extension(&repo_name, latest_tag, &manifest, extension_type)
.ok_or_else(|| {
RegistryError::NotFound(format!("Invalid extension metadata for {}", repo_name))
})
}
/// List all versions for an extension
pub async fn list_versions(
&self,
extension_type: ExtensionType,
name: &str,
) -> Result<Vec<ExtensionVersion>> {
let repo_name = self.format_repository_name(extension_type, name);
debug!("Fetching versions for: {}", repo_name);
let tags = self.list_tags(&repo_name).await?;
let mut versions = Vec::new();
for tag in tags.tags {
if let Ok(manifest) = self.get_manifest(&repo_name, &tag).await {
let size = manifest.layers.iter().map(|l| l.size).sum();
versions.push(ExtensionVersion {
version: tag.clone(),
published_at: Utc::now(), /* OCI doesn't provide creation time without
* additional metadata */
download_url: None,
checksum: Some(manifest.config.digest.clone()),
size: Some(size),
});
}
}
Ok(versions)
}
/// Pull extension artifact
pub async fn download_extension(
&self,
extension_type: ExtensionType,
name: &str,
tag: &str,
) -> Result<Bytes> {
let repo_name = self.format_repository_name(extension_type, name);
debug!("Pulling extension: {} tag {}", repo_name, tag);
let manifest = self.get_manifest(&repo_name, tag).await?;
// Download first layer (assuming single-layer artifacts)
let layer = manifest
.layers
.first()
.ok_or_else(|| RegistryError::Oci("No layers found in manifest".to_string()))?;
self.download_blob(&repo_name, &layer.digest).await
}
/// List catalog (all repositories)
async fn list_catalog(&self) -> Result<OciCatalog> {
let url = format!("https://{}/v2/_catalog", self.registry);
let mut request = self.client.get(&url);
if let Some(ref token) = self.auth_token {
request = request.header("Authorization", format!("Bearer {}", token));
}
let response = request
.send()
.await
.map_err(|e| RegistryError::Oci(format!("Request failed: {}", e)))?;
if !response.status().is_success() {
return Err(RegistryError::Oci(format!(
"Failed to list catalog: {}",
response.status()
)));
}
response
.json()
.await
.map_err(|e| RegistryError::Oci(format!("Failed to parse response: {}", e)))
}
/// List tags for repository
async fn list_tags(&self, repository: &str) -> Result<OciTagsList> {
let url = format!("https://{}/v2/{}/tags/list", self.registry, repository);
let mut request = self.client.get(&url);
if let Some(ref token) = self.auth_token {
request = request.header("Authorization", format!("Bearer {}", token));
}
let response = request
.send()
.await
.map_err(|e| RegistryError::Oci(format!("Request failed: {}", e)))?;
if response.status().as_u16() == 404 {
return Err(RegistryError::NotFound(format!(
"Repository not found: {}",
repository
)));
}
if !response.status().is_success() {
return Err(RegistryError::Oci(format!(
"Failed to list tags: {}",
response.status()
)));
}
response
.json()
.await
.map_err(|e| RegistryError::Oci(format!("Failed to parse response: {}", e)))
}
/// Get manifest for tag
async fn get_manifest(&self, repository: &str, tag: &str) -> Result<OciManifest> {
let url = format!(
"https://{}/v2/{}/manifests/{}",
self.registry, repository, tag
);
let mut request = self
.client
.get(&url)
.header("Accept", "application/vnd.oci.image.manifest.v1+json");
if let Some(ref token) = self.auth_token {
request = request.header("Authorization", format!("Bearer {}", token));
}
let response = request
.send()
.await
.map_err(|e| RegistryError::Oci(format!("Request failed: {}", e)))?;
if response.status().as_u16() == 404 {
return Err(RegistryError::NotFound(format!(
"Manifest not found: {}:{}",
repository, tag
)));
}
if !response.status().is_success() {
return Err(RegistryError::Oci(format!(
"Failed to get manifest: {}",
response.status()
)));
}
response
.json()
.await
.map_err(|e| RegistryError::Oci(format!("Failed to parse manifest: {}", e)))
}
/// Download blob by digest
async fn download_blob(&self, repository: &str, digest: &str) -> Result<Bytes> {
let url = format!(
"https://{}/v2/{}/blobs/{}",
self.registry, repository, digest
);
let mut request = self.client.get(&url);
if let Some(ref token) = self.auth_token {
request = request.header("Authorization", format!("Bearer {}", token));
}
let response = request
.send()
.await
.map_err(|e| RegistryError::Oci(format!("Download failed: {}", e)))?;
if !response.status().is_success() {
return Err(RegistryError::Oci(format!(
"Download failed: {}",
response.status()
)));
}
response
.bytes()
.await
.map_err(|e| RegistryError::Oci(format!("Failed to read response: {}", e)))
}
/// Parse repository name into extension type and name
fn parse_repository_name(&self, repo_name: &str) -> Option<(ExtensionType, String)> {
let name = repo_name.strip_prefix(&format!("{}/", self.namespace))?;
name.strip_suffix("-provider")
.map(|base_name| (ExtensionType::Provider, base_name.to_string()))
.or_else(|| {
name.strip_suffix("-taskserv")
.map(|base_name| (ExtensionType::Taskserv, base_name.to_string()))
})
.or_else(|| {
name.strip_suffix("-cluster")
.map(|base_name| (ExtensionType::Cluster, base_name.to_string()))
})
}
/// Format repository name from extension type and name
fn format_repository_name(&self, extension_type: ExtensionType, name: &str) -> String {
let suffix = match extension_type {
ExtensionType::Provider => "-provider",
ExtensionType::Taskserv => "-taskserv",
ExtensionType::Cluster => "-cluster",
};
format!("{}/{}{}", self.namespace, name, suffix)
}
/// Convert OCI manifest to Extension
fn manifest_to_extension(
&self,
repo_name: &str,
tag: &str,
manifest: &OciManifest,
extension_type: ExtensionType,
) -> Option<Extension> {
let (_, name) = self.parse_repository_name(repo_name)?;
let description = manifest
.annotations
.as_ref()
.and_then(|a| a.get("org.opencontainers.image.description"))
.cloned()
.unwrap_or_default();
let author = manifest
.annotations
.as_ref()
.and_then(|a| a.get("org.opencontainers.image.authors"))
.cloned();
let repository = manifest
.annotations
.as_ref()
.and_then(|a| a.get("org.opencontainers.image.url"))
.cloned();
let size = manifest.layers.iter().map(|l| l.size).sum();
Some(Extension {
name,
extension_type,
version: tag.to_string(),
description,
author,
repository,
source: ExtensionSource::Oci,
published_at: Utc::now(),
download_url: None,
checksum: Some(manifest.config.digest.clone()),
size: Some(size),
tags: None,
})
}
/// Check health of OCI connection
pub async fn health_check(&self) -> Result<()> {
let url = format!("https://{}/v2/", self.registry);
let mut request = self.client.get(&url).timeout(Duration::from_secs(5));
if let Some(ref token) = self.auth_token {
request = request.header("Authorization", format!("Bearer {}", token));
}
let response = request
.send()
.await
.map_err(|e| RegistryError::Oci(format!("Health check failed: {}", e)))?;
if response.status().is_success() || response.status().as_u16() == 401 {
// 401 means registry is up but auth is required
Ok(())
} else {
Err(RegistryError::Oci(format!(
"Health check returned: {}",
response.status()
)))
}
}
}
/// ExtensionClient trait implementation for OCI
#[async_trait]
impl ExtensionClient for OciClient {
fn backend_id(&self) -> &str {
&self.backend_id
}
fn backend_type(&self) -> BackendType {
BackendType::Oci
}
async fn list_extensions(
&self,
extension_type: Option<ExtensionType>,
) -> Result<Vec<Extension>> {
OciClient::list_extensions(self, extension_type).await
}
async fn get_extension(&self, extension_type: ExtensionType, name: &str) -> Result<Extension> {
OciClient::get_extension(self, extension_type, name).await
}
async fn list_versions(
&self,
extension_type: ExtensionType,
name: &str,
) -> Result<Vec<ExtensionVersion>> {
OciClient::list_versions(self, extension_type, name).await
}
async fn download_extension(
&self,
extension_type: ExtensionType,
name: &str,
version: &str,
) -> Result<Bytes> {
OciClient::download_extension(self, extension_type, name, version).await
}
async fn health_check(&self) -> Result<()> {
OciClient::health_check(self).await
}
}
/// DistributionClient trait implementation for OCI
#[async_trait]
impl DistributionClient for OciClient {
async fn get_manifest(&self, repo_name: &str, tag: &str) -> Result<ManifestInfo> {
let manifest = self.get_manifest(repo_name, tag).await?;
let total_size: u64 =
manifest.layers.iter().map(|l| l.size).sum::<u64>() + manifest.config.size;
Ok(ManifestInfo {
config_digest: manifest.config.digest,
layers: manifest
.layers
.into_iter()
.map(|l| LayerInfo {
digest: l.digest,
size: l.size,
media_type: l.media_type,
})
.collect(),
total_size,
})
}
async fn list_catalog(&self) -> Result<Vec<String>> {
let catalog = self.list_catalog().await?;
Ok(catalog.repositories)
}
async fn get_digest(&self, repo_name: &str, tag: &str) -> Result<String> {
let manifest = self.get_manifest(repo_name, tag).await?;
Ok(manifest.config.digest)
}
async fn verify_artifact(&self, repo_name: &str, tag: &str, digest: &str) -> Result<bool> {
let manifest = self.get_manifest(repo_name, tag).await?;
Ok(manifest.config.digest == digest)
}
}

5
crates/catalog-registry/src/oci/mod.rs Normal file
View file

@ -0,0 +1,5 @@
pub mod client;
pub mod models;
pub use client::OciClient;
pub use models::{OciAuthToken, OciCatalog, OciDescriptor, OciManifest, OciTagsList};

48
crates/catalog-registry/src/oci/models.rs Normal file
View file

@ -0,0 +1,48 @@
use serde::{Deserialize, Serialize};
/// OCI manifest
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct OciManifest {
#[serde(rename = "schemaVersion")]
pub schema_version: u32,
#[serde(rename = "mediaType")]
pub media_type: String,
pub config: OciDescriptor,
pub layers: Vec<OciDescriptor>,
#[serde(skip_serializing_if = "Option::is_none")]
pub annotations: Option<std::collections::HashMap<String, String>>,
}
/// OCI descriptor
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct OciDescriptor {
#[serde(rename = "mediaType")]
pub media_type: String,
pub digest: String,
pub size: u64,
#[serde(skip_serializing_if = "Option::is_none")]
pub annotations: Option<std::collections::HashMap<String, String>>,
}
/// OCI catalog response
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct OciCatalog {
pub repositories: Vec<String>,
}
/// OCI tags list
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct OciTagsList {
pub name: String,
pub tags: Vec<String>,
}
/// OCI authentication token
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct OciAuthToken {
pub token: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub access_token: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub expires_in: Option<u64>,
}

91
crates/catalog-registry/src/registry.rs Normal file
View file

@ -0,0 +1,91 @@
//! Registry management and federation support
use std::collections::HashMap;
use serde::{Deserialize, Serialize};
/// Registry catalog response
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Catalog {
pub repositories: Vec<String>,
}
/// Image tags list
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TagsList {
pub name: String,
pub tags: Vec<String>,
}
/// Registry configuration
#[derive(Debug, Clone)]
pub struct RegistryConfig {
/// Primary registry URL
pub primary_url: String,
/// Upstream registries for federation (fallback)
pub upstream: Vec<String>,
/// Storage backend (memory, filesystem, s3)
pub storage: StorageBackend,
/// Enable authentication
pub auth_enabled: bool,
}
/// Storage backend type
#[derive(Debug, Clone)]
pub enum StorageBackend {
Memory,
Filesystem(String),
S3 { bucket: String, region: String },
}
impl RegistryConfig {
/// Create default in-memory configuration
pub fn default_memory() -> Self {
Self {
primary_url: "http://localhost:8084".to_string(),
upstream: vec![],
storage: StorageBackend::Memory,
auth_enabled: false,
}
}
}
/// Registry federation for distributed extension management
pub struct RegistryFederation {
config: RegistryConfig,
#[allow(dead_code)]
cache: HashMap<String, Vec<u8>>,
}
impl RegistryFederation {
/// Create new federation
pub fn new(config: RegistryConfig) -> Self {
Self {
config,
cache: HashMap::new(),
}
}
/// Get registry configuration
pub fn config(&self) -> &RegistryConfig {
&self.config
}
/// Check upstream registries for blob
pub async fn pull_from_upstream(&mut self, digest: &str) -> Result<Vec<u8>, String> {
// In Phase 4D, upstream federation delegates to actual registry proxying
// Full implementation with HTTP client to upstream registries comes in Phase 4+
Err(format!("Blob not found in upstream: {}", digest))
}
}
/// Repository metadata
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RepositoryMetadata {
pub name: String,
pub description: String,
pub owner: String,
pub created_at: String,
pub updated_at: String,
pub tags_count: u32,
}

215
crates/catalog-registry/src/service.rs Normal file
View file

@ -0,0 +1,215 @@
//! Core catalog registry service implementation
use std::collections::HashMap;
use std::net::SocketAddr;
use std::sync::Arc;
use anyhow::Result;
use serde::{Deserialize, Serialize};
use sha2::Digest;
use tokio::sync::RwLock;
/// OCI image manifest
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ImageManifest {
/// Schema version
pub schema_version: u32,
/// Media type
pub media_type: String,
/// Image config digest (sha256:...)
pub config: ManifestConfig,
/// Image layers
pub layers: Vec<Layer>,
}
/// Image configuration reference
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ManifestConfig {
/// Digest (sha256:...)
pub digest: String,
/// Size in bytes
pub size: u64,
/// Media type
pub media_type: String,
}
/// Image layer
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Layer {
/// Layer digest (sha256:...)
pub digest: String,
/// Size in bytes
pub size: u64,
/// Media type
pub media_type: String,
}
/// Push blob request
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PushBlobRequest {
/// Blob content
pub content: Vec<u8>,
/// Content digest for validation
pub digest: String,
}
/// Push blob response
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PushBlobResponse {
/// Blob digest
pub digest: String,
/// Blob size in bytes
pub size: u64,
}
/// Pull manifest response
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PullManifestResponse {
/// Image manifest
pub manifest: ImageManifest,
/// Manifest digest
pub digest: String,
/// Content type
pub content_type: String,
}
/// Extension metadata
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ExtensionMetadata {
/// Extension name
pub name: String,
/// Extension version
pub version: String,
/// Extension description
pub description: String,
/// Supported platforms (e.g., ["linux/amd64", "linux/arm64"])
pub platforms: Vec<String>,
}
/// Core catalog registry service
pub struct CatalogRegistry {
/// Server address
addr: SocketAddr,
/// Stored blobs (digest -> content)
blobs: Arc<RwLock<HashMap<String, Vec<u8>>>>,
/// Stored manifests (repo:tag -> manifest)
manifests: Arc<RwLock<HashMap<String, ImageManifest>>>,
/// Extension metadata registry
extensions: Arc<RwLock<HashMap<String, ExtensionMetadata>>>,
}
impl CatalogRegistry {
/// Create new catalog registry
pub fn new(addr: SocketAddr) -> Self {
Self {
addr,
blobs: Arc::new(RwLock::new(HashMap::new())),
manifests: Arc::new(RwLock::new(HashMap::new())),
extensions: Arc::new(RwLock::new(HashMap::new())),
}
}
/// Get service address
pub fn addr(&self) -> SocketAddr {
self.addr
}
/// Check if blob exists
pub async fn blob_exists(&self, digest: &str) -> Result<bool> {
let blobs = self.blobs.read().await;
Ok(blobs.contains_key(digest))
}
/// Push blob to registry
pub async fn push_blob(&self, req: PushBlobRequest) -> Result<PushBlobResponse> {
let size = req.content.len() as u64;
let mut blobs = self.blobs.write().await;
blobs.insert(req.digest.clone(), req.content);
Ok(PushBlobResponse {
digest: req.digest,
size,
})
}
/// Pull blob from registry
pub async fn pull_blob(&self, digest: &str) -> Result<Vec<u8>> {
let blobs = self.blobs.read().await;
blobs
.get(digest)
.cloned()
.ok_or_else(|| anyhow::anyhow!("Blob not found: {}", digest))
}
/// Check if manifest exists
pub async fn manifest_exists(&self, reference: &str) -> Result<bool> {
let manifests = self.manifests.read().await;
Ok(manifests.contains_key(reference))
}
/// Put image manifest
pub async fn put_manifest(&self, reference: String, manifest: ImageManifest) -> Result<String> {
let digest = format!(
"sha256:{}",
hex::encode(sha2::Sha256::digest(&serde_json::to_vec(&manifest)?))
);
let mut manifests = self.manifests.write().await;
manifests.insert(reference, manifest);
Ok(digest)
}
/// Get image manifest
pub async fn get_manifest(&self, reference: &str) -> Result<PullManifestResponse> {
let manifests = self.manifests.read().await;
let manifest = manifests
.get(reference)
.cloned()
.ok_or_else(|| anyhow::anyhow!("Manifest not found: {}", reference))?;
let digest = format!(
"sha256:{}",
hex::encode(sha2::Sha256::digest(&serde_json::to_vec(&manifest)?))
);
Ok(PullManifestResponse {
manifest,
digest,
content_type: "application/vnd.docker.distribution.manifest.v2+json".to_string(),
})
}
/// Register extension
pub async fn register_extension(&self, metadata: ExtensionMetadata) -> Result<()> {
let mut extensions = self.extensions.write().await;
extensions.insert(metadata.name.clone(), metadata);
Ok(())
}
/// Get extension metadata
pub async fn get_extension(&self, name: &str) -> Result<ExtensionMetadata> {
let extensions = self.extensions.read().await;
extensions
.get(name)
.cloned()
.ok_or_else(|| anyhow::anyhow!("Extension not found: {}", name))
}
/// List all extensions
pub async fn list_extensions(&self) -> Result<Vec<ExtensionMetadata>> {
let extensions = self.extensions.read().await;
Ok(extensions.values().cloned().collect())
}
/// Health check endpoint
pub async fn health_check(&self) -> Result<()> {
Ok(())
}
}
impl Default for CatalogRegistry {
fn default() -> Self {
Self::new("127.0.0.1:8084".parse().unwrap())
}
}

188
crates/catalog-registry/tests/integration_test.rs Normal file
View file

@ -0,0 +1,188 @@
use axum::body::Body;
use axum::http::{Request, StatusCode};
use catalog_registry::config::OciConfig;
use catalog_registry::{build_routes, AppState, Config};
use http_body_util::BodyExt;
use tower::ServiceExt;
/// Create a minimal test config with a mock OCI backend
fn create_test_config() -> Config {
Config {
server: catalog_registry::config::ServerConfig::default(),
gitea: None,
// Use OCI as test backend (doesn't require file validation for auth_token_path)
oci: Some(OciConfig {
id: Some("test-oci".to_string()),
registry: "localhost:5000".to_string(),
namespace: "test".to_string(),
auth_token_path: None,
timeout_seconds: 30,
verify_ssl: false,
}),
sources: catalog_registry::config::SourcesConfig::default(),
distributions: catalog_registry::config::DistributionsConfig::default(),
cache: catalog_registry::config::CacheConfig::default(),
}
}
#[tokio::test]
#[ignore] // Requires OCI registry or Gitea service to be running
#[ignore] // Requires OCI registry service to be running
async fn test_health_check() {
let config = create_test_config();
let state = AppState::new(config, None)
.await
.expect("Failed to create app state");
let app = build_routes(state);
let response = app
.oneshot(
Request::builder()
.uri("/api/v1/health")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let body = response.into_body().collect().await.unwrap().to_bytes();
let health: serde_json::Value = serde_json::from_slice(&body).unwrap();
assert!(health.get("status").is_some());
assert!(health.get("version").is_some());
assert!(health.get("uptime").is_some());
}
#[tokio::test]
#[ignore] // Requires OCI registry or Gitea service to be running
async fn test_list_extensions_empty() {
let config = create_test_config();
let state = AppState::new(config, None)
.await
.expect("Failed to create app state");
let app = build_routes(state);
let response = app
.oneshot(
Request::builder()
.uri("/api/v1/extensions")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let body = response.into_body().collect().await.unwrap().to_bytes();
let extensions: Vec<serde_json::Value> = serde_json::from_slice(&body).unwrap();
// Should be empty when no backends configured
assert_eq!(extensions.len(), 0);
}
#[tokio::test]
#[ignore] // Requires OCI registry or Gitea service to be running
async fn test_get_nonexistent_extension() {
let config = create_test_config();
let state = AppState::new(config, None)
.await
.expect("Failed to create app state");
let app = build_routes(state);
let response = app
.oneshot(
Request::builder()
.uri("/api/v1/extensions/provider/nonexistent")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::NOT_FOUND);
}
#[tokio::test]
#[ignore] // Requires OCI registry or Gitea service to be running
async fn test_metrics_endpoint() {
let config = create_test_config();
let state = AppState::new(config, None)
.await
.expect("Failed to create app state");
let app = build_routes(state);
let response = app
.oneshot(
Request::builder()
.uri("/api/v1/metrics")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let body = response.into_body().collect().await.unwrap().to_bytes();
let metrics = String::from_utf8(body.to_vec()).unwrap();
assert!(metrics.contains("http_requests_total"));
}
#[tokio::test]
#[ignore] // Requires OCI registry or Gitea service to be running
async fn test_cache_stats_endpoint() {
let config = create_test_config();
let state = AppState::new(config, None)
.await
.expect("Failed to create app state");
let app = build_routes(state);
let response = app
.oneshot(
Request::builder()
.uri("/api/v1/cache/stats")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let body = response.into_body().collect().await.unwrap().to_bytes();
let stats: serde_json::Value = serde_json::from_slice(&body).unwrap();
assert!(stats.get("list_entries").is_some());
assert!(stats.get("metadata_entries").is_some());
assert!(stats.get("total_entries").is_some());
}
#[tokio::test]
#[ignore] // Requires OCI registry or Gitea service to be running
async fn test_invalid_extension_type() {
let config = create_test_config();
let state = AppState::new(config, None)
.await
.expect("Failed to create app state");
let app = build_routes(state);
let response = app
.oneshot(
Request::builder()
.uri("/api/v1/extensions/invalid_type/test")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
}

29
crates/contract-tests/Cargo.toml Normal file
View file

@ -0,0 +1,29 @@
[package]
authors.workspace = true
edition.workspace = true
license.workspace = true
name = "contract-tests"
repository.workspace = true
version.workspace = true
description = "G3 — CLI↔HTTP↔MCP contract tests for the provisioning-core tool Registry"
publish = false
[lib]
path = "src/lib.rs"
[dependencies]
provisioning-core = { path = "../provisioning-core" }
provisioning-daemon = { path = "../provisioning-daemon" }
provisioning-mcp = { path = "../mcp-server" }
tokio = { workspace = true, features = ["full"] }
axum = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
anyhow = { workspace = true }
async-trait = { workspace = true }
reqwest = { workspace = true }
jsonschema = "0.28"
[dev-dependencies]
rustls = { version = "0.23", features = ["aws_lc_rs"] }

287
crates/contract-tests/src/lib.rs Normal file
View file

@ -0,0 +1,287 @@
//! G3 — CLI↔HTTP↔MCP contract-test harness.
//!
//! Provides fixture tools, a three-tier dispatcher, and envelope-normalisation
//! helpers so the invariant "same tool + same params → same payload through all
//! three surfaces" can be asserted in an integration test without touching
//! filesystem, orchestrator, or provider CLIs.
use async_trait::async_trait;
use provisioning_core::{
Environment, Registry, ToolError,
protocol::ToolCategory,
tool::{Context, Tool},
};
use serde_json::{Value, json};
use std::sync::Arc;
// ── Fixture tools ─────────────────────────────────────────────────────────────
/// Returns the input params verbatim — schema allows any object.
pub struct EchoTool;
#[async_trait]
impl Tool for EchoTool {
fn name(&self) -> &'static str { "ct_echo" }
fn description(&self) -> &'static str { "Echo the input parameters verbatim" }
fn schema(&self) -> Value {
json!({
"type": "object",
"properties": {
"message": { "type": "string" }
},
"required": ["message"],
"additionalProperties": true
})
}
fn category(&self) -> ToolCategory { ToolCategory::Read }
async fn invoke(&self, params: Value, _ctx: &Context) -> Result<Value, ToolError> {
let message = params.get("message").and_then(|v| v.as_str()).ok_or_else(|| {
ToolError::invalid_param("message", "required string field missing")
})?;
Ok(json!({
"items": [{ "message": message }],
"total": 1
}))
}
}
/// Deterministic three-item listing — same output regardless of params.
pub struct ListingTool;
#[async_trait]
impl Tool for ListingTool {
fn name(&self) -> &'static str { "ct_listing" }
fn description(&self) -> &'static str { "Return a canonical three-item listing" }
fn schema(&self) -> Value {
json!({ "type": "object", "properties": {}, "additionalProperties": false })
}
fn category(&self) -> ToolCategory { ToolCategory::Read }
async fn invoke(&self, _params: Value, _ctx: &Context) -> Result<Value, ToolError> {
Ok(json!({
"items": [
{ "id": "a", "value": 1 },
{ "id": "b", "value": 2 },
{ "id": "c", "value": 3 }
],
"total": 3
}))
}
}
/// Always fails with `InvalidParam` — used to validate error-path contract.
pub struct FailingTool;
#[async_trait]
impl Tool for FailingTool {
fn name(&self) -> &'static str { "ct_fail" }
fn description(&self) -> &'static str { "Always fails with InvalidParam" }
fn schema(&self) -> Value {
json!({ "type": "object", "properties": {}, "additionalProperties": true })
}
fn category(&self) -> ToolCategory { ToolCategory::Read }
async fn invoke(&self, _params: Value, _ctx: &Context) -> Result<Value, ToolError> {
Err(ToolError::invalid_param("expected", "fixture always rejects"))
}
}
/// Catalog parity fixture — mirrors the `catalog_validate` (ADR-047) response
/// shape so CLI/HTTP/MCP agreement on the catalog surface is asserted without
/// depending on a real manifest on disk or the nickel binary.
pub struct CatalogValidateFixture;
#[async_trait]
impl Tool for CatalogValidateFixture {
fn name(&self) -> &'static str { "ct_catalog" }
fn description(&self) -> &'static str {
"Return a canonical catalog_validate-shaped payload"
}
fn schema(&self) -> Value {
json!({
"type": "object",
"required": ["manifest"],
"properties": { "manifest": { "type": "string" } }
})
}
fn category(&self) -> ToolCategory { ToolCategory::Read }
async fn invoke(&self, _params: Value, _ctx: &Context) -> Result<Value, ToolError> {
Ok(json!({
"valid": true,
"manifest": "catalog/providers/hetzner/manifest.ncl",
"contract": { "id": "catalog-manifest", "version": ">=1.0.0, <2.0.0" }
}))
}
}
/// Build a Registry populated with the fixture tools.
pub fn make_fixture_registry() -> Registry {
let mut reg = Registry::new();
reg.register(Arc::new(EchoTool)).expect("register echo");
reg.register(Arc::new(ListingTool)).expect("register listing");
reg.register(Arc::new(FailingTool)).expect("register fail");
reg.register(Arc::new(CatalogValidateFixture)).expect("register catalog");
reg
}
// ── Output-shape schema (the contract) ────────────────────────────────────────
/// JSON Schema that all `Read`-category list-style tools must satisfy.
/// Generated once at build time so each contract-test case can validate
/// without re-parsing.
pub fn listing_output_schema() -> Value {
json!({
"type": "object",
"required": ["items", "total"],
"properties": {
"items": { "type": "array" },
"total": { "type": "integer", "minimum": 0 }
},
"additionalProperties": true
})
}
// ── Three-tier dispatcher ─────────────────────────────────────────────────────
pub struct TierOutcome {
/// Inner payload after envelope unwrap. `None` on error.
pub payload: Option<Value>,
/// Error code on failure, `None` on success. Each tier maps ToolError
/// to its own code namespace — the normaliser strips that difference.
pub error_code: Option<i32>,
/// Error message (trimmed of tier-specific prefixes).
pub error_message: Option<String>,
}
impl TierOutcome {
pub fn ok(payload: Value) -> Self {
Self { payload: Some(payload), error_code: None, error_message: None }
}
pub fn err(code: i32, message: impl Into<String>) -> Self {
Self { payload: None, error_code: Some(code), error_message: Some(message.into()) }
}
}
/// Tier A — direct Registry invocation. The reference tier: what the other
/// two surfaces must agree with after envelope unwrapping.
pub async fn tier_registry(
registry: &Registry,
env: Arc<Environment>,
tool: &str,
params: Value,
) -> TierOutcome {
let ctx = Context::new(env);
match registry.invoke(tool, params, &ctx).await {
Ok(v) => TierOutcome::ok(v),
Err(e) => {
let code = match &e {
ToolError::NotFound(_) => -32001,
ToolError::InvalidParam { .. } => -32602,
ToolError::Unauthorized(_) => -32003,
_ => -32000,
};
TierOutcome::err(code, e.to_string())
}
}
}
/// Tier B — HTTP daemon. The axum router is reused directly on a TcpListener
/// bound to 127.0.0.1:0 for deterministic port isolation.
pub async fn tier_http(base_url: &str, tool: &str, params: Value) -> TierOutcome {
let client = reqwest::Client::new();
let url = format!("{}/api/v1/tools/{}", base_url, tool);
let body = json!({ "params": params });
let resp = match client.post(&url).json(&body).send().await {
Ok(r) => r,
Err(e) => return TierOutcome::err(-32000, format!("http transport: {e}")),
};
let status = resp.status();
let body: Value = match resp.json().await {
Ok(v) => v,
Err(e) => return TierOutcome::err(-32000, format!("http body: {e}")),
};
if status.is_success() {
match body.get("result").cloned() {
Some(v) => TierOutcome::ok(v),
None => TierOutcome::err(-32000, "missing result field in http response"),
}
} else {
let code = body
.get("code")
.and_then(|v| v.as_i64())
.map(|c| c as i32)
.unwrap_or(-32000);
let msg = body
.get("error")
.and_then(|v| v.as_str())
.unwrap_or("<missing error>")
.to_string();
TierOutcome::err(code, msg)
}
}
/// Tier C — MCP surface via McpServer::handle_request.
/// MCP wraps success payloads in `{content: [{type:"text", text: "<json>"}]}`;
/// the unwrap parses the text field back into JSON.
pub async fn tier_mcp(server: &provisioning_mcp_server::registry_server::McpServer, tool: &str, params: Value) -> TierOutcome {
let rpc = json!({
"jsonrpc": "2.0",
"id": 1,
"method": "tools/call",
"params": { "name": tool, "arguments": params }
});
let resp = server.handle_request(rpc).await;
if let Some(err) = resp.get("error") {
let code = err.get("code").and_then(|v| v.as_i64()).map(|c| c as i32).unwrap_or(-32000);
let msg = err.get("message").and_then(|v| v.as_str()).unwrap_or("<missing>").to_string();
return TierOutcome::err(code, msg);
}
let content = match resp.get("result").and_then(|r| r.get("content")) {
Some(c) => c,
None => return TierOutcome::err(-32000, "mcp response missing result.content"),
};
let text = content
.get(0)
.and_then(|c| c.get("text"))
.and_then(|t| t.as_str());
let text = match text {
Some(t) => t,
None => return TierOutcome::err(-32000, "mcp content[0].text missing"),
};
match serde_json::from_str::<Value>(text) {
Ok(v) => TierOutcome::ok(v),
Err(e) => TierOutcome::err(-32000, format!("mcp text parse: {e}")),
}
}
// ── Payload normalisation ─────────────────────────────────────────────────────
/// Strip fields that are allowed to diverge between tiers: trace ids, timestamps,
/// uuids. The contract is on semantic payload, not byte equality.
pub fn normalise(v: &Value) -> Value {
match v {
Value::Object(map) => {
let mut out = serde_json::Map::new();
for (k, val) in map {
if matches!(k.as_str(), "trace_id" | "span_id" | "timestamp" | "generated_at" | "request_id")
{
continue;
}
out.insert(k.clone(), normalise(val));
}
Value::Object(out)
}
Value::Array(arr) => Value::Array(arr.iter().map(normalise).collect()),
_ => v.clone(),
}
}

218
crates/contract-tests/tests/g3_contract.rs Normal file
View file

@ -0,0 +1,218 @@
//! G3 — CLI↔HTTP↔MCP contract test.
//!
//! For each fixture tool, invoke via:
//! Tier A — Registry directly (reference)
//! Tier B — HTTP daemon (axum::serve on 127.0.0.1:0)
//! Tier C — MCP server (handle_request in-process)
//!
//! Assert:
//! 1. All tiers that succeed produce equal normalised payloads
//! 2. The payload validates against the listing output schema
//! 3. All tiers that fail produce the same error code
//!
//! The normaliser strips volatile fields (trace_id, timestamp, …) so the
//! contract is on semantic equivalence, not byte-for-byte equality.
use contract_tests::{
listing_output_schema, make_fixture_registry, normalise,
tier_http, tier_mcp, tier_registry,
};
use jsonschema::Validator;
use provisioning_mcp_server::registry_server::McpServer;
use provisioning_core::Environment;
use provisioning_daemon::{AppState, http::router};
use serde_json::{Value, json};
use std::sync::{Arc, OnceLock};
// ── Test harness ──────────────────────────────────────────────────────────────
static CRYPTO: OnceLock<()> = OnceLock::new();
fn init_crypto() {
CRYPTO.get_or_init(|| {
let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
});
}
struct Harness {
env: Arc<Environment>,
registry: Arc<provisioning_core::Registry>,
mcp: McpServer,
http_base: String,
_daemon_task: tokio::task::JoinHandle<()>,
}
async fn boot() -> Harness {
init_crypto();
// Shared fixture registry for all tiers.
let registry = Arc::new(make_fixture_registry());
let env = Arc::new(Environment::default());
// Tier B — bind daemon on ephemeral port.
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("bind");
let addr = listener.local_addr().expect("local_addr");
let http_base = format!("http://{addr}");
// AppState owns its own Arc<Registry>; we build it from a fresh fixture
// registry so the daemon and the reference tier see identical tool sets.
let daemon_registry = make_fixture_registry();
let app = AppState::new(daemon_registry, Environment::default());
let app_router = router(app);
let task = tokio::spawn(async move {
axum::serve(listener, app_router).await.expect("daemon serve");
});
// Wait for the daemon to accept connections (fast — local TCP).
for _ in 0..40 {
if reqwest::get(format!("{}/health", http_base)).await.is_ok() {
break;
}
tokio::time::sleep(std::time::Duration::from_millis(25)).await;
}
// Tier C — in-process MCP server with its own fixture registry.
let mcp = McpServer::new(make_fixture_registry(), Environment::default(), "g3-test", "0");
Harness { env, registry, mcp, http_base, _daemon_task: task }
}
fn validate_listing(v: &Value) {
let schema = listing_output_schema();
let validator = Validator::new(&schema).expect("compile schema");
validator
.validate(v)
.unwrap_or_else(|errors| panic!("listing schema validation failed: {errors}"));
}
// ── Success contract ──────────────────────────────────────────────────────────
#[tokio::test]
async fn listing_tool_agrees_across_three_tiers() {
let h = boot().await;
let params = json!({});
let a = tier_registry(&h.registry, h.env.clone(), "ct_listing", params.clone()).await;
let b = tier_http(&h.http_base, "ct_listing", params.clone()).await;
let c = tier_mcp(&h.mcp, "ct_listing", params.clone()).await;
let a_payload = a.payload.expect("tier A registry must succeed");
let b_payload = b.payload.expect("tier B http must succeed");
let c_payload = c.payload.expect("tier C mcp must succeed");
validate_listing(&a_payload);
validate_listing(&b_payload);
validate_listing(&c_payload);
assert_eq!(normalise(&a_payload), normalise(&b_payload), "A↔B diverge");
assert_eq!(normalise(&a_payload), normalise(&c_payload), "A↔C diverge");
assert_eq!(normalise(&b_payload), normalise(&c_payload), "B↔C diverge");
}
#[tokio::test]
async fn catalog_tool_agrees_across_three_tiers() {
let h = boot().await;
let params = json!({ "manifest": "catalog/providers/hetzner/manifest.ncl" });
let a = tier_registry(&h.registry, h.env.clone(), "ct_catalog", params.clone()).await;
let b = tier_http(&h.http_base, "ct_catalog", params.clone()).await;
let c = tier_mcp(&h.mcp, "ct_catalog", params.clone()).await;
let a_payload = a.payload.expect("tier A registry must succeed");
let b_payload = b.payload.expect("tier B http must succeed");
let c_payload = c.payload.expect("tier C mcp must succeed");
assert_eq!(normalise(&a_payload), normalise(&b_payload), "A↔B diverge");
assert_eq!(normalise(&a_payload), normalise(&c_payload), "A↔C diverge");
assert_eq!(normalise(&b_payload), normalise(&c_payload), "B↔C diverge");
// Catalog response shape — valid flag + declared contract reference
assert_eq!(a_payload["valid"], true);
assert_eq!(a_payload["contract"]["id"], "catalog-manifest");
assert_eq!(b_payload["contract"]["id"], "catalog-manifest");
assert_eq!(c_payload["contract"]["id"], "catalog-manifest");
}
#[tokio::test]
async fn echo_tool_agrees_across_three_tiers() {
let h = boot().await;
let params = json!({ "message": "contract" });
let a = tier_registry(&h.registry, h.env.clone(), "ct_echo", params.clone()).await;
let b = tier_http(&h.http_base, "ct_echo", params.clone()).await;
let c = tier_mcp(&h.mcp, "ct_echo", params.clone()).await;
let a_payload = a.payload.expect("A must succeed");
let b_payload = b.payload.expect("B must succeed");
let c_payload = c.payload.expect("C must succeed");
validate_listing(&a_payload);
validate_listing(&b_payload);
validate_listing(&c_payload);
assert_eq!(normalise(&a_payload), normalise(&b_payload));
assert_eq!(normalise(&a_payload), normalise(&c_payload));
// Exact content check — echo should round-trip the message
assert_eq!(a_payload["items"][0]["message"], "contract");
assert_eq!(b_payload["items"][0]["message"], "contract");
assert_eq!(c_payload["items"][0]["message"], "contract");
}
// ── Failure contract ──────────────────────────────────────────────────────────
#[tokio::test]
async fn invalid_param_error_code_agrees_across_tiers() {
let h = boot().await;
let params = json!({}); // echo requires "message" → InvalidParam
let a = tier_registry(&h.registry, h.env.clone(), "ct_echo", params.clone()).await;
let b = tier_http(&h.http_base, "ct_echo", params.clone()).await;
let c = tier_mcp(&h.mcp, "ct_echo", params.clone()).await;
// All three must classify as InvalidParam → -32602
assert_eq!(a.error_code, Some(-32602), "A: {:?}", a.error_message);
assert_eq!(b.error_code, Some(-32602), "B: {:?}", b.error_message);
assert_eq!(c.error_code, Some(-32602), "C: {:?}", c.error_message);
}
#[tokio::test]
async fn failing_tool_error_code_agrees_across_tiers() {
let h = boot().await;
let params = json!({});
let a = tier_registry(&h.registry, h.env.clone(), "ct_fail", params.clone()).await;
let b = tier_http(&h.http_base, "ct_fail", params.clone()).await;
let c = tier_mcp(&h.mcp, "ct_fail", params.clone()).await;
assert_eq!(a.error_code, Some(-32602));
assert_eq!(b.error_code, Some(-32602));
assert_eq!(c.error_code, Some(-32602));
}
// ── tools/list contract ───────────────────────────────────────────────────────
#[tokio::test]
async fn tools_list_count_agrees_across_surfaces() {
let h = boot().await;
// Registry baseline
let reg_count = h.registry.len();
// HTTP
let http_body: Value = reqwest::get(format!("{}/api/v1/tools", h.http_base))
.await
.expect("http list")
.json()
.await
.expect("parse");
let http_count = http_body["tools"].as_array().expect("tools array").len();
// MCP
let rpc = json!({"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}});
let mcp_resp = h.mcp.handle_request(rpc).await;
let mcp_count = mcp_resp["result"]["tools"].as_array().expect("mcp tools").len();
assert_eq!(reg_count, http_count, "HTTP count diverges from registry");
assert_eq!(reg_count, mcp_count, "MCP count diverges from registry");
}

109
crates/control-center-ui/Cargo.toml Normal file
View file

@ -0,0 +1,109 @@
[package]
authors.workspace = true
autobins = false
description = "Control Center UI - Leptos CSR App for Cloud Infrastructure Management"
edition.workspace = true
license.workspace = true
name = "control-center-ui"
repository.workspace = true
version.workspace = true
[lib]
crate-type = ["cdylib"]
name = "control_center_ui"
path = "src/main.rs"
[dependencies]
# ============================================================================
# WORKSPACE DEPENDENCIES
# ============================================================================
# Serialization
chrono = { workspace = true, features = ["wasm-bindgen"] }
serde = { workspace = true }
serde_json = { workspace = true }
uuid = { workspace = true, features = ["js"] }
# Error handling and async
futures = { workspace = true }
thiserror = { workspace = true }
# Logging and debugging
tracing = { workspace = true }
# Security and cryptography
aes-gcm = { workspace = true, features = ["aes", "std"] }
base64 = { workspace = true }
hmac = { workspace = true }
rand = { workspace = true }
regex = { workspace = true }
sha2 = { workspace = true }
# ============================================================================
# WASM-SPECIFIC DEPENDENCIES
# ============================================================================
# Leptos Framework with CSR features
leptos = { workspace = true }
leptos_meta = { workspace = true }
leptos_router = { workspace = true }
# WASM utilities
wasm-bindgen = { workspace = true }
# ============================================================================
# ADDITIONAL WORKSPACE DEPENDENCIES
# ============================================================================
# URL handling
url = { workspace = true }
# Icons and UI utilities
icondata = { workspace = true }
leptos_icons = { workspace = true }
# Authentication and cryptography
image = { workspace = true }
qrcode = { workspace = true }
totp-rs = { workspace = true }
# Serialization utilities
serde-wasm-bindgen = { workspace = true }
# Logging for WASM
console_error_panic_hook = { workspace = true }
tracing-wasm = { workspace = true }
# HTTP client and networking
gloo-net = { workspace = true }
gloo-storage = { workspace = true }
gloo-timers = { workspace = true }
gloo-utils = { workspace = true }
# Chart.js bindings and canvas utilities
plotters = { workspace = true }
plotters-canvas = { workspace = true }
# WASM utilities
js-sys = { workspace = true }
wasm-bindgen-futures = { workspace = true }
# Random number generation (WASM-specific override with js feature)
getrandom = { workspace = true, features = ["wasm_js"] }
# HTTP client
reqwest = { workspace = true, features = ["json"] }
# Tokio with time features
tokio = { workspace = true, features = ["time"] }
# Web APIs (WASM browser APIs)
web-sys = { workspace = true }
# Profile configurations moved to workspace root
[package.metadata.wasm-pack.profile.release]
wasm-opt = ['-Oz', '--enable-mutable-globals']
[package.metadata.wasm-pack.profile.dev]
wasm-opt = false

368
crates/control-center-ui/README.md Normal file
View file

@ -0,0 +1,368 @@
# Control Center UI - Audit Log Viewer
A comprehensive React-based audit log viewer for the Cedar Policy Engine with advanced search, real-time streaming,
compliance reporting, and visualization capabilities.
## 🚀 Features
### 🔍 Advanced Search & Filtering
- **Multi-dimensional Filters**: Date range, users, actions, resources, severity, compliance frameworks
- **Real-time Search**: Debounced search with instant results
- **Saved Searches**: Save and reuse complex filter combinations
- **Quick Filters**: One-click access to common time ranges and filters
- **Correlation Search**: Find logs by request ID, session ID, or trace correlation
### 📊 High-Performance Data Display
- **Virtual Scrolling**: Handle millions of log entries with smooth scrolling
- **Infinite Loading**: Automatic pagination with optimized data fetching
- **Column Sorting**: Sort by any field with persistent state
- **Bulk Selection**: Select multiple logs for batch operations
- **Responsive Design**: Works seamlessly on desktop, tablet, and mobile
### 🔴 Real-time Streaming
- **WebSocket Integration**: Live log updates without page refresh
- **Connection Management**: Automatic reconnection with exponential backoff
- **Real-time Indicators**: Visual status of live connection
- **Message Queuing**: Handles high-volume log streams efficiently
- **Alert Notifications**: Critical events trigger immediate notifications
### 📋 Detailed Log Inspection
- **JSON Viewer**: Syntax-highlighted JSON with collapsible sections
- **Multi-tab Interface**: Overview, Context, Metadata, Compliance, Raw JSON
- **Sensitive Data Toggle**: Hide/show sensitive information
- **Copy Utilities**: One-click copying of IDs, values, and entire records
- **Deep Linking**: Direct URLs to specific log entries
### 📤 Export & Reporting
- **Multiple Formats**: CSV, JSON, PDF export with customizable fields
- **Template System**: Pre-built templates for different report types
- **Batch Export**: Export filtered results or selected logs
- **Progress Tracking**: Real-time export progress indication
- **Custom Fields**: Choose exactly which data to include
### 🛡️ Compliance Management
- **Framework Support**: SOC2, HIPAA, PCI DSS, GDPR compliance templates
- **Report Generation**: Automated compliance reports with evidence
- **Finding Tracking**: Track violations and remediation status
- **Attestation Management**: Digital signatures and certifications
- **Template Library**: Customizable report templates for different frameworks
### 🔗 Log Correlation & Tracing
- **Request Tracing**: Follow request flows across services
- **Session Analysis**: View all activity for a user session
- **Dependency Mapping**: Understand log relationships and causality
- **Timeline Views**: Chronological visualization of related events
### 📈 Visualization & Analytics
- **Dashboard Metrics**: Real-time statistics and KPIs
- **Timeline Charts**: Visual representation of log patterns
- **Geographic Distribution**: Location-based log analysis
- **Severity Trends**: Track security event patterns over time
- **User Activity**: Monitor user behavior and access patterns
## 🛠 Technology Stack
### Frontend Framework
- **React 18.3.1**: Modern React with hooks and concurrent features
- **TypeScript 5.5.4**: Type-safe development with advanced types
- **Vite 5.4.1**: Lightning-fast build tool and dev server
### UI Components & Styling
- **TailwindCSS 3.4.9**: Utility-first CSS framework
- **DaisyUI 4.4.19**: Beautiful component library built on Tailwind
- **Framer Motion 11.3.24**: Smooth animations and transitions
- **Lucide React 0.427.0**: Beautiful, customizable icons
### Data Management
- **TanStack Query 5.51.23**: Powerful data fetching and caching
- **TanStack Table 8.20.1**: Headless table utilities for complex data
- **TanStack Virtual 3.8.4**: Virtual scrolling for performance
- **Zustand 4.5.4**: Lightweight state management
### Forms & Validation
- **React Hook Form 7.52.2**: Performant forms with minimal re-renders
- **React Select 5.8.0**: Flexible select components with search
### Real-time & Networking
- **Native WebSocket API**: Direct WebSocket integration
- **Custom Hooks**: Reusable WebSocket management with reconnection
### Export & Reporting
- **jsPDF 2.5.1**: Client-side PDF generation
- **jsPDF AutoTable 3.8.2**: Table formatting for PDF reports
- **Native Blob API**: File download and export functionality
### Date & Time
- **date-fns 3.6.0**: Modern date utility library with tree shaking
## 📁 Project Structure
```bash
src/
├── components/audit/ # Audit log components
│ ├── AuditLogViewer.tsx # Main viewer component
│ ├── SearchFilters.tsx # Advanced search interface
│ ├── VirtualizedLogTable.tsx # High-performance table
│ ├── LogDetailModal.tsx # Detailed log inspection
│ ├── ExportModal.tsx # Export functionality
│ ├── ComplianceReportGenerator.tsx # Compliance reports
│ └── RealTimeIndicator.tsx # WebSocket status
├── hooks/ # Custom React hooks
│ └── useWebSocket.ts # WebSocket management
├── services/ # API integration
│ └── api.ts # Audit API client
├── types/ # TypeScript definitions
│ └── audit.ts # Audit-specific types
├── utils/ # Utility functions
├── store/ # State management
└── styles/ # CSS and styling
```
## 🔧 Setup and Development
### Prerequisites
- **Node.js 18+** and **npm 9+**
- **Control Center backend** running on `http://localhost:8080`
### Installation
```bash
# Clone the repository
git clone <repository-url>
cd control-center-ui
# Install dependencies
npm install
# Start development server
npm run dev
```
The application will be available at `http://localhost:3000`
### Building for Production
```bash
# Type check
npm run type-check
# Build for production
npm run build
# Preview production build
npm run preview
```
## 🌐 API Integration
The UI integrates with the Control Center backend and expects the following endpoints:
- `GET /audit/logs` - Fetch audit logs with filtering and pagination
- `GET /audit/logs/{id}` - Get specific log entry details
- `POST /audit/search` - Advanced search functionality
- `GET /audit/saved-searches` - Manage saved search queries
- `POST /audit/export` - Export logs in various formats (CSV, JSON, PDF)
- `GET /compliance/reports` - Compliance report management
- `POST /compliance/reports/generate` - Generate compliance reports
- `WS /audit/stream` - Real-time log streaming via WebSocket
- `GET /health` - Health check endpoint
### WebSocket Integration
Real-time log streaming is implemented using WebSocket connections:
```bash
import { useWebSocket } from './hooks/useWebSocket';
const { isConnected, lastMessage } = useWebSocket({
url: 'ws://localhost:8080/ws/audit',
onNewAuditLog: (log) => {
// Handle new log entry in real-time
updateLogsList(log);
}
});
```
## ✅ Features Implemented
### Core Audit Log Viewer System
- ✅ **Advanced Search Filters**: Multi-dimensional filtering with date range, users, actions, resources, severity, compliance frameworks
- ✅ **Virtual Scrolling Component**: High-performance rendering capable of handling millions of log entries
- ✅ **Real-time Log Streaming**: WebSocket integration with automatic reconnection and live status indicators
- ✅ **Detailed Log Modal**: Multi-tab interface with JSON syntax highlighting, sensitive data toggle, and copy utilities
- ✅ **Export Functionality**: Support for CSV, JSON, and PDF formats with customizable fields and templates
- ✅ **Saved Search Queries**: User preference system for saving and reusing complex search combinations
### Compliance & Security Features
- ✅ **Compliance Report Generator**: Automated report generation with SOC2, HIPAA, PCI DSS, and GDPR templates
- ✅ **Violation Tracking**: Remediation workflow system with task management and progress tracking
- ✅ **Timeline Visualization**: Chronological visualization of audit trails with correlation mapping
- ✅ **Request ID Correlation**: Cross-service request tracing and session analysis
- ✅ **Attestation Management**: Digital signature system for compliance certifications
- ✅ **Log Retention Management**: Archival policies and retention period management
### Performance & User Experience
- ✅ **Dashboard Analytics**: Real-time metrics including success rates, critical events, and compliance scores
- ✅ **Responsive Design**: Mobile-first design that works across all device sizes
- ✅ **Loading States**: Comprehensive loading indicators and skeleton screens
- ✅ **Error Handling**: Robust error boundaries with user-friendly error messages
- ✅ **Keyboard Shortcuts**: Accessibility features and keyboard navigation support
## 🎨 Styling and Theming
### TailwindCSS Configuration
The application uses a comprehensive TailwindCSS setup with:
- **DaisyUI Components**: Pre-built, accessible UI components
- **Custom Color Palette**: Primary, secondary, success, warning, error themes
- **Custom Animations**: Smooth transitions and loading states
- **Dark/Light Themes**: Automatic theme switching with system preference detection
- **Responsive Grid System**: Mobile-first responsive design
### Component Design System
- **Consistent Spacing**: Standardized margin and padding scales
- **Typography Scale**: Hierarchical text sizing and weights
- **Icon System**: Comprehensive icon library with consistent styling
- **Form Controls**: Validated, accessible form components
- **Data Visualization**: Charts and metrics with consistent styling
## 📱 Performance Optimization
### Virtual Scrolling
- Renders only visible rows for optimal performance
- Handles datasets with millions of entries smoothly
- Maintains smooth scrolling with momentum preservation
- Automatic cleanup of off-screen elements
### Efficient Data Fetching
- Infinite queries with intelligent pagination
- Aggressive caching with TanStack Query
- Optimistic updates for better user experience
- Background refetching for fresh data
### Bundle Optimization
- Code splitting by route and feature
- Tree shaking for minimal bundle size
- Lazy loading of heavy components
- Optimized production builds
## 🔒 Security Considerations
### Data Protection
- Sensitive data masking in UI components
- Secure WebSocket connections (WSS in production)
- Content Security Policy headers for XSS protection
- Input sanitization for search queries
### API Security
- JWT token authentication support (when implemented)
- Request rate limiting awareness
- Secure file downloads with proper headers
- CORS configuration for cross-origin requests
## 🚀 Deployment
### Docker Deployment
```bash
FROM node:18-alpine as builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
RUN npm run build
FROM nginx:alpine
COPY --from=builder /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/nginx.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]
```
### Kubernetes Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: control-center-ui
spec:
replicas: 3
selector:
matchLabels:
app: control-center-ui
template:
metadata:
labels:
app: control-center-ui
spec:
containers:
- name: control-center-ui
image: control-center-ui:latest
ports:
- containerPort: 80
env:
- name: VITE_API_BASE_URL
value: "https://api.example.com"
```
## 🤝 Contributing
### Development Guidelines
- Follow TypeScript strict mode conventions
- Use existing component patterns and design system
- Maintain accessibility standards (WCAG 2.1 AA)
- Add proper error boundaries for robust error handling
- Write meaningful commit messages following conventional commits
### Code Style
- Use Prettier for consistent code formatting
- Follow ESLint rules for code quality
- Use semantic HTML elements for accessibility
- Maintain consistent naming conventions
- Document complex logic with comments
## 📄 License
This project follows the same license as the parent Control Center repository.
## 🆘 Support
For questions, issues, or contributions:
1. Check existing issues in the repository
2. Review the comprehensive documentation
3. Create detailed bug reports or feature requests
4. Follow the established contribution guidelines
---
Built with ❤️ for comprehensive audit log management, compliance monitoring, and security analytics.

33
crates/control-center-ui/REFERENCE.md Normal file
View file

@ -0,0 +1,33 @@
# Control Center UI Reference
This directory will reference the existing control center UI implementation.
## Current Implementation Location
`/Users/Akasha/repo-cnz/src/control-center-ui/`
## Implementation Details
- **Language**: Web frontend (likely React/Vue/Leptos)
- **Purpose**: Web interface for system management
- **Features**:
- Dashboard and monitoring UI
- Configuration management interface
- System administration controls
## Integration Status
- **Current**: Fully functional in original location
- **New Structure**: Reference established
- **Migration**: Planned for future phase
## Usage
The control center UI remains fully functional at its original location.
```bash
cd /Users/Akasha/repo-cnz/src/control-center-ui
# Use existing UI development commands
```
See original implementation for development setup and usage instructions.

Some files were not shown because too many files have changed in this diff Show more