provisioning/docs/src/operations/services-local-deployment.md

635 lines
15 KiB
Markdown
Raw Normal View History

# Platform Services - Local Deployment Inventory
## Overview
Complete inventory of all 10 provisioning platform services with deployment options, endpoints, and configuration details for local development and testing.
**Quick Facts**:
- **10 total platform services**
- **8 HTTP-based services** with REST API
- **1 binary protocol service** (MCP - Model Context Protocol)
- **1 WASM application** (Control Center UI)
- **Ports**: 3000, 8000, 8081-8083, 8100, 8300, 8400, 8600, 9090
## Table of Contents
- [Services Summary](#services-summary)
- [Port Quick Reference](#port-quick-reference)
- [Services by Category](#services-by-category)
- [Detailed Service Documentation](#detailed-service-documentation)
---
## Services Summary
| # | Service | Binary | Port | Size | Purpose |
|---|---------|--------|------|------|---------|
| 1 | orchestrator | provisioning-orchestrator | 9090 | 33 MB | Batch workflows, task queue, rollback |
| 2 | extension-registry | extension-registry | 8082 | 2.2 MB | OCI-compliant registry proxy |
| 3 | control-center | provisioning-control-center | 8000 | TBD | JWT auth, user mgmt, secrets, WebSocket |
| 4 | provisioning-daemon | provisioning-daemon | 8100 | 7.9 MB | Nushell execution, config rendering |
| 5 | ai-service | ai-service | 8083 | 3.8 MB | RAG, MCP tools, extension DAGs |
| 6 | provisioning-rag | provisioning-rag | 8300 | 1.9 MB | Vector search, semantic retrieval |
| 7 | mcp-server | provisioning-mcp-server | 8400 | TBD | Infrastructure automation tools (binary protocol) |
| 8 | vault-service | provisioning-vault-service | 8081 | TBD | KMS, encryption, secrets (Age/Cosmian) |
| 9 | detector | provisioning-detector | 8600 | TBD | Infrastructure detection & discovery |
| 10 | control-center-ui | control-center-ui | 3000 | N/A | Web dashboard (WASM/Leptos) |
---
## Port Quick Reference
```
3000 control-center-ui (WASM app)
8000 control-center (JWT auth, secrets)
8001 [reserved]
8081 vault-service (KMS)
8082 extension-registry (OCI)
8083 ai-service (RAG, MCP)
8100 provisioning-daemon (Nushell)
8300 provisioning-rag (Vector DB)
8400 mcp-server (binary protocol)
8600 detector (detection)
9090 orchestrator (workflows)
```
---
## Services by Category
### Core Services (Required)
- **orchestrator** (9090) - Batch workflows, task queue, rollback
- **control-center** (8000) - Authentication, authorization
- **extension-registry** (8082) - Extension management
### Platform Services
- **provisioning-daemon** (8100) - Nushell execution
- **vault-service** (8081) - Secrets, KMS
- **ai-service** (8083) - AI capabilities
### AI & Knowledge
- **provisioning-rag** (8300) - RAG engine
- **mcp-server** (8400) - Infrastructure tools
### Utilities
- **detector** (8600) - Infrastructure detection
- **control-center-ui** (3000) - Web dashboard
---
## Detailed Service Documentation
### 1. orchestrator
**Binary**: provisioning-orchestrator
**Port**: 9090
**Size**: 33 MB
**Purpose**: Manages distributed task execution, batch workflows, cluster provisioning, and disaster recovery.
**Key Features**:
- Batch workflow execution with task queue
- State management and snapshots
- Checkpoint creation and rollback
- Compliance and audit logging
- System metrics and health monitoring
**Key Endpoints**:
```
GET /api/v1/health Health check
GET /tasks List all tasks
GET /tasks/{id} Get task status
POST /batch/execute Execute batch operation
GET /batch/operations List operations
POST /rollback/checkpoints Create checkpoint
GET /rollback/checkpoints List checkpoints
POST /rollback/execute Execute rollback
GET /state/system/health System health
GET /state/system/metrics System metrics
```
**Health Check**:
```bash
curl http://localhost:9090/api/v1/health
```
**Invocation**:
```bash
./provisioning-orchestrator --port 9090
```
---
### 2. extension-registry
**Binary**: extension-registry
**Port**: 8082
**Size**: 2.2 MB
**Purpose**: OCI v2 API-compliant registry proxy for managing extensions.
**Key Features**:
- OCI v2 API compliance
- Extension metadata management
- Blob and manifest handling
- Catalog browsing
**Key Endpoints**:
```
GET /api/v1/health Health check
GET /extensions List all extensions
GET /extensions/:name Get extension metadata
POST /extensions Register extension
GET /v2/_catalog OCI catalog
GET /v2/:name/tags/list List tags
GET /v2/:name/manifests/:ref Pull manifest
PUT /v2/:name/manifests/:ref Push manifest
```
**Health Check**:
```bash
curl http://localhost:8082/api/v1/health
```
**Invocation**:
```bash
./extension-registry --port 8082 --host 127.0.0.1
```
---
### 3. Control Center
**Binary**: provisioning-control-center
**Port**: 8000
**Size**: TBD
**Purpose**: JWT authentication, user management, secrets management, WebSocket real-time events.
**Key Features**:
- JWT token generation and validation
- User CRUD operations
- Role-based access control
- Real-time WebSocket events
- Secrets management with versioning
- Multi-factor authentication (TOTP, WebAuthn)
- Audit logging
**Tech Stack**:
- Framework: Axum
- Database: SurrealDB, SQLx
- Auth: JWT, Argon2
- Crypto: AES-GCM, HMAC, RSA, SHA2
**Key Endpoints** (Public):
```
POST /auth/login User authentication
POST /auth/refresh Refresh token
```
**Key Endpoints** (Protected):
```
POST /auth/logout Logout
GET /permissions List permissions
GET /deployments List deployments
POST /deployments Create deployment
GET /secrets List secrets
POST /secrets Create secret
GET /secrets/:path Get secret
PUT /secrets/:path Update secret
DELETE /secrets/:path Delete secret
GET /ws WebSocket (real-time events)
```
**Health Check**:
```bash
curl http://localhost:8000/health
```
**Invocation**:
```bash
./provisioning-control-center --port 8000 --config config/control-center.toml
```
---
### 4. Provisioning Daemon
**Binary**: provisioning-daemon
**Port**: 8100
**Size**: 7.9 MB
**Purpose**: Runtime service for executing Nushell scripts and rendering configuration templates.
**Key Features**:
- Execute Nushell scripts
- Render Nickel/TOML templates
- Configuration validation
- Operation execution framework
**Tech Stack**:
- Framework: Axum
- Core: daemon-cli library
- Languages: Nickel, TOML, JSON
**CLI Arguments**:
```bash
-c, --config <PATH> Config file
--config-dir <DIR> Config directory
-m, --mode <MODE> Mode: solo|multiuser|cicd|enterprise
-v, --verbose Verbose logging
--validate-config Validate and exit
--show-config Show config and exit
```
**Key Endpoints**:
```
GET /api/v1/health Health check
POST /execute Execute Nushell script
POST /render Render template
GET /operations List operations
```
**Health Check**:
```bash
curl http://localhost:8100/api/v1/health
```
**Invocation**:
```bash
./provisioning-daemon --config config/daemon.toml --mode solo
```
---
### 5. AI Service
**Binary**: ai-service
**Port**: 8083
**Size**: 3.8 MB
**Purpose**: HTTP service for AI capabilities including RAG, MCP tools, extension DAGs.
**Key Features**:
- Retrieval-Augmented Generation (RAG)
- MCP tool invocation
- Extension dependency graphs
- Best practice recommendations
**Tech Stack**:
- Framework: Axum
- RAG: RAG crate
- MCP: mcp-server crate
- LLM/Embeddings: Stratum
**CLI Arguments**:
```bash
-c, --config <PATH> Config file
--config-dir <DIR> Config directory
-m, --mode <MODE> Mode
-H, --host <HOST> Bind host (default: 127.0.0.1)
-p, --port <PORT> Bind port (default: 8083)
```
**Key Endpoints**:
```
POST /api/v1/ai/mcp/tool Call MCP tool
POST /api/v1/ai/ask RAG question answering
GET /api/v1/ai/dag/extensions Get DAG
GET /api/v1/ai/knowledge/best-practices Best practices
GET /health Health check
```
**Health Check**:
```bash
curl http://localhost:8083/health
```
**Invocation**:
```bash
./ai-service --port 8083 --config config/ai-service.toml
```
---
### 6. Provisioning RAG
**Binary**: provisioning-rag
**Port**: 8300
**Size**: 1.9 MB
**Purpose**: RAG engine with semantic search, hybrid search, conversation tracking.
**Key Features**:
- Semantic document search
- Hybrid search (BM25 + vector)
- Conversation tracking
- Batch query processing
- Response caching (LRU)
**Tech Stack**:
- Framework: Axum
- RAG: Rig + rig-surrealdb
- Vector DB: SurrealDB (HNSW)
- Embeddings: Stratum
- Hybrid: BM25 + semantic
**Key Endpoints**:
```
POST /query Semantic search
POST /conversations Start conversation
GET /conversations/{id} Get conversation
POST /conversations/{id}/turn Add turn
POST /batch/query Batch processing
GET /batch/{id}/status Batch status
GET /health Health check
GET /metrics Metrics
```
**Health Check**:
```bash
curl http://localhost:8300/health
```
**Invocation**:
```bash
./provisioning-rag --config config/rag.toml --mode solo
```
---
### 7. MCP Server
**Binary**: provisioning-mcp-server
**Port**: 8400 (reserved - uses binary protocol)
**⚠️ IMPORTANT**: Uses Model Context Protocol (binary), NOT HTTP.
**Purpose**: Infrastructure automation tools, AI query integration, status/metrics retrieval.
**Key Features**:
- Infrastructure automation
- AI query integration
- Status and metrics
- Log retrieval
- Documentation finder
- Troubleshooting
**Tech Stack**:
- Protocol: Model Context Protocol (MCP)
- SDK: rust-mcp-sdk
- File Ops: walkdir
- Config: TOML, JSON, YAML
**Entry Point**: `simple_main.rs` (main.rs disabled)
**MCP Tools**:
```
provision_create_server Create server
provision_deploy_taskserv Deploy taskserv
provision_cluster_create Create cluster
ai_query AI query
get_infrastructure_status Infra status
get_system_metrics Metrics
get_logs Logs
```
**Health Check**: N/A (binary protocol)
**Invocation**: (Used by MCP clients, not directly)
```bash
./provisioning-mcp-server --config config/mcp-server.toml --mode solo
```
---
### 8. Vault Service
**Binary**: provisioning-vault-service
**Port**: 8081
**Size**: TBD
**Purpose**: KMS for secrets encryption, decryption, key generation, rotation.
**Key Features**:
- Encryption/decryption
- Key generation
- Key rotation
- Multi-backend (Age, Cosmian, RustyVault, SecretumVault)
- Environment-specific (dev/prod/enterprise)
**Tech Stack**:
- Framework: Axum
- Crypto: Age, Cosmian KMS, RustyVault
- Config: TOML + env vars
**Supported Backends**:
```
Age dev File-based encryption
Cosmian KMS prod Cloud-hosted SaaS
RustyVault enterprise Self-hosted
SecretumVault enterprise Commercial
```
**Environment Variables**:
```bash
KMS_BIND_ADDR Bind address (default: 0.0.0.0:8081)
KMS_CONFIG_PATH Config file
PROVISIONING_ENV Mode: dev|prod|enterprise
AGE_PUBLIC_KEY_PATH Age public key (dev)
AGE_PRIVATE_KEY_PATH Age private key (dev)
COSMIAN_KMS_URL Cosmian URL (prod - REQUIRED)
COSMIAN_API_KEY Cosmian key (prod - REQUIRED)
```
**Key Endpoints**:
```
GET /api/v1/kms/health Health check
GET /api/v1/kms/status KMS status
POST /api/v1/kms/encrypt Encrypt data
POST /api/v1/kms/decrypt Decrypt data
POST /api/v1/kms/generate-key Generate key
POST /api/v1/kms/rotate-key Rotate key
```
**Health Check**:
```bash
curl http://localhost:8081/api/v1/kms/health
```
**Invocation**:
```bash
# Dev mode
export PROVISIONING_ENV=dev
./provisioning-vault-service
# Prod mode
export PROVISIONING_ENV=prod
export COSMIAN_KMS_URL=https://cosmian.example.com
export COSMIAN_API_KEY=your-key
./provisioning-vault-service
```
---
### 9. Detector
**Binary**: provisioning-detector
**Port**: 8600
**Size**: TBD
**Purpose**: Infrastructure detection and system discovery.
**Key Features**:
- Infrastructure capability detection
- System environment analysis
- Cloud provider detection
- Resource availability checking
- Compatibility checking
**Tech Stack** (Minimal):
- CLI: clap
- Serialization: serde_json
- Error Handling: anyhow, thiserror
- File Ops: walkdir
- Regex: regex
- Time: chrono
**Note**: CLI-based tool or library with optional CLI interface.
**Expected CLI Interface**:
```bash
./provisioning-detector [OPTIONS] [COMMAND]
```
**Health Check**: N/A
**Invocation**:
```bash
./provisioning-detector detect --config config/detector.toml
```
---
### 10. Control Center UI
**Build Artifact**: control-center-ui
**Port**: 3000
**Size**: N/A (WASM/JavaScript)
**⚠️ IMPORTANT**: NOT a Rust binary. WASM application running in browser.
**Purpose**: Web dashboard for provisioning platform management.
**Key Features**:
- Real-time dashboard (WebSocket)
- User authentication
- Secrets management UI
- Rule editor
- Deployment management
- MFA setup (TOTP, WebAuthn)
- Multi-language support
- Charts and metrics
- Real-time notifications
**Tech Stack** (WASM):
- Framework: Leptos (CSR - Client-Side Rendering)
- Target: WebAssembly
- Build: wasm-pack
- Routing: leptos_router
- UI: leptos_icons
- Charts: plotters-canvas
- Crypto: AES-GCM, HMAC, SHA2
- Auth: TOTP, WebAuthn
- Network: gloo-net
- Storage: gloo-storage
- Logging: tracing-wasm
**Build Instructions**:
```bash
# Install wasm-pack
cargo install wasm-pack
# Build WASM
cd provisioning/platform/crates/control-center-ui
wasm-pack build --target web --release
# Output: dist/ directory
```
**Development Server**:
```bash
# Option 1: wasm-pack serve
wasm-pack serve
# Option 2: Python
cd dist && python -m http.server 3000
# Option 3: Node.js
npx http-server dist -p 3000
```
**Deployment**: Requires web server (nginx, Apache, CDN).
**Connection to Backend**:
```
Control Center UI (port 3000)
├── HTTP → Control Center (port 8000)
│ POST /auth/login
│ GET /secrets
│ GET /deployments
└── WebSocket → Control Center (port 8000)
ws://localhost:8000/ws
```
**Health Check**: N/A (static WASM)
**Browser Support**: Chrome/Edge/Firefox/Safari (WebAssembly required)
**Access**:
```bash
# After building and serving:
# Open http://localhost:3000 in browser
# Login with credentials
# Access dashboard
```
---
## Additional Information
### Service Dependencies
- **Control Center** → SurrealDB, Redis (optional)
- **AI Service** → RAG service, LLM providers
- **RAG Engine** → SurrealDB, LLM/Embeddings
- **Vault** → Backend KMS (Cosmian, RustyVault, etc.)
- **MCP Server** → RAG, other services (API)
- **Daemon** → daemon-cli library (prov-ecosystem)
### Health Checks Summary
| Service | Endpoint | Type | Timeout |
|---------|----------|------|---------|
| control-center | /health | HTTP | 10s |
| orchestrator | /api/v1/health | HTTP | 10s |
| extension-registry | /api/v1/health | HTTP | 10s |
| ai-service | /health | HTTP | 10s |
| provisioning-daemon | /api/v1/health | HTTP | 10s |
| provisioning-rag | /health | HTTP | 10s |
| vault-service | /api/v1/kms/health | HTTP | 10s |
| mcp-server | N/A | Binary Protocol | N/A |
| detector | N/A | CLI/Library | N/A |
| control-center-ui | N/A | WASM | N/A |
---
**Last Updated**: 2025-02-03
**Total Services**: 10
**Endpoints Documented**: 50+
**Status**: ✅ Complete Inventory