docs: add complete platform services documentation
- Add Platform Services Inventory with all 10 services and 50+ endpoints - Add Local Services Setup Guide with build, config, and troubleshooting - Add start-local-binaries.nu automation script with dependency resolution - Update SUMMARY.md navigation for new operational documentation This resolves layout convention violations by: - Moving root markdown files to operations/ directory - Using lowercase kebab-case for all filenames - Consolidating operational documentation - Updating navigation index Coverage: - All 10 services documented: vault, registry, control-center, rag, ai, mcp, daemon, orchestrator, detector, ui - Service dependencies tracked and visualized - Local development workflows documented - Nushell automation script with core/all/custom service groups
This commit is contained in:
Jesús Pérez
parent
ef4e428485
commit
334f351fc5
4 changed files with 2081 additions and 0 deletions
|
|
@ -27,6 +27,7 @@
|
||||||
# User Guides
|
# User Guides
|
||||||
|
|
||||||
- [Guides Overview](guides/README.md)
|
- [Guides Overview](guides/README.md)
|
||||||
|
- [TTY Command Workflow Guide](guides/tty-command-workflow-guide.md)
|
||||||
- [From Scratch Guide](guides/from-scratch.md)
|
- [From Scratch Guide](guides/from-scratch.md)
|
||||||
- [Workspace Management](guides/workspace-management.md)
|
- [Workspace Management](guides/workspace-management.md)
|
||||||
- [Multi-Cloud Deployment](guides/multi-cloud-deployment.md)
|
- [Multi-Cloud Deployment](guides/multi-cloud-deployment.md)
|
||||||
|
|
@ -40,11 +41,14 @@
|
||||||
- [Infrastructure Overview](infrastructure/README.md)
|
- [Infrastructure Overview](infrastructure/README.md)
|
||||||
- [Nickel Guide](infrastructure/nickel-guide.md)
|
- [Nickel Guide](infrastructure/nickel-guide.md)
|
||||||
- [Configuration System](infrastructure/configuration-system.md)
|
- [Configuration System](infrastructure/configuration-system.md)
|
||||||
|
- [CLI Unified Configuration](infrastructure/cli-unified-configuration.md)
|
||||||
|
- [Orchestrator Storage Backends](infrastructure/orchestrator-storage-backends.md)
|
||||||
- [Schemas Reference](infrastructure/schemas-reference.md)
|
- [Schemas Reference](infrastructure/schemas-reference.md)
|
||||||
- [Providers](infrastructure/providers.md)
|
- [Providers](infrastructure/providers.md)
|
||||||
- [Task Services](infrastructure/task-services.md)
|
- [Task Services](infrastructure/task-services.md)
|
||||||
- [Clusters](infrastructure/clusters.md)
|
- [Clusters](infrastructure/clusters.md)
|
||||||
- [Batch Workflows](infrastructure/batch-workflows.md)
|
- [Batch Workflows](infrastructure/batch-workflows.md)
|
||||||
|
- [Docker Build System](infrastructure/docker-builds.md)
|
||||||
- [Version Management](infrastructure/version-management.md)
|
- [Version Management](infrastructure/version-management.md)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -54,6 +58,7 @@
|
||||||
- [Features Overview](features/README.md)
|
- [Features Overview](features/README.md)
|
||||||
- [Workspace Management](features/workspace-management.md)
|
- [Workspace Management](features/workspace-management.md)
|
||||||
- [CLI Architecture](features/cli-architecture.md)
|
- [CLI Architecture](features/cli-architecture.md)
|
||||||
|
- [TTY Command Flow Management](features/tty-command-flow-management.md)
|
||||||
- [Configuration System](features/configuration-system.md)
|
- [Configuration System](features/configuration-system.md)
|
||||||
- [Batch Workflows](features/batch-workflows.md)
|
- [Batch Workflows](features/batch-workflows.md)
|
||||||
- [Orchestrator](features/orchestrator.md)
|
- [Orchestrator](features/orchestrator.md)
|
||||||
|
|
@ -70,8 +75,11 @@
|
||||||
# Operations
|
# Operations
|
||||||
|
|
||||||
- [Operations Overview](operations/README.md)
|
- [Operations Overview](operations/README.md)
|
||||||
|
- [Deployment Methods](operations/deployment-methods.md)
|
||||||
- [Deployment Modes](operations/deployment-modes.md)
|
- [Deployment Modes](operations/deployment-modes.md)
|
||||||
- [Service Management](operations/service-management.md)
|
- [Service Management](operations/service-management.md)
|
||||||
|
- [Platform Services Inventory](operations/services-local-deployment.md)
|
||||||
|
- [Local Services Setup](operations/local-services-setup.md)
|
||||||
- [Monitoring](operations/monitoring.md)
|
- [Monitoring](operations/monitoring.md)
|
||||||
- [Backup & Recovery](operations/backup-recovery.md)
|
- [Backup & Recovery](operations/backup-recovery.md)
|
||||||
- [Upgrade](operations/upgrade.md)
|
- [Upgrade](operations/upgrade.md)
|
||||||
|
|
@ -85,6 +93,7 @@
|
||||||
- [Security Overview](security/README.md)
|
- [Security Overview](security/README.md)
|
||||||
- [Authentication](security/authentication.md)
|
- [Authentication](security/authentication.md)
|
||||||
- [Authorization](security/authorization.md)
|
- [Authorization](security/authorization.md)
|
||||||
|
- [Control Center Security](security/control-center-security.md)
|
||||||
- [Multi-Factor Authentication](security/mfa.md)
|
- [Multi-Factor Authentication](security/mfa.md)
|
||||||
- [Audit Logging](security/audit-logging.md)
|
- [Audit Logging](security/audit-logging.md)
|
||||||
- [KMS Guide](security/kms-guide.md)
|
- [KMS Guide](security/kms-guide.md)
|
||||||
|
|
@ -101,6 +110,8 @@
|
||||||
# Development
|
# Development
|
||||||
|
|
||||||
- [Development Overview](development/README.md)
|
- [Development Overview](development/README.md)
|
||||||
|
- [Extension Loading Guide](development/extension-loading-guide.md)
|
||||||
|
- [TTY Wrapper Development](development/tty-wrapper-development.md)
|
||||||
- [Extension Development](development/extension-development.md)
|
- [Extension Development](development/extension-development.md)
|
||||||
- [Provider Development](development/provider-development.md)
|
- [Provider Development](development/provider-development.md)
|
||||||
- [Plugin Development](development/plugin-development.md)
|
- [Plugin Development](development/plugin-development.md)
|
||||||
|
|
@ -148,6 +159,7 @@
|
||||||
# Troubleshooting
|
# Troubleshooting
|
||||||
|
|
||||||
- [Troubleshooting Overview](troubleshooting/README.md)
|
- [Troubleshooting Overview](troubleshooting/README.md)
|
||||||
|
- [Known Issues](troubleshooting/known-issues.md)
|
||||||
- [Common Issues](troubleshooting/common-issues.md)
|
- [Common Issues](troubleshooting/common-issues.md)
|
||||||
- [Debug Guide](troubleshooting/debug-guide.md)
|
- [Debug Guide](troubleshooting/debug-guide.md)
|
||||||
- [Logs Analysis](troubleshooting/logs-analysis.md)
|
- [Logs Analysis](troubleshooting/logs-analysis.md)
|
||||||
|
|
|
||||||
851
docs/src/operations/local-services-setup.md
Normal file
851
docs/src/operations/local-services-setup.md
Normal file
|
|
@ -0,0 +1,851 @@
|
||||||
|
# Local Services Setup Guide
|
||||||
|
|
||||||
|
Local development setup for running all 10 provisioning platform services locally. This guide covers building, configuring, and running services on your development machine.
|
||||||
|
|
||||||
|
**Audience**: Developers setting up local development environment
|
||||||
|
**Prerequisites**: Rust 1.80+, Nushell 0.110.0+, Docker/Podman, Git
|
||||||
|
**Estimated setup time**: 30-45 minutes
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
1. [Prerequisites](#prerequisites)
|
||||||
|
2. [Repository Structure](#repository-structure)
|
||||||
|
3. [Build All Services](#build-all-services)
|
||||||
|
4. [Service Dependencies](#service-dependencies)
|
||||||
|
5. [Configuration Files](#configuration-files)
|
||||||
|
6. [Database & Storage Setup](#database--storage-setup)
|
||||||
|
7. [Security Keys Setup](#security-keys-setup)
|
||||||
|
8. [Running Services](#running-services)
|
||||||
|
9. [Troubleshooting](#troubleshooting)
|
||||||
|
10. [Development Workflows](#development-workflows)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
### Required Tools
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Rust toolchain (1.80 or later)
|
||||||
|
rustc --version # Should be 1.80+
|
||||||
|
cargo --version
|
||||||
|
|
||||||
|
# Nushell scripting runtime
|
||||||
|
nu --version # Should be 0.110.0+
|
||||||
|
|
||||||
|
# Container runtime (choose one)
|
||||||
|
docker --version
|
||||||
|
# OR
|
||||||
|
podman --version
|
||||||
|
|
||||||
|
# Git version control
|
||||||
|
git --version
|
||||||
|
|
||||||
|
# Optional but recommended
|
||||||
|
k9s --version # Kubernetes dashboard
|
||||||
|
kind --version # Local Kubernetes cluster (for k8s tests)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Environment Setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Set working directory to project root
|
||||||
|
cd /Users/Akasha/project-provisioning
|
||||||
|
|
||||||
|
# Set NICKEL import path for IaC validation
|
||||||
|
export NICKEL_IMPORT_PATH=/Users/Akasha/project-provisioning/provisioning
|
||||||
|
|
||||||
|
# Add scripts to PATH for convenience
|
||||||
|
export PATH="$PWD/provisioning/scripts:$PATH"
|
||||||
|
```
|
||||||
|
|
||||||
|
### System Requirements
|
||||||
|
|
||||||
|
- **CPU**: 8+ cores recommended
|
||||||
|
- **RAM**: 16+ GB (for running all 10 services simultaneously)
|
||||||
|
- **Disk**: 30+ GB free (for builds and artifacts)
|
||||||
|
- **macOS**: 12.0+ or Linux (Ubuntu 22.04+) recommended
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Repository Structure
|
||||||
|
|
||||||
|
### Core Provisioning Platform
|
||||||
|
|
||||||
|
```
|
||||||
|
provisioning/
|
||||||
|
├── core/ # Core CLI and libraries
|
||||||
|
│ ├── cli/ # CLI implementation
|
||||||
|
│ ├── libs/ # Shared libraries
|
||||||
|
│ └── plugins/ # Plugin system
|
||||||
|
├── platform/ # Platform services (10 crates)
|
||||||
|
│ ├── crates/
|
||||||
|
│ │ ├── orchestrator/ # Batch workflow orchestrator
|
||||||
|
│ │ ├── control-center/ # Control plane + auth
|
||||||
|
│ │ ├── extension-registry/ # OCI registry
|
||||||
|
│ │ ├── provisioning-daemon/ # Nushell execution
|
||||||
|
│ │ ├── ai-service/ # RAG + MCP tools
|
||||||
|
│ │ ├── provisioning-rag/ # Vector search
|
||||||
|
│ │ ├── mcp-server/ # Infrastructure automation
|
||||||
|
│ │ ├── vault-service/ # KMS + encryption
|
||||||
|
│ │ ├── detector/ # Infrastructure detection
|
||||||
|
│ │ └── control-center-ui/ # Web UI (Leptos/WASM)
|
||||||
|
│ └── Cargo.workspace.toml # Workspace definition
|
||||||
|
├── schemas/ # Nickel IaC schemas
|
||||||
|
├── extensions/ # Provider extensions
|
||||||
|
├── config/ # Configuration templates
|
||||||
|
└── scripts/ # Helper scripts
|
||||||
|
```
|
||||||
|
|
||||||
|
### Workspace Infrastructure
|
||||||
|
|
||||||
|
```
|
||||||
|
workspaces/
|
||||||
|
├── docker-desktop/ # Docker Desktop K8s config
|
||||||
|
├── kind-cluster/ # Kind K8s cluster config
|
||||||
|
├── local-services/ # Local service definitions
|
||||||
|
└── README.md # Workspace documentation
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Build All Services
|
||||||
|
|
||||||
|
### Quick Start: Build Everything
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Navigate to workspace
|
||||||
|
cd provisioning/platform
|
||||||
|
|
||||||
|
# Build all services in release mode (optimized)
|
||||||
|
cargo build --release --workspace
|
||||||
|
|
||||||
|
# Build specific service
|
||||||
|
cargo build --release -p orchestrator
|
||||||
|
|
||||||
|
# Build with development profile (faster, debug symbols)
|
||||||
|
cargo build --workspace
|
||||||
|
```
|
||||||
|
|
||||||
|
### Build Time Estimates
|
||||||
|
|
||||||
|
- **Full workspace rebuild**: 8-12 minutes (first time)
|
||||||
|
- **Incremental rebuild**: 1-3 minutes (after changes)
|
||||||
|
- **Single service**: 2-5 minutes
|
||||||
|
- **Clean rebuild**: 15-20 minutes
|
||||||
|
|
||||||
|
### Build Optimization
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Use parallel compilation (adjust for your CPU)
|
||||||
|
export CARGO_BUILD_JOBS=8
|
||||||
|
|
||||||
|
# Enable incremental compilation
|
||||||
|
export CARGO_INCREMENTAL=1
|
||||||
|
|
||||||
|
# Use sccache for caching (speeds up rebuilds)
|
||||||
|
cargo install sccache
|
||||||
|
export RUSTC_WRAPPER=sccache
|
||||||
|
|
||||||
|
# Rebuild with these optimizations
|
||||||
|
cargo build --release --workspace
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify Build Success
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check compiled binaries
|
||||||
|
ls -lh provisioning/platform/target/release/
|
||||||
|
|
||||||
|
# Expected binaries (examples):
|
||||||
|
# orchestrator
|
||||||
|
# control-center
|
||||||
|
# extension-registry
|
||||||
|
# provisioning-daemon
|
||||||
|
# ai-service
|
||||||
|
# vault-service
|
||||||
|
# mcp-server
|
||||||
|
# detector
|
||||||
|
# provisioning-rag
|
||||||
|
# control-center-ui
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Service Dependencies
|
||||||
|
|
||||||
|
### Service Startup Order (Critical)
|
||||||
|
|
||||||
|
Services have dependencies and must start in specific order:
|
||||||
|
|
||||||
|
```
|
||||||
|
1. vault-service (port 8081) # Auth dependencies
|
||||||
|
└─ Required by: All services
|
||||||
|
|
||||||
|
2. extension-registry (port 8082) # OCI registry
|
||||||
|
└─ Required by: orchestrator, provisioning-daemon
|
||||||
|
|
||||||
|
3. control-center (port 8000) # Core control plane
|
||||||
|
├─ Requires: vault-service
|
||||||
|
└─ Required by: Most services
|
||||||
|
|
||||||
|
4. provisioning-rag (port 8300) # Vector search DB
|
||||||
|
└─ Required by: ai-service
|
||||||
|
|
||||||
|
5. ai-service (port 8083) # RAG + MCP
|
||||||
|
├─ Requires: provisioning-rag, vault-service
|
||||||
|
└─ Required by: orchestrator
|
||||||
|
|
||||||
|
6. mcp-server (port 8400) # Infrastructure automation
|
||||||
|
├─ Requires: vault-service
|
||||||
|
└─ Used by: orchestrator
|
||||||
|
|
||||||
|
7. orchestrator (port 9090) # Batch workflows
|
||||||
|
├─ Requires: extension-registry, control-center, ai-service
|
||||||
|
└─ Core service
|
||||||
|
|
||||||
|
8. provisioning-daemon (port 8100) # Script execution
|
||||||
|
├─ Requires: vault-service
|
||||||
|
└─ Executes: Nushell scripts
|
||||||
|
|
||||||
|
9. detector (port 8600) # Infrastructure detection
|
||||||
|
├─ Requires: vault-service
|
||||||
|
└─ Optional for: Multi-cloud detection
|
||||||
|
|
||||||
|
10. control-center-ui (port 3000) # Web UI
|
||||||
|
├─ Requires: control-center
|
||||||
|
└─ Access point: http://localhost:3000
|
||||||
|
```
|
||||||
|
|
||||||
|
### Dependency Graph
|
||||||
|
|
||||||
|
```
|
||||||
|
vault-service (8081)
|
||||||
|
↓
|
||||||
|
├─→ control-center (8000)
|
||||||
|
│ ├─→ provisioning-daemon (8100)
|
||||||
|
│ └─→ orchestrator (9090)
|
||||||
|
│
|
||||||
|
├─→ ai-service (8083)
|
||||||
|
│ └─ provisioning-rag (8300)
|
||||||
|
│
|
||||||
|
├─→ mcp-server (8400)
|
||||||
|
│
|
||||||
|
└─→ detector (8600)
|
||||||
|
|
||||||
|
extension-registry (8082)
|
||||||
|
└─→ orchestrator (9090)
|
||||||
|
|
||||||
|
control-center-ui (3000)
|
||||||
|
└─→ control-center (8000)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Startup Script
|
||||||
|
|
||||||
|
Use the provided script to start services in correct order:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start core services only (vault, registry, control-center)
|
||||||
|
provisioning-scripts/start-local-binaries.nu --services core
|
||||||
|
|
||||||
|
# Start all services
|
||||||
|
provisioning-scripts/start-local-binaries.nu --services all
|
||||||
|
|
||||||
|
# Start specific services
|
||||||
|
provisioning-scripts/start-local-binaries.nu --services custom orchestrator control-center ai-service
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Configuration Files
|
||||||
|
|
||||||
|
### Service Configuration Locations
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Configuration directory
|
||||||
|
~/.config/provisioning/
|
||||||
|
|
||||||
|
# Create required directories
|
||||||
|
mkdir -p ~/.config/provisioning/{dev,local}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Nickel Schema Files (Source of Truth)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Nickel IaC schemas - Use ONLY these, never hardcode
|
||||||
|
provisioning/schemas/
|
||||||
|
├── main.ncl # Main schema entry point
|
||||||
|
├── platform/
|
||||||
|
│ ├── schemas/ # Service schemas (validated)
|
||||||
|
│ │ ├── orchestrator.ncl
|
||||||
|
│ │ ├── control-center.ncl
|
||||||
|
│ │ ├── ai-service.ncl
|
||||||
|
│ │ └── ...
|
||||||
|
│ └── defaults/ # Default configurations
|
||||||
|
│ ├── orchestrator-defaults.ncl
|
||||||
|
│ ├── service-defaults.ncl
|
||||||
|
│ └── ...
|
||||||
|
└── common/
|
||||||
|
└── helpers.ncl # Composition utilities
|
||||||
|
```
|
||||||
|
|
||||||
|
### Load Hierarchy (High to Low)
|
||||||
|
|
||||||
|
1. **CLI Arguments** - `--config`, `--port`, etc.
|
||||||
|
2. **Environment Variables** - `PROVISIONING_*`
|
||||||
|
3. **User Configuration** - `~/.config/provisioning/user-config.ncl`
|
||||||
|
4. **Infrastructure Configuration** - Nickel schemas
|
||||||
|
5. **System Defaults** - `provisioning/config/defaults.toml`
|
||||||
|
|
||||||
|
### Example: Loading Orchestrator Config
|
||||||
|
|
||||||
|
```nickel
|
||||||
|
# Load schemas
|
||||||
|
let orchestrator_schema = import "schemas/platform/schemas/orchestrator.ncl" in
|
||||||
|
let defaults = import "schemas/platform/defaults/orchestrator-defaults.ncl" in
|
||||||
|
let user_overrides = import "~/.config/provisioning/orchestrator-config.ncl" in
|
||||||
|
let helpers = import "schemas/platform/common/helpers.ncl" in
|
||||||
|
|
||||||
|
# Compose config (deep merge: defaults → user overrides)
|
||||||
|
let config = helpers.compose_config defaults user_overrides in
|
||||||
|
config | orchestrator_schema.OrchestratorConfig
|
||||||
|
```
|
||||||
|
|
||||||
|
### Deep Merge vs Shallow Merge
|
||||||
|
|
||||||
|
⚠️ **CRITICAL**: Always use `helpers.compose_config` (deep merge), NEVER use `&` operator (shallow):
|
||||||
|
|
||||||
|
```nickel
|
||||||
|
# ❌ WRONG - Shallow merge (loses nested fields)
|
||||||
|
{server = {port = 9000}} & {server = {workers = 4}}
|
||||||
|
# Result: {server = {workers = 4}} ← port lost!
|
||||||
|
|
||||||
|
# ✅ CORRECT - Deep merge (preserves nested fields)
|
||||||
|
helpers.compose_config
|
||||||
|
{server = {port = 9000}}
|
||||||
|
{server = {workers = 4}}
|
||||||
|
# Result: {server = {port = 9000, workers = 4}}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Database & Storage Setup
|
||||||
|
|
||||||
|
### PostgreSQL (if required)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start PostgreSQL in Docker (optional)
|
||||||
|
docker run -d \
|
||||||
|
--name provisioning-postgres \
|
||||||
|
-e POSTGRES_PASSWORD=dev-password \
|
||||||
|
-e POSTGRES_USER=provisioning \
|
||||||
|
-e POSTGRES_DB=provisioning_db \
|
||||||
|
-p 5432:5432 \
|
||||||
|
postgres:16-alpine
|
||||||
|
|
||||||
|
# Wait for postgres to be ready
|
||||||
|
sleep 5
|
||||||
|
|
||||||
|
# Verify connection
|
||||||
|
psql -h localhost -U provisioning -d provisioning_db -c "SELECT 1"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Vector Database (for ai-service/provisioning-rag)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Option 1: Use in-memory (for development)
|
||||||
|
# Set environment: VECTOR_DB_MODE=memory
|
||||||
|
|
||||||
|
# Option 2: Use Postgres with pgvector extension
|
||||||
|
# Set environment: VECTOR_DB_MODE=postgres
|
||||||
|
|
||||||
|
# Option 3: Use external vector service
|
||||||
|
# Set environment: VECTOR_DB_URL=http://localhost:8300
|
||||||
|
```
|
||||||
|
|
||||||
|
### Redis Cache (for orchestrator)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start Redis (optional, for distributed caching)
|
||||||
|
docker run -d \
|
||||||
|
--name provisioning-redis \
|
||||||
|
-p 6379:6379 \
|
||||||
|
redis:7-alpine
|
||||||
|
|
||||||
|
# Verify connection
|
||||||
|
redis-cli ping # Should return PONG
|
||||||
|
```
|
||||||
|
|
||||||
|
### Cleanup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Stop all containers
|
||||||
|
docker stop provisioning-postgres provisioning-redis
|
||||||
|
|
||||||
|
# Remove containers
|
||||||
|
docker rm provisioning-postgres provisioning-redis
|
||||||
|
|
||||||
|
# Remove images
|
||||||
|
docker rmi postgres:16-alpine redis:7-alpine
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Security Keys Setup
|
||||||
|
|
||||||
|
### Generate Required Keys
|
||||||
|
|
||||||
|
All services require cryptographic keys for encryption, signing, and authentication.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create keys directory
|
||||||
|
mkdir -p ~/.provisioning/keys/{dev,prod}
|
||||||
|
cd ~/.provisioning/keys/dev
|
||||||
|
|
||||||
|
# Install Age (required for encryption)
|
||||||
|
# On macOS:
|
||||||
|
brew install age
|
||||||
|
|
||||||
|
# Or download: https://github.com/FiloSottile/age/releases
|
||||||
|
```
|
||||||
|
|
||||||
|
### Age Encryption Keys (Master Keys)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate age keypair for development
|
||||||
|
age-keygen -o age-dev-key.txt
|
||||||
|
|
||||||
|
# Output format:
|
||||||
|
# age secret key: AGE-...[secret]
|
||||||
|
# Public key: age...[public]
|
||||||
|
|
||||||
|
# Store securely
|
||||||
|
chmod 600 age-dev-key.txt
|
||||||
|
export AGE_KEY=$(cat age-dev-key.txt | grep "^# public key:" | cut -d' ' -f4)
|
||||||
|
|
||||||
|
# For SOPS (encrypted config files)
|
||||||
|
echo "age: $AGE_KEY" > .sops.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### Service API Keys
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate random API keys for services
|
||||||
|
openssl rand -hex 32 > api-keys.txt
|
||||||
|
|
||||||
|
# Expected format:
|
||||||
|
# orchestrator-key: 8f7a9c2b1e4d6f3a9b2c1d4e6f7a8b9c
|
||||||
|
# control-center-key: 9c2b1e4d6f3a9b2c1d4e6f7a8b9c2b1e
|
||||||
|
```
|
||||||
|
|
||||||
|
### JWT Signing Keys
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate JWT keys (for control-center auth)
|
||||||
|
openssl genrsa -out jwt-private.pem 2048
|
||||||
|
openssl rsa -in jwt-private.pem -pubout -out jwt-public.pem
|
||||||
|
|
||||||
|
# Verify keys
|
||||||
|
ls -lh jwt-*.pem
|
||||||
|
```
|
||||||
|
|
||||||
|
### Load Keys into Environment
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create development environment file
|
||||||
|
cat > ~/.config/provisioning/dev-env.sh << 'EOF'
|
||||||
|
#!/bin/bash
|
||||||
|
export AGE_KEY=$(cat ~/.provisioning/keys/dev/age-dev-key.txt | grep "^# secret key:" | cut -d' ' -f4)
|
||||||
|
export PROVISIONING_API_KEY=$(cat ~/.provisioning/keys/dev/api-keys.txt | head -1 | cut -d: -f2)
|
||||||
|
export JWT_PRIVATE_KEY=$(cat ~/.provisioning/keys/dev/jwt-private.pem)
|
||||||
|
export JWT_PUBLIC_KEY=$(cat ~/.provisioning/keys/dev/jwt-public.pem)
|
||||||
|
EOF
|
||||||
|
|
||||||
|
chmod +x ~/.config/provisioning/dev-env.sh
|
||||||
|
source ~/.config/provisioning/dev-env.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Security Best Practices
|
||||||
|
|
||||||
|
- ❌ **Never commit keys** to Git (add to `.gitignore`)
|
||||||
|
- ✅ **Use SOPS** for encrypted config files
|
||||||
|
- ✅ **Rotate keys regularly** (monthly for dev, quarterly for prod)
|
||||||
|
- ✅ **Use separate keys** for each environment
|
||||||
|
- ✅ **Store in secure location** (use vault for production)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Running Services
|
||||||
|
|
||||||
|
### Method 1: Using Startup Script (Recommended)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start all services
|
||||||
|
./provisioning/scripts/start-local-binaries.nu --services all
|
||||||
|
|
||||||
|
# Start core services only (vault, registry, control-center)
|
||||||
|
./provisioning/scripts/start-local-binaries.nu --services core
|
||||||
|
|
||||||
|
# Start custom service set
|
||||||
|
./provisioning/scripts/start-local-binaries.nu --services custom vault-service control-center ai-service
|
||||||
|
|
||||||
|
# Available flags:
|
||||||
|
# --services <set> : core|all|custom (followed by service names)
|
||||||
|
# --debug : Enable debug logging
|
||||||
|
# --health-check : Enable health monitoring
|
||||||
|
# --config <file> : Load custom config file
|
||||||
|
```
|
||||||
|
|
||||||
|
### Method 2: Manual Start (for Debugging)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Terminal 1: Start vault-service
|
||||||
|
cd provisioning/platform
|
||||||
|
cargo run --release -p vault-service
|
||||||
|
|
||||||
|
# Terminal 2: Start control-center
|
||||||
|
cd provisioning/platform
|
||||||
|
cargo run --release -p control-center -- --config ~/.config/provisioning/control-center.toml
|
||||||
|
|
||||||
|
# Terminal 3: Start orchestrator
|
||||||
|
cd provisioning/platform
|
||||||
|
cargo run --release -p orchestrator
|
||||||
|
|
||||||
|
# Terminal 4+: Start other services similarly
|
||||||
|
```
|
||||||
|
|
||||||
|
### Health Checks
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check if services are running
|
||||||
|
curl -s http://localhost:8000/health
|
||||||
|
curl -s http://localhost:8081/health
|
||||||
|
curl -s http://localhost:9090/health
|
||||||
|
|
||||||
|
# Check all services at once
|
||||||
|
./provisioning/scripts/check-health.nu --verbose
|
||||||
|
|
||||||
|
# Monitor service status
|
||||||
|
watch -n 2 './provisioning/scripts/check-health.nu'
|
||||||
|
```
|
||||||
|
|
||||||
|
### View Logs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Using startup script
|
||||||
|
./provisioning/scripts/start-local-binaries.nu --logs
|
||||||
|
|
||||||
|
# Manual logs (if running in background)
|
||||||
|
tail -f ~/.provisioning/logs/orchestrator.log
|
||||||
|
tail -f ~/.provisioning/logs/control-center.log
|
||||||
|
|
||||||
|
# Stream all logs
|
||||||
|
tail -f ~/.provisioning/logs/*.log
|
||||||
|
```
|
||||||
|
|
||||||
|
### Stop All Services
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Using startup script
|
||||||
|
./provisioning/scripts/start-local-binaries.nu --stop
|
||||||
|
|
||||||
|
# Manual stop
|
||||||
|
pkill -f "cargo run"
|
||||||
|
pkill orchestrator control-center provisioning-daemon ai-service
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Service Won't Start
|
||||||
|
|
||||||
|
**Problem**: Service fails to start with port binding error
|
||||||
|
**Cause**: Port already in use
|
||||||
|
**Solution**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Find process using port (example: port 8000)
|
||||||
|
lsof -i :8000
|
||||||
|
# Kill process
|
||||||
|
kill -9 <PID>
|
||||||
|
|
||||||
|
# Or change service port (set in config)
|
||||||
|
export CONTROL_CENTER_PORT=8001
|
||||||
|
```
|
||||||
|
|
||||||
|
### Build Failures
|
||||||
|
|
||||||
|
**Problem**: `cargo build` fails with missing dependencies
|
||||||
|
**Solution**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Update dependencies
|
||||||
|
cargo update
|
||||||
|
|
||||||
|
# Clean and rebuild
|
||||||
|
cargo clean
|
||||||
|
cargo build --release --workspace
|
||||||
|
|
||||||
|
# Check for toolchain issues
|
||||||
|
rustup update
|
||||||
|
rustc --version # Verify 1.80+
|
||||||
|
```
|
||||||
|
|
||||||
|
**Problem**: WASM build for UI fails
|
||||||
|
**Solution**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Install WASM target
|
||||||
|
rustup target add wasm32-unknown-unknown
|
||||||
|
|
||||||
|
# Rebuild UI
|
||||||
|
cargo build --release -p control-center-ui --target wasm32-unknown-unknown
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration Errors
|
||||||
|
|
||||||
|
**Problem**: Nickel schema validation fails
|
||||||
|
**Solution**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Set NICKEL_IMPORT_PATH
|
||||||
|
export NICKEL_IMPORT_PATH=/Users/Akasha/project-provisioning/provisioning
|
||||||
|
|
||||||
|
# Validate schema
|
||||||
|
nickel export schemas/main.ncl > /tmp/schema.json
|
||||||
|
|
||||||
|
# Check specific config
|
||||||
|
nickel export schemas/platform/schemas/orchestrator.ncl > /tmp/orchestrator-schema.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### Connection Issues
|
||||||
|
|
||||||
|
**Problem**: Services can't communicate
|
||||||
|
**Solution**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check network connectivity
|
||||||
|
telnet localhost 8000
|
||||||
|
telnet localhost 8081
|
||||||
|
|
||||||
|
# Verify service is listening
|
||||||
|
netstat -an | grep 8000
|
||||||
|
|
||||||
|
# Check firewall rules
|
||||||
|
sudo pfctl -s all | grep localhost
|
||||||
|
```
|
||||||
|
|
||||||
|
### Database Connection Errors
|
||||||
|
|
||||||
|
**Problem**: PostgreSQL connection refused
|
||||||
|
**Solution**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check if postgres is running
|
||||||
|
docker ps | grep postgres
|
||||||
|
|
||||||
|
# Check logs
|
||||||
|
docker logs provisioning-postgres
|
||||||
|
|
||||||
|
# Restart postgres
|
||||||
|
docker restart provisioning-postgres
|
||||||
|
|
||||||
|
# Wait for startup
|
||||||
|
sleep 5
|
||||||
|
```
|
||||||
|
|
||||||
|
### Log Analysis
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Find errors in logs
|
||||||
|
grep ERROR ~/.provisioning/logs/*.log
|
||||||
|
|
||||||
|
# Check specific service
|
||||||
|
grep -A5 "vault-service" ~/.provisioning/logs/*.log
|
||||||
|
|
||||||
|
# Count errors by service
|
||||||
|
for file in ~/.provisioning/logs/*.log; do
|
||||||
|
echo "$file: $(grep ERROR $file | wc -l) errors"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Development Workflows
|
||||||
|
|
||||||
|
### Common Development Tasks
|
||||||
|
|
||||||
|
#### Adding a New Endpoint to Control Center
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Edit the service code
|
||||||
|
nano provisioning/platform/crates/control-center/src/main.rs
|
||||||
|
|
||||||
|
# 2. Rebuild
|
||||||
|
cargo build -p control-center
|
||||||
|
|
||||||
|
# 3. Test
|
||||||
|
cargo test -p control-center
|
||||||
|
|
||||||
|
# 4. Run and test endpoint
|
||||||
|
curl -X POST http://localhost:8000/api/v1/new-endpoint
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Modifying Nickel Schema
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Edit schema
|
||||||
|
nano provisioning/schemas/platform/schemas/orchestrator.ncl
|
||||||
|
|
||||||
|
# 2. Validate schema
|
||||||
|
export NICKEL_IMPORT_PATH=$PWD/provisioning
|
||||||
|
nickel export provisioning/schemas/platform/schemas/orchestrator.ncl
|
||||||
|
|
||||||
|
# 3. Regenerate TOML (if used)
|
||||||
|
nickel export provisioning/schemas/main.ncl > provisioning/config/generated.toml
|
||||||
|
|
||||||
|
# 4. Test with services
|
||||||
|
./provisioning/scripts/start-local-binaries.nu --services core
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Running Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run all tests
|
||||||
|
cargo test --workspace
|
||||||
|
|
||||||
|
# Run tests for specific service
|
||||||
|
cargo test -p orchestrator
|
||||||
|
|
||||||
|
# Run with output
|
||||||
|
cargo test --workspace -- --nocapture
|
||||||
|
|
||||||
|
# Run benchmarks
|
||||||
|
cargo bench --workspace
|
||||||
|
|
||||||
|
# Run with coverage (requires tarpaulin)
|
||||||
|
cargo install cargo-tarpaulin
|
||||||
|
cargo tarpaulin --workspace
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Code Formatting & Linting
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Format code
|
||||||
|
cargo fmt --all
|
||||||
|
|
||||||
|
# Check formatting
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
|
||||||
|
# Run clippy (lint)
|
||||||
|
cargo clippy --all -- -D warnings
|
||||||
|
|
||||||
|
# Fix clippy warnings automatically
|
||||||
|
cargo clippy --fix --allow-dirty --workspace
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Profiling Performance
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run with perf (Linux)
|
||||||
|
perf record -g cargo run --release -p orchestrator
|
||||||
|
perf report
|
||||||
|
|
||||||
|
# Or use flamegraph
|
||||||
|
cargo install flamegraph
|
||||||
|
cargo flamegraph -p orchestrator
|
||||||
|
|
||||||
|
# macOS profiling
|
||||||
|
sudo dtrace -c "cargo run --release -p orchestrator" -n 'syscall:::entry { @[execname] = count() }'
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick Reference
|
||||||
|
|
||||||
|
### Essential Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build everything
|
||||||
|
cd provisioning/platform && cargo build --release --workspace
|
||||||
|
|
||||||
|
# Start services
|
||||||
|
./provisioning/scripts/start-local-binaries.nu --services all
|
||||||
|
|
||||||
|
# Run tests
|
||||||
|
cargo test --workspace
|
||||||
|
|
||||||
|
# Format code
|
||||||
|
cargo fmt --all && cargo clippy --fix --allow-dirty --workspace
|
||||||
|
|
||||||
|
# Check health
|
||||||
|
./provisioning/scripts/check-health.nu
|
||||||
|
|
||||||
|
# View logs
|
||||||
|
tail -f ~/.provisioning/logs/*.log
|
||||||
|
|
||||||
|
# Stop all services
|
||||||
|
pkill -f "cargo run"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Service Ports Quick Reference
|
||||||
|
|
||||||
|
| Service | Port | Protocol |
|
||||||
|
|---------|------|----------|
|
||||||
|
| control-center | 8000 | HTTP/WebSocket |
|
||||||
|
| vault-service | 8081 | gRPC |
|
||||||
|
| extension-registry | 8082 | HTTP |
|
||||||
|
| ai-service | 8083 | HTTP |
|
||||||
|
| provisioning-daemon | 8100 | gRPC |
|
||||||
|
| provisioning-rag | 8300 | REST |
|
||||||
|
| mcp-server | 8400 | Binary |
|
||||||
|
| detector | 8600 | HTTP |
|
||||||
|
| orchestrator | 9090 | HTTP |
|
||||||
|
| control-center-ui | 3000 | HTTP (WASM) |
|
||||||
|
|
||||||
|
### Configuration File Locations
|
||||||
|
|
||||||
|
| Component | Config | Location |
|
||||||
|
|-----------|--------|----------|
|
||||||
|
| Orchestrator | TOML | `~/.config/provisioning/orchestrator.toml` |
|
||||||
|
| Control Center | TOML | `~/.config/provisioning/control-center.toml` |
|
||||||
|
| User Config | Nickel | `~/.config/provisioning/user-config.ncl` |
|
||||||
|
| Development Keys | Age | `~/.provisioning/keys/dev/` |
|
||||||
|
| Logs | Log files | `~/.provisioning/logs/` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Getting Help
|
||||||
|
|
||||||
|
### Resources
|
||||||
|
|
||||||
|
- **Documentation**: `provisioning/docs/src/`
|
||||||
|
- **Examples**: `provisioning/examples/`
|
||||||
|
- **Tests**: `provisioning/platform/crates/*/tests/`
|
||||||
|
- **Guides**: `provisioning/docs/src/guides/`
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
See [Troubleshooting](#troubleshooting) section above for:
|
||||||
|
- Service startup problems
|
||||||
|
- Build failures
|
||||||
|
- Configuration errors
|
||||||
|
- Connection issues
|
||||||
|
- Database problems
|
||||||
|
|
||||||
|
### Next Steps
|
||||||
|
|
||||||
|
1. Complete [Prerequisites](#prerequisites)
|
||||||
|
2. Follow [Build All Services](#build-all-services)
|
||||||
|
3. Setup [Security Keys](#security-keys-setup)
|
||||||
|
4. Run services using [Running Services](#running-services)
|
||||||
|
5. Verify health checks and logs
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Last Updated**: 2025-02-03
|
||||||
|
**Audience**: Local Development
|
||||||
|
**Status**: Production Ready
|
||||||
634
docs/src/operations/services-local-deployment.md
Normal file
634
docs/src/operations/services-local-deployment.md
Normal file
|
|
@ -0,0 +1,634 @@
|
||||||
|
# Platform Services - Local Deployment Inventory
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Complete inventory of all 10 provisioning platform services with deployment options, endpoints, and configuration details for local development and testing.
|
||||||
|
|
||||||
|
**Quick Facts**:
|
||||||
|
- **10 total platform services**
|
||||||
|
- **8 HTTP-based services** with REST API
|
||||||
|
- **1 binary protocol service** (MCP - Model Context Protocol)
|
||||||
|
- **1 WASM application** (Control Center UI)
|
||||||
|
- **Ports**: 3000, 8000, 8081-8083, 8100, 8300, 8400, 8600, 9090
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [Services Summary](#services-summary)
|
||||||
|
- [Port Quick Reference](#port-quick-reference)
|
||||||
|
- [Services by Category](#services-by-category)
|
||||||
|
- [Detailed Service Documentation](#detailed-service-documentation)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Services Summary
|
||||||
|
|
||||||
|
| # | Service | Binary | Port | Size | Purpose |
|
||||||
|
|---|---------|--------|------|------|---------|
|
||||||
|
| 1 | orchestrator | provisioning-orchestrator | 9090 | 33 MB | Batch workflows, task queue, rollback |
|
||||||
|
| 2 | extension-registry | extension-registry | 8082 | 2.2 MB | OCI-compliant registry proxy |
|
||||||
|
| 3 | control-center | provisioning-control-center | 8000 | TBD | JWT auth, user mgmt, secrets, WebSocket |
|
||||||
|
| 4 | provisioning-daemon | provisioning-daemon | 8100 | 7.9 MB | Nushell execution, config rendering |
|
||||||
|
| 5 | ai-service | ai-service | 8083 | 3.8 MB | RAG, MCP tools, extension DAGs |
|
||||||
|
| 6 | provisioning-rag | provisioning-rag | 8300 | 1.9 MB | Vector search, semantic retrieval |
|
||||||
|
| 7 | mcp-server | provisioning-mcp-server | 8400 | TBD | Infrastructure automation tools (binary protocol) |
|
||||||
|
| 8 | vault-service | provisioning-vault-service | 8081 | TBD | KMS, encryption, secrets (Age/Cosmian) |
|
||||||
|
| 9 | detector | provisioning-detector | 8600 | TBD | Infrastructure detection & discovery |
|
||||||
|
| 10 | control-center-ui | control-center-ui | 3000 | N/A | Web dashboard (WASM/Leptos) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Port Quick Reference
|
||||||
|
|
||||||
|
```
|
||||||
|
3000 control-center-ui (WASM app)
|
||||||
|
8000 control-center (JWT auth, secrets)
|
||||||
|
8001 [reserved]
|
||||||
|
8081 vault-service (KMS)
|
||||||
|
8082 extension-registry (OCI)
|
||||||
|
8083 ai-service (RAG, MCP)
|
||||||
|
8100 provisioning-daemon (Nushell)
|
||||||
|
8300 provisioning-rag (Vector DB)
|
||||||
|
8400 mcp-server (binary protocol)
|
||||||
|
8600 detector (detection)
|
||||||
|
9090 orchestrator (workflows)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Services by Category
|
||||||
|
|
||||||
|
### Core Services (Required)
|
||||||
|
|
||||||
|
- **orchestrator** (9090) - Batch workflows, task queue, rollback
|
||||||
|
- **control-center** (8000) - Authentication, authorization
|
||||||
|
- **extension-registry** (8082) - Extension management
|
||||||
|
|
||||||
|
### Platform Services
|
||||||
|
|
||||||
|
- **provisioning-daemon** (8100) - Nushell execution
|
||||||
|
- **vault-service** (8081) - Secrets, KMS
|
||||||
|
- **ai-service** (8083) - AI capabilities
|
||||||
|
|
||||||
|
### AI & Knowledge
|
||||||
|
|
||||||
|
- **provisioning-rag** (8300) - RAG engine
|
||||||
|
- **mcp-server** (8400) - Infrastructure tools
|
||||||
|
|
||||||
|
### Utilities
|
||||||
|
|
||||||
|
- **detector** (8600) - Infrastructure detection
|
||||||
|
- **control-center-ui** (3000) - Web dashboard
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Detailed Service Documentation
|
||||||
|
|
||||||
|
### 1. orchestrator
|
||||||
|
|
||||||
|
**Binary**: provisioning-orchestrator
|
||||||
|
**Port**: 9090
|
||||||
|
**Size**: 33 MB
|
||||||
|
|
||||||
|
**Purpose**: Manages distributed task execution, batch workflows, cluster provisioning, and disaster recovery.
|
||||||
|
|
||||||
|
**Key Features**:
|
||||||
|
- Batch workflow execution with task queue
|
||||||
|
- State management and snapshots
|
||||||
|
- Checkpoint creation and rollback
|
||||||
|
- Compliance and audit logging
|
||||||
|
- System metrics and health monitoring
|
||||||
|
|
||||||
|
**Key Endpoints**:
|
||||||
|
```
|
||||||
|
GET /api/v1/health Health check
|
||||||
|
GET /tasks List all tasks
|
||||||
|
GET /tasks/{id} Get task status
|
||||||
|
POST /batch/execute Execute batch operation
|
||||||
|
GET /batch/operations List operations
|
||||||
|
POST /rollback/checkpoints Create checkpoint
|
||||||
|
GET /rollback/checkpoints List checkpoints
|
||||||
|
POST /rollback/execute Execute rollback
|
||||||
|
GET /state/system/health System health
|
||||||
|
GET /state/system/metrics System metrics
|
||||||
|
```
|
||||||
|
|
||||||
|
**Health Check**:
|
||||||
|
```bash
|
||||||
|
curl http://localhost:9090/api/v1/health
|
||||||
|
```
|
||||||
|
|
||||||
|
**Invocation**:
|
||||||
|
```bash
|
||||||
|
./provisioning-orchestrator --port 9090
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2. extension-registry
|
||||||
|
|
||||||
|
**Binary**: extension-registry
|
||||||
|
**Port**: 8082
|
||||||
|
**Size**: 2.2 MB
|
||||||
|
|
||||||
|
**Purpose**: OCI v2 API-compliant registry proxy for managing extensions.
|
||||||
|
|
||||||
|
**Key Features**:
|
||||||
|
- OCI v2 API compliance
|
||||||
|
- Extension metadata management
|
||||||
|
- Blob and manifest handling
|
||||||
|
- Catalog browsing
|
||||||
|
|
||||||
|
**Key Endpoints**:
|
||||||
|
```
|
||||||
|
GET /api/v1/health Health check
|
||||||
|
GET /extensions List all extensions
|
||||||
|
GET /extensions/:name Get extension metadata
|
||||||
|
POST /extensions Register extension
|
||||||
|
GET /v2/_catalog OCI catalog
|
||||||
|
GET /v2/:name/tags/list List tags
|
||||||
|
GET /v2/:name/manifests/:ref Pull manifest
|
||||||
|
PUT /v2/:name/manifests/:ref Push manifest
|
||||||
|
```
|
||||||
|
|
||||||
|
**Health Check**:
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8082/api/v1/health
|
||||||
|
```
|
||||||
|
|
||||||
|
**Invocation**:
|
||||||
|
```bash
|
||||||
|
./extension-registry --port 8082 --host 127.0.0.1
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 3. Control Center
|
||||||
|
|
||||||
|
**Binary**: provisioning-control-center
|
||||||
|
**Port**: 8000
|
||||||
|
**Size**: TBD
|
||||||
|
|
||||||
|
**Purpose**: JWT authentication, user management, secrets management, WebSocket real-time events.
|
||||||
|
|
||||||
|
**Key Features**:
|
||||||
|
- JWT token generation and validation
|
||||||
|
- User CRUD operations
|
||||||
|
- Role-based access control
|
||||||
|
- Real-time WebSocket events
|
||||||
|
- Secrets management with versioning
|
||||||
|
- Multi-factor authentication (TOTP, WebAuthn)
|
||||||
|
- Audit logging
|
||||||
|
|
||||||
|
**Tech Stack**:
|
||||||
|
- Framework: Axum
|
||||||
|
- Database: SurrealDB, SQLx
|
||||||
|
- Auth: JWT, Argon2
|
||||||
|
- Crypto: AES-GCM, HMAC, RSA, SHA2
|
||||||
|
|
||||||
|
**Key Endpoints** (Public):
|
||||||
|
```
|
||||||
|
POST /auth/login User authentication
|
||||||
|
POST /auth/refresh Refresh token
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key Endpoints** (Protected):
|
||||||
|
```
|
||||||
|
POST /auth/logout Logout
|
||||||
|
GET /permissions List permissions
|
||||||
|
GET /deployments List deployments
|
||||||
|
POST /deployments Create deployment
|
||||||
|
GET /secrets List secrets
|
||||||
|
POST /secrets Create secret
|
||||||
|
GET /secrets/:path Get secret
|
||||||
|
PUT /secrets/:path Update secret
|
||||||
|
DELETE /secrets/:path Delete secret
|
||||||
|
GET /ws WebSocket (real-time events)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Health Check**:
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8000/health
|
||||||
|
```
|
||||||
|
|
||||||
|
**Invocation**:
|
||||||
|
```bash
|
||||||
|
./provisioning-control-center --port 8000 --config config/control-center.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4. Provisioning Daemon
|
||||||
|
|
||||||
|
**Binary**: provisioning-daemon
|
||||||
|
**Port**: 8100
|
||||||
|
**Size**: 7.9 MB
|
||||||
|
|
||||||
|
**Purpose**: Runtime service for executing Nushell scripts and rendering configuration templates.
|
||||||
|
|
||||||
|
**Key Features**:
|
||||||
|
- Execute Nushell scripts
|
||||||
|
- Render Nickel/TOML templates
|
||||||
|
- Configuration validation
|
||||||
|
- Operation execution framework
|
||||||
|
|
||||||
|
**Tech Stack**:
|
||||||
|
- Framework: Axum
|
||||||
|
- Core: daemon-cli library
|
||||||
|
- Languages: Nickel, TOML, JSON
|
||||||
|
|
||||||
|
**CLI Arguments**:
|
||||||
|
```bash
|
||||||
|
-c, --config <PATH> Config file
|
||||||
|
--config-dir <DIR> Config directory
|
||||||
|
-m, --mode <MODE> Mode: solo|multiuser|cicd|enterprise
|
||||||
|
-v, --verbose Verbose logging
|
||||||
|
--validate-config Validate and exit
|
||||||
|
--show-config Show config and exit
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key Endpoints**:
|
||||||
|
```
|
||||||
|
GET /api/v1/health Health check
|
||||||
|
POST /execute Execute Nushell script
|
||||||
|
POST /render Render template
|
||||||
|
GET /operations List operations
|
||||||
|
```
|
||||||
|
|
||||||
|
**Health Check**:
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8100/api/v1/health
|
||||||
|
```
|
||||||
|
|
||||||
|
**Invocation**:
|
||||||
|
```bash
|
||||||
|
./provisioning-daemon --config config/daemon.toml --mode solo
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 5. AI Service
|
||||||
|
|
||||||
|
**Binary**: ai-service
|
||||||
|
**Port**: 8083
|
||||||
|
**Size**: 3.8 MB
|
||||||
|
|
||||||
|
**Purpose**: HTTP service for AI capabilities including RAG, MCP tools, extension DAGs.
|
||||||
|
|
||||||
|
**Key Features**:
|
||||||
|
- Retrieval-Augmented Generation (RAG)
|
||||||
|
- MCP tool invocation
|
||||||
|
- Extension dependency graphs
|
||||||
|
- Best practice recommendations
|
||||||
|
|
||||||
|
**Tech Stack**:
|
||||||
|
- Framework: Axum
|
||||||
|
- RAG: RAG crate
|
||||||
|
- MCP: mcp-server crate
|
||||||
|
- LLM/Embeddings: Stratum
|
||||||
|
|
||||||
|
**CLI Arguments**:
|
||||||
|
```bash
|
||||||
|
-c, --config <PATH> Config file
|
||||||
|
--config-dir <DIR> Config directory
|
||||||
|
-m, --mode <MODE> Mode
|
||||||
|
-H, --host <HOST> Bind host (default: 127.0.0.1)
|
||||||
|
-p, --port <PORT> Bind port (default: 8083)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key Endpoints**:
|
||||||
|
```
|
||||||
|
POST /api/v1/ai/mcp/tool Call MCP tool
|
||||||
|
POST /api/v1/ai/ask RAG question answering
|
||||||
|
GET /api/v1/ai/dag/extensions Get DAG
|
||||||
|
GET /api/v1/ai/knowledge/best-practices Best practices
|
||||||
|
GET /health Health check
|
||||||
|
```
|
||||||
|
|
||||||
|
**Health Check**:
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8083/health
|
||||||
|
```
|
||||||
|
|
||||||
|
**Invocation**:
|
||||||
|
```bash
|
||||||
|
./ai-service --port 8083 --config config/ai-service.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 6. Provisioning RAG
|
||||||
|
|
||||||
|
**Binary**: provisioning-rag
|
||||||
|
**Port**: 8300
|
||||||
|
**Size**: 1.9 MB
|
||||||
|
|
||||||
|
**Purpose**: RAG engine with semantic search, hybrid search, conversation tracking.
|
||||||
|
|
||||||
|
**Key Features**:
|
||||||
|
- Semantic document search
|
||||||
|
- Hybrid search (BM25 + vector)
|
||||||
|
- Conversation tracking
|
||||||
|
- Batch query processing
|
||||||
|
- Response caching (LRU)
|
||||||
|
|
||||||
|
**Tech Stack**:
|
||||||
|
- Framework: Axum
|
||||||
|
- RAG: Rig + rig-surrealdb
|
||||||
|
- Vector DB: SurrealDB (HNSW)
|
||||||
|
- Embeddings: Stratum
|
||||||
|
- Hybrid: BM25 + semantic
|
||||||
|
|
||||||
|
**Key Endpoints**:
|
||||||
|
```
|
||||||
|
POST /query Semantic search
|
||||||
|
POST /conversations Start conversation
|
||||||
|
GET /conversations/{id} Get conversation
|
||||||
|
POST /conversations/{id}/turn Add turn
|
||||||
|
POST /batch/query Batch processing
|
||||||
|
GET /batch/{id}/status Batch status
|
||||||
|
GET /health Health check
|
||||||
|
GET /metrics Metrics
|
||||||
|
```
|
||||||
|
|
||||||
|
**Health Check**:
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8300/health
|
||||||
|
```
|
||||||
|
|
||||||
|
**Invocation**:
|
||||||
|
```bash
|
||||||
|
./provisioning-rag --config config/rag.toml --mode solo
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 7. MCP Server
|
||||||
|
|
||||||
|
**Binary**: provisioning-mcp-server
|
||||||
|
**Port**: 8400 (reserved - uses binary protocol)
|
||||||
|
|
||||||
|
**⚠️ IMPORTANT**: Uses Model Context Protocol (binary), NOT HTTP.
|
||||||
|
|
||||||
|
**Purpose**: Infrastructure automation tools, AI query integration, status/metrics retrieval.
|
||||||
|
|
||||||
|
**Key Features**:
|
||||||
|
- Infrastructure automation
|
||||||
|
- AI query integration
|
||||||
|
- Status and metrics
|
||||||
|
- Log retrieval
|
||||||
|
- Documentation finder
|
||||||
|
- Troubleshooting
|
||||||
|
|
||||||
|
**Tech Stack**:
|
||||||
|
- Protocol: Model Context Protocol (MCP)
|
||||||
|
- SDK: rust-mcp-sdk
|
||||||
|
- File Ops: walkdir
|
||||||
|
- Config: TOML, JSON, YAML
|
||||||
|
|
||||||
|
**Entry Point**: `simple_main.rs` (main.rs disabled)
|
||||||
|
|
||||||
|
**MCP Tools**:
|
||||||
|
```
|
||||||
|
provision_create_server Create server
|
||||||
|
provision_deploy_taskserv Deploy taskserv
|
||||||
|
provision_cluster_create Create cluster
|
||||||
|
ai_query AI query
|
||||||
|
get_infrastructure_status Infra status
|
||||||
|
get_system_metrics Metrics
|
||||||
|
get_logs Logs
|
||||||
|
```
|
||||||
|
|
||||||
|
**Health Check**: N/A (binary protocol)
|
||||||
|
|
||||||
|
**Invocation**: (Used by MCP clients, not directly)
|
||||||
|
```bash
|
||||||
|
./provisioning-mcp-server --config config/mcp-server.toml --mode solo
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 8. Vault Service
|
||||||
|
|
||||||
|
**Binary**: provisioning-vault-service
|
||||||
|
**Port**: 8081
|
||||||
|
**Size**: TBD
|
||||||
|
|
||||||
|
**Purpose**: KMS for secrets encryption, decryption, key generation, rotation.
|
||||||
|
|
||||||
|
**Key Features**:
|
||||||
|
- Encryption/decryption
|
||||||
|
- Key generation
|
||||||
|
- Key rotation
|
||||||
|
- Multi-backend (Age, Cosmian, RustyVault, SecretumVault)
|
||||||
|
- Environment-specific (dev/prod/enterprise)
|
||||||
|
|
||||||
|
**Tech Stack**:
|
||||||
|
- Framework: Axum
|
||||||
|
- Crypto: Age, Cosmian KMS, RustyVault
|
||||||
|
- Config: TOML + env vars
|
||||||
|
|
||||||
|
**Supported Backends**:
|
||||||
|
```
|
||||||
|
Age dev File-based encryption
|
||||||
|
Cosmian KMS prod Cloud-hosted SaaS
|
||||||
|
RustyVault enterprise Self-hosted
|
||||||
|
SecretumVault enterprise Commercial
|
||||||
|
```
|
||||||
|
|
||||||
|
**Environment Variables**:
|
||||||
|
```bash
|
||||||
|
KMS_BIND_ADDR Bind address (default: 0.0.0.0:8081)
|
||||||
|
KMS_CONFIG_PATH Config file
|
||||||
|
PROVISIONING_ENV Mode: dev|prod|enterprise
|
||||||
|
AGE_PUBLIC_KEY_PATH Age public key (dev)
|
||||||
|
AGE_PRIVATE_KEY_PATH Age private key (dev)
|
||||||
|
COSMIAN_KMS_URL Cosmian URL (prod - REQUIRED)
|
||||||
|
COSMIAN_API_KEY Cosmian key (prod - REQUIRED)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key Endpoints**:
|
||||||
|
```
|
||||||
|
GET /api/v1/kms/health Health check
|
||||||
|
GET /api/v1/kms/status KMS status
|
||||||
|
POST /api/v1/kms/encrypt Encrypt data
|
||||||
|
POST /api/v1/kms/decrypt Decrypt data
|
||||||
|
POST /api/v1/kms/generate-key Generate key
|
||||||
|
POST /api/v1/kms/rotate-key Rotate key
|
||||||
|
```
|
||||||
|
|
||||||
|
**Health Check**:
|
||||||
|
```bash
|
||||||
|
curl http://localhost:8081/api/v1/kms/health
|
||||||
|
```
|
||||||
|
|
||||||
|
**Invocation**:
|
||||||
|
```bash
|
||||||
|
# Dev mode
|
||||||
|
export PROVISIONING_ENV=dev
|
||||||
|
./provisioning-vault-service
|
||||||
|
|
||||||
|
# Prod mode
|
||||||
|
export PROVISIONING_ENV=prod
|
||||||
|
export COSMIAN_KMS_URL=https://cosmian.example.com
|
||||||
|
export COSMIAN_API_KEY=your-key
|
||||||
|
./provisioning-vault-service
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 9. Detector
|
||||||
|
|
||||||
|
**Binary**: provisioning-detector
|
||||||
|
**Port**: 8600
|
||||||
|
**Size**: TBD
|
||||||
|
|
||||||
|
**Purpose**: Infrastructure detection and system discovery.
|
||||||
|
|
||||||
|
**Key Features**:
|
||||||
|
- Infrastructure capability detection
|
||||||
|
- System environment analysis
|
||||||
|
- Cloud provider detection
|
||||||
|
- Resource availability checking
|
||||||
|
- Compatibility checking
|
||||||
|
|
||||||
|
**Tech Stack** (Minimal):
|
||||||
|
- CLI: clap
|
||||||
|
- Serialization: serde_json
|
||||||
|
- Error Handling: anyhow, thiserror
|
||||||
|
- File Ops: walkdir
|
||||||
|
- Regex: regex
|
||||||
|
- Time: chrono
|
||||||
|
|
||||||
|
**Note**: CLI-based tool or library with optional CLI interface.
|
||||||
|
|
||||||
|
**Expected CLI Interface**:
|
||||||
|
```bash
|
||||||
|
./provisioning-detector [OPTIONS] [COMMAND]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Health Check**: N/A
|
||||||
|
|
||||||
|
**Invocation**:
|
||||||
|
```bash
|
||||||
|
./provisioning-detector detect --config config/detector.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 10. Control Center UI
|
||||||
|
|
||||||
|
**Build Artifact**: control-center-ui
|
||||||
|
**Port**: 3000
|
||||||
|
**Size**: N/A (WASM/JavaScript)
|
||||||
|
|
||||||
|
**⚠️ IMPORTANT**: NOT a Rust binary. WASM application running in browser.
|
||||||
|
|
||||||
|
**Purpose**: Web dashboard for provisioning platform management.
|
||||||
|
|
||||||
|
**Key Features**:
|
||||||
|
- Real-time dashboard (WebSocket)
|
||||||
|
- User authentication
|
||||||
|
- Secrets management UI
|
||||||
|
- Rule editor
|
||||||
|
- Deployment management
|
||||||
|
- MFA setup (TOTP, WebAuthn)
|
||||||
|
- Multi-language support
|
||||||
|
- Charts and metrics
|
||||||
|
- Real-time notifications
|
||||||
|
|
||||||
|
**Tech Stack** (WASM):
|
||||||
|
- Framework: Leptos (CSR - Client-Side Rendering)
|
||||||
|
- Target: WebAssembly
|
||||||
|
- Build: wasm-pack
|
||||||
|
- Routing: leptos_router
|
||||||
|
- UI: leptos_icons
|
||||||
|
- Charts: plotters-canvas
|
||||||
|
- Crypto: AES-GCM, HMAC, SHA2
|
||||||
|
- Auth: TOTP, WebAuthn
|
||||||
|
- Network: gloo-net
|
||||||
|
- Storage: gloo-storage
|
||||||
|
- Logging: tracing-wasm
|
||||||
|
|
||||||
|
**Build Instructions**:
|
||||||
|
```bash
|
||||||
|
# Install wasm-pack
|
||||||
|
cargo install wasm-pack
|
||||||
|
|
||||||
|
# Build WASM
|
||||||
|
cd provisioning/platform/crates/control-center-ui
|
||||||
|
wasm-pack build --target web --release
|
||||||
|
|
||||||
|
# Output: dist/ directory
|
||||||
|
```
|
||||||
|
|
||||||
|
**Development Server**:
|
||||||
|
```bash
|
||||||
|
# Option 1: wasm-pack serve
|
||||||
|
wasm-pack serve
|
||||||
|
|
||||||
|
# Option 2: Python
|
||||||
|
cd dist && python -m http.server 3000
|
||||||
|
|
||||||
|
# Option 3: Node.js
|
||||||
|
npx http-server dist -p 3000
|
||||||
|
```
|
||||||
|
|
||||||
|
**Deployment**: Requires web server (nginx, Apache, CDN).
|
||||||
|
|
||||||
|
**Connection to Backend**:
|
||||||
|
```
|
||||||
|
Control Center UI (port 3000)
|
||||||
|
├── HTTP → Control Center (port 8000)
|
||||||
|
│ POST /auth/login
|
||||||
|
│ GET /secrets
|
||||||
|
│ GET /deployments
|
||||||
|
└── WebSocket → Control Center (port 8000)
|
||||||
|
ws://localhost:8000/ws
|
||||||
|
```
|
||||||
|
|
||||||
|
**Health Check**: N/A (static WASM)
|
||||||
|
|
||||||
|
**Browser Support**: Chrome/Edge/Firefox/Safari (WebAssembly required)
|
||||||
|
|
||||||
|
**Access**:
|
||||||
|
```bash
|
||||||
|
# After building and serving:
|
||||||
|
# Open http://localhost:3000 in browser
|
||||||
|
# Login with credentials
|
||||||
|
# Access dashboard
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Additional Information
|
||||||
|
|
||||||
|
### Service Dependencies
|
||||||
|
|
||||||
|
- **Control Center** → SurrealDB, Redis (optional)
|
||||||
|
- **AI Service** → RAG service, LLM providers
|
||||||
|
- **RAG Engine** → SurrealDB, LLM/Embeddings
|
||||||
|
- **Vault** → Backend KMS (Cosmian, RustyVault, etc.)
|
||||||
|
- **MCP Server** → RAG, other services (API)
|
||||||
|
- **Daemon** → daemon-cli library (prov-ecosystem)
|
||||||
|
|
||||||
|
### Health Checks Summary
|
||||||
|
|
||||||
|
| Service | Endpoint | Type | Timeout |
|
||||||
|
|---------|----------|------|---------|
|
||||||
|
| control-center | /health | HTTP | 10s |
|
||||||
|
| orchestrator | /api/v1/health | HTTP | 10s |
|
||||||
|
| extension-registry | /api/v1/health | HTTP | 10s |
|
||||||
|
| ai-service | /health | HTTP | 10s |
|
||||||
|
| provisioning-daemon | /api/v1/health | HTTP | 10s |
|
||||||
|
| provisioning-rag | /health | HTTP | 10s |
|
||||||
|
| vault-service | /api/v1/kms/health | HTTP | 10s |
|
||||||
|
| mcp-server | N/A | Binary Protocol | N/A |
|
||||||
|
| detector | N/A | CLI/Library | N/A |
|
||||||
|
| control-center-ui | N/A | WASM | N/A |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Last Updated**: 2025-02-03
|
||||||
|
**Total Services**: 10
|
||||||
|
**Endpoints Documented**: 50+
|
||||||
|
**Status**: ✅ Complete Inventory
|
||||||
584
scripts/start-local-binaries.nu
Executable file
584
scripts/start-local-binaries.nu
Executable file
|
|
@ -0,0 +1,584 @@
|
||||||
|
#!/usr/bin/env nu
|
||||||
|
# Start local provisioning platform services
|
||||||
|
# Usage: ./start-local-binaries.nu [FLAGS] [--services <set> [services...]]
|
||||||
|
#
|
||||||
|
# Services sets:
|
||||||
|
# core - vault-service, extension-registry, control-center
|
||||||
|
# all - All 10 services
|
||||||
|
# custom - Specified services (pass as arguments after --services custom)
|
||||||
|
#
|
||||||
|
# Examples:
|
||||||
|
# ./start-local-binaries.nu --services core
|
||||||
|
# ./start-local-binaries.nu --services all
|
||||||
|
# ./start-local-binaries.nu --services custom orchestrator control-center ai-service
|
||||||
|
|
||||||
|
# Color constants for terminal output
|
||||||
|
const COLOR_RESET = "\u{1b}[0m"
|
||||||
|
const COLOR_GREEN = "\u{1b}[32m"
|
||||||
|
const COLOR_YELLOW = "\u{1b}[33m"
|
||||||
|
const COLOR_RED = "\u{1b}[31m"
|
||||||
|
const COLOR_BLUE = "\u{1b}[34m"
|
||||||
|
const COLOR_CYAN = "\u{1b}[36m"
|
||||||
|
|
||||||
|
# Service registry with metadata
|
||||||
|
const SERVICES_REGISTRY = {
|
||||||
|
"vault-service": {
|
||||||
|
port: 8081,
|
||||||
|
protocol: "gRPC",
|
||||||
|
description: "Key management and encryption service",
|
||||||
|
depends_on: [],
|
||||||
|
binary: "vault-service"
|
||||||
|
},
|
||||||
|
"extension-registry": {
|
||||||
|
port: 8082,
|
||||||
|
protocol: "HTTP",
|
||||||
|
description: "OCI container registry for extensions",
|
||||||
|
depends_on: [],
|
||||||
|
binary: "extension-registry"
|
||||||
|
},
|
||||||
|
"control-center": {
|
||||||
|
port: 8000,
|
||||||
|
protocol: "HTTP/WebSocket",
|
||||||
|
description: "Core control plane with JWT auth",
|
||||||
|
depends_on: ["vault-service"],
|
||||||
|
binary: "control-center"
|
||||||
|
},
|
||||||
|
"provisioning-rag": {
|
||||||
|
port: 8300,
|
||||||
|
protocol: "REST",
|
||||||
|
description: "Vector search and RAG database",
|
||||||
|
depends_on: [],
|
||||||
|
binary: "provisioning-rag"
|
||||||
|
},
|
||||||
|
"ai-service": {
|
||||||
|
port: 8083,
|
||||||
|
protocol: "HTTP",
|
||||||
|
description: "AI service with RAG and MCP tools",
|
||||||
|
depends_on: ["provisioning-rag", "vault-service"],
|
||||||
|
binary: "ai-service"
|
||||||
|
},
|
||||||
|
"mcp-server": {
|
||||||
|
port: 8400,
|
||||||
|
protocol: "Binary",
|
||||||
|
description: "Infrastructure automation server",
|
||||||
|
depends_on: ["vault-service"],
|
||||||
|
binary: "mcp-server"
|
||||||
|
},
|
||||||
|
"provisioning-daemon": {
|
||||||
|
port: 8100,
|
||||||
|
protocol: "gRPC",
|
||||||
|
description: "Nushell script execution daemon",
|
||||||
|
depends_on: ["vault-service"],
|
||||||
|
binary: "provisioning-daemon"
|
||||||
|
},
|
||||||
|
"orchestrator": {
|
||||||
|
port: 9090,
|
||||||
|
protocol: "HTTP",
|
||||||
|
description: "Batch workflow orchestrator",
|
||||||
|
depends_on: ["extension-registry", "control-center", "ai-service"],
|
||||||
|
binary: "orchestrator"
|
||||||
|
},
|
||||||
|
"detector": {
|
||||||
|
port: 8600,
|
||||||
|
protocol: "HTTP",
|
||||||
|
description: "Infrastructure detection service",
|
||||||
|
depends_on: ["vault-service"],
|
||||||
|
binary: "detector"
|
||||||
|
},
|
||||||
|
"control-center-ui": {
|
||||||
|
port: 3000,
|
||||||
|
protocol: "HTTP (WASM)",
|
||||||
|
description: "Web UI dashboard (Leptos/WASM)",
|
||||||
|
depends_on: ["control-center"],
|
||||||
|
binary: "control-center-ui"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Service group definitions
|
||||||
|
const SERVICE_GROUPS = {
|
||||||
|
"core": ["vault-service", "extension-registry", "control-center"],
|
||||||
|
"all": [
|
||||||
|
"vault-service",
|
||||||
|
"extension-registry",
|
||||||
|
"control-center",
|
||||||
|
"provisioning-rag",
|
||||||
|
"ai-service",
|
||||||
|
"mcp-server",
|
||||||
|
"provisioning-daemon",
|
||||||
|
"orchestrator",
|
||||||
|
"detector",
|
||||||
|
"control-center-ui"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
# Utility functions
|
||||||
|
|
||||||
|
def log_info [message: string] {
|
||||||
|
print $"($COLOR_BLUE)ℹ($COLOR_RESET) ($message)"
|
||||||
|
}
|
||||||
|
|
||||||
|
def log_success [message: string] {
|
||||||
|
print $"($COLOR_GREEN)✓($COLOR_RESET) ($message)"
|
||||||
|
}
|
||||||
|
|
||||||
|
def log_warning [message: string] {
|
||||||
|
print $"($COLOR_YELLOW)⚠($COLOR_RESET) ($message)"
|
||||||
|
}
|
||||||
|
|
||||||
|
def log_error [message: string] {
|
||||||
|
print $"($COLOR_RED)✗($COLOR_RESET) ($message)"
|
||||||
|
}
|
||||||
|
|
||||||
|
def log_section [title: string] {
|
||||||
|
print $"($COLOR_CYAN)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━($COLOR_RESET)"
|
||||||
|
print $"($COLOR_CYAN)($title)($COLOR_RESET)"
|
||||||
|
print $"($COLOR_CYAN)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━($COLOR_RESET)"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Check if a port is available
|
||||||
|
def is_port_available [port: int] -> bool {
|
||||||
|
try {
|
||||||
|
let result = (
|
||||||
|
if ($nu.os-info.name == "macos") {
|
||||||
|
lsof -i -P -n | grep LISTEN | grep -c $":($port) " | into int
|
||||||
|
} else {
|
||||||
|
netstat -an | grep LISTEN | grep -c ":($port) " | into int
|
||||||
|
}
|
||||||
|
)
|
||||||
|
$result == 0
|
||||||
|
} catch {
|
||||||
|
true # Assume available if check fails
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Resolve service startup order (respecting dependencies)
|
||||||
|
def resolve_startup_order [services: list<string>] -> list<string> {
|
||||||
|
let mut ordered = []
|
||||||
|
let mut remaining = $services
|
||||||
|
let mut max_iterations = 100
|
||||||
|
let mut iterations = 0
|
||||||
|
|
||||||
|
while ($remaining | length > 0) and ($iterations < $max_iterations) {
|
||||||
|
let mut found_any = false
|
||||||
|
|
||||||
|
for service in $remaining {
|
||||||
|
let deps = ($SERVICES_REGISTRY | get $service).depends_on
|
||||||
|
let all_deps_satisfied = (
|
||||||
|
$deps | all { |dep| $ordered | any { |s| $s == $dep } }
|
||||||
|
)
|
||||||
|
|
||||||
|
if $all_deps_satisfied {
|
||||||
|
$ordered = ($ordered | append $service)
|
||||||
|
$remaining = ($remaining | filter { |s| $s != $service })
|
||||||
|
$found_any = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if not $found_any {
|
||||||
|
log_error $"Circular dependency detected or missing dependencies for: ($remaining | str join ', ')"
|
||||||
|
return []
|
||||||
|
}
|
||||||
|
|
||||||
|
$iterations = $iterations + 1
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($remaining | length > 0) {
|
||||||
|
log_error $"Failed to resolve startup order for: ($remaining | str join ', ')"
|
||||||
|
return []
|
||||||
|
}
|
||||||
|
|
||||||
|
$ordered
|
||||||
|
}
|
||||||
|
|
||||||
|
# Parse command line arguments
|
||||||
|
def parse_arguments [args: list<string>] -> record {
|
||||||
|
let mut config = {
|
||||||
|
services_set: "core",
|
||||||
|
custom_services: [],
|
||||||
|
debug: false,
|
||||||
|
health_check: false,
|
||||||
|
logs: false,
|
||||||
|
stop: false,
|
||||||
|
config_file: null,
|
||||||
|
verbose: false
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut i = 0
|
||||||
|
while $i < ($args | length) {
|
||||||
|
let arg = $args | get $i
|
||||||
|
|
||||||
|
if $arg == "--services" {
|
||||||
|
$i = $i + 1
|
||||||
|
if $i < ($args | length) {
|
||||||
|
let set = $args | get $i
|
||||||
|
$config.services_set = $set
|
||||||
|
|
||||||
|
# If custom, collect remaining service names
|
||||||
|
if $set == "custom" {
|
||||||
|
$i = $i + 1
|
||||||
|
while $i < ($args | length) and not ($args | get $i | str starts-with "-") {
|
||||||
|
$config.custom_services = ($config.custom_services | append ($args | get $i))
|
||||||
|
$i = $i + 1
|
||||||
|
}
|
||||||
|
$i = $i - 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else if $arg == "--debug" {
|
||||||
|
$config.debug = true
|
||||||
|
} else if $arg == "--health-check" {
|
||||||
|
$config.health_check = true
|
||||||
|
} else if $arg == "--logs" {
|
||||||
|
$config.logs = true
|
||||||
|
} else if $arg == "--stop" {
|
||||||
|
$config.stop = true
|
||||||
|
} else if $arg == "--config" {
|
||||||
|
$i = $i + 1
|
||||||
|
if $i < ($args | length) {
|
||||||
|
$config.config_file = ($args | get $i)
|
||||||
|
}
|
||||||
|
} else if $arg == "--verbose" {
|
||||||
|
$config.verbose = true
|
||||||
|
}
|
||||||
|
|
||||||
|
$i = $i + 1
|
||||||
|
}
|
||||||
|
|
||||||
|
$config
|
||||||
|
}
|
||||||
|
|
||||||
|
# Determine which services to start
|
||||||
|
def get_services_to_start [config: record] -> list<string> {
|
||||||
|
if $config.services_set == "custom" {
|
||||||
|
$config.custom_services
|
||||||
|
} else if $config.services_set == "all" {
|
||||||
|
$SERVICE_GROUPS.all
|
||||||
|
} else {
|
||||||
|
$SERVICE_GROUPS.core
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Check if service is already running
|
||||||
|
def is_service_running [service_name: string] -> bool {
|
||||||
|
let port = ($SERVICES_REGISTRY | get $service_name).port
|
||||||
|
|
||||||
|
try {
|
||||||
|
if ($nu.os-info.name == "macos") {
|
||||||
|
let result = (lsof -i -P -n | grep LISTEN | grep -c $":($port) " | into int)
|
||||||
|
$result > 0
|
||||||
|
} else {
|
||||||
|
let result = (netstat -an | grep LISTEN | grep -c ":($port) " | into int)
|
||||||
|
$result > 0
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Start a single service
|
||||||
|
def start_service [service_name: string, config: record, index: int] -> bool {
|
||||||
|
let service_info = $SERVICES_REGISTRY | get $service_name
|
||||||
|
let port = $service_info.port
|
||||||
|
let binary = $service_info.binary
|
||||||
|
let protocol = $service_info.protocol
|
||||||
|
|
||||||
|
if (is_service_running $service_name) {
|
||||||
|
log_warning $"($service_name) is already running on port ($port)"
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
if not (is_port_available $port) {
|
||||||
|
log_error $"Port ($port) is not available for ($service_name)"
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
# Show progress
|
||||||
|
print ""
|
||||||
|
print $"[$COLOR_YELLOW($index)($COLOR_RESET)] Starting ($service_name) on port ($port) ($COLOR_GREEN)($protocol)($COLOR_RESET)"
|
||||||
|
print $" → $($service_info.description)"
|
||||||
|
|
||||||
|
# Prepare environment variables
|
||||||
|
let mut env_vars = {}
|
||||||
|
|
||||||
|
match $service_name {
|
||||||
|
"vault-service" => {
|
||||||
|
$env_vars = {
|
||||||
|
VAULT_PORT: ($port | into string),
|
||||||
|
RUST_LOG: (if $config.debug { "debug" } else { "info" })
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"control-center" => {
|
||||||
|
$env_vars = {
|
||||||
|
CONTROL_CENTER_PORT: ($port | into string),
|
||||||
|
CONTROL_CENTER_CONFIG: ($config.config_file ?? "~/.config/provisioning/control-center.toml"),
|
||||||
|
RUST_LOG: (if $config.debug { "debug" } else { "info" })
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"orchestrator" => {
|
||||||
|
$env_vars = {
|
||||||
|
ORCHESTRATOR_PORT: ($port | into string),
|
||||||
|
REGISTRY_URL: "http://localhost:8082",
|
||||||
|
RUST_LOG: (if $config.debug { "debug" } else { "info" })
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"ai-service" => {
|
||||||
|
$env_vars = {
|
||||||
|
AI_SERVICE_PORT: ($port | into string),
|
||||||
|
RAG_URL: "http://localhost:8300",
|
||||||
|
RUST_LOG: (if $config.debug { "debug" } else { "info" })
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"control-center-ui" => {
|
||||||
|
$env_vars = {
|
||||||
|
UI_PORT: ($port | into string),
|
||||||
|
CONTROL_CENTER_URL: "http://localhost:8000",
|
||||||
|
RUST_LOG: (if $config.debug { "debug" } else { "info" })
|
||||||
|
}
|
||||||
|
},
|
||||||
|
_ => {
|
||||||
|
$env_vars = {
|
||||||
|
RUST_LOG: (if $config.debug { "debug" } else { "info" })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Create logs directory
|
||||||
|
let log_dir = "~/.provisioning/logs"
|
||||||
|
try {
|
||||||
|
mkdir ($log_dir | path expand)
|
||||||
|
} catch { }
|
||||||
|
|
||||||
|
let log_file = $"($log_dir)/($service_name).log"
|
||||||
|
|
||||||
|
# Start service in background
|
||||||
|
try {
|
||||||
|
if ($nu.os-info.name == "macos") {
|
||||||
|
let cmd = if ($service_name == "control-center-ui") {
|
||||||
|
# UI service uses different build
|
||||||
|
$"cargo run --release -p control-center-ui 2>&1 | tee ($log_file)"
|
||||||
|
} else {
|
||||||
|
$"cargo run --release -p ($service_name) 2>&1 | tee ($log_file)"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Set environment and run in background
|
||||||
|
with-env $env_vars {
|
||||||
|
nohup bash -c $cmd > /dev/null 2>&1 &
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
let cmd = if ($service_name == "control-center-ui") {
|
||||||
|
$"cargo run --release -p control-center-ui 2>&1 | tee ($log_file)"
|
||||||
|
} else {
|
||||||
|
$"cargo run --release -p ($service_name) 2>&1 | tee ($log_file)"
|
||||||
|
}
|
||||||
|
|
||||||
|
with-env $env_vars {
|
||||||
|
nohup sh -c $cmd > /dev/null 2>&1 &
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Wait for service to be ready
|
||||||
|
sleep 2s
|
||||||
|
|
||||||
|
# Check if service is running
|
||||||
|
if (is_service_running $service_name) {
|
||||||
|
log_success $"($service_name) started successfully"
|
||||||
|
return true
|
||||||
|
} else {
|
||||||
|
log_error $"($service_name) failed to start (port not responding)"
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
log_error $"Failed to start ($service_name): ($in)"
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Health check for a service
|
||||||
|
def health_check [service_name: string] -> bool {
|
||||||
|
let service_info = $SERVICES_REGISTRY | get $service_name
|
||||||
|
let port = $service_info.port
|
||||||
|
|
||||||
|
try {
|
||||||
|
let response = (
|
||||||
|
curl -s -f $"http://localhost:($port)/health" 2>/dev/null ||
|
||||||
|
curl -s -f $"http://localhost:($port)/ping" 2>/dev/null ||
|
||||||
|
curl -s -f $"http://localhost:($port)/" 2>/dev/null
|
||||||
|
)
|
||||||
|
true
|
||||||
|
} catch {
|
||||||
|
false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Stop all services
|
||||||
|
def stop_all_services [] {
|
||||||
|
log_section "Stopping all services"
|
||||||
|
|
||||||
|
# Kill all cargo processes
|
||||||
|
try {
|
||||||
|
let result = (
|
||||||
|
if ($nu.os-info.name == "macos") {
|
||||||
|
pkill -f "cargo run" 2>/dev/null
|
||||||
|
} else {
|
||||||
|
pkill -f "cargo run" 2>/dev/null
|
||||||
|
}
|
||||||
|
)
|
||||||
|
log_success "Stopped all cargo processes"
|
||||||
|
} catch {
|
||||||
|
log_warning "No cargo processes found"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Stop Docker containers if running
|
||||||
|
try {
|
||||||
|
for container in ["provisioning-postgres", "provisioning-redis"] {
|
||||||
|
docker stop $container 2>/dev/null
|
||||||
|
log_success $"Stopped Docker container: ($container)"
|
||||||
|
}
|
||||||
|
} catch { }
|
||||||
|
|
||||||
|
print ""
|
||||||
|
log_success "All services stopped"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Show service status
|
||||||
|
def show_status [services: list<string>] {
|
||||||
|
log_section "Service Status"
|
||||||
|
|
||||||
|
for service in $services {
|
||||||
|
let service_info = $SERVICES_REGISTRY | get $service
|
||||||
|
let is_running = (is_service_running $service)
|
||||||
|
let status = (if $is_running { $"($COLOR_GREEN)✓ RUNNING($COLOR_RESET)" } else { $"($COLOR_RED)✗ STOPPED($COLOR_RESET)" })
|
||||||
|
|
||||||
|
print $"($service): $status (port $($service_info.port))"
|
||||||
|
}
|
||||||
|
|
||||||
|
print ""
|
||||||
|
}
|
||||||
|
|
||||||
|
# Main execution
|
||||||
|
def main [args: list<string>] {
|
||||||
|
let config = (parse_arguments $args)
|
||||||
|
|
||||||
|
# Handle stop action
|
||||||
|
if $config.stop {
|
||||||
|
(stop_all_services)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
# Validate services
|
||||||
|
let services_to_start = (get_services_to_start $config)
|
||||||
|
|
||||||
|
if ($services_to_start | length == 0) {
|
||||||
|
log_error "No services to start"
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
# Validate all services exist
|
||||||
|
for service in $services_to_start {
|
||||||
|
if not ($SERVICES_REGISTRY | has $service) {
|
||||||
|
log_error $"Unknown service: ($service)"
|
||||||
|
log_info "Available services: $([$SERVICES_REGISTRY | keys | join ', '])"
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Show startup configuration
|
||||||
|
log_section "Provisioning Platform - Local Services"
|
||||||
|
log_info $"Starting ($($services_to_start | length)) services"
|
||||||
|
log_info $"Service set: ($config.services_set)"
|
||||||
|
|
||||||
|
if $config.debug {
|
||||||
|
log_info "Debug mode: ENABLED"
|
||||||
|
}
|
||||||
|
|
||||||
|
if $config.health_check {
|
||||||
|
log_info "Health checks: ENABLED"
|
||||||
|
}
|
||||||
|
|
||||||
|
print ""
|
||||||
|
|
||||||
|
# Resolve startup order
|
||||||
|
let startup_order = (resolve_startup_order $services_to_start)
|
||||||
|
|
||||||
|
if ($startup_order | length == 0) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
# Change to provisioning/platform directory
|
||||||
|
let original_dir = (pwd)
|
||||||
|
let platform_dir = $"($original_dir)/provisioning/platform"
|
||||||
|
|
||||||
|
if not (test -d $platform_dir) {
|
||||||
|
log_error "provisioning/platform directory not found"
|
||||||
|
log_info "Make sure you're running this script from the project root"
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
cd $platform_dir
|
||||||
|
|
||||||
|
# Start services
|
||||||
|
let mut failed_services = []
|
||||||
|
|
||||||
|
for idx in 0..($startup_order | length) {
|
||||||
|
let service = $startup_order | get $idx
|
||||||
|
let success = (start_service $service $config ($idx + 1))
|
||||||
|
|
||||||
|
if not $success {
|
||||||
|
$failed_services = ($failed_services | append $service)
|
||||||
|
}
|
||||||
|
|
||||||
|
# Wait between services (dependencies need time to start)
|
||||||
|
if $idx < (($startup_order | length) - 1) {
|
||||||
|
sleep 2s
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
print ""
|
||||||
|
log_section "Startup Summary"
|
||||||
|
|
||||||
|
# Show status
|
||||||
|
(show_status $startup_order)
|
||||||
|
|
||||||
|
# Health checks if enabled
|
||||||
|
if $config.health_check {
|
||||||
|
print ""
|
||||||
|
log_info "Running health checks..."
|
||||||
|
|
||||||
|
for service in $startup_order {
|
||||||
|
let healthy = (health_check $service)
|
||||||
|
let status = (if $healthy { $"($COLOR_GREEN)✓ HEALTHY($COLOR_RESET)" } else { $"($COLOR_YELLOW)⚠ NOT RESPONDING($COLOR_RESET)" })
|
||||||
|
print $" ($service): $status"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Summary
|
||||||
|
print ""
|
||||||
|
if ($failed_services | length > 0) {
|
||||||
|
log_warning $"($($failed_services | length)) services failed to start"
|
||||||
|
for service in $failed_services {
|
||||||
|
print $" - ($service)"
|
||||||
|
}
|
||||||
|
log_info "Check logs: ~/.provisioning/logs/"
|
||||||
|
} else {
|
||||||
|
log_success "All services started successfully!"
|
||||||
|
}
|
||||||
|
|
||||||
|
print ""
|
||||||
|
log_info "Service URLs:"
|
||||||
|
print " Control Center: http://localhost:8000"
|
||||||
|
print " Control Center UI: http://localhost:3000"
|
||||||
|
print " Orchestrator: http://localhost:9090"
|
||||||
|
print " AI Service: http://localhost:8083"
|
||||||
|
print " Vault Service: grpc://localhost:8081"
|
||||||
|
|
||||||
|
print ""
|
||||||
|
log_info "Logs location: ~/.provisioning/logs/"
|
||||||
|
log_info "Stop all services: ./start-local-binaries.nu --stop"
|
||||||
|
|
||||||
|
print ""
|
||||||
|
|
||||||
|
cd $original_dir
|
||||||
|
}
|
||||||
|
|
||||||
|
# Run main with arguments
|
||||||
|
main $nu.env.ARGS
|
||||||
Loading…
Add table
Reference in a new issue