# Platform Deployment Guide\n\n**Version**: 1.0.0\n**Last Updated**: 2026-01-05\n**Target Audience**: DevOps Engineers, Platform Operators\n**Status**: Production Ready\n\nPractical guide for deploying the 9-service provisioning platform in any environment using mode-based configuration.\n\n## Table of Contents\n\n1. [Prerequisites](#prerequisites)\n2. [Deployment Modes](#deployment-modes)\n3. [Quick Start](#quick-start)\n4. [Solo Mode Deployment](#solo-mode-deployment)\n5. [Multiuser Mode Deployment](#multiuser-mode-deployment)\n6. [CICD Mode Deployment](#cicd-mode-deployment)\n7. [Enterprise Mode Deployment](#enterprise-mode-deployment)\n8. [Service Management](#service-management)\n9. [Health Checks & Monitoring](#health-checks--monitoring)\n10. [Troubleshooting](#troubleshooting)\n\n---\n\n## Prerequisites\n\n### Required Software\n\n- **Rust**: 1.70+ (for building services)\n- **Nickel**: Latest (for config validation)\n- **Nushell**: 0.109.1+ (for scripts)\n- **Cargo**: Included with Rust\n- **Git**: For cloning and pulling updates\n\n### Required Tools (Mode-Dependent)\n\n| Tool | Solo | Multiuser | CICD | Enterprise |\n| ------ | ------ | ----------- | ------ | ------------ |\n| Docker/Podman | No | Optional | Yes | Yes |\n| SurrealDB | No | Yes | No | No |\n| Etcd | No | No | No | Yes |\n| PostgreSQL | No | Optional | No | Optional |\n| OpenAI/Anthropic API | No | Optional | Yes | Yes |\n\n### System Requirements\n\n| Resource | Solo | Multiuser | CICD | Enterprise |\n| ---------- | ------ | ----------- | ------ | ------------ |\n| CPU Cores | 2+ | 4+ | 8+ | 16+ |\n| Memory | 2 GB | 4 GB | 8 GB | 16 GB |\n| Disk | 10 GB | 50 GB | 100 GB | 500 GB |\n| Network | Local | Local/Cloud | Cloud | HA Cloud |\n\n### Directory Structure\n\n```\n# Ensure base directories exist\nmkdir -p provisioning/schemas/platform\nmkdir -p provisioning/platform/logs\nmkdir -p provisioning/platform/data\nmkdir -p provisioning/.typedialog/platform\nmkdir -p provisioning/config/runtime\n```\n\n---\n\n## Deployment Modes\n\n### Mode Selection Matrix\n\n| Requirement | Recommended Mode |\n| ------------- | ------------------ |\n| Development & testing | **solo** |\n| Team environment (2-10 people) | **multiuser** |\n| CI/CD pipelines & automation | **cicd** |\n| Production with HA | **enterprise** |\n\n### Mode Characteristics\n\n#### Solo Mode\n\n**Use Case**: Development, testing, demonstration\n\n**Characteristics**:\n- All services run locally with minimal resources\n- Filesystem-based storage (no external databases)\n- No TLS/SSL required\n- Embedded/in-memory backends\n- Single machine only\n\n**Services Configuration**:\n- 2-4 workers per service\n- 30-60 second timeouts\n- No replication or clustering\n- Debug-level logging enabled\n\n**Startup Time**: ~2-5 minutes\n**Data Persistence**: Local files only\n\n---\n\n#### Multiuser Mode\n\n**Use Case**: Team environments, shared infrastructure\n\n**Characteristics**:\n- Shared database backends (SurrealDB)\n- Multiple concurrent users\n- CORS and multi-user features enabled\n- Optional TLS support\n- 2-4 machines (or containerized)\n\n**Services Configuration**:\n- 4-6 workers per service\n- 60-120 second timeouts\n- Basic replication available\n- Info-level logging\n\n**Startup Time**: ~3-8 minutes (database dependent)\n**Data Persistence**: SurrealDB (shared)\n\n---\n\n#### CICD Mode\n\n**Use Case**: CI/CD pipelines, ephemeral environments\n\n**Characteristics**:\n- Ephemeral storage (memory, temporary)\n- High throughput\n- RAG system disabled\n- Minimal logging\n- Stateless services\n\n**Services Configuration**:\n- 8-12 workers per service\n- 10-30 second timeouts\n- No persistence\n- Warn-level logging\n\n**Startup Time**: ~1-2 minutes\n**Data Persistence**: None (ephemeral)\n\n---\n\n#### Enterprise Mode\n\n**Use Case**: Production, high availability, compliance\n\n**Characteristics**:\n- Distributed, replicated backends\n- High availability (HA) clustering\n- TLS/SSL encryption\n- Audit logging\n- Full monitoring and observability\n\n**Services Configuration**:\n- 16-32 workers per service\n- 120-300 second timeouts\n- Active replication across 3+ nodes\n- Info-level logging with audit trails\n\n**Startup Time**: ~5-15 minutes (cluster initialization)\n**Data Persistence**: Replicated across cluster\n\n---\n\n## Quick Start\n\n### 1. Clone Repository\n\n```\ngit clone https://github.com/your-org/project-provisioning.git\ncd project-provisioning\n```\n\n### 2. Select Deployment Mode\n\nChoose your mode based on use case:\n\n```\n# For development\nexport DEPLOYMENT_MODE=solo\n\n# For team environments\nexport DEPLOYMENT_MODE=multiuser\n\n# For CI/CD\nexport DEPLOYMENT_MODE=cicd\n\n# For production\nexport DEPLOYMENT_MODE=enterprise\n```\n\n### 3. Set Environment Variables\n\nAll services use mode-specific TOML configs automatically loaded via environment variables:\n\n```\n# Vault Service\nexport VAULT_MODE=$DEPLOYMENT_MODE\n\n# Extension Registry\nexport REGISTRY_MODE=$DEPLOYMENT_MODE\n\n# RAG System\nexport RAG_MODE=$DEPLOYMENT_MODE\n\n# AI Service\nexport AI_SERVICE_MODE=$DEPLOYMENT_MODE\n\n# Provisioning Daemon\nexport DAEMON_MODE=$DEPLOYMENT_MODE\n```\n\n### 4. Build All Services\n\n```\n# Build all platform crates\ncargo build --release -p vault-service \\n                      -p extension-registry \\n                      -p provisioning-rag \\n                      -p ai-service \\n                      -p provisioning-daemon \\n                      -p orchestrator \\n                      -p control-center \\n                      -p mcp-server \\n                      -p installer\n```\n\n### 5. Start Services (Order Matters)\n\n```\n# Start in dependency order:\n\n# 1. Core infrastructure (KMS, storage)\ncargo run --release -p vault-service &\n\n# 2. Configuration and extensions\ncargo run --release -p extension-registry &\n\n# 3. AI/RAG layer\ncargo run --release -p provisioning-rag &\ncargo run --release -p ai-service &\n\n# 4. Orchestration layer\ncargo run --release -p orchestrator &\ncargo run --release -p control-center &\ncargo run --release -p mcp-server &\n\n# 5. Background operations\ncargo run --release -p provisioning-daemon &\n\n# 6. Installer (optional, for new deployments)\ncargo run --release -p installer &\n```\n\n### 6. Verify Services\n\n```\n# Check all services are running\npgrep -l "vault-service|extension-registry|provisioning-rag|ai-service"\n\n# Test endpoints\ncurl http://localhost:8200/health   # Vault\ncurl http://localhost:8081/health   # Registry\ncurl http://localhost:8083/health   # RAG\ncurl http://localhost:8082/health   # AI Service\ncurl http://localhost:9090/health   # Orchestrator\ncurl http://localhost:8080/health   # Control Center\n```\n\n---\n\n## Solo Mode Deployment\n\n**Perfect for**: Development, testing, learning\n\n### Step 1: Verify Solo Configuration Files\n\n```\n# Check that solo schemas are available\nls -la provisioning/schemas/platform/defaults/deployment/solo-defaults.ncl\n\n# Available schemas for each service:\n# - provisioning/schemas/platform/schemas/vault-service.ncl\n# - provisioning/schemas/platform/schemas/extension-registry.ncl\n# - provisioning/schemas/platform/schemas/rag.ncl\n# - provisioning/schemas/platform/schemas/ai-service.ncl\n# - provisioning/schemas/platform/schemas/provisioning-daemon.ncl\n```\n\n### Step 2: Set Solo Environment Variables\n\n```\n# Set all services to solo mode\nexport VAULT_MODE=solo\nexport REGISTRY_MODE=solo\nexport RAG_MODE=solo\nexport AI_SERVICE_MODE=solo\nexport DAEMON_MODE=solo\n\n# Verify settings\necho $VAULT_MODE  # Should output: solo\n```\n\n### Step 3: Build Services\n\n```\n# Build in release mode for better performance\ncargo build --release\n```\n\n### Step 4: Create Local Data Directories\n\n```\n# Create storage directories for solo mode\nmkdir -p /tmp/provisioning-solo/{vault,registry,rag,ai,daemon}\nchmod 755 /tmp/provisioning-solo/{vault,registry,rag,ai,daemon}\n```\n\n### Step 5: Start Services\n\n```\n# Start each service in a separate terminal or use tmux:\n\n# Terminal 1: Vault\ncargo run --release -p vault-service\n\n# Terminal 2: Registry\ncargo run --release -p extension-registry\n\n# Terminal 3: RAG\ncargo run --release -p provisioning-rag\n\n# Terminal 4: AI Service\ncargo run --release -p ai-service\n\n# Terminal 5: Orchestrator\ncargo run --release -p orchestrator\n\n# Terminal 6: Control Center\ncargo run --release -p control-center\n\n# Terminal 7: Daemon\ncargo run --release -p provisioning-daemon\n```\n\n### Step 6: Test Services\n\n```\n# Wait 10-15 seconds for services to start, then test\n\n# Check service health\ncurl -s http://localhost:8200/health | jq .\ncurl -s http://localhost:8081/health | jq .\ncurl -s http://localhost:8083/health | jq .\n\n# Try a simple operation\ncurl -X GET http://localhost:9090/api/v1/health\n```\n\n### Step 7: Verify Persistence (Optional)\n\n```\n# Check that data is stored locally\nls -la /tmp/provisioning-solo/vault/\nls -la /tmp/provisioning-solo/registry/\n\n# Data should accumulate as you use the services\n```\n\n### Cleanup\n\n```\n# Stop all services\npkill -f "cargo run --release"\n\n# Remove temporary data (optional)\nrm -rf /tmp/provisioning-solo\n```\n\n---\n\n## Multiuser Mode Deployment\n\n**Perfect for**: Team environments, shared infrastructure\n\n### Prerequisites\n\n- **SurrealDB**: Running and accessible at `http://surrealdb:8000`\n- **Network Access**: All machines can reach SurrealDB\n- **DNS/Hostnames**: Services accessible via hostnames (not just localhost)\n\n### Step 1: Deploy SurrealDB\n\n```\n# Using Docker (recommended)\ndocker run -d \\n  --name surrealdb \\n  -p 8000:8000 \\n  surrealdb/surrealdb:latest \\n  start --user root --pass root\n\n# Or using native installation:\nsurreal start --user root --pass root\n```\n\n### Step 2: Verify SurrealDB Connectivity\n\n```\n# Test SurrealDB connection\ncurl -s http://localhost:8000/health\n\n# Should return: {"version":"v1.x.x"}\n```\n\n### Step 3: Set Multiuser Environment Variables\n\n```\n# Configure all services for multiuser mode\nexport VAULT_MODE=multiuser\nexport REGISTRY_MODE=multiuser\nexport RAG_MODE=multiuser\nexport AI_SERVICE_MODE=multiuser\nexport DAEMON_MODE=multiuser\n\n# Set database connection\nexport SURREALDB_URL=http://surrealdb:8000\nexport SURREALDB_USER=root\nexport SURREALDB_PASS=root\n\n# Set service hostnames (if not localhost)\nexport VAULT_SERVICE_HOST=vault.internal\nexport REGISTRY_HOST=registry.internal\nexport RAG_HOST=rag.internal\n```\n\n### Step 4: Build Services\n\n```\ncargo build --release\n```\n\n### Step 5: Create Shared Data Directories\n\n```\n# Create directories on shared storage (NFS, etc.)\nmkdir -p /mnt/provisioning-data/{vault,registry,rag,ai}\nchmod 755 /mnt/provisioning-data/{vault,registry,rag,ai}\n\n# Or use local directories if on separate machines\nmkdir -p /var/lib/provisioning/{vault,registry,rag,ai}\n```\n\n### Step 6: Start Services on Multiple Machines\n\n```\n# Machine 1: Infrastructure services\nssh ops@machine1\nexport VAULT_MODE=multiuser\ncargo run --release -p vault-service &\ncargo run --release -p extension-registry &\n\n# Machine 2: AI services\nssh ops@machine2\nexport RAG_MODE=multiuser\nexport AI_SERVICE_MODE=multiuser\ncargo run --release -p provisioning-rag &\ncargo run --release -p ai-service &\n\n# Machine 3: Orchestration\nssh ops@machine3\ncargo run --release -p orchestrator &\ncargo run --release -p control-center &\n\n# Machine 4: Background tasks\nssh ops@machine4\nexport DAEMON_MODE=multiuser\ncargo run --release -p provisioning-daemon &\n```\n\n### Step 7: Test Multi-Machine Setup\n\n```\n# From any machine, test cross-machine connectivity\ncurl -s http://machine1:8200/health\ncurl -s http://machine2:8083/health\ncurl -s http://machine3:9090/health\n\n# Test integration\ncurl -X POST http://machine3:9090/api/v1/provision \\n  -H "Content-Type: application/json" \\n  -d '{"workspace": "test"}'\n```\n\n### Step 8: Enable User Access\n\n```\n# Create shared credentials\nexport VAULT_TOKEN=s.xxxxxxxxxxx\n\n# Configure TLS (optional but recommended)\n# Update configs to use https:// URLs\nexport VAULT_MODE=multiuser\n# Edit provisioning/schemas/platform/schemas/vault-service.ncl\n# Add TLS configuration in the schema definition\n# See: provisioning/schemas/platform/validators/ for constraints\n```\n\n### Monitoring Multiuser Deployment\n\n```\n# Check all services are connected to SurrealDB\nfor host in machine1 machine2 machine3 machine4; do\n  ssh ops@$host "curl -s http://localhost/api/v1/health | jq .database_connected"\ndone\n\n# Monitor SurrealDB\ncurl -s http://surrealdb:8000/version\n```\n\n---\n\n## CICD Mode Deployment\n\n**Perfect for**: GitHub Actions, GitLab CI, Jenkins, cloud automation\n\n### Step 1: Understand Ephemeral Nature\n\nCICD mode services:\n- Don't persist data between runs\n- Use in-memory storage\n- Have RAG disabled\n- Optimize for startup speed\n- Suitable for containerized deployments\n\n### Step 2: Set CICD Environment Variables\n\n```\n# Use cicd mode for all services\nexport VAULT_MODE=cicd\nexport REGISTRY_MODE=cicd\nexport RAG_MODE=cicd\nexport AI_SERVICE_MODE=cicd\nexport DAEMON_MODE=cicd\n\n# Disable TLS (not needed in CI)\nexport CI_ENVIRONMENT=true\n```\n\n### Step 3: Containerize Services (Optional)\n\n```\n# Dockerfile for CICD deployments\nFROM rust:1.75-slim\n\nWORKDIR /app\nCOPY . .\n\n# Build all services\nRUN cargo build --release\n\n# Set CICD mode\nENV VAULT_MODE=cicd\nENV REGISTRY_MODE=cicd\nENV RAG_MODE=cicd\nENV AI_SERVICE_MODE=cicd\n\n# Expose ports\nEXPOSE 8200 8081 8083 8082 9090 8080\n\n# Run services\nCMD ["sh", "-c", "\\n  cargo run --release -p vault-service & \\n  cargo run --release -p extension-registry & \\n  cargo run --release -p provisioning-rag & \\n  cargo run --release -p ai-service & \\n  cargo run --release -p orchestrator & \\n  wait"]\n```\n\n### Step 4: GitHub Actions Example\n\n```\nname: CICD Platform Deployment\n\non:\n  push:\n    branches: [main, develop]\n\njobs:\n  test-deployment:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v3\n\n      - name: Install Rust\n        uses: actions-rs/toolchain@v1\n        with:\n          toolchain: 1.75\n          profile: minimal\n\n      - name: Set CICD Mode\n        run: |\n          echo "VAULT_MODE=cicd" >> $GITHUB_ENV\n          echo "REGISTRY_MODE=cicd" >> $GITHUB_ENV\n          echo "RAG_MODE=cicd" >> $GITHUB_ENV\n          echo "AI_SERVICE_MODE=cicd" >> $GITHUB_ENV\n          echo "DAEMON_MODE=cicd" >> $GITHUB_ENV\n\n      - name: Build Services\n        run: cargo build --release\n\n      - name: Run Integration Tests\n        run: |\n          # Start services in background\n          cargo run --release -p vault-service &\n          cargo run --release -p extension-registry &\n          cargo run --release -p orchestrator &\n\n          # Wait for startup\n          sleep 10\n\n          # Run tests\n          cargo test --release\n\n      - name: Health Checks\n        run: |\n          curl -f http://localhost:8200/health\n          curl -f http://localhost:8081/health\n          curl -f http://localhost:9090/health\n\n  deploy:\n    needs: test-deployment\n    runs-on: ubuntu-latest\n    if: github.ref == 'refs/heads/main'\n    steps:\n      - uses: actions/checkout@v3\n      - name: Deploy to Production\n        run: |\n          # Deploy production enterprise cluster\n          ./scripts/deploy-enterprise.sh\n```\n\n### Step 5: Run CICD Tests\n\n```\n# Simulate CI environment locally\nexport VAULT_MODE=cicd\nexport CI_ENVIRONMENT=true\n\n# Build\ncargo build --release\n\n# Run short-lived services for testing\ntimeout 30 cargo run --release -p vault-service &\ntimeout 30 cargo run --release -p extension-registry &\ntimeout 30 cargo run --release -p orchestrator &\n\n# Run tests while services are running\nsleep 5\ncargo test --release\n\n# Services auto-cleanup after timeout\n```\n\n---\n\n## Enterprise Mode Deployment\n\n**Perfect for**: Production, high availability, compliance\n\n### Prerequisites\n\n- **3+ Machines**: Minimum 3 for HA\n- **Etcd Cluster**: For distributed consensus\n- **Load Balancer**: HAProxy, nginx, or cloud LB\n- **TLS Certificates**: Valid certificates for all services\n- **Monitoring**: Prometheus, ELK, or cloud monitoring\n- **Backup System**: Daily snapshots to S3 or similar\n\n### Step 1: Deploy Infrastructure\n\n#### 1.1 Deploy Etcd Cluster\n\n```\n# Node 1, 2, 3\netcd --name=node-1 \\n     --listen-client-urls=http://0.0.0.0:2379 \\n     --advertise-client-urls=http://node-1.internal:2379 \\n     --initial-cluster="node-1=http://node-1.internal:2380,node-2=http://node-2.internal:2380,node-3=http://node-3.internal:2380" \\n     --initial-cluster-state=new\n\n# Verify cluster\netcdctl --endpoints=http://localhost:2379 member list\n```\n\n#### 1.2 Deploy Load Balancer\n\n```\n# HAProxy configuration for vault-service (example)\nfrontend vault_frontend\n    bind *:8200\n    mode tcp\n    default_backend vault_backend\n\nbackend vault_backend\n    mode tcp\n    balance roundrobin\n    server vault-1 10.0.1.10:8200 check\n    server vault-2 10.0.1.11:8200 check\n    server vault-3 10.0.1.12:8200 check\n```\n\n#### 1.3 Configure TLS\n\n```\n# Generate certificates (or use existing)\nmkdir -p /etc/provisioning/tls\n\n# For each service:\nopenssl req -x509 -newkey rsa:4096 \\n  -keyout /etc/provisioning/tls/vault-key.pem \\n  -out /etc/provisioning/tls/vault-cert.pem \\n  -days 365 -nodes \\n  -subj "/CN=vault.provisioning.prod"\n\n# Set permissions\nchmod 600 /etc/provisioning/tls/*-key.pem\nchmod 644 /etc/provisioning/tls/*-cert.pem\n```\n\n### Step 2: Set Enterprise Environment Variables\n\n```\n# All machines: Set enterprise mode\nexport VAULT_MODE=enterprise\nexport REGISTRY_MODE=enterprise\nexport RAG_MODE=enterprise\nexport AI_SERVICE_MODE=enterprise\nexport DAEMON_MODE=enterprise\n\n# Database cluster\nexport SURREALDB_URL="ws://surrealdb-cluster.internal:8000"\nexport SURREALDB_REPLICAS=3\n\n# Etcd cluster\nexport ETCD_ENDPOINTS="http://node-1.internal:2379,http://node-2.internal:2379,http://node-3.internal:2379"\n\n# TLS configuration\nexport TLS_CERT_PATH=/etc/provisioning/tls\nexport TLS_VERIFY=true\nexport TLS_CA_CERT=/etc/provisioning/tls/ca.crt\n\n# Monitoring\nexport PROMETHEUS_URL=http://prometheus.internal:9090\nexport METRICS_ENABLED=true\nexport AUDIT_LOG_ENABLED=true\n```\n\n### Step 3: Deploy Services Across Cluster\n\n```\n# Ansible playbook (simplified)\n---\n- hosts: provisioning_cluster\n  tasks:\n    - name: Build services\n      shell: cargo build --release\n\n    - name: Start vault-service (machine 1-3)\n      shell: "cargo run --release -p vault-service"\n      when: "'vault' in group_names"\n\n    - name: Start orchestrator (machine 2-3)\n      shell: "cargo run --release -p orchestrator"\n      when: "'orchestrator' in group_names"\n\n    - name: Start daemon (machine 3)\n      shell: "cargo run --release -p provisioning-daemon"\n      when: "'daemon' in group_names"\n\n    - name: Verify cluster health\n      uri:\n        url: "https://{{ inventory_hostname }}:9090/health"\n        validate_certs: yes\n```\n\n### Step 4: Monitor Cluster Health\n\n```\n# Check cluster status\ncurl -s https://vault.internal:8200/health | jq .state\n\n# Check replication\ncurl -s https://orchestrator.internal:9090/api/v1/cluster/status\n\n# Monitor etcd\netcdctl --endpoints=https://node-1.internal:2379 endpoint health\n\n# Check leader election\netcdctl --endpoints=https://node-1.internal:2379 election list\n```\n\n### Step 5: Enable Monitoring & Alerting\n\n```\n# Prometheus configuration\nglobal:\n  scrape_interval: 30s\n  evaluation_interval: 30s\n\nscrape_configs:\n  - job_name: 'vault-service'\n    scheme: https\n    tls_config:\n      ca_file: /etc/provisioning/tls/ca.crt\n    static_configs:\n      - targets: ['vault-1.internal:8200', 'vault-2.internal:8200', 'vault-3.internal:8200']\n\n  - job_name: 'orchestrator'\n    scheme: https\n    static_configs:\n      - targets: ['orch-1.internal:9090', 'orch-2.internal:9090', 'orch-3.internal:9090']\n```\n\n### Step 6: Backup & Recovery\n\n```\n# Daily backup script\n#!/bin/bash\nBACKUP_DIR="/mnt/provisioning-backups"\nDATE=$(date +%Y%m%d_%H%M%S)\n\n# Backup etcd\netcdctl --endpoints=https://node-1.internal:2379 \\n  snapshot save "$BACKUP_DIR/etcd-$DATE.db"\n\n# Backup SurrealDB\ncurl -X POST https://surrealdb.internal:8000/backup \\n  -H "Authorization: Bearer $SURREALDB_TOKEN" \\n  > "$BACKUP_DIR/surreal-$DATE.sql"\n\n# Upload to S3\naws s3 cp "$BACKUP_DIR/etcd-$DATE.db" \\n  s3://provisioning-backups/etcd/\n\n# Cleanup old backups (keep 30 days)\nfind "$BACKUP_DIR" -mtime +30 -delete\n```\n\n---\n\n## Service Management\n\n### Starting Services\n\n#### Individual Service Startup\n\n```\n# Start one service\nexport VAULT_MODE=enterprise\ncargo run --release -p vault-service\n\n# In another terminal\nexport REGISTRY_MODE=enterprise\ncargo run --release -p extension-registry\n```\n\n#### Batch Startup\n\n```\n# Start all services (dependency order)\n#!/bin/bash\nset -e\n\nMODE=${1:-solo}\nexport VAULT_MODE=$MODE\nexport REGISTRY_MODE=$MODE\nexport RAG_MODE=$MODE\nexport AI_SERVICE_MODE=$MODE\nexport DAEMON_MODE=$MODE\n\necho "Starting provisioning platform in $MODE mode..."\n\n# Core services first\necho "Starting infrastructure..."\ncargo run --release -p vault-service &\nVAULT_PID=$!\n\necho "Starting extension registry..."\ncargo run --release -p extension-registry &\nREGISTRY_PID=$!\n\n# AI layer\necho "Starting AI services..."\ncargo run --release -p provisioning-rag &\nRAG_PID=$!\n\ncargo run --release -p ai-service &\nAI_PID=$!\n\n# Orchestration\necho "Starting orchestration..."\ncargo run --release -p orchestrator &\nORCH_PID=$!\n\necho "All services started. PIDs: $VAULT_PID $REGISTRY_PID $RAG_PID $AI_PID $ORCH_PID"\n```\n\n### Stopping Services\n\n```\n# Stop all services gracefully\npkill -SIGTERM -f "cargo run --release -p"\n\n# Wait for graceful shutdown\nsleep 5\n\n# Force kill if needed\npkill -9 -f "cargo run --release -p"\n\n# Verify all stopped\npgrep -f "cargo run --release -p" && echo "Services still running" || echo "All stopped"\n```\n\n### Restarting Services\n\n```\n# Restart single service\npkill -SIGTERM vault-service\nsleep 2\ncargo run --release -p vault-service &\n\n# Restart all services\n./scripts/restart-all.sh $MODE\n\n# Restart with config reload\nexport VAULT_MODE=multiuser\npkill -SIGTERM vault-service\nsleep 2\ncargo run --release -p vault-service &\n```\n\n### Checking Service Status\n\n```\n# Check running processes\npgrep -a "cargo run --release"\n\n# Check listening ports\nnetstat -tlnp | grep -E "8200|8081|8083|8082|9090|8080"\n\n# Or using ss (modern alternative)\nss -tlnp | grep -E "8200|8081|8083|8082|9090|8080"\n\n# Health endpoint checks\nfor service in vault registry rag ai orchestrator; do\n  echo "=== $service ==="\n  curl -s http://localhost:${port[$service]}/health | jq .\ndone\n```\n\n---\n\n## Health Checks & Monitoring\n\n### Manual Health Verification\n\n```\n# Vault Service\ncurl -s http://localhost:8200/health | jq .\n# Expected: {"status":"ok","uptime":123.45}\n\n# Extension Registry\ncurl -s http://localhost:8081/health | jq .\n\n# RAG System\ncurl -s http://localhost:8083/health | jq .\n# Expected: {"status":"ok","embeddings":"ready","vector_db":"connected"}\n\n# AI Service\ncurl -s http://localhost:8082/health | jq .\n\n# Orchestrator\ncurl -s http://localhost:9090/health | jq .\n\n# Control Center\ncurl -s http://localhost:8080/health | jq .\n```\n\n### Service Integration Tests\n\n```\n# Test vault <-> registry integration\ncurl -X POST http://localhost:8200/api/encrypt \\n  -H "Content-Type: application/json" \\n  -d '{"plaintext":"secret"}' | jq .\n\n# Test RAG system\ncurl -X POST http://localhost:8083/api/ingest \\n  -H "Content-Type: application/json" \\n  -d '{"document":"test.md","content":"# Test"}' | jq .\n\n# Test orchestrator\ncurl -X GET http://localhost:9090/api/v1/status | jq .\n\n# End-to-end workflow\ncurl -X POST http://localhost:9090/api/v1/provision \\n  -H "Content-Type: application/json" \\n  -d '{\n    "workspace": "test",\n    "services": ["vault", "registry"],\n    "mode": "solo"\n  }' | jq .\n```\n\n### Monitoring Dashboards\n\n#### Prometheus Metrics\n\n```\n# Query service uptime\ncurl -s 'http://prometheus:9090/api/v1/query?query=up' | jq .\n\n# Query request rate\ncurl -s 'http://prometheus:9090/api/v1/query?query=rate(http_requests_total[5m])' | jq .\n\n# Query error rate\ncurl -s 'http://prometheus:9090/api/v1/query?query=rate(http_errors_total[5m])' | jq .\n```\n\n#### Log Aggregation\n\n```\n# Follow vault logs\ntail -f /var/log/provisioning/vault-service.log\n\n# Follow all service logs\ntail -f /var/log/provisioning/*.log\n\n# Search for errors\ngrep -r "ERROR" /var/log/provisioning/\n\n# Follow with filtering\ntail -f /var/log/provisioning/orchestrator.log | grep -E "ERROR|WARN"\n```\n\n### Alerting\n\n```\n# AlertManager configuration\ngroups:\n  - name: provisioning\n    rules:\n      - alert: ServiceDown\n        expr: up{job=~"vault|registry|rag|orchestrator"} == 0\n        for: 5m\n        annotations:\n          summary: "{{ $labels.job }} is down"\n\n      - alert: HighErrorRate\n        expr: rate(http_errors_total[5m]) > 0.05\n        annotations:\n          summary: "High error rate detected"\n\n      - alert: DiskSpaceWarning\n        expr: node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.2\n        annotations:\n          summary: "Disk space below 20%"\n```\n\n---\n\n## Troubleshooting\n\n### Service Won't Start\n\n**Problem**: `error: failed to bind to port 8200`\n\n**Solutions**:\n```\n# Check if port is in use\nlsof -i :8200\nss -tlnp | grep 8200\n\n# Kill existing process\npkill -9 -f vault-service\n\n# Or use different port\nexport VAULT_SERVER_PORT=8201\ncargo run --release -p vault-service\n```\n\n### Configuration Loading Fails\n\n**Problem**: `error: failed to load config from mode file`\n\n**Solutions**:\n```\n# Verify schemas exist\nls -la provisioning/schemas/platform/schemas/vault-service.ncl\n\n# Validate schema syntax\nnickel typecheck provisioning/schemas/platform/schemas/vault-service.ncl\n\n# Check defaults are present\nnickel typecheck provisioning/schemas/platform/defaults/vault-service-defaults.ncl\n\n# Verify deployment mode overlay exists\nls -la provisioning/schemas/platform/defaults/deployment/$VAULT_MODE-defaults.ncl\n\n# Run service with explicit mode\nexport VAULT_MODE=solo\ncargo run --release -p vault-service\n```\n\n### Database Connection Issues\n\n**Problem**: `error: failed to connect to database`\n\n**Solutions**:\n```\n# Verify database is running\ncurl http://surrealdb:8000/health\netcdctl --endpoints=http://etcd:2379 endpoint health\n\n# Check connectivity\nnc -zv surrealdb 8000\nnc -zv etcd 2379\n\n# Update connection string\nexport SURREALDB_URL=ws://surrealdb:8000\nexport ETCD_ENDPOINTS=http://etcd:2379\n\n# Restart service with new config\npkill -9 vault-service\ncargo run --release -p vault-service\n```\n\n### Service Crashes on Startup\n\n**Problem**: Service exits with code 1 or 139\n\n**Solutions**:\n```\n# Run with verbose logging\nRUST_LOG=debug cargo run -p vault-service 2>&1 | head -50\n\n# Check system resources\nfree -h\ndf -h\n\n# Check for core dumps\ncoredumpctl list\n\n# Run under debugger (if crash suspected)\nrust-gdb --args target/release/vault-service\n```\n\n### High Memory Usage\n\n**Problem**: Service consuming > expected memory\n\n**Solutions**:\n```\n# Check memory usage\nps aux | grep vault-service | grep -v grep\n\n# Monitor over time\nwatch -n 1 'ps aux | grep vault-service | grep -v grep'\n\n# Reduce worker count\nexport VAULT_SERVER_WORKERS=2\ncargo run --release -p vault-service\n\n# Check for memory leaks\nvalgrind --leak-check=full target/release/vault-service\n```\n\n### Network/DNS Issues\n\n**Problem**: `error: failed to resolve hostname`\n\n**Solutions**:\n```\n# Test DNS resolution\nnslookup vault.internal\ndig vault.internal\n\n# Test connectivity to service\ncurl -v http://vault.internal:8200/health\n\n# Add to /etc/hosts if needed\necho "10.0.1.10 vault.internal" >> /etc/hosts\n\n# Check network interface\nip addr show\nnetstat -nr\n```\n\n### Data Persistence Issues\n\n**Problem**: Data lost after restart\n\n**Solutions**:\n```\n# Verify backup exists\nls -la /mnt/provisioning-backups/\nls -la /var/lib/provisioning/\n\n# Check disk space\ndf -h /var/lib/provisioning\n\n# Verify file permissions\nls -l /var/lib/provisioning/vault/\nchmod 755 /var/lib/provisioning/vault/*\n\n# Restore from backup\n./scripts/restore-backup.sh /mnt/provisioning-backups/vault-20260105.sql\n```\n\n### Debugging Checklist\n\nWhen troubleshooting, use this systematic approach:\n\n```\n# 1. Check service is running\npgrep -f vault-service || echo "Service not running"\n\n# 2. Check port is listening\nss -tlnp | grep 8200 || echo "Port not listening"\n\n# 3. Check logs for errors\ntail -20 /var/log/provisioning/vault-service.log | grep -i error\n\n# 4. Test HTTP endpoint\ncurl -i http://localhost:8200/health\n\n# 5. Check dependencies\ncurl http://surrealdb:8000/health\netcdctl --endpoints=http://etcd:2379 endpoint health\n\n# 6. Check schema definition\nnickel typecheck provisioning/schemas/platform/schemas/vault-service.ncl\n\n# 7. Verify environment variables\nenv | grep -E "VAULT_|SURREALDB_|ETCD_"\n\n# 8. Check system resources\nfree -h && df -h && top -bn1 | head -10\n```\n\n---\n\n## Configuration Updates\n\n### Updating Service Configuration\n\n```\n# 1. Edit the schema definition\nvim provisioning/schemas/platform/schemas/vault-service.ncl\n\n# 2. Update defaults if needed\nvim provisioning/schemas/platform/defaults/vault-service-defaults.ncl\n\n# 3. Validate syntax\nnickel typecheck provisioning/schemas/platform/schemas/vault-service.ncl\n\n# 4. Re-export configuration from schemas\n./provisioning/.typedialog/platform/scripts/generate-configs.nu vault-service multiuser\n\n# 5. Restart affected service (no downtime for clients)\npkill -SIGTERM vault-service\nsleep 2\ncargo run --release -p vault-service &\n\n# 4. Verify configuration loaded\ncurl http://localhost:8200/api/config | jq .\n```\n\n### Mode Migration\n\n```\n# Migrate from solo to multiuser:\n\n# 1. Stop services\npkill -SIGTERM -f "cargo run"\nsleep 5\n\n# 2. Backup current data\ntar -czf /backup/provisioning-solo-$(date +%s).tar.gz /var/lib/provisioning/\n\n# 3. Set new mode\nexport VAULT_MODE=multiuser\nexport REGISTRY_MODE=multiuser\nexport RAG_MODE=multiuser\n\n# 4. Start services with new config\ncargo run --release -p vault-service &\ncargo run --release -p extension-registry &\n\n# 5. Verify new mode\ncurl http://localhost:8200/api/config | jq .deployment_mode\n```\n\n---\n\n## Production Checklist\n\nBefore deploying to production:\n\n- [ ] All services compiled in release mode (`--release`)\n- [ ] TLS certificates installed and valid\n- [ ] Database cluster deployed and healthy\n- [ ] Load balancer configured and routing traffic\n- [ ] Monitoring and alerting configured\n- [ ] Backup system tested and working\n- [ ] High availability verified (failover tested)\n- [ ] Security hardening applied (firewall rules, etc.)\n- [ ] Documentation updated for your environment\n- [ ] Team trained on deployment procedures\n- [ ] Runbooks created for common operations\n- [ ] Disaster recovery plan tested\n\n---\n\n## Getting Help\n\n### Community Resources\n\n- **GitHub Issues**: Report bugs at `github.com/your-org/provisioning/issues`\n- **Documentation**: Full docs at `provisioning/docs/`\n- **Slack Channel**: `#provisioning-platform`\n\n### Internal Support\n\n- **Platform Team**: [platform@your-org.com](mailto:platform@your-org.com)\n- **On-Call**: Check PagerDuty for active rotation\n- **Escalation**: Contact infrastructure leadership\n\n### Useful Commands Reference\n\n```\n# View all available commands\ncargo run -- --help\n\n# View service schemas\nls -la provisioning/schemas/platform/schemas/\nls -la provisioning/schemas/platform/defaults/\n\n# List running services\nps aux | grep cargo\n\n# Monitor service logs in real-time\njournalctl -fu provisioning-vault\n\n# Generate diagnostics bundle\n./scripts/generate-diagnostics.sh > /tmp/diagnostics-$(date +%s).tar.gz\n```
