# Local Services Setup Guide

Local development setup for running all 10 provisioning platform services locally. This guide covers building, configuring, and running services on your development machine.

**Audience**: Developers setting up local development environment  
**Prerequisites**: Rust 1.80+, Nushell 0.110.0+, Docker/Podman, Git  
**Estimated setup time**: 30-45 minutes

---

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [Repository Structure](#repository-structure)
3. [Build All Services](#build-all-services)
4. [Service Dependencies](#service-dependencies)
5. [Configuration Files](#configuration-files)
6. [Database & Storage Setup](#database--storage-setup)
7. [Security Keys Setup](#security-keys-setup)
8. [Running Services](#running-services)
9. [Troubleshooting](#troubleshooting)
10. [Development Workflows](#development-workflows)

---

## Prerequisites

### Required Tools

```bash
# Rust toolchain (1.80 or later)
rustc --version  # Should be 1.80+
cargo --version

# Nushell scripting runtime
nu --version     # Should be 0.110.0+

# Container runtime (choose one)
docker --version
# OR
podman --version

# Git version control
git --version

# Optional but recommended
k9s --version    # Kubernetes dashboard
kind --version   # Local Kubernetes cluster (for k8s tests)
```

### Environment Setup

```bash
# Set working directory to project root
cd /Users/Akasha/project-provisioning

# Set NICKEL import path for IaC validation
export NICKEL_IMPORT_PATH=/Users/Akasha/project-provisioning/provisioning

# Add scripts to PATH for convenience
export PATH="$PWD/provisioning/scripts:$PATH"
```

### System Requirements

- **CPU**: 8+ cores recommended
- **RAM**: 16+ GB (for running all 10 services simultaneously)
- **Disk**: 30+ GB free (for builds and artifacts)
- **macOS**: 12.0+ or Linux (Ubuntu 22.04+) recommended

---

## Repository Structure

### Core Provisioning Platform

```
provisioning/
├── core/                    # Core CLI and libraries
│   ├── cli/                 # CLI implementation
│   ├── libs/                # Shared libraries
│   └── plugins/             # Plugin system
├── platform/                # Platform services (10 crates)
│   ├── crates/
│   │   ├── orchestrator/    # Batch workflow orchestrator
│   │   ├── control-center/  # Control plane + auth
│   │   ├── extension-registry/  # OCI registry
│   │   ├── provisioning-daemon/ # Nushell execution
│   │   ├── ai-service/      # RAG + MCP tools
│   │   ├── provisioning-rag/    # Vector search
│   │   ├── mcp-server/      # Infrastructure automation
│   │   ├── vault-service/   # KMS + encryption
│   │   ├── detector/        # Infrastructure detection
│   │   └── control-center-ui/   # Web UI (Leptos/WASM)
│   └── Cargo.workspace.toml # Workspace definition
├── schemas/                 # Nickel IaC schemas
├── extensions/              # Provider extensions
├── config/                  # Configuration templates
└── scripts/                 # Helper scripts
```

### Workspace Infrastructure

```
workspaces/
├── docker-desktop/          # Docker Desktop K8s config
├── kind-cluster/            # Kind K8s cluster config
├── local-services/          # Local service definitions
└── README.md                # Workspace documentation
```

---

## Build All Services

### Quick Start: Build Everything

```bash
# Navigate to workspace
cd provisioning/platform

# Build all services in release mode (optimized)
cargo build --release --workspace

# Build specific service
cargo build --release -p orchestrator

# Build with development profile (faster, debug symbols)
cargo build --workspace
```

### Build Time Estimates

- **Full workspace rebuild**: 8-12 minutes (first time)
- **Incremental rebuild**: 1-3 minutes (after changes)
- **Single service**: 2-5 minutes
- **Clean rebuild**: 15-20 minutes

### Build Optimization

```bash
# Use parallel compilation (adjust for your CPU)
export CARGO_BUILD_JOBS=8

# Enable incremental compilation
export CARGO_INCREMENTAL=1

# Use sccache for caching (speeds up rebuilds)
cargo install sccache
export RUSTC_WRAPPER=sccache

# Rebuild with these optimizations
cargo build --release --workspace
```

### Verify Build Success

```bash
# Check compiled binaries
ls -lh provisioning/platform/target/release/

# Expected binaries (examples):
# orchestrator
# control-center
# extension-registry
# provisioning-daemon
# ai-service
# vault-service
# mcp-server
# detector
# provisioning-rag
# control-center-ui
```

---

## Service Dependencies

### Service Startup Order (Critical)

Services have dependencies and must start in specific order:

```
1. vault-service (port 8081)          # Auth dependencies
   └─ Required by: All services

2. extension-registry (port 8082)     # OCI registry
   └─ Required by: orchestrator, provisioning-daemon

3. control-center (port 8000)         # Core control plane
   ├─ Requires: vault-service
   └─ Required by: Most services

4. provisioning-rag (port 8300)       # Vector search DB
   └─ Required by: ai-service

5. ai-service (port 8083)             # RAG + MCP
   ├─ Requires: provisioning-rag, vault-service
   └─ Required by: orchestrator

6. mcp-server (port 8400)             # Infrastructure automation
   ├─ Requires: vault-service
   └─ Used by: orchestrator

7. orchestrator (port 9090)           # Batch workflows
   ├─ Requires: extension-registry, control-center, ai-service
   └─ Core service

8. provisioning-daemon (port 8100)    # Script execution
   ├─ Requires: vault-service
   └─ Executes: Nushell scripts

9. detector (port 8600)               # Infrastructure detection
   ├─ Requires: vault-service
   └─ Optional for: Multi-cloud detection

10. control-center-ui (port 3000)     # Web UI
    ├─ Requires: control-center
    └─ Access point: http://localhost:3000
```

### Dependency Graph

```
vault-service (8081)
    ↓
    ├─→ control-center (8000)
    │       ├─→ provisioning-daemon (8100)
    │       └─→ orchestrator (9090)
    │
    ├─→ ai-service (8083)
    │   └─ provisioning-rag (8300)
    │
    ├─→ mcp-server (8400)
    │
    └─→ detector (8600)

extension-registry (8082)
    └─→ orchestrator (9090)

control-center-ui (3000)
    └─→ control-center (8000)
```

### Startup Script

Use the provided script to start services in correct order:

```bash
# Start core services only (vault, registry, control-center)
provisioning-scripts/start-local-binaries.nu --services core

# Start all services
provisioning-scripts/start-local-binaries.nu --services all

# Start specific services
provisioning-scripts/start-local-binaries.nu --services custom orchestrator control-center ai-service
```

---

## Configuration Files

### Service Configuration Locations

```bash
# Configuration directory
~/.config/provisioning/

# Create required directories
mkdir -p ~/.config/provisioning/{dev,local}
```

### Nickel Schema Files (Source of Truth)

```bash
# Nickel IaC schemas - Use ONLY these, never hardcode
provisioning/schemas/
├── main.ncl                 # Main schema entry point
├── platform/
│   ├── schemas/             # Service schemas (validated)
│   │   ├── orchestrator.ncl
│   │   ├── control-center.ncl
│   │   ├── ai-service.ncl
│   │   └── ...
│   └── defaults/            # Default configurations
│       ├── orchestrator-defaults.ncl
│       ├── service-defaults.ncl
│       └── ...
└── common/
    └── helpers.ncl          # Composition utilities
```

### Load Hierarchy (High to Low)

1. **CLI Arguments** - `--config`, `--port`, etc.
2. **Environment Variables** - `PROVISIONING_*`
3. **User Configuration** - `~/.config/provisioning/user-config.ncl`
4. **Infrastructure Configuration** - Nickel schemas
5. **System Defaults** - `provisioning/config/defaults.toml`

### Example: Loading Orchestrator Config

```nickel
# Load schemas
let orchestrator_schema = import "schemas/platform/schemas/orchestrator.ncl" in
let defaults = import "schemas/platform/defaults/orchestrator-defaults.ncl" in
let user_overrides = import "~/.config/provisioning/orchestrator-config.ncl" in
let helpers = import "schemas/platform/common/helpers.ncl" in

# Compose config (deep merge: defaults → user overrides)
let config = helpers.compose_config defaults user_overrides in
config | orchestrator_schema.OrchestratorConfig
```

### Deep Merge vs Shallow Merge

⚠️ **CRITICAL**: Always use `helpers.compose_config` (deep merge), NEVER use `&` operator (shallow):

```nickel
# ❌ WRONG - Shallow merge (loses nested fields)
{server = {port = 9000}} & {server = {workers = 4}}
# Result: {server = {workers = 4}}  ← port lost!

# ✅ CORRECT - Deep merge (preserves nested fields)
helpers.compose_config 
  {server = {port = 9000}} 
  {server = {workers = 4}}
# Result: {server = {port = 9000, workers = 4}}
```

---

## Database & Storage Setup

### PostgreSQL (if required)

```bash
# Start PostgreSQL in Docker (optional)
docker run -d \
  --name provisioning-postgres \
  -e POSTGRES_PASSWORD=dev-password \
  -e POSTGRES_USER=provisioning \
  -e POSTGRES_DB=provisioning_db \
  -p 5432:5432 \
  postgres:16-alpine

# Wait for postgres to be ready
sleep 5

# Verify connection
psql -h localhost -U provisioning -d provisioning_db -c "SELECT 1"
```

### Vector Database (for ai-service/provisioning-rag)

```bash
# Option 1: Use in-memory (for development)
# Set environment: VECTOR_DB_MODE=memory

# Option 2: Use Postgres with pgvector extension
# Set environment: VECTOR_DB_MODE=postgres

# Option 3: Use external vector service
# Set environment: VECTOR_DB_URL=http://localhost:8300
```

### Redis Cache (for orchestrator)

```bash
# Start Redis (optional, for distributed caching)
docker run -d \
  --name provisioning-redis \
  -p 6379:6379 \
  redis:7-alpine

# Verify connection
redis-cli ping  # Should return PONG
```

### Cleanup

```bash
# Stop all containers
docker stop provisioning-postgres provisioning-redis

# Remove containers
docker rm provisioning-postgres provisioning-redis

# Remove images
docker rmi postgres:16-alpine redis:7-alpine
```

---

## Security Keys Setup

### Generate Required Keys

All services require cryptographic keys for encryption, signing, and authentication.

```bash
# Create keys directory
mkdir -p ~/.provisioning/keys/{dev,prod}
cd ~/.provisioning/keys/dev

# Install Age (required for encryption)
# On macOS:
brew install age

# Or download: https://github.com/FiloSottile/age/releases
```

### Age Encryption Keys (Master Keys)

```bash
# Generate age keypair for development
age-keygen -o age-dev-key.txt

# Output format:
# age secret key: AGE-...[secret]
# Public key: age...[public]

# Store securely
chmod 600 age-dev-key.txt
export AGE_KEY=$(cat age-dev-key.txt | grep "^# public key:" | cut -d' ' -f4)

# For SOPS (encrypted config files)
echo "age: $AGE_KEY" > .sops.yaml
```

### Service API Keys

```bash
# Generate random API keys for services
openssl rand -hex 32 > api-keys.txt

# Expected format:
# orchestrator-key: 8f7a9c2b1e4d6f3a9b2c1d4e6f7a8b9c
# control-center-key: 9c2b1e4d6f3a9b2c1d4e6f7a8b9c2b1e
```

### JWT Signing Keys

```bash
# Generate JWT keys (for control-center auth)
openssl genrsa -out jwt-private.pem 2048
openssl rsa -in jwt-private.pem -pubout -out jwt-public.pem

# Verify keys
ls -lh jwt-*.pem
```

### Load Keys into Environment

```bash
# Create development environment file
cat > ~/.config/provisioning/dev-env.sh << 'EOF'
#!/bin/bash
export AGE_KEY=$(cat ~/.provisioning/keys/dev/age-dev-key.txt | grep "^# secret key:" | cut -d' ' -f4)
export PROVISIONING_API_KEY=$(cat ~/.provisioning/keys/dev/api-keys.txt | head -1 | cut -d: -f2)
export JWT_PRIVATE_KEY=$(cat ~/.provisioning/keys/dev/jwt-private.pem)
export JWT_PUBLIC_KEY=$(cat ~/.provisioning/keys/dev/jwt-public.pem)
EOF

chmod +x ~/.config/provisioning/dev-env.sh
source ~/.config/provisioning/dev-env.sh
```

### Security Best Practices

- ❌ **Never commit keys** to Git (add to `.gitignore`)
- ✅ **Use SOPS** for encrypted config files
- ✅ **Rotate keys regularly** (monthly for dev, quarterly for prod)
- ✅ **Use separate keys** for each environment
- ✅ **Store in secure location** (use vault for production)

---

## Running Services

### Method 1: Using Startup Script (Recommended)

```bash
# Start all services
./provisioning/scripts/start-local-binaries.nu --services all

# Start core services only (vault, registry, control-center)
./provisioning/scripts/start-local-binaries.nu --services core

# Start custom service set
./provisioning/scripts/start-local-binaries.nu --services custom vault-service control-center ai-service

# Available flags:
#   --services <set>    : core|all|custom (followed by service names)
#   --debug             : Enable debug logging
#   --health-check      : Enable health monitoring
#   --config <file>     : Load custom config file
```

### Method 2: Manual Start (for Debugging)

```bash
# Terminal 1: Start vault-service
cd provisioning/platform
cargo run --release -p vault-service

# Terminal 2: Start control-center
cd provisioning/platform
cargo run --release -p control-center -- --config ~/.config/provisioning/control-center.toml

# Terminal 3: Start orchestrator
cd provisioning/platform
cargo run --release -p orchestrator

# Terminal 4+: Start other services similarly
```

### Health Checks

```bash
# Check if services are running
curl -s http://localhost:8000/health
curl -s http://localhost:8081/health
curl -s http://localhost:9090/health

# Check all services at once
./provisioning/scripts/check-health.nu --verbose

# Monitor service status
watch -n 2 './provisioning/scripts/check-health.nu'
```

### View Logs

```bash
# Using startup script
./provisioning/scripts/start-local-binaries.nu --logs

# Manual logs (if running in background)
tail -f ~/.provisioning/logs/orchestrator.log
tail -f ~/.provisioning/logs/control-center.log

# Stream all logs
tail -f ~/.provisioning/logs/*.log
```

### Stop All Services

```bash
# Using startup script
./provisioning/scripts/start-local-binaries.nu --stop

# Manual stop
pkill -f "cargo run"
pkill orchestrator control-center provisioning-daemon ai-service
```

---

## Troubleshooting

### Service Won't Start

**Problem**: Service fails to start with port binding error  
**Cause**: Port already in use  
**Solution**:

```bash
# Find process using port (example: port 8000)
lsof -i :8000
# Kill process
kill -9 <PID>

# Or change service port (set in config)
export CONTROL_CENTER_PORT=8001
```

### Build Failures

**Problem**: `cargo build` fails with missing dependencies  
**Solution**:

```bash
# Update dependencies
cargo update

# Clean and rebuild
cargo clean
cargo build --release --workspace

# Check for toolchain issues
rustup update
rustc --version  # Verify 1.80+
```

**Problem**: WASM build for UI fails  
**Solution**:

```bash
# Install WASM target
rustup target add wasm32-unknown-unknown

# Rebuild UI
cargo build --release -p control-center-ui --target wasm32-unknown-unknown
```

### Configuration Errors

**Problem**: Nickel schema validation fails  
**Solution**:

```bash
# Set NICKEL_IMPORT_PATH
export NICKEL_IMPORT_PATH=/Users/Akasha/project-provisioning/provisioning

# Validate schema
nickel export schemas/main.ncl > /tmp/schema.json

# Check specific config
nickel export schemas/platform/schemas/orchestrator.ncl > /tmp/orchestrator-schema.json
```

### Connection Issues

**Problem**: Services can't communicate  
**Solution**:

```bash
# Check network connectivity
telnet localhost 8000
telnet localhost 8081

# Verify service is listening
netstat -an | grep 8000

# Check firewall rules
sudo pfctl -s all | grep localhost
```

### Database Connection Errors

**Problem**: PostgreSQL connection refused  
**Solution**:

```bash
# Check if postgres is running
docker ps | grep postgres

# Check logs
docker logs provisioning-postgres

# Restart postgres
docker restart provisioning-postgres

# Wait for startup
sleep 5
```

### Log Analysis

```bash
# Find errors in logs
grep ERROR ~/.provisioning/logs/*.log

# Check specific service
grep -A5 "vault-service" ~/.provisioning/logs/*.log

# Count errors by service
for file in ~/.provisioning/logs/*.log; do
  echo "$file: $(grep ERROR $file | wc -l) errors"
done
```

---

## Development Workflows

### Common Development Tasks

#### Adding a New Endpoint to Control Center

```bash
# 1. Edit the service code
nano provisioning/platform/crates/control-center/src/main.rs

# 2. Rebuild
cargo build -p control-center

# 3. Test
cargo test -p control-center

# 4. Run and test endpoint
curl -X POST http://localhost:8000/api/v1/new-endpoint
```

#### Modifying Nickel Schema

```bash
# 1. Edit schema
nano provisioning/schemas/platform/schemas/orchestrator.ncl

# 2. Validate schema
export NICKEL_IMPORT_PATH=$PWD/provisioning
nickel export provisioning/schemas/platform/schemas/orchestrator.ncl

# 3. Regenerate TOML (if used)
nickel export provisioning/schemas/main.ncl > provisioning/config/generated.toml

# 4. Test with services
./provisioning/scripts/start-local-binaries.nu --services core
```

#### Running Tests

```bash
# Run all tests
cargo test --workspace

# Run tests for specific service
cargo test -p orchestrator

# Run with output
cargo test --workspace -- --nocapture

# Run benchmarks
cargo bench --workspace

# Run with coverage (requires tarpaulin)
cargo install cargo-tarpaulin
cargo tarpaulin --workspace
```

#### Code Formatting & Linting

```bash
# Format code
cargo fmt --all

# Check formatting
cargo fmt --all -- --check

# Run clippy (lint)
cargo clippy --all -- -D warnings

# Fix clippy warnings automatically
cargo clippy --fix --allow-dirty --workspace
```

#### Profiling Performance

```bash
# Run with perf (Linux)
perf record -g cargo run --release -p orchestrator
perf report

# Or use flamegraph
cargo install flamegraph
cargo flamegraph -p orchestrator

# macOS profiling
sudo dtrace -c "cargo run --release -p orchestrator" -n 'syscall:::entry { @[execname] = count() }'
```

---

## Quick Reference

### Essential Commands

```bash
# Build everything
cd provisioning/platform && cargo build --release --workspace

# Start services
./provisioning/scripts/start-local-binaries.nu --services all

# Run tests
cargo test --workspace

# Format code
cargo fmt --all && cargo clippy --fix --allow-dirty --workspace

# Check health
./provisioning/scripts/check-health.nu

# View logs
tail -f ~/.provisioning/logs/*.log

# Stop all services
pkill -f "cargo run"
```

### Service Ports Quick Reference

| Service | Port | Protocol |
|---------|------|----------|
| control-center | 8000 | HTTP/WebSocket |
| vault-service | 8081 | gRPC |
| extension-registry | 8082 | HTTP |
| ai-service | 8083 | HTTP |
| provisioning-daemon | 8100 | gRPC |
| provisioning-rag | 8300 | REST |
| mcp-server | 8400 | Binary |
| detector | 8600 | HTTP |
| orchestrator | 9090 | HTTP |
| control-center-ui | 3000 | HTTP (WASM) |

### Configuration File Locations

| Component | Config | Location |
|-----------|--------|----------|
| Orchestrator | TOML | `~/.config/provisioning/orchestrator.toml` |
| Control Center | TOML | `~/.config/provisioning/control-center.toml` |
| User Config | Nickel | `~/.config/provisioning/user-config.ncl` |
| Development Keys | Age | `~/.provisioning/keys/dev/` |
| Logs | Log files | `~/.provisioning/logs/` |

---

## Getting Help

### Resources

- **Documentation**: `provisioning/docs/src/`
- **Examples**: `provisioning/examples/`
- **Tests**: `provisioning/platform/crates/*/tests/`
- **Guides**: `provisioning/docs/src/guides/`

### Common Issues

See [Troubleshooting](#troubleshooting) section above for:
- Service startup problems
- Build failures
- Configuration errors
- Connection issues
- Database problems

### Next Steps

1. Complete [Prerequisites](#prerequisites)
2. Follow [Build All Services](#build-all-services)
3. Setup [Security Keys](#security-keys-setup)
4. Run services using [Running Services](#running-services)
5. Verify health checks and logs

---

**Last Updated**: 2025-02-03  
**Audience**: Local Development  
**Status**: Production Ready
