Provisioning Platform Documentation

Last Updated: 2025-01-02 (Phase 3.A Cleanup Complete) Status: ✅ Primary documentation source (145 files consolidated)

Welcome to the comprehensive documentation for the Provisioning Platform - a modern, cloud-native infrastructure automation system built with Nushell, Nickel, and Rust.

Note: Architecture Decision Records (ADRs) and design documentation are in docs/ directory. This location contains user-facing, operational, and product documentation.

Quick Navigation

🚀 Getting Started

📚 User Guides

🏗️ Architecture

📋 Architecture Decision Records (ADRs)

🔌 API Documentation

🛠️ Development

🐛 Troubleshooting

📖 How-To Guides

🔐 Configuration

📦 Quick References

Documentation Structure

provisioning/docs/src/
├── README.md (this file)          # Documentation hub
├── getting-started/               # Getting started guides
│   ├── installation-guide.md
│   ├── getting-started.md
│   └── quickstart-cheatsheet.md
├── architecture/                  # System architecture
│   ├── adr/                       # Architecture Decision Records
│   ├── design-principles.md
│   ├── integration-patterns.md
│   ├── system-overview.md
│   └── ... (and 10+ more architecture docs)
├── infrastructure/                # Infrastructure guides
│   ├── cli-reference.md
│   ├── workspace-setup.md
│   ├── workspace-switching-guide.md
│   └── infrastructure-management.md
├── api-reference/                 # API documentation
│   ├── rest-api.md
│   ├── websocket.md
│   ├── integration-examples.md
│   └── sdks.md
├── development/                   # Developer guides
│   ├── README.md
│   ├── implementation-guide.md
│   ├── quick-provider-guide.md
│   ├── taskserv-developer-guide.md
│   └── ... (15+ more developer docs)
├── guides/                        # How-to guides
│   ├── from-scratch.md
│   ├── update-infrastructure.md
│   └── customize-infrastructure.md
├── operations/                    # Operations guides
│   ├── service-management-guide.md
│   ├── coredns-guide.md
│   └── ... (more operations docs)
├── security/                      # Security docs
├── integration/                   # Integration guides
├── testing/                       # Testing docs
├── configuration/                 # Configuration docs
├── troubleshooting/               # Troubleshooting guides
└── quick-reference/               # Quick references

Key Concepts

Infrastructure as Code (IaC)

The provisioning platform uses declarative configuration to manage infrastructure. Instead of manually creating resources, you define what you want in Nickel configuration files, and the system makes it happen.

Mode-Based Architecture

The system supports four operational modes:

• Solo: Single developer local development
• Multi-user: Team collaboration with shared services
• CI/CD: Automated pipeline execution
• Enterprise: Production deployment with strict compliance

Extension System

Extensibility through:

• Providers: Cloud platform integrations (AWS, UpCloud, Local)
• Task Services: Infrastructure components (Kubernetes, databases, etc.)
• Clusters: Complete deployment configurations

OCI-Native Distribution

Extensions and packages distributed as OCI artifacts, enabling:

• Industry-standard packaging
• Efficient caching and bandwidth
• Version pinning and rollback
• Air-gapped deployments

Documentation by Role

For New Users

1. Start with Installation Guide
2. Read Getting Started
3. Follow From Scratch Guide
4. Reference Quickstart Cheatsheet

For Developers

1. Review System Overview
2. Study Design Principles
3. Read relevant ADRs
4. Follow Development Guide
5. Reference Nickel Quick Reference

For Operators

1. Understand Mode System
2. Learn Service Management
3. Review Infrastructure Management
4. Study OCI Registry

For Architects

1. Read System Overview
2. Study all ADRs
3. Review Integration Patterns
4. Understand Multi-Repo Architecture

System Capabilities

✅ Infrastructure Automation

• Multi-cloud support (AWS, UpCloud, Local)
• Declarative configuration with Nickel
• Automated dependency resolution
• Batch operations with rollback

✅ Workflow Orchestration

• Hybrid Rust/Nushell orchestration
• Checkpoint-based recovery
• Parallel execution with limits
• Real-time monitoring

✅ Test Environments

• Containerized testing
• Multi-node cluster simulation
• Topology templates
• Automated cleanup

✅ Mode-Based Operation

• Solo: Local development
• Multi-user: Team collaboration
• CI/CD: Automated pipelines
• Enterprise: Production deployment

✅ Extension Management

• OCI-native distribution
• Automatic dependency resolution
• Version management
• Local and remote sources

Key Achievements

🚀 Batch Workflow System (v3.1.0)

• Provider-agnostic batch operations
• Mixed provider support (UpCloud + AWS + local)
• Dependency resolution with soft/hard dependencies
• Real-time monitoring and rollback

🏗️ Hybrid Orchestrator (v3.0.0)

• Solves Nushell deep call stack limitations
• Preserves all business logic
• REST API for external integration
• Checkpoint-based state management

⚙️ Configuration System (v2.0.0)

• Migrated from ENV to config-driven
• Hierarchical configuration loading
• Variable interpolation
• True IaC without hardcoded fallbacks

🎯 Modular CLI (v3.2.0)

• 84% reduction in main file size
• Domain-driven handlers
• 80+ shortcuts
• Bi-directional help system

🧪 Test Environment Service (v3.4.0)

• Automated containerized testing
• Multi-node cluster topologies
• CI/CD integration ready
• Template-based configurations

🔄 Workspace Switching (v2.0.5)

• Centralized workspace management
• Single-command workspace switching
• Active workspace tracking
• User preference system

Technology Stack

Support

Getting Help

• Documentation: You’re reading it!
• Quick Reference: Run provisioning sc or provisioning guide quickstart
• Help System: Run provisioning help or provisioning <command> help
• Interactive Shell: Run provisioning nu for Nushell REPL

Reporting Issues

• Check Troubleshooting Guide
• Review FAQ
• Enable debug mode: provisioning --debug <command>
• Check logs: provisioning platform logs <service>

Contributing

This project welcomes contributions! See Development Guide for:

• Development setup
• Code style guidelines
• Testing requirements
• Pull request process

License

[Add license information]

Version History

Maintained By: Provisioning Team Last Review: 2025-10-06 Next Review: 2026-01-06

Installation Guide

This guide will help you install Infrastructure Automation on your machine and get it ready for use.

What You’ll Learn

• System requirements and prerequisites
• Different installation methods
• How to verify your installation
• Setting up your environment
• Troubleshooting common installation issues

System Requirements

Operating System Support

• Linux: Any modern distribution (Ubuntu 20.04+, CentOS 8+, Debian 11+)
• macOS: 11.0+ (Big Sur and newer)
• Windows: Windows 10/11 with WSL2

Hardware Requirements

Architecture Support

• x86_64 (Intel/AMD 64-bit) - Full support
• ARM64 (Apple Silicon, ARM servers) - Full support

Prerequisites

Before installation, ensure you have:

1. Administrative privileges - Required for system-wide installation
2. Internet connection - For downloading dependencies
3. Terminal/Command line access - Basic command line knowledge helpful

Pre-installation Checklist

# Check your system
uname -a                    # View system information
df -h                      # Check available disk space
curl --version             # Verify internet connectivity

Installation Methods

Method 1: Package Installation (Recommended)

This is the easiest method for most users.

Step 1: Download the Package

# Download the latest release package
wget https://releases.example.com/provisioning-latest.tar.gz

# Or using curl
curl -LO https://releases.example.com/provisioning-latest.tar.gz

Step 2: Extract and Install

# Extract the package
tar xzf provisioning-latest.tar.gz

# Navigate to extracted directory
cd provisioning-*

# Run the installation script
sudo ./install-provisioning

The installer will:

• Install to /usr/local/provisioning
• Create a global command at /usr/local/bin/provisioning
• Install all required dependencies
• Set up configuration templates

Method 2: Container Installation

For containerized environments or testing.

Using Docker

# Pull the provisioning container
docker pull provisioning:latest

# Create a container with persistent storage
docker run -it --name provisioning-setup \
  -v ~/provisioning-data:/data \
  provisioning:latest

# Install to host system (optional)
docker cp provisioning-setup:/usr/local/provisioning ./
sudo cp -r ./provisioning /usr/local/
sudo ln -sf /usr/local/provisioning/bin/provisioning /usr/local/bin/provisioning

Using Podman

# Similar to Docker but with Podman
podman pull provisioning:latest
podman run -it --name provisioning-setup \
  -v ~/provisioning-data:/data \
  provisioning:latest

Method 3: Source Installation

For developers or custom installations.

Prerequisites for Source Installation

• Git - For cloning the repository
• Build tools - Compiler toolchain for your platform

Installation Steps

# Clone the repository
git clone https://github.com/your-org/provisioning.git
cd provisioning

# Run installation from source
./distro/from-repo.sh

# Or if you have development environment
./distro/pack-install.sh

Method 4: Manual Installation

For advanced users who want complete control.

# Create installation directory
sudo mkdir -p /usr/local/provisioning

# Copy files (assumes you have the source)
sudo cp -r ./* /usr/local/provisioning/

# Create global command
sudo ln -sf /usr/local/provisioning/core/nulib/provisioning /usr/local/bin/provisioning

# Install dependencies manually
./install-dependencies.sh

Installation Process Details

What Gets Installed

The installation process sets up:

1. Core System Files

/usr/local/provisioning/
├── core/                 # Core provisioning logic
├── providers/            # Cloud provider integrations
├── taskservs/           # Infrastructure services
├── cluster/             # Cluster configurations
├── schemas/             # Configuration schemas (Nickel)
├── templates/           # Template files
└── resources/           # Project resources

2. Required Tools

3. Nushell Plugins

• nu_plugin_tera - Template rendering

4. Configuration Files

• User configuration templates
• Environment-specific configs
• Default settings and schemas

Post-Installation Verification

Basic Verification

# Check if provisioning command is available
provisioning --version

# Verify installation
provisioning env

# Show comprehensive environment info
provisioning allenv

Expected output should show:

✅ Provisioning v1.0.0 installed
✅ All dependencies available
✅ Configuration loaded successfully

Tool Verification

# Check individual tools
nu --version              # Should show Nushell 0.109.0+
nickel version            # Should show Nickel 1.5+
sops --version           # Should show SOPS 3.10.2
age --version            # Should show Age 1.2.1
k9s version              # Should show K9s 0.50.6

Plugin Verification

# Start Nushell and check plugins
nu -c "version | get installed_plugins"

# Should include:
# - nu_plugin_tera (template rendering)

Configuration Verification

# Validate configuration
provisioning validate config

# Should show:
# ✅ Configuration validation passed!

Environment Setup

Shell Configuration

Add to your shell profile (~/.bashrc, ~/.zshrc, or ~/.profile):

# Add provisioning to PATH
export PATH="/usr/local/bin:$PATH"

# Optional: Set default provisioning directory
export PROVISIONING="/usr/local/provisioning"

Configuration Initialization

# Initialize user configuration
provisioning init config

# This creates ~/.provisioning/config.user.toml

First-Time Setup

# Set up your first workspace
mkdir -p ~/provisioning-workspace
cd ~/provisioning-workspace

# Initialize workspace
provisioning init config dev

# Verify setup
provisioning env

Platform-Specific Instructions

Linux (Ubuntu/Debian)

# Install system dependencies
sudo apt update
sudo apt install -y curl wget tar

# Proceed with standard installation
wget https://releases.example.com/provisioning-latest.tar.gz
tar xzf provisioning-latest.tar.gz
cd provisioning-*
sudo ./install-provisioning

Linux (RHEL/CentOS/Fedora)

# Install system dependencies
sudo dnf install -y curl wget tar
# or for older versions: sudo yum install -y curl wget tar

# Proceed with standard installation

macOS

# Using Homebrew (if available)
brew install curl wget

# Or download directly
curl -LO https://releases.example.com/provisioning-latest.tar.gz
tar xzf provisioning-latest.tar.gz
cd provisioning-*
sudo ./install-provisioning

Windows (WSL2)

# In WSL2 terminal
sudo apt update
sudo apt install -y curl wget tar

# Proceed with Linux installation steps
wget https://releases.example.com/provisioning-latest.tar.gz
# ... continue as Linux

Configuration Examples

Basic Configuration

Create ~/.provisioning/config.user.toml:

[core]
name = "my-provisioning"

[paths]
base = "/usr/local/provisioning"
infra = "~/provisioning-workspace"

[debug]
enabled = false
log_level = "info"

[providers]
default = "local"

[output]
format = "yaml"

Development Configuration

For developers, use enhanced debugging:

[debug]
enabled = true
log_level = "debug"
check = true

[cache]
enabled = false  # Disable caching during development

Upgrade and Migration

Upgrading from Previous Version

# Backup current installation
sudo cp -r /usr/local/provisioning /usr/local/provisioning.backup

# Download new version
wget https://releases.example.com/provisioning-latest.tar.gz

# Extract and install
tar xzf provisioning-latest.tar.gz
cd provisioning-*
sudo ./install-provisioning

# Verify upgrade
provisioning --version

Migrating Configuration

# Backup your configuration
cp -r ~/.provisioning ~/.provisioning.backup

# Initialize new configuration
provisioning init config

# Manually merge important settings from backup

Troubleshooting Installation Issues

Common Installation Problems

Permission Denied Errors

# Problem: Cannot write to /usr/local
# Solution: Use sudo
sudo ./install-provisioning

# Or install to user directory
./install-provisioning --prefix=$HOME/provisioning
export PATH="$HOME/provisioning/bin:$PATH"

Missing Dependencies

# Problem: curl/wget not found
# Ubuntu/Debian solution:
sudo apt install -y curl wget tar

# RHEL/CentOS solution:
sudo dnf install -y curl wget tar

Download Failures

# Problem: Cannot download package
# Solution: Check internet connection and try alternative
ping google.com

# Try alternative download method
curl -LO --retry 3 https://releases.example.com/provisioning-latest.tar.gz

# Or use wget with retries
wget --tries=3 https://releases.example.com/provisioning-latest.tar.gz

Extraction Failures

# Problem: Archive corrupted
# Solution: Verify and re-download
sha256sum provisioning-latest.tar.gz  # Check against published hash

# Re-download if hash doesn't match
rm provisioning-latest.tar.gz
wget https://releases.example.com/provisioning-latest.tar.gz

Tool Installation Failures

# Problem: Nushell installation fails
# Solution: Check architecture and OS compatibility
uname -m    # Should show x86_64 or arm64
uname -s    # Should show Linux, Darwin, etc.

# Try manual tool installation
./install-dependencies.sh --verbose

Verification Failures

Command Not Found

# Problem: 'provisioning' command not found
# Check installation path
ls -la /usr/local/bin/provisioning

# If missing, create symlink
sudo ln -sf /usr/local/provisioning/core/nulib/provisioning /usr/local/bin/provisioning

# Add to PATH if needed
export PATH="/usr/local/bin:$PATH"
echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc

Plugin Errors

# Problem: Plugin command not found
# Solution: Ensure plugin is properly registered

# Check available plugins
nu -c "version | get installed_plugins"

# If plugin missing, reload Nushell:
exec nu

Configuration Errors

# Problem: Configuration validation fails
# Solution: Initialize with template
provisioning init config

# Or validate and show errors
provisioning validate config --detailed

Getting Help

If you encounter issues not covered here:

1. Check logs: provisioning --debug env
2. Validate configuration: provisioning validate config
3. Check system compatibility: provisioning version --verbose
4. Consult troubleshooting guide: docs/user/troubleshooting-guide.md

Next Steps

After successful installation:

1. Complete the Getting Started Guide: docs/user/getting-started.md
2. Set up your first workspace: docs/user/workspace-setup.md
3. Learn about configuration: docs/user/configuration.md
4. Try example tutorials: docs/user/examples/

Your provisioning is now ready to manage cloud infrastructure!

Installation Validation & Bootstrap Guide

Objective: Validate your provisioning installation, run bootstrap to initialize the workspace, and verify all components are working correctly.

Expected Duration: 30-45 minutes

Prerequisites: Fresh clone of provisioning repository at /Users/Akasha/project-provisioning

Section 1: Prerequisites Verification

Before running the bootstrap script, verify that your system has all required dependencies.

Step 1.1: Check System Requirements

Run these commands to verify your system meets minimum requirements:

# Check OS
uname -s
# Expected: Darwin (macOS), Linux, or WSL2

# Check CPU cores
sysctl -n hw.physicalcpu  # macOS
# OR
nproc  # Linux
# Expected: 2 or more cores

# Check RAM
sysctl -n hw.memsize | awk '{print int($1 / 1024 / 1024 / 1024) " GB"}' # macOS
# OR
grep MemTotal /proc/meminfo | awk '{print int($2 / 1024 / 1024) " GB"}'  # Linux
# Expected: 2 GB or more (4 GB+ recommended)

# Check free disk space
df -h | grep -E '^/dev|^Filesystem'
# Expected: At least 2 GB free (10 GB+ recommended)

Success Criteria:

• OS is macOS, Linux, or WSL2
• CPU: 2+ cores available
• RAM: 2 GB minimum, 4+ GB recommended
• Disk: 2 GB free minimum

Step 1.2: Verify Nushell Installation

Nushell is required for bootstrap and CLI operations:

command -v nu
# Expected output: /path/to/nu

nu --version
# Expected output: 0.109.0 or higher

If Nushell is not installed:

# macOS (using Homebrew)
brew install nushell

# Linux (Debian/Ubuntu)
sudo apt-get update && sudo apt-get install nushell

# Linux (RHEL/CentOS)
sudo yum install nushell

# Or install from source: https://nushell.sh/book/installation.html

Step 1.3: Verify Nickel Installation

Nickel is required for configuration validation:

command -v nickel
# Expected output: /path/to/nickel

nickel --version
# Expected output: nickel 1.x.x or higher

If Nickel is not installed:

# Install via Cargo (requires Rust)
cargo install nickel-lang-cli

# Or: https://nickel-lang.org/

Step 1.4: Verify Docker Installation

Docker is required for running containerized services:

command -v docker
# Expected output: /path/to/docker

docker --version
# Expected output: Docker version 20.10 or higher

If Docker is not installed:

Visit Docker installation guide and install for your OS.

Step 1.5: Check Provisioning Binary

Verify the provisioning CLI binary exists:

ls -la /Users/Akasha/project-provisioning/provisioning/core/cli/provisioning
# Expected: -rwxr-xr-x (executable)

file /Users/Akasha/project-provisioning/provisioning/core/cli/provisioning
# Expected: ELF 64-bit or similar binary format

If binary is not executable:

chmod +x /Users/Akasha/project-provisioning/provisioning/core/cli/provisioning

Prerequisites Checklist

[ ] OS is macOS, Linux, or WSL2
[ ] CPU: 2+ cores available
[ ] RAM: 2 GB minimum installed
[ ] Disk: 2+ GB free space
[ ] Nushell 0.109.0+ installed
[ ] Nickel 1.x.x installed
[ ] Docker 20.10+ installed
[ ] Provisioning binary exists and is executable

Section 2: Bootstrap Installation

The bootstrap script automates 7 stages of installation and initialization. Run it from the project root directory.

Step 2.1: Navigate to Project Root

cd /Users/Akasha/project-provisioning

Step 2.2: Run Bootstrap Script

./provisioning/bootstrap/install.sh

Bootstrap Output

You should see output similar to this:

╔════════════════════════════════════════════════════════════════╗
║              PROVISIONING BOOTSTRAP (Bash)                     ║
╚════════════════════════════════════════════════════════════════╝

📊 Stage 1: System Detection
─────────────────────────────────────────────────────────────────
  OS: Darwin
  Architecture: arm64 (or x86_64)
  CPU Cores: 8
  Memory: 16 GB
  ✅ System requirements met

📦 Stage 2: Checking Dependencies
─────────────────────────────────────────────────────────────────
  Versions:
    Docker: Docker version 28.5.2
    Rust: rustc 1.75.0
    Nushell: 0.109.1
  ✅ All dependencies found

📁 Stage 3: Creating Directory Structure
─────────────────────────────────────────────────────────────────
  ✅ Directory structure created

⚙️  Stage 4: Validating Configuration
─────────────────────────────────────────────────────────────────
  ✅ Configuration syntax valid

📤 Stage 5: Exporting Configuration to TOML
─────────────────────────────────────────────────────────────────
  ✅ Configuration exported

🚀 Stage 6: Initializing Orchestrator Service
─────────────────────────────────────────────────────────────────
  ✅ Orchestrator started

✅ Stage 7: Verification
─────────────────────────────────────────────────────────────────
  ✅ All configuration files generated
  ✅ All required directories created

╔════════════════════════════════════════════════════════════════╗
║                   BOOTSTRAP COMPLETE ✅                        ║
╚════════════════════════════════════════════════════════════════╝

📍 Next Steps:

1. Verify configuration:
   cat /Users/Akasha/project-provisioning/workspaces/workspace_librecloud/config/config.ncl

2. Check orchestrator is running:
   curl http://localhost:9090/health

3. Start provisioning:
   provisioning server create --infra sgoyol --name web-01

What Bootstrap Does

The bootstrap script automatically:

1. Detects your system (OS, CPU, RAM, architecture)
2. Verifies dependencies (Docker, Rust, Nushell)
3. Creates workspace directories (config, state, cache)
4. Validates Nickel configuration (syntax checking)
5. Exports configuration (Nickel → TOML files)
6. Initializes orchestrator (starts service in background)
7. Verifies installation (checks all files created)

Section 3: Installation Validation

After bootstrap completes, verify that all components are working correctly.

Step 3.1: Verify Workspace Directories

Bootstrap should have created workspace directories. Verify they exist:

cd /Users/Akasha/project-provisioning

# Check all required directories
ls -la workspaces/workspace_librecloud/.orchestrator/data/queue/
ls -la workspaces/workspace_librecloud/.kms/
ls -la workspaces/workspace_librecloud/.providers/
ls -la workspaces/workspace_librecloud/.taskservs/
ls -la workspaces/workspace_librecloud/.clusters/

Expected Output:

total 0
drwxr-xr-x  2 user  group  64 Jan  7 10:30 .

(directories exist and are accessible)

Step 3.2: Verify Generated Configuration Files

Bootstrap should have exported Nickel configuration to TOML format:

# Check generated files exist
ls -la workspaces/workspace_librecloud/config/generated/

# View workspace configuration
cat workspaces/workspace_librecloud/config/generated/workspace.toml

# View provider configuration
cat workspaces/workspace_librecloud/config/generated/providers/upcloud.toml

# View orchestrator configuration
cat workspaces/workspace_librecloud/config/generated/platform/orchestrator.toml

Expected Output:

config/
├── generated/
│   ├── workspace.toml
│   ├── providers/
│   │   └── upcloud.toml
│   └── platform/
│       └── orchestrator.toml

Step 3.3: Type-Check Nickel Configuration

Verify Nickel configuration files have valid syntax:

cd /Users/Akasha/project-provisioning/workspaces/workspace_librecloud

# Type-check main workspace config
nickel typecheck config/config.ncl
# Expected: No output (success) or clear error messages

# Type-check infrastructure configs
nickel typecheck infra/wuji/main.ncl
nickel typecheck infra/sgoyol/main.ncl

# Use workspace utility for comprehensive validation
nu workspace.nu validate
# Expected: ✓ All files validated successfully

# Type-check all Nickel files
nu workspace.nu typecheck

Expected Output:

✓ All files validated successfully
✓ infra/wuji/main.ncl
✓ infra/sgoyol/main.ncl

Step 3.4: Verify Orchestrator Service

The orchestrator service manages workflows and deployments:

# Check if orchestrator is running (health check)
curl http://localhost:9090/health
# Expected: {"status": "healthy"} or similar response

# If health check fails, check orchestrator logs
tail -f /Users/Akasha/project-provisioning/provisioning/platform/orchestrator/data/orchestrator.log

# Alternative: Check if orchestrator process is running
ps aux | grep orchestrator
# Expected: Running orchestrator process visible

Expected Output:

{
  "status": "healthy",
  "uptime": "0:05:23"
}

If Orchestrator Failed to Start:

Check logs and restart manually:

cd /Users/Akasha/project-provisioning/provisioning/platform/orchestrator

# Check log file
cat data/orchestrator.log

# Or start orchestrator manually
./scripts/start-orchestrator.nu --background

# Verify it's running
curl http://localhost:9090/health

Step 3.5: Install Provisioning CLI (Optional)

You can install the provisioning CLI globally for easier access:

# Option A: System-wide installation (requires sudo)
cd /Users/Akasha/project-provisioning
sudo ./scripts/install-provisioning.sh

# Verify installation
provisioning --version
provisioning help

# Option B: Add to PATH temporarily (current session only)
export PATH="$PATH:/Users/Akasha/project-provisioning/provisioning/core/cli"

# Verify
provisioning --version

Expected Output:

provisioning version 1.0.0

Usage: provisioning [OPTIONS] COMMAND

Commands:
  server     - Server management
  workspace  - Workspace management
  config     - Configuration management
  help       - Show help information

Installation Validation Checklist

[ ] Workspace directories created (.orchestrator, .kms, .providers, .taskservs, .clusters)
[ ] Generated TOML files exist in config/generated/
[ ] Nickel type-checking passes (no errors)
[ ] Workspace utility validation passes
[ ] Orchestrator responding to health check
[ ] Orchestrator process running
[ ] Provisioning CLI accessible and working

Section 4: Troubleshooting

This section covers common issues and solutions.

Issue: “Nushell not found”

Symptoms:

./provisioning/bootstrap/install.sh: line X: nu: command not found

Solution:

1. Install Nushell (see Step 1.2)
2. Verify installation: nu --version
3. Retry bootstrap script

Issue: “Nickel configuration validation failed”

Symptoms:

⚙️  Stage 4: Validating Configuration
Error: Nickel configuration validation failed

Solution:

1. Check Nickel syntax: nickel typecheck config/config.ncl
2. Review error message for specific issue
3. Edit config file: vim config/config.ncl
4. Run bootstrap again

Issue: “Docker not installed”

Symptoms:

❌ Docker is required but not installed

Solution:

1. Install Docker: Docker installation guide
2. Verify: docker --version
3. Retry bootstrap script

Issue: “Configuration export failed”

Symptoms:

⚠️ Configuration export encountered issues (may continue)

Solution:

1. Check Nushell library paths: nu -c "use provisioning/core/nulib/lib_provisioning/config/export.nu *"
2. Verify export library exists: ls provisioning/core/nulib/lib_provisioning/config/export.nu
3. Re-export manually:

   cd /Users/Akasha/project-provisioning
   nu -c "
     use provisioning/core/nulib/lib_provisioning/config/export.nu *
     export-all-configs 'workspaces/workspace_librecloud'
   "
   

Issue: “Orchestrator didn’t start”

Symptoms:

🚀 Stage 6: Initializing Orchestrator Service
⚠️ Orchestrator may not have started (check logs)

curl http://localhost:9090/health
# Connection refused

Solution:

1. Check for port conflicts: lsof -i :9090
2. If port 9090 is in use, either:
   • Stop the conflicting service
   • Change orchestrator port in configuration
3. Check logs: tail -f provisioning/platform/orchestrator/data/orchestrator.log
4. Start manually: cd provisioning/platform/orchestrator && ./scripts/start-orchestrator.nu --background
5. Verify: curl http://localhost:9090/health

Issue: “Sudo password prompt during bootstrap”

Symptoms:

Stage 3: Creating Directory Structure
[sudo] password for user:

Solution:

• This is normal if creating directories in system locations
• Enter your sudo password when prompted
• Or: Run bootstrap from home directory instead

Issue: “Permission denied” on binary

Symptoms:

bash: ./provisioning/bootstrap/install.sh: Permission denied

Solution:

# Make script executable
chmod +x /Users/Akasha/project-provisioning/provisioning/bootstrap/install.sh

# Retry
./provisioning/bootstrap/install.sh

Section 5: Next Steps

After successful installation validation, you can:

Option 1: Deploy workspace_librecloud

To deploy infrastructure to UpCloud:

# Read workspace deployment guide
cat workspaces/workspace_librecloud/docs/deployment-guide.md

# Or: From workspace directory
cd workspaces/workspace_librecloud
cat docs/deployment-guide.md

Option 2: Create a New Workspace

To create a new workspace for different infrastructure:

provisioning workspace init my_workspace --template minimal

Option 3: Explore Available Modules

Discover what’s available to deploy:

# List available task services
provisioning mod discover taskservs

# List available providers
provisioning mod discover providers

# List available clusters
provisioning mod discover clusters

Section 6: Verification Checklist

After completing all steps, verify with this final checklist:

Prerequisites Verified:
  [ ] OS is macOS, Linux, or WSL2
  [ ] CPU: 2+ cores
  [ ] RAM: 2+ GB available
  [ ] Disk: 2+ GB free
  [ ] Nushell 0.109.0+ installed
  [ ] Nickel 1.x.x installed
  [ ] Docker 20.10+ installed
  [ ] Provisioning binary executable

Bootstrap Completed:
  [ ] All 7 stages completed successfully
  [ ] No error messages in output
  [ ] Installation log shows success

Installation Validated:
  [ ] Workspace directories exist
  [ ] Generated TOML files exist
  [ ] Nickel type-checking passes
  [ ] Workspace validation passes
  [ ] Orchestrator health check passes
  [ ] Provisioning CLI works (if installed)

Ready to Deploy:
  [ ] No errors in validation steps
  [ ] All services responding correctly
  [ ] Configuration properly exported

Getting Help

If you encounter issues not covered here:

1. Check logs: tail -f provisioning/platform/orchestrator/data/orchestrator.log
2. Enable debug mode: provisioning --debug <command>
3. Review bootstrap output: Scroll up to see detailed error messages
4. Check documentation: provisioning help or provisioning guide <topic>
5. Workspace guide: cat workspaces/workspace_librecloud/docs/deployment-guide.md

Summary

This guide covers:

• ✅ Prerequisites verification (Nushell, Nickel, Docker)
• ✅ Bootstrap installation (7-stage automated process)
• ✅ Installation validation (directories, configs, services)
• ✅ Troubleshooting common issues
• ✅ Next steps for deployment

You now have a fully installed and validated provisioning system ready for workspace deployment.

Getting Started Guide

Welcome to Infrastructure Automation. This guide will walk you through your first steps with infrastructure automation, from basic setup to deploying your first infrastructure.

What You’ll Learn

• Essential concepts and terminology
• How to configure your first environment
• Creating and managing infrastructure
• Basic server and service management
• Common workflows and best practices

Prerequisites

Before starting this guide, ensure you have:

• ✅ Completed the Installation Guide
• ✅ Verified your installation with provisioning --version
• ✅ Basic familiarity with command-line interfaces

Essential Concepts

Infrastructure as Code (IaC)

Provisioning uses declarative configuration to manage infrastructure. Instead of manually creating resources, you define what you want in configuration files, and the system makes it happen.

You describe → System creates → Infrastructure exists

Key Components

Configuration Languages

• Nickel: Primary configuration language for infrastructure definitions (type-safe, validated)
• TOML: User preferences and system settings
• YAML: Kubernetes manifests and service definitions

First-Time Setup

Step 1: Initialize Your Configuration

Create your personal configuration:

# Initialize user configuration
provisioning init config

# This creates ~/.provisioning/config.user.toml

Step 2: Verify Your Environment

# Check your environment setup
provisioning env

# View comprehensive configuration
provisioning allenv

You should see output like:

✅ Configuration loaded successfully
✅ All required tools available
📁 Base path: /usr/local/provisioning
🏠 User config: ~/.provisioning/config.user.toml

Step 3: Explore Available Resources

# List available providers
provisioning list providers

# List available task services
provisioning list taskservs

# List available clusters
provisioning list clusters

Your First Infrastructure

Let’s create a simple local infrastructure to learn the basics.

Step 1: Create a Workspace

# Create a new workspace directory
mkdir ~/my-first-infrastructure
cd ~/my-first-infrastructure

# Initialize workspace
provisioning generate infra --new local-demo

This creates:

local-demo/
├── config/
│   └── config.ncl     # Master Nickel configuration
├── infra/
│   └── default/
│       ├── main.ncl   # Infrastructure definition
│       └── servers.ncl # Server configurations
└── docs/              # Auto-generated guides

Step 2: Examine the Configuration

# View the generated configuration
provisioning show settings --infra local-demo

Step 3: Validate the Configuration

# Validate syntax and structure
provisioning validate config --infra local-demo

# Should show: ✅ Configuration validation passed!

Step 4: Deploy Infrastructure (Check Mode)

# Dry run - see what would be created
provisioning server create --infra local-demo --check

# This shows planned changes without making them

Step 5: Create Your Infrastructure

# Create the actual infrastructure
provisioning server create --infra local-demo

# Wait for completion
provisioning server list --infra local-demo

Working with Services

Installing Your First Service

Let’s install a containerized service:

# Install Docker/containerd
provisioning taskserv create containerd --infra local-demo

# Verify installation
provisioning taskserv list --infra local-demo

Installing Kubernetes

For container orchestration:

# Install Kubernetes
provisioning taskserv create kubernetes --infra local-demo

# This may take several minutes...

Checking Service Status

# Show all services on your infrastructure
provisioning show servers --infra local-demo

# Show specific service details
provisioning show servers web-01 taskserv kubernetes --infra local-demo

Understanding Commands

Command Structure

All commands follow this pattern:

provisioning [global-options] <command> [command-options] [arguments]

Global Options

Essential Commands

Working with Multiple Environments

Environment Concepts

The system supports multiple environments:

• dev - Development and testing
• test - Integration testing
• prod - Production deployment

Switching Environments

# Set environment for this session
export PROVISIONING_ENV=dev
provisioning env

# Or specify per command
provisioning --environment dev server create

Environment-Specific Configuration

Create environment configs:

# Development environment
provisioning init config dev

# Production environment
provisioning init config prod

Common Workflows

Workflow 1: Development Environment

# 1. Create development workspace
mkdir ~/dev-environment
cd ~/dev-environment

# 2. Generate infrastructure
provisioning generate infra --new dev-setup

# 3. Customize for development
# Edit settings.ncl to add development tools

# 4. Deploy
provisioning server create --infra dev-setup --check
provisioning server create --infra dev-setup

# 5. Install development services
provisioning taskserv create kubernetes --infra dev-setup
provisioning taskserv create containerd --infra dev-setup

Workflow 2: Service Updates

# Check for service updates
provisioning taskserv check-updates

# Update specific service
provisioning taskserv update kubernetes --infra dev-setup

# Verify update
provisioning taskserv versions kubernetes

Workflow 3: Infrastructure Scaling

# Add servers to existing infrastructure
# Edit settings.ncl to add more servers

# Apply changes
provisioning server create --infra dev-setup

# Install services on new servers
provisioning taskserv create containerd --infra dev-setup

Interactive Mode

Starting Interactive Shell

# Start Nushell with provisioning loaded
provisioning nu

In the interactive shell, you have access to all provisioning functions:

# Inside Nushell session
use lib_provisioning *

# Check environment
show_env

# List available functions
help commands | where name =~ "provision"

Useful Interactive Commands

# Show detailed server information
find_servers "web-*" | table

# Get cost estimates
servers_walk_by_costs $settings "" false false "stdout"

# Check task service status
taskservs_list | where status == "running"

Configuration Management

Understanding Configuration Files

1. System Defaults: config.defaults.toml - System-wide defaults
2. User Config: ~/.provisioning/config.user.toml - Your preferences
3. Environment Config: config.{env}.toml - Environment-specific settings
4. Infrastructure Config: settings.ncl - Infrastructure definitions

Configuration Hierarchy

Infrastructure settings.ncl
    ↓ (overrides)
Environment config.{env}.toml
    ↓ (overrides)
User config.user.toml
    ↓ (overrides)
System config.defaults.toml

Customizing Your Configuration

# Edit user configuration
provisioning sops ~/.provisioning/config.user.toml

# Or using your preferred editor
nano ~/.provisioning/config.user.toml

Example customizations:

[debug]
enabled = true        # Enable debug mode by default
log_level = "debug"   # Verbose logging

[providers]
default = "aws"       # Use AWS as default provider

[output]
format = "json"       # Prefer JSON output

Monitoring and Observability

Checking System Status

# Overall system health
provisioning env

# Infrastructure status
provisioning show servers --infra dev-setup

# Service status
provisioning taskserv list --infra dev-setup

Logging and Debugging

# Enable debug mode for troubleshooting
provisioning --debug server create --infra dev-setup --check

# View logs for specific operations
provisioning show logs --infra dev-setup

Cost Monitoring

# Show cost estimates
provisioning show cost --infra dev-setup

# Detailed cost breakdown
provisioning server price --infra dev-setup

Best Practices

1. Configuration Management

• ✅ Use version control for infrastructure definitions
• ✅ Test changes in development before production
• ✅ Use --check mode to preview changes
• ✅ Keep user configuration separate from infrastructure

2. Security

• ✅ Use SOPS for encrypting sensitive data
• ✅ Regular key rotation for cloud providers
• ✅ Principle of least privilege for access
• ✅ Audit infrastructure changes

3. Operational Excellence

• ✅ Monitor infrastructure costs regularly
• ✅ Keep services updated
• ✅ Document custom configurations
• ✅ Plan for disaster recovery

4. Development Workflow

# 1. Always validate before applying
provisioning validate config --infra my-infra

# 2. Use check mode first
provisioning server create --infra my-infra --check

# 3. Apply changes incrementally
provisioning server create --infra my-infra

# 4. Verify results
provisioning show servers --infra my-infra

Getting Help

Built-in Help System

# General help
provisioning help

# Command-specific help
provisioning server help
provisioning taskserv help
provisioning cluster help

# Show available options
provisioning generate help

Command Reference

For complete command documentation, see: CLI Reference

Troubleshooting

If you encounter issues, see: Troubleshooting Guide

Real-World Example

Let’s walk through a complete example of setting up a web application infrastructure:

Step 1: Plan Your Infrastructure

# Create project workspace
mkdir ~/webapp-infrastructure
cd ~/webapp-infrastructure

# Generate base infrastructure
provisioning generate infra --new webapp

Step 2: Customize Configuration

Edit webapp/settings.ncl to define:

• 2 web servers for load balancing
• 1 database server
• Load balancer configuration

Step 3: Deploy Base Infrastructure

# Validate configuration
provisioning validate config --infra webapp

# Preview deployment
provisioning server create --infra webapp --check

# Deploy servers
provisioning server create --infra webapp

Step 4: Install Services

# Install container runtime on all servers
provisioning taskserv create containerd --infra webapp

# Install load balancer on web servers
provisioning taskserv create haproxy --infra webapp

# Install database on database server
provisioning taskserv create postgresql --infra webapp

Step 5: Deploy Application

# Create application cluster
provisioning cluster create webapp --infra webapp

# Verify deployment
provisioning show servers --infra webapp
provisioning cluster list --infra webapp

Next Steps

Now that you understand the basics:

1. Set up your workspace: Workspace Setup Guide
2. Learn about infrastructure management: Infrastructure Management Guide
3. Understand configuration: Configuration Guide
4. Explore examples: Examples and Tutorials

You’re ready to start building and managing cloud infrastructure with confidence!

Provisioning Platform Quick Reference

Version: 3.5.0 Last Updated: 2025-10-09

Quick Navigation

• Plugin Commands - Native Nushell plugins (10-50x faster)
• CLI Shortcuts - 80+ command shortcuts
• Infrastructure Commands - Servers, taskservs, clusters
• Orchestration Commands - Workflows, batch operations
• Configuration Commands - Config, validation, environment
• Workspace Commands - Multi-workspace management
• Security Commands - Auth, MFA, secrets, compliance
• Common Workflows - Complete deployment examples
• Debug and Check Mode - Testing and troubleshooting
• Output Formats - JSON, YAML, table formatting

Plugin Commands

Native Nushell plugins for high-performance operations. 10-50x faster than HTTP API.

Authentication Plugin (nu_plugin_auth)

# Login (password prompted securely)
auth login admin

# Login with custom URL
auth login admin --url https://control-center.example.com

# Verify current session
auth verify
# Returns: { active: true, user: "admin", role: "Admin", expires_at: "...", mfa_verified: true }

# List active sessions
auth sessions

# Logout
auth logout

# MFA enrollment
auth mfa enroll totp       # TOTP (Google Authenticator, Authy)
auth mfa enroll webauthn   # WebAuthn (YubiKey, Touch ID, Windows Hello)

# MFA verification
auth mfa verify --code 123456
auth mfa verify --code ABCD-EFGH-IJKL  # Backup code

Installation:

cd provisioning/core/plugins/nushell-plugins
cargo build --release -p nu_plugin_auth
plugin add target/release/nu_plugin_auth

KMS Plugin (nu_plugin_kms)

Performance: 10x faster encryption (~5 ms vs ~50 ms HTTP)

# Encrypt with auto-detected backend
kms encrypt "secret data"
# vault:v1:abc123...

# Encrypt with specific backend
kms encrypt "data" --backend rustyvault --key provisioning-main
kms encrypt "data" --backend age --key age1xxxxxxxxx
kms encrypt "data" --backend aws --key alias/provisioning

# Encrypt with context (AAD for additional security)
kms encrypt "data" --context "user=admin,env=production"

# Decrypt (auto-detects backend from format)
kms decrypt "vault:v1:abc123..."
kms decrypt "-----BEGIN AGE ENCRYPTED FILE-----..."

# Decrypt with context (must match encryption context)
kms decrypt "vault:v1:abc123..." --context "user=admin,env=production"

# Generate data encryption key
kms generate-key
kms generate-key --spec AES256

# Check backend status
kms status

Supported Backends:

• rustyvault: High-performance (~5 ms) - Production
• age: Local encryption (~3 ms) - Development
• cosmian: Cloud KMS (~30 ms)
• aws: AWS KMS (~50 ms)
• vault: HashiCorp Vault (~40 ms)

Installation:

cargo build --release -p nu_plugin_kms
plugin add target/release/nu_plugin_kms

# Set backend environment
export RUSTYVAULT_ADDR="http://localhost:8200"
export RUSTYVAULT_TOKEN="hvs.xxxxx"

Orchestrator Plugin (nu_plugin_orchestrator)

Performance: 30-50x faster queries (~1 ms vs ~30-50 ms HTTP)

# Get orchestrator status (direct file access, ~1 ms)
orch status
# { active_tasks: 5, completed_tasks: 120, health: "healthy" }

# Validate workflow Nickel file (~10 ms vs ~100 ms HTTP)
orch validate workflows/deploy.ncl
orch validate workflows/deploy.ncl --strict

# List tasks (direct file read, ~5 ms)
orch tasks
orch tasks --status running
orch tasks --status failed --limit 10

Installation:

cargo build --release -p nu_plugin_orchestrator
plugin add target/release/nu_plugin_orchestrator

Plugin Performance Comparison

CLI Shortcuts

Infrastructure Shortcuts

# Server shortcuts
provisioning s              # server (same as 'provisioning server')
provisioning s create       # Create servers
provisioning s delete       # Delete servers
provisioning s list         # List servers
provisioning s ssh web-01   # SSH into server

# Taskserv shortcuts
provisioning t              # taskserv (same as 'provisioning taskserv')
provisioning task           # taskserv (alias)
provisioning t create kubernetes
provisioning t delete kubernetes
provisioning t list
provisioning t generate kubernetes
provisioning t check-updates

# Cluster shortcuts
provisioning cl             # cluster (same as 'provisioning cluster')
provisioning cl create buildkit
provisioning cl delete buildkit
provisioning cl list

# Infrastructure shortcuts
provisioning i              # infra (same as 'provisioning infra')
provisioning infras         # infra (alias)
provisioning i list
provisioning i validate

Orchestration Shortcuts

# Workflow shortcuts
provisioning wf             # workflow (same as 'provisioning workflow')
provisioning flow           # workflow (alias)
provisioning wf list
provisioning wf status <task_id>
provisioning wf monitor <task_id>
provisioning wf stats
provisioning wf cleanup

# Batch shortcuts
provisioning bat            # batch (same as 'provisioning batch')
provisioning batch submit workflows/example.ncl
provisioning bat list
provisioning bat status <workflow_id>
provisioning bat monitor <workflow_id>
provisioning bat rollback <workflow_id>
provisioning bat cancel <workflow_id>
provisioning bat stats

# Orchestrator shortcuts
provisioning orch           # orchestrator (same as 'provisioning orchestrator')
provisioning orch start
provisioning orch stop
provisioning orch status
provisioning orch health
provisioning orch logs

Development Shortcuts

# Module shortcuts
provisioning mod            # module (same as 'provisioning module')
provisioning mod discover taskserv
provisioning mod discover provider
provisioning mod discover cluster
provisioning mod load taskserv workspace kubernetes
provisioning mod list taskserv workspace
provisioning mod unload taskserv workspace kubernetes
provisioning mod sync-kcl

# Layer shortcuts
provisioning lyr            # layer (same as 'provisioning layer')
provisioning lyr explain
provisioning lyr show
provisioning lyr test
provisioning lyr stats

# Version shortcuts
provisioning version check
provisioning version show
provisioning version updates
provisioning version apply <name> <version>
provisioning version taskserv <name>

# Package shortcuts
provisioning pack core
provisioning pack provider upcloud
provisioning pack list
provisioning pack clean

Workspace Shortcuts

# Workspace shortcuts
provisioning ws             # workspace (same as 'provisioning workspace')
provisioning ws init
provisioning ws create <name>
provisioning ws validate
provisioning ws info
provisioning ws list
provisioning ws migrate
provisioning ws switch <name>  # Switch active workspace
provisioning ws active         # Show active workspace

# Template shortcuts
provisioning tpl            # template (same as 'provisioning template')
provisioning tmpl           # template (alias)
provisioning tpl list
provisioning tpl types
provisioning tpl show <name>
provisioning tpl apply <name>
provisioning tpl validate <name>

Configuration Shortcuts

# Environment shortcuts
provisioning e              # env (same as 'provisioning env')
provisioning val            # validate (same as 'provisioning validate')
provisioning st             # setup (same as 'provisioning setup')
provisioning config         # setup (alias)

# Show shortcuts
provisioning show settings
provisioning show servers
provisioning show config

# Initialization
provisioning init <name>

# All environment
provisioning allenv         # Show all config and environment

Utility Shortcuts

# List shortcuts
provisioning l              # list (same as 'provisioning list')
provisioning ls             # list (alias)
provisioning list           # list (full)

# SSH operations
provisioning ssh <server>

# SOPS operations
provisioning sops <file>    # Edit encrypted file

# Cache management
provisioning cache clear
provisioning cache stats

# Provider operations
provisioning providers list
provisioning providers info <name>

# Nushell session
provisioning nu             # Start Nushell with provisioning library loaded

# QR code generation
provisioning qr <data>

# Nushell information
provisioning nuinfo

# Plugin management
provisioning plugin         # plugin (same as 'provisioning plugin')
provisioning plugins        # plugin (alias)
provisioning plugin list
provisioning plugin test nu_plugin_kms

Generation Shortcuts

# Generate shortcuts
provisioning g              # generate (same as 'provisioning generate')
provisioning gen            # generate (alias)
provisioning g server
provisioning g taskserv <name>
provisioning g cluster <name>
provisioning g infra --new <name>
provisioning g new <type> <name>

Action Shortcuts

# Common actions
provisioning c              # create (same as 'provisioning create')
provisioning d              # delete (same as 'provisioning delete')
provisioning u              # update (same as 'provisioning update')

# Pricing shortcuts
provisioning price          # Show server pricing
provisioning cost           # price (alias)
provisioning costs          # price (alias)

# Create server + taskservs (combo command)
provisioning cst            # create-server-task
provisioning csts           # create-server-task (alias)

Infrastructure Commands

Server Management

# Create servers
provisioning server create
provisioning server create --check  # Dry-run mode
provisioning server create --yes    # Skip confirmation

# Delete servers
provisioning server delete
provisioning server delete --check
provisioning server delete --yes

# List servers
provisioning server list
provisioning server list --infra wuji
provisioning server list --out json

# SSH into server
provisioning server ssh web-01
provisioning server ssh db-01

# Show pricing
provisioning server price
provisioning server price --provider upcloud

Taskserv Management

# Create taskserv
provisioning taskserv create kubernetes
provisioning taskserv create kubernetes --check
provisioning taskserv create kubernetes --infra wuji

# Delete taskserv
provisioning taskserv delete kubernetes
provisioning taskserv delete kubernetes --check

# List taskservs
provisioning taskserv list
provisioning taskserv list --infra wuji

# Generate taskserv configuration
provisioning taskserv generate kubernetes
provisioning taskserv generate kubernetes --out yaml

# Check for updates
provisioning taskserv check-updates
provisioning taskserv check-updates --taskserv kubernetes

Cluster Management

# Create cluster
provisioning cluster create buildkit
provisioning cluster create buildkit --check
provisioning cluster create buildkit --infra wuji

# Delete cluster
provisioning cluster delete buildkit
provisioning cluster delete buildkit --check

# List clusters
provisioning cluster list
provisioning cluster list --infra wuji

Orchestration Commands

Workflow Management

# Submit server creation workflow
nu -c "use core/nulib/workflows/server_create.nu *; server_create_workflow 'wuji' '' [] --check"

# Submit taskserv workflow
nu -c "use core/nulib/workflows/taskserv.nu *; taskserv create 'kubernetes' 'wuji' --check"

# Submit cluster workflow
nu -c "use core/nulib/workflows/cluster.nu *; cluster create 'buildkit' 'wuji' --check"

# List all workflows
provisioning workflow list
nu -c "use core/nulib/workflows/management.nu *; workflow list"

# Get workflow statistics
provisioning workflow stats
nu -c "use core/nulib/workflows/management.nu *; workflow stats"

# Monitor workflow in real-time
provisioning workflow monitor <task_id>
nu -c "use core/nulib/workflows/management.nu *; workflow monitor <task_id>"

# Check orchestrator health
provisioning workflow orchestrator
nu -c "use core/nulib/workflows/management.nu *; workflow orchestrator"

# Get specific workflow status
provisioning workflow status <task_id>
nu -c "use core/nulib/workflows/management.nu *; workflow status <task_id>"

Batch Operations

# Submit batch workflow from Nickel
provisioning batch submit workflows/example_batch.ncl
nu -c "use core/nulib/workflows/batch.nu *; batch submit workflows/example_batch.ncl"

# Monitor batch workflow progress
provisioning batch monitor <workflow_id>
nu -c "use core/nulib/workflows/batch.nu *; batch monitor <workflow_id>"

# List batch workflows with filtering
provisioning batch list
provisioning batch list --status Running
nu -c "use core/nulib/workflows/batch.nu *; batch list --status Running"

# Get detailed batch status
provisioning batch status <workflow_id>
nu -c "use core/nulib/workflows/batch.nu *; batch status <workflow_id>"

# Initiate rollback for failed workflow
provisioning batch rollback <workflow_id>
nu -c "use core/nulib/workflows/batch.nu *; batch rollback <workflow_id>"

# Cancel running batch
provisioning batch cancel <workflow_id>

# Show batch workflow statistics
provisioning batch stats
nu -c "use core/nulib/workflows/batch.nu *; batch stats"

Orchestrator Management

# Start orchestrator in background
cd provisioning/platform/orchestrator
./scripts/start-orchestrator.nu --background

# Check orchestrator status
./scripts/start-orchestrator.nu --check
provisioning orchestrator status

# Stop orchestrator
./scripts/start-orchestrator.nu --stop
provisioning orchestrator stop

# View logs
tail -f provisioning/platform/orchestrator/data/orchestrator.log
provisioning orchestrator logs

Configuration Commands

Environment and Validation

# Show environment variables
provisioning env

# Show all environment and configuration
provisioning allenv

# Validate configuration
provisioning validate config
provisioning validate infra

# Setup wizard
provisioning setup

Configuration Files

# System defaults
less provisioning/config/config.defaults.toml

# User configuration
vim workspace/config/local-overrides.toml

# Environment-specific configs
vim workspace/config/dev-defaults.toml
vim workspace/config/test-defaults.toml
vim workspace/config/prod-defaults.toml

# Infrastructure-specific config
vim workspace/infra/<name>/config.toml

HTTP Configuration

# Configure HTTP client behavior
# In workspace/config/local-overrides.toml:
[http]
use_curl = true  # Use curl instead of ureq

Workspace Commands

Workspace Management

# List all workspaces
provisioning workspace list

# Show active workspace
provisioning workspace active

# Switch to another workspace
provisioning workspace switch <name>
provisioning workspace activate <name>  # alias

# Register new workspace
provisioning workspace register <name> <path>
provisioning workspace register <name> <path> --activate

# Remove workspace from registry
provisioning workspace remove <name>
provisioning workspace remove <name> --force

# Initialize new workspace
provisioning workspace init
provisioning workspace init --name production

# Create new workspace
provisioning workspace create <name>

# Validate workspace
provisioning workspace validate

# Show workspace info
provisioning workspace info

# Migrate workspace
provisioning workspace migrate

User Preferences

# View user preferences
provisioning workspace preferences

# Set user preference
provisioning workspace set-preference editor vim
provisioning workspace set-preference output_format yaml
provisioning workspace set-preference confirm_delete true

# Get user preference
provisioning workspace get-preference editor

User Config Location:

• macOS: ~/Library/Application Support/provisioning/user_config.yaml
• Linux: ~/.config/provisioning/user_config.yaml
• Windows: %APPDATA%\provisioning\user_config.yaml

Security Commands

Authentication (via CLI)

# Login
provisioning login admin

# Logout
provisioning logout

# Show session status
provisioning auth status

# List active sessions
provisioning auth sessions

Multi-Factor Authentication (MFA)

# Enroll in TOTP (Google Authenticator, Authy)
provisioning mfa totp enroll

# Enroll in WebAuthn (YubiKey, Touch ID, Windows Hello)
provisioning mfa webauthn enroll

# Verify MFA code
provisioning mfa totp verify --code 123456
provisioning mfa webauthn verify

# List registered devices
provisioning mfa devices

Secrets Management

# Generate AWS STS credentials (15 min-12h TTL)
provisioning secrets generate aws --ttl 1hr

# Generate SSH key pair (Ed25519)
provisioning secrets generate ssh --ttl 4hr

# List active secrets
provisioning secrets list

# Revoke secret
provisioning secrets revoke <secret_id>

# Cleanup expired secrets
provisioning secrets cleanup

SSH Temporal Keys

# Connect to server with temporal key
provisioning ssh connect server01 --ttl 1hr

# Generate SSH key pair only
provisioning ssh generate --ttl 4hr

# List active SSH keys
provisioning ssh list

# Revoke SSH key
provisioning ssh revoke <key_id>

KMS Operations (via CLI)

# Encrypt configuration file
provisioning kms encrypt secure.yaml

# Decrypt configuration file
provisioning kms decrypt secure.yaml.enc

# Encrypt entire config directory
provisioning config encrypt workspace/infra/production/

# Decrypt config directory
provisioning config decrypt workspace/infra/production/

Break-Glass Emergency Access

# Request emergency access
provisioning break-glass request "Production database outage"

# Approve emergency request (requires admin)
provisioning break-glass approve <request_id> --reason "Approved by CTO"

# List break-glass sessions
provisioning break-glass list

# Revoke break-glass session
provisioning break-glass revoke <session_id>

Compliance and Audit

# Generate compliance report
provisioning compliance report
provisioning compliance report --standard gdpr
provisioning compliance report --standard soc2
provisioning compliance report --standard iso27001

# GDPR operations
provisioning compliance gdpr export <user_id>
provisioning compliance gdpr delete <user_id>
provisioning compliance gdpr rectify <user_id>

# Incident management
provisioning compliance incident create "Security breach detected"
provisioning compliance incident list
provisioning compliance incident update <incident_id> --status investigating

# Audit log queries
provisioning audit query --user alice --action deploy --from 24h
provisioning audit export --format json --output audit-logs.json

Common Workflows

Complete Deployment from Scratch

# 1. Initialize workspace
provisioning workspace init --name production

# 2. Validate configuration
provisioning validate config

# 3. Create infrastructure definition
provisioning generate infra --new production

# 4. Create servers (check mode first)
provisioning server create --infra production --check

# 5. Create servers (actual deployment)
provisioning server create --infra production --yes

# 6. Install Kubernetes
provisioning taskserv create kubernetes --infra production --check
provisioning taskserv create kubernetes --infra production

# 7. Deploy cluster services
provisioning cluster create production --check
provisioning cluster create production

# 8. Verify deployment
provisioning server list --infra production
provisioning taskserv list --infra production

# 9. SSH to servers
provisioning server ssh k8s-master-01

Multi-Environment Deployment

# Deploy to dev
provisioning server create --infra dev --check
provisioning server create --infra dev
provisioning taskserv create kubernetes --infra dev

# Deploy to staging
provisioning server create --infra staging --check
provisioning server create --infra staging
provisioning taskserv create kubernetes --infra staging

# Deploy to production (with confirmation)
provisioning server create --infra production --check
provisioning server create --infra production
provisioning taskserv create kubernetes --infra production

Update Infrastructure

# 1. Check for updates
provisioning taskserv check-updates

# 2. Update specific taskserv (check mode)
provisioning taskserv update kubernetes --check

# 3. Apply update
provisioning taskserv update kubernetes

# 4. Verify update
provisioning taskserv list --infra production | where name == kubernetes

Encrypted Secrets Deployment

# 1. Authenticate
auth login admin
auth mfa verify --code 123456

# 2. Encrypt secrets
kms encrypt (open secrets/production.yaml) --backend rustyvault | save secrets/production.enc

# 3. Deploy with encrypted secrets
provisioning cluster create production --secrets secrets/production.enc

# 4. Verify deployment
orch tasks --status completed

Debug and Check Mode

Debug Mode

Enable verbose logging with --debug or -x flag:

# Server creation with debug output
provisioning server create --debug
provisioning server create -x

# Taskserv creation with debug
provisioning taskserv create kubernetes --debug

# Show detailed error traces
provisioning --debug taskserv create kubernetes

Check Mode (Dry Run)

Preview changes without applying them with --check or -c flag:

# Check what servers would be created
provisioning server create --check
provisioning server create -c

# Check taskserv installation
provisioning taskserv create kubernetes --check

# Check cluster creation
provisioning cluster create buildkit --check

# Combine with debug for detailed preview
provisioning server create --check --debug

Auto-Confirm Mode

Skip confirmation prompts with --yes or -y flag:

# Auto-confirm server creation
provisioning server create --yes
provisioning server create -y

# Auto-confirm deletion
provisioning server delete --yes

Wait Mode

Wait for operations to complete with --wait or -w flag:

# Wait for server creation to complete
provisioning server create --wait

# Wait for taskserv installation
provisioning taskserv create kubernetes --wait

Infrastructure Selection

Specify target infrastructure with --infra or -i flag:

# Create servers in specific infrastructure
provisioning server create --infra production
provisioning server create -i production

# List servers in specific infrastructure
provisioning server list --infra production

Output Formats

JSON Output

# Output as JSON
provisioning server list --out json
provisioning taskserv list --out json

# Pipeline JSON output
provisioning server list --out json | jq '.[] | select(.status == "running")'

YAML Output

# Output as YAML
provisioning server list --out yaml
provisioning taskserv list --out yaml

# Pipeline YAML output
provisioning server list --out yaml | yq '.[] | select(.status == "running")'

Table Output (Default)

# Output as table (default)
provisioning server list
provisioning server list --out table

# Pretty-printed table
provisioning server list | table

Text Output

# Output as plain text
provisioning server list --out text

Performance Tips

Use Plugins for Frequent Operations

# ❌ Slow: HTTP API (50 ms per call)
for i in 1..100 { http post http://localhost:9998/encrypt { data: "secret" } }

# ✅ Fast: Plugin (5 ms per call, 10x faster)
for i in 1..100 { kms encrypt "secret" }

Batch Operations

# Use batch workflows for multiple operations
provisioning batch submit workflows/multi-cloud-deploy.ncl

Check Mode for Testing

# Always test with --check first
provisioning server create --check
provisioning server create  # Only after verification

Help System

Command-Specific Help

# Show help for specific command
provisioning help server
provisioning help taskserv
provisioning help cluster
provisioning help workflow
provisioning help batch

# Show help for command category
provisioning help infra
provisioning help orch
provisioning help dev
provisioning help ws
provisioning help config

Bi-Directional Help

# All these work identically:
provisioning help workspace
provisioning workspace help
provisioning ws help
provisioning help ws

General Help

# Show all commands
provisioning help
provisioning --help

# Show version
provisioning version
provisioning --version

Quick Reference: Common Flags

Plugin Installation Quick Reference

# Build all plugins (one-time setup)
cd provisioning/core/plugins/nushell-plugins
cargo build --release --all

# Register plugins
plugin add target/release/nu_plugin_auth
plugin add target/release/nu_plugin_kms
plugin add target/release/nu_plugin_orchestrator

# Verify installation
plugin list | where name =~ "auth|kms|orch"
auth --help
kms --help
orch --help

# Set environment
export RUSTYVAULT_ADDR="http://localhost:8200"
export RUSTYVAULT_TOKEN="hvs.xxxxx"
export CONTROL_CENTER_URL="http://localhost:3000"

Related Documentation

• Complete Plugin Guide: docs/user/PLUGIN_INTEGRATION_GUIDE.md
• Plugin Reference: docs/user/NUSHELL_PLUGINS_GUIDE.md
• From Scratch Guide: docs/guides/from-scratch.md
• Update Infrastructure: Update Guide
• Customize Infrastructure: Customize Guide
• CLI Architecture: CLI Reference
• Security System: Security Architecture

For fastest access to this guide: provisioning sc

Last Updated: 2025-10-09 Maintained By: Platform Team

Setup Quick Start - 5 Minutes to Deployment

Goal: Get provisioning running in 5 minutes with a working example

Step 1: Check Prerequisites (30 seconds)

# Check Nushell
nu --version   # Should be 0.109.0+

# Check deployment tool
docker --version    # OR
kubectl version     # OR
ssh -V              # OR
systemctl --version

Step 2: Install Provisioning (1 minute)

# Option A: Using installer script
curl -sSL https://install.provisioning.dev | bash

# Option B: From source
git clone https://github.com/project-provisioning/provisioning
cd provisioning
./scripts/install.sh

Step 3: Initialize System (2 minutes)

# Run interactive setup
provisioning setup system --interactive

# Follow the prompts:
# - Press Enter for defaults
# - Select your deployment tool
# - Enter provider credentials (if using cloud)

Step 4: Create Your First Workspace (1 minute)

# Create workspace
provisioning setup workspace myapp

# Verify it was created
provisioning workspace list

Step 5: Deploy Your First Server (1 minute)

# Activate workspace
provisioning workspace activate myapp

# Check configuration
provisioning setup validate

# Deploy server (dry-run first)
provisioning server create --check

# Deploy for real
provisioning server create --yes

Verify Everything Works

# Check health
provisioning platform health

# Check servers
provisioning server list

# SSH into server (if applicable)
provisioning server ssh <server-name>

Common Commands Cheat Sheet

# Workspace management
provisioning workspace list              # List all workspaces
provisioning workspace activate prod     # Switch workspace
provisioning workspace create dev        # Create new workspace

# Server management
provisioning server list                 # List servers
provisioning server create               # Create server
provisioning server delete <name>        # Delete server
provisioning server ssh <name>           # SSH into server

# Configuration
provisioning setup validate              # Validate configuration
provisioning setup update platform       # Update platform settings

# System info
provisioning info                        # System information
provisioning capability check            # Check capabilities
provisioning platform health             # Check platform health

Troubleshooting Quick Fixes

Setup wizard won’t start

# Check Nushell
nu --version

# Check permissions
chmod +x $(which provisioning)

Configuration error

# Validate configuration
provisioning setup validate --verbose

# Check paths
provisioning info paths

Deployment fails

# Dry-run to see what would happen
provisioning server create --check

# Check platform status
provisioning platform status

What’s Next

After basic setup:

1. Configure Provider: Add cloud provider credentials
2. Create More Workspaces: Dev, staging, production
3. Deploy Services: Web servers, databases, etc.
4. Set Up Monitoring: Health checks, logging
5. Automate Deployments: CI/CD integration

Need Help

# Get help
provisioning help

# Setup help
provisioning help setup

# Specific command help
provisioning <command> --help

# View documentation
provisioning guide system-setup

Key Files

Your configuration is in:

macOS: ~/Library/Application Support/provisioning/ Linux: ~/.config/provisioning/

Important files:

• system.toml - System configuration
• user_preferences.toml - User settings
• workspaces/*/ - Workspace definitions

Ready to dive deeper? Check out the Full Setup Guide

Provisioning Setup System Guide

Version: 1.0.0 Last Updated: 2025-12-09 Status: Production Ready

Quick Start

Prerequisites

• Nushell 0.109.0+
• bash
• One deployment tool: Docker, Kubernetes, SSH, or systemd
• Optional: KCL, SOPS, Age

30-Second Setup

# Install provisioning
curl -sSL https://install.provisioning.dev | bash

# Run setup wizard
provisioning setup system --interactive

# Create workspace
provisioning setup workspace myproject

# Start deploying
provisioning server create

Configuration Paths

macOS: ~/Library/Application Support/provisioning/ Linux: ~/.config/provisioning/ Windows: %APPDATA%/provisioning/

Directory Structure

provisioning/
├── system.toml                  # System info (immutable)
├── user_preferences.toml        # User settings (editable)
├── platform/                    # Platform services
├── providers/                   # Provider configs
└── workspaces/                  # Workspace definitions
    └── myproject/
        ├── config/
        ├── infra/
        └── auth.token

Setup Wizard

Run the interactive setup wizard:

provisioning setup system --interactive

The wizard guides you through:

1. Welcome & Prerequisites Check
2. Operating System Detection
3. Configuration Path Selection
4. Platform Services Setup
5. Provider Selection
6. Security Configuration
7. Review & Confirmation

Configuration Management

Hierarchy (highest to lowest priority)

1. Runtime Arguments (--flag value)
2. Environment Variables (PROVISIONING_*)
3. Workspace Configuration
4. Workspace Authentication Token
5. User Preferences (user_preferences.toml)
6. Platform Configurations (platform/*.toml)
7. Provider Configurations (providers/*.toml)
8. System Configuration (system.toml)
9. Built-in Defaults

Configuration Files

• system.toml - System information (OS, architecture, paths)
• user_preferences.toml - User preferences (editor, format, etc.)
• platform/*.toml - Service endpoints and configuration
• providers/*.toml - Cloud provider settings

Multiple Workspaces

Create and manage multiple isolated environments:

# Create workspace
provisioning setup workspace dev
provisioning setup workspace prod

# List workspaces
provisioning workspace list

# Activate workspace
provisioning workspace activate prod

Configuration Updates

Update any setting:

# Update platform configuration
provisioning setup platform --config new-config.toml

# Update provider settings
provisioning setup provider upcloud --config upcloud-config.toml

# Validate changes
provisioning setup validate

Backup & Restore

# Backup current configuration
provisioning setup backup --path ./backup.tar.gz

# Restore from backup
provisioning setup restore --path ./backup.tar.gz

# Migrate from old setup
provisioning setup migrate --from-existing

Troubleshooting

“Command not found: provisioning”

export PATH="/usr/local/bin:$PATH"

“Nushell not found”

curl -sSL https://raw.githubusercontent.com/nushell/nushell/main/install.sh | bash

“Cannot write to directory”

chmod 755 ~/Library/Application\ Support/provisioning/

Check required tools

provisioning setup validate --check-tools

FAQ

Q: Do I need all optional tools? A: No. You need at least one deployment tool (Docker, Kubernetes, SSH, or systemd).

Q: Can I use provisioning without Docker? A: Yes. Provisioning supports Docker, Kubernetes, SSH, systemd, or combinations.

Q: How do I update configuration? A: provisioning setup update <category>

Q: Can I have multiple workspaces? A: Yes, unlimited workspaces.

Q: Is my configuration secure? A: Yes. Credentials stored securely, never in config files.

Q: Can I share workspaces with my team? A: Yes, via GitOps - configurations in Git, secrets in secure storage.

Getting Help

# General help
provisioning help

# Setup help
provisioning help setup

# Specific command help
provisioning setup system --help

Next Steps

1. Installation Guide
2. Workspace Setup
3. Provider Configuration
4. From Scratch Guide

Status: Production Ready ✅ Version: 1.0.0 Last Updated: 2025-12-09

Quick Start

This guide has moved to a multi-chapter format for better readability.

📖 Navigate to Quick Start Guide

Please see the complete quick start guide here:

• Prerequisites - System requirements and setup
• Installation - Install provisioning platform
• First Deployment - Deploy your first infrastructure
• Verification - Verify your deployment

Quick Commands

# Check system status
provisioning status

# Get next step suggestions
provisioning next

# View interactive guide
provisioning guide from-scratch

For the complete step-by-step walkthrough, start with Prerequisites.

Prerequisites

Before installing the Provisioning Platform, ensure your system meets the following requirements.

Hardware Requirements

Minimum Requirements (Solo Mode)

• CPU: 2 cores
• RAM: 4 GB
• Disk: 20 GB available space
• Network: Internet connection for downloading dependencies

Recommended Requirements (Multi-User Mode)

• CPU: 4 cores
• RAM: 8 GB
• Disk: 50 GB available space
• Network: Reliable internet connection

Production Requirements (Enterprise Mode)

• CPU: 16 cores
• RAM: 32 GB
• Disk: 500 GB available space (SSD recommended)
• Network: High-bandwidth connection with static IP

Operating System

Supported Platforms

• macOS: 12.0 (Monterey) or later
• Linux:
  • Ubuntu 22.04 LTS or later
  • Fedora 38 or later
  • Debian 12 (Bookworm) or later
  • RHEL 9 or later

Platform-Specific Notes

macOS:

• Xcode Command Line Tools required
• Homebrew recommended for package management

Linux:

• systemd-based distribution recommended
• sudo access required for some operations

Required Software

Core Dependencies

Optional Dependencies

Installation Verification

Before proceeding, verify your system has the core dependencies installed:

Nushell

# Check Nushell version
nu --version

# Expected output: 0.107.1 or higher

Nickel

# Check Nickel version
nickel --version

# Expected output: 1.15.0 or higher

Docker

# Check Docker version
docker --version

# Check Docker is running
docker ps

# Expected: Docker version 20.10+ and connection successful

SOPS

# Check SOPS version
sops --version

# Expected output: 3.10.2 or higher

Age

# Check Age version
age --version

# Expected output: 1.2.1 or higher

Installing Missing Dependencies

macOS (using Homebrew)

# Install Homebrew if not already installed
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install Nushell
brew install nushell

# Install Nickel
brew install nickel

# Install Docker Desktop
brew install --cask docker

# Install SOPS
brew install sops

# Install Age
brew install age

# Optional: Install extras
brew install k9s glow bat

Ubuntu/Debian

# Update package list
sudo apt update

# Install prerequisites
sudo apt install -y curl git build-essential

# Install Nushell (from GitHub releases)
curl -LO https://github.com/nushell/nushell/releases/download/0.107.1/nu-0.107.1-x86_64-linux-musl.tar.gz
tar xzf nu-0.107.1-x86_64-linux-musl.tar.gz
sudo mv nu /usr/local/bin/

# Install Nickel (using Rust cargo)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
cargo install nickel

# Install Docker
sudo apt install -y docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

# Install SOPS
curl -LO https://github.com/getsops/sops/releases/download/v3.10.2/sops-v3.10.2.linux.amd64
chmod +x sops-v3.10.2.linux.amd64
sudo mv sops-v3.10.2.linux.amd64 /usr/local/bin/sops

# Install Age
sudo apt install -y age

Fedora/RHEL

# Install Nushell
sudo dnf install -y nushell

# Install Nickel (using Rust cargo)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
cargo install nickel

# Install Docker
sudo dnf install -y docker
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

# Install SOPS
sudo dnf install -y sops

# Install Age
sudo dnf install -y age

Network Requirements

Firewall Ports

If running platform services, ensure these ports are available:

External Connectivity

The platform requires outbound internet access to:

• Download dependencies and updates
• Pull container images
• Access cloud provider APIs (AWS, UpCloud)
• Fetch extension packages

Cloud Provider Credentials (Optional)

If you plan to use cloud providers, prepare credentials:

AWS

• AWS Access Key ID
• AWS Secret Access Key
• Configured via ~/.aws/credentials or environment variables

UpCloud

• UpCloud username
• UpCloud password
• Configured via environment variables or config files

Next Steps

Once all prerequisites are met, proceed to: → Installation

Installation

This guide walks you through installing the Provisioning Platform on your system.

Overview

The installation process involves:

1. Cloning the repository
2. Installing Nushell plugins
3. Setting up configuration
4. Initializing your first workspace

Estimated time: 15-20 minutes

Step 1: Clone the Repository

# Clone the repository
git clone https://github.com/provisioning/provisioning-platform.git
cd provisioning-platform

# Checkout the latest stable release (optional)
git checkout tags/v3.5.0

Step 2: Install Nushell Plugins

The platform uses multiple Nushell plugins for enhanced functionality.

Install nu_plugin_tera (Template Rendering)

# Install from crates.io
cargo install nu_plugin_tera

# Register with Nushell
nu -c "plugin add ~/.cargo/bin/nu_plugin_tera; plugin use tera"

Verify Plugin Installation

# Start Nushell
nu

# List installed plugins
plugin list

# Expected output should include:
# - tera

Step 3: Add CLI to PATH

Make the provisioning command available globally:

# Option 1: Symlink to /usr/local/bin (recommended)
sudo ln -s "$(pwd)/provisioning/core/cli/provisioning" /usr/local/bin/provisioning

# Option 2: Add to PATH in your shell profile
echo 'export PATH="$PATH:'"$(pwd)"'/provisioning/core/cli"' >> ~/.bashrc  # or ~/.zshrc
source ~/.bashrc  # or ~/.zshrc

# Verify installation
provisioning --version

Step 4: Generate Age Encryption Keys

Generate keys for encrypting sensitive configuration:

# Create Age key directory
mkdir -p ~/.config/provisioning/age

# Generate private key
age-keygen -o ~/.config/provisioning/age/private_key.txt

# Extract public key
age-keygen -y ~/.config/provisioning/age/private_key.txt > ~/.config/provisioning/age/public_key.txt

# Secure the keys
chmod 600 ~/.config/provisioning/age/private_key.txt
chmod 644 ~/.config/provisioning/age/public_key.txt

Step 5: Configure Environment

Set up basic environment variables:

# Create environment file
cat > ~/.provisioning/env << 'ENVEOF'
# Provisioning Environment Configuration
export PROVISIONING_ENV=dev
export PROVISIONING_PATH=$(pwd)
export PROVISIONING_KAGE=~/.config/provisioning/age
ENVEOF

# Source the environment
source ~/.provisioning/env

# Add to shell profile for persistence
echo 'source ~/.provisioning/env' >> ~/.bashrc  # or ~/.zshrc

Step 6: Initialize Workspace

Create your first workspace:

# Initialize a new workspace
provisioning workspace init my-first-workspace

# Expected output:
# ✓ Workspace 'my-first-workspace' created successfully
# ✓ Configuration template generated
# ✓ Workspace activated

# Verify workspace
provisioning workspace list

Step 7: Validate Installation

Run the installation verification:

# Check system configuration
provisioning validate config

# Check all dependencies
provisioning env

# View detailed environment
provisioning allenv

Expected output should show:

• ✅ All core dependencies installed
• ✅ Age keys configured
• ✅ Workspace initialized
• ✅ Configuration valid

Optional: Install Platform Services

If you plan to use platform services (orchestrator, control center, etc.):

# Build platform services
cd provisioning/platform

# Build orchestrator
cd orchestrator
cargo build --release
cd ..

# Build control center
cd control-center
cargo build --release
cd ..

# Build KMS service
cd kms-service
cargo build --release
cd ..

# Verify builds
ls */target/release/

Optional: Install Platform with Installer

Use the interactive installer for a guided setup:

# Build the installer
cd provisioning/platform/installer
cargo build --release

# Run interactive installer
./target/release/provisioning-installer

# Or headless installation
./target/release/provisioning-installer --headless --mode solo --yes

Troubleshooting

Nushell Plugin Not Found

If plugins aren’t recognized:

# Rebuild plugin registry
nu -c "plugin list; plugin use tera"

Permission Denied

If you encounter permission errors:

# Ensure proper ownership
sudo chown -R $USER:$USER ~/.config/provisioning

# Check PATH
echo $PATH | grep provisioning

Age Keys Not Found

If encryption fails:

# Verify keys exist
ls -la ~/.config/provisioning/age/

# Regenerate if needed
age-keygen -o ~/.config/provisioning/age/private_key.txt

Next Steps

Once installation is complete, proceed to: → First Deployment

Additional Resources

• Detailed Installation Guide
• Workspace Management
• Troubleshooting Guide

First Deployment

This guide walks you through deploying your first infrastructure using the Provisioning Platform.

Overview

In this chapter, you’ll:

1. Configure a simple infrastructure
2. Create your first server
3. Install a task service (Kubernetes)
4. Verify the deployment

Estimated time: 10-15 minutes

Step 1: Configure Infrastructure

Create a basic infrastructure configuration:

# Generate infrastructure template
provisioning generate infra --new my-infra

# This creates: workspace/infra/my-infra/
# - config.toml (infrastructure settings)
# - settings.ncl (Nickel configuration)

Step 2: Edit Configuration

Edit the generated configuration:

# Edit with your preferred editor
$EDITOR workspace/infra/my-infra/settings.ncl

Example configuration:

import provisioning.settings as cfg

# Infrastructure settings
infra_settings = cfg.InfraSettings {
    name = "my-infra"
    provider = "local"  # Start with local provider
    environment = "development"
}

# Server configuration
servers = [
    {
        hostname = "dev-server-01"
        cores = 2
        memory = 4096  # MB
        disk = 50  # GB
    }
]

Step 3: Create Server (Check Mode)

First, run in check mode to see what would happen:

# Check mode - no actual changes
provisioning server create --infra my-infra --check

# Expected output:
# ✓ Validation passed
# ⚠ Check mode: No changes will be made
# 
# Would create:
# - Server: dev-server-01 (2 cores, 4 GB RAM, 50 GB disk)

Step 4: Create Server (Real)

If check mode looks good, create the server:

# Create server
provisioning server create --infra my-infra

# Expected output:
# ✓ Creating server: dev-server-01
# ✓ Server created successfully
# ✓ IP Address: 192.168.1.100
# ✓ SSH access: ssh user@192.168.1.100

Step 5: Verify Server

Check server status:

# List all servers
provisioning server list

# Get detailed server info
provisioning server info dev-server-01

# SSH to server (optional)
provisioning server ssh dev-server-01

Step 6: Install Kubernetes (Check Mode)

Install a task service on the server:

# Check mode first
provisioning taskserv create kubernetes --infra my-infra --check

# Expected output:
# ✓ Validation passed
# ⚠ Check mode: No changes will be made
#
# Would install:
# - Kubernetes v1.28.0
# - Required dependencies: containerd, etcd
# - On servers: dev-server-01

Step 7: Install Kubernetes (Real)

Proceed with installation:

# Install Kubernetes
provisioning taskserv create kubernetes --infra my-infra --wait

# This will:
# 1. Check dependencies
# 2. Install containerd
# 3. Install etcd
# 4. Install Kubernetes
# 5. Configure and start services

# Monitor progress
provisioning workflow monitor <task-id>

Step 8: Verify Installation

Check that Kubernetes is running:

# List installed task services
provisioning taskserv list --infra my-infra

# Check Kubernetes status
provisioning server ssh dev-server-01
kubectl get nodes  # On the server
exit

# Or remotely
provisioning server exec dev-server-01 -- kubectl get nodes

Common Deployment Patterns

Pattern 1: Multiple Servers

Create multiple servers at once:

servers = [
    {hostname = "web-01", cores = 2, memory = 4096},
    {hostname = "web-02", cores = 2, memory = 4096},
    {hostname = "db-01", cores = 4, memory = 8192}
]

provisioning server create --infra my-infra --servers web-01,web-02,db-01

Pattern 2: Server with Multiple Task Services

Install multiple services on one server:

provisioning taskserv create kubernetes,cilium,postgres --infra my-infra --servers web-01

Pattern 3: Complete Cluster

Deploy a complete cluster configuration:

provisioning cluster create buildkit --infra my-infra

Deployment Workflow

The typical deployment workflow:

# 1. Initialize workspace
provisioning workspace init production

# 2. Generate infrastructure
provisioning generate infra --new prod-infra

# 3. Configure (edit settings.ncl)
$EDITOR workspace/infra/prod-infra/settings.ncl

# 4. Validate configuration
provisioning validate config --infra prod-infra

# 5. Create servers (check mode)
provisioning server create --infra prod-infra --check

# 6. Create servers (real)
provisioning server create --infra prod-infra

# 7. Install task services
provisioning taskserv create kubernetes --infra prod-infra --wait

# 8. Deploy cluster (if needed)
provisioning cluster create my-cluster --infra prod-infra

# 9. Verify
provisioning server list
provisioning taskserv list

Troubleshooting

Server Creation Fails

# Check logs
provisioning server logs dev-server-01

# Try with debug mode
provisioning --debug server create --infra my-infra

Task Service Installation Fails

# Check task service logs
provisioning taskserv logs kubernetes

# Retry installation
provisioning taskserv create kubernetes --infra my-infra --force

SSH Connection Issues

# Verify SSH key
ls -la ~/.ssh/

# Test SSH manually
ssh -v user@<server-ip>

# Use provisioning SSH helper
provisioning server ssh dev-server-01 --debug

Next Steps

Now that you’ve completed your first deployment: → Verification - Verify your deployment is working correctly

Additional Resources

• Complete Deployment Guide
• Infrastructure Management
• Troubleshooting Guide

Verification

This guide helps you verify that your Provisioning Platform deployment is working correctly.

Overview

After completing your first deployment, verify:

1. System configuration
2. Server accessibility
3. Task service health
4. Platform services (if installed)

Step 1: Verify Configuration

Check that all configuration is valid:

# Validate all configuration
provisioning validate config

# Expected output:
# ✓ Configuration valid
# ✓ No errors found
# ✓ All required fields present

# Check environment variables
provisioning env

# View complete configuration
provisioning allenv

Step 2: Verify Servers

Check that servers are accessible and healthy:

# List all servers
provisioning server list

# Expected output:
# ┌───────────────┬──────────┬───────┬────────┬──────────────┬──────────┐
# │ Hostname      │ Provider │ Cores │ Memory │ IP Address   │ Status   │
# ├───────────────┼──────────┼───────┼────────┼──────────────┼──────────┤
# │ dev-server-01 │ local    │ 2     │ 4096   │ 192.168.1.100│ running  │
# └───────────────┴──────────┴───────┴────────┴──────────────┴──────────┘

# Check server details
provisioning server info dev-server-01

# Test SSH connectivity
provisioning server ssh dev-server-01 -- echo "SSH working"

Step 3: Verify Task Services

Check installed task services:

# List task services
provisioning taskserv list

# Expected output:
# ┌────────────┬─────────┬────────────────┬──────────┐
# │ Name       │ Version │ Server         │ Status   │
# ├────────────┼─────────┼────────────────┼──────────┤
# │ containerd │ 1.7.0   │ dev-server-01  │ running  │
# │ etcd       │ 3.5.0   │ dev-server-01  │ running  │
# │ kubernetes │ 1.28.0  │ dev-server-01  │ running  │
# └────────────┴─────────┴────────────────┴──────────┘

# Check specific task service
provisioning taskserv status kubernetes

# View task service logs
provisioning taskserv logs kubernetes --tail 50

Step 4: Verify Kubernetes (If Installed)

If you installed Kubernetes, verify it’s working:

# Check Kubernetes nodes
provisioning server ssh dev-server-01 -- kubectl get nodes

# Expected output:
# NAME            STATUS   ROLES           AGE   VERSION
# dev-server-01   Ready    control-plane   10m   v1.28.0

# Check Kubernetes pods
provisioning server ssh dev-server-01 -- kubectl get pods -A

# All pods should be Running or Completed

Step 5: Verify Platform Services (Optional)

If you installed platform services:

Orchestrator

# Check orchestrator health
curl http://localhost:8080/health

# Expected:
# {"status":"healthy","version":"0.1.0"}

# List tasks
curl http://localhost:8080/tasks

Control Center

# Check control center health
curl http://localhost:9090/health

# Test policy evaluation
curl -X POST http://localhost:9090/policies/evaluate \
  -H "Content-Type: application/json" \
  -d '{"principal":{"id":"test"},"action":{"id":"read"},"resource":{"id":"test"}}'

KMS Service

# Check KMS health
curl http://localhost:8082/api/v1/kms/health

# Test encryption
echo "test" | provisioning kms encrypt

Step 6: Run Health Checks

Run comprehensive health checks:

# Check all components
provisioning health check

# Expected output:
# ✓ Configuration: OK
# ✓ Servers: 1/1 healthy
# ✓ Task Services: 3/3 running
# ✓ Platform Services: 3/3 healthy
# ✓ Network Connectivity: OK
# ✓ Encryption Keys: OK

Step 7: Verify Workflows

If you used workflows:

# List all workflows
provisioning workflow list

# Check specific workflow
provisioning workflow status <workflow-id>

# View workflow stats
provisioning workflow stats

Common Verification Checks

DNS Resolution (If CoreDNS Installed)

# Test DNS resolution
dig @localhost test.provisioning.local

# Check CoreDNS status
provisioning server ssh dev-server-01 -- systemctl status coredns

Network Connectivity

# Test server-to-server connectivity
provisioning server ssh dev-server-01 -- ping -c 3 dev-server-02

# Check firewall rules
provisioning server ssh dev-server-01 -- sudo iptables -L

Storage and Resources

# Check disk usage
provisioning server ssh dev-server-01 -- df -h

# Check memory usage
provisioning server ssh dev-server-01 -- free -h

# Check CPU usage
provisioning server ssh dev-server-01 -- top -bn1 | head -20

Troubleshooting Failed Verifications

Configuration Validation Failed

# View detailed error
provisioning validate config --verbose

# Check specific infrastructure
provisioning validate config --infra my-infra

Server Unreachable

# Check server logs
provisioning server logs dev-server-01

# Try debug mode
provisioning --debug server ssh dev-server-01

Task Service Not Running

# Check service logs
provisioning taskserv logs kubernetes

# Restart service
provisioning taskserv restart kubernetes --infra my-infra

Platform Service Down

# Check service status
provisioning platform status orchestrator

# View service logs
provisioning platform logs orchestrator --tail 100

# Restart service
provisioning platform restart orchestrator

Performance Verification

Response Time Tests

# Measure server response time
time provisioning server info dev-server-01

# Measure task service response time
time provisioning taskserv list

# Measure workflow submission time
time provisioning workflow submit test-workflow.ncl

Resource Usage

# Check platform resource usage
docker stats  # If using Docker

# Check system resources
provisioning system resources

Security Verification

Encryption

# Verify encryption keys
ls -la ~/.config/provisioning/age/

# Test encryption/decryption
echo "test" | provisioning kms encrypt | provisioning kms decrypt

Authentication (If Enabled)

# Test login
provisioning login --username admin

# Verify token
provisioning whoami

# Test MFA (if enabled)
provisioning mfa verify <code>

Verification Checklist

Use this checklist to ensure everything is working:

• Configuration validation passes
• All servers are accessible via SSH
• All servers show “running” status
• All task services show “running” status
• Kubernetes nodes are “Ready” (if installed)
• Kubernetes pods are “Running” (if installed)
• Platform services respond to health checks
• Encryption/decryption works
• Workflows can be submitted and complete
• No errors in logs
• Resource usage is within expected limits

Next Steps

Once verification is complete:

• User Guide - Learn advanced features
• Quick Reference - Command shortcuts
• Infrastructure Management - Day-to-day operations
• Troubleshooting - Common issues and solutions

Additional Resources

• Complete From-Scratch Guide
• Service Management Guide
• Test Environment Guide

Congratulations! You’ve successfully deployed and verified your first Provisioning Platform infrastructure!

Platform Service Configuration

After verifying your installation, the next step is to configure the platform services. This guide walks you through setting up your provisioning platform for deployment.

What You’ll Learn

• Understanding platform services and configuration modes
• Setting up platform configurations with setup-platform-config.sh
• Choosing the right deployment mode for your use case
• Configuring services interactively or with quick mode
• Running platform services with your configuration

Prerequisites

Before configuring platform services, ensure you have:

• ✅ Completed Installation Steps
• ✅ Verified installation with Verification
• ✅ Nickel 0.10+ (for configuration language)
• ✅ Nushell 0.109+ (for scripts)
• ✅ TypeDialog (optional, for interactive configuration)

Platform Services Overview

The provisioning platform consists of 8 core services:

Deployment Modes

Choose a deployment mode based on your needs:

Step 1: Initialize Configuration Script

The configuration system is managed by a standalone script that doesn’t require the main installer:

# Navigate to the provisioning directory
cd /path/to/project-provisioning

# Verify the setup script exists
ls -la provisioning/scripts/setup-platform-config.sh

# Make script executable
chmod +x provisioning/scripts/setup-platform-config.sh

Step 2: Choose Configuration Method

Method A: Interactive TypeDialog Configuration (Recommended)

TypeDialog provides an interactive form-based configuration interface available in multiple backends (web, TUI, CLI).

Quick Interactive Setup (All Services at Once)

# Run interactive setup - prompts for choices
./provisioning/scripts/setup-platform-config.sh

# Follow the prompts to:
# 1. Choose action (TypeDialog, Quick Mode, Clean, List)
# 2. Select service (or all services)
# 3. Choose deployment mode
# 4. Select backend (web, tui, cli)

Configure Specific Service with TypeDialog

# Configure orchestrator in solo mode with web UI
./provisioning/scripts/setup-platform-config.sh \
  --service orchestrator \
  --mode solo \
  --backend web

# TypeDialog opens browser → User fills form → Config generated

When to use TypeDialog:

• First-time setup with visual form guidance
• Updating configuration with validation
• Multiple services needing coordinated changes
• Team environments where UI is preferred

Method B: Quick Mode Configuration (Fastest)

Quick mode automatically creates all service configurations from defaults overlaid with mode-specific tuning.

# Quick setup for solo development mode
./provisioning/scripts/setup-platform-config.sh --quick-mode --mode solo

# Quick setup for enterprise production
./provisioning/scripts/setup-platform-config.sh --quick-mode --mode enterprise

# Result: All 8 services configured immediately with appropriate resource limits

When to use Quick Mode:

• Initial setup with standard defaults
• Switching deployment modes
• CI/CD automated setup
• Scripted/programmatic configuration

Method C: Manual Nickel Configuration

For advanced users who prefer editing configuration files directly:

# View schema definition
cat provisioning/schemas/platform/schemas/orchestrator.ncl

# View default values
cat provisioning/schemas/platform/defaults/orchestrator-defaults.ncl

# View mode overlay
cat provisioning/schemas/platform/defaults/deployment/solo-defaults.ncl

# Edit configuration directly
vim provisioning/config/runtime/orchestrator.solo.ncl

# Validate Nickel syntax
nickel typecheck provisioning/config/runtime/orchestrator.solo.ncl

# Regenerate TOML from edited config (CRITICAL STEP)
./provisioning/scripts/setup-platform-config.sh --generate-toml

When to use Manual Edit:

• Advanced customization beyond form options
• Programmatic configuration generation
• Integration with CI/CD systems
• Custom workspace-specific overrides

Step 3: Understand Configuration Layers

The configuration system uses layered composition:

1. Schema (Type contract)
   ↓ Defines valid fields and constraints

2. Service Defaults (Base values)
   ↓ Default configuration for each service

3. Mode Overlay (Mode-specific tuning)
   ↓ solo, multiuser, cicd, or enterprise settings

4. User Customization (Overrides)
   ↓ User-specific or workspace-specific changes

5. Runtime Config (Final result)
   ↓ provisioning/config/runtime/orchestrator.solo.ncl

6. TOML Export (Service consumption)
   ↓ provisioning/config/runtime/generated/orchestrator.solo.toml

All layers are automatically composed and validated.

Step 4: Verify Generated Configuration

After running the setup script, verify the configuration was created:

# List generated runtime configurations
ls -la provisioning/config/runtime/

# Check generated TOML files
ls -la provisioning/config/runtime/generated/

# Verify TOML is valid
cat provisioning/config/runtime/generated/orchestrator.solo.toml | head -20

You should see files for all 8 services in both the runtime directory (Nickel format) and the generated directory (TOML format).

Step 5: Run Platform Services

After successful configuration, services can be started:

Running a Single Service

# Set deployment mode
export ORCHESTRATOR_MODE=solo

# Run the orchestrator service
cd provisioning/platform
cargo run -p orchestrator

Running Multiple Services

# Terminal 1: Vault Service (secrets management)
export VAULT_MODE=solo
cargo run -p vault-service

# Terminal 2: Orchestrator (main service)
export ORCHESTRATOR_MODE=solo
cargo run -p orchestrator

# Terminal 3: Control Center (web UI)
export CONTROL_CENTER_MODE=solo
cargo run -p control-center

# Access web UI at http://localhost:8080 (default)

Docker-Based Deployment

# Start all services in Docker (requires docker-compose.yml)
cd provisioning/platform/infrastructure/docker
docker-compose -f docker-compose.solo.yml up

# Or for enterprise mode
docker-compose -f docker-compose.enterprise.yml up

Step 6: Verify Services Are Running

# Check orchestrator status
curl http://localhost:9000/health

# Check control center web UI
open http://localhost:8080

# View service logs
export ORCHESTRATOR_MODE=solo
cargo run -p orchestrator -- --log-level debug

Customizing Configuration

Scenario: Change Deployment Mode

If you need to switch from solo to multiuser mode:

# Option 1: Re-run setup with new mode
./provisioning/scripts/setup-platform-config.sh --quick-mode --mode multiuser

# Option 2: Interactive update via TypeDialog
./provisioning/scripts/setup-platform-config.sh --service orchestrator --mode multiuser --backend web

# Result: All configurations updated for multiuser mode
#         Services read from provisioning/config/runtime/generated/orchestrator.multiuser.toml

Scenario: Manual Configuration Edit

If you need fine-grained control:

# 1. Edit the Nickel configuration directly
vim provisioning/config/runtime/orchestrator.solo.ncl

# 2. Make your changes (for example, change port, add environment variables)

# 3. Validate syntax
nickel typecheck provisioning/config/runtime/orchestrator.solo.ncl

# 4. CRITICAL: Regenerate TOML (services won't see changes without this)
./provisioning/scripts/setup-platform-config.sh --generate-toml

# 5. Verify TOML was updated
stat provisioning/config/runtime/generated/orchestrator.solo.toml

# 6. Restart service with new configuration
pkill orchestrator
export ORCHESTRATOR_MODE=solo
cargo run -p orchestrator

Scenario: Workspace-Specific Overrides

For workspace-specific customization:

# Create workspace override file
mkdir -p workspace_myworkspace/config
cat > workspace_myworkspace/config/platform-overrides.ncl <<'EOF'
# Workspace-specific settings
{
  orchestrator = {
    server.port = 9999,  # Custom port
    workspace.name = "myworkspace"
  },

  control_center = {
    workspace.name = "myworkspace"
  }
}
EOF

# Generate config with workspace overrides
./provisioning/scripts/setup-platform-config.sh --workspace workspace_myworkspace

# Configuration system merges: defaults + mode overlay + workspace overrides

Available Configuration Commands

# List all available modes
./provisioning/scripts/setup-platform-config.sh --list-modes
# Output: solo, multiuser, cicd, enterprise

# List all configurable services
./provisioning/scripts/setup-platform-config.sh --list-services
# Output: orchestrator, control-center, mcp-server, vault-service, extension-registry, rag, ai-service, provisioning-daemon

# List current configurations
./provisioning/scripts/setup-platform-config.sh --list-configs
# Output: Shows current runtime configurations and their status

# Clean all runtime configurations (use with caution)
./provisioning/scripts/setup-platform-config.sh --clean
# Removes: provisioning/config/runtime/*.ncl
#          provisioning/config/runtime/generated/*.toml

Configuration File Locations

Public Definitions (Part of repository)

provisioning/schemas/platform/
├── schemas/              # Type contracts (Nickel)
├── defaults/             # Base configuration values
│   └── deployment/       # Mode-specific: solo, multiuser, cicd, enterprise
├── validators/           # Business logic validation
├── templates/            # Configuration generation templates
└── constraints/          # Validation limits

Private Runtime Configs (Gitignored)

provisioning/config/runtime/              # User-specific deployments
├── orchestrator.solo.ncl                 # Editable config
├── orchestrator.multiuser.ncl
└── generated/                            # Auto-generated, don't edit
    ├── orchestrator.solo.toml            # For Rust services
    └── orchestrator.multiuser.toml

Examples (Reference)

provisioning/config/examples/
├── orchestrator.solo.example.ncl         # Solo mode reference
└── orchestrator.enterprise.example.ncl   # Enterprise mode reference

Troubleshooting Configuration

Issue: Script Fails with “Nickel not found”

# Install Nickel
# macOS
brew install nickel

# Linux
cargo install nickel --version 0.10

# Verify installation
nickel --version
# Expected: 0.10.0 or higher

Issue: Configuration Won’t Generate TOML

# Check Nickel syntax
nickel typecheck provisioning/config/runtime/orchestrator.solo.ncl

# If errors found, view detailed message
nickel typecheck -i provisioning/config/runtime/orchestrator.solo.ncl

# Try manual export
nickel export --format toml provisioning/config/runtime/orchestrator.solo.ncl

Issue: Service Can’t Read Configuration

# Verify TOML file exists
ls -la provisioning/config/runtime/generated/orchestrator.solo.toml

# Verify file is valid TOML
head -20 provisioning/config/runtime/generated/orchestrator.solo.toml

# Check service is looking in right location
echo $ORCHESTRATOR_MODE  # Should be set to 'solo', 'multiuser', etc.

# Verify environment variable is correct
export ORCHESTRATOR_MODE=solo
cargo run -p orchestrator --verbose

Issue: Services Won’t Start After Config Change

# If you edited .ncl file manually, TOML must be regenerated
./provisioning/scripts/setup-platform-config.sh --generate-toml

# Verify new TOML was created
stat provisioning/config/runtime/generated/orchestrator.solo.toml

# Check modification time (should be recent)
ls -lah provisioning/config/runtime/generated/orchestrator.solo.toml

Important Notes

🔒 Runtime Configurations Are Private

Files in provisioning/config/runtime/ are gitignored because:

• May contain encrypted secrets or credentials
• Deployment-specific (different per environment)
• User-customized (each developer/machine has different needs)

📘 Schemas Are Public

Files in provisioning/schemas/platform/ are version-controlled because:

• Define product structure and constraints
• Part of official releases
• Source of truth for configuration format
• Shared across the team

🔄 Configuration Is Idempotent

The setup script is safe to run multiple times:

# Safe: Updates only what's needed
./provisioning/scripts/setup-platform-config.sh --quick-mode --mode enterprise

# Safe: Doesn't overwrite without --clean
./provisioning/scripts/setup-platform-config.sh --generate-toml

# Only deletes on explicit request
./provisioning/scripts/setup-platform-config.sh --clean

⚠️ Installer Status

The full provisioning installer (provisioning/scripts/install.sh) is not yet implemented. Currently:

• ✅ Configuration setup script is standalone and ready to use
• ⏳ Full installer integration is planned for future release
• ✅ Manual workflow works perfectly without installer
• ✅ CI/CD integration available now

Next Steps

After completing platform configuration:

1. Run Services: Start your platform services with configured settings
2. Access Web UI: Open Control Center at http://localhost:8080 (default)
3. Create First Infrastructure: Deploy your first servers and clusters
4. Set Up Extensions: Configure providers and task services for your needs
5. Backup Configuration: Back up runtime configs to private repository

Additional Resources

• Setup Status & Current System Status - Quick reference for system readiness
• Configuration README - Detailed configuration management guide
• Setup Script Documentation - Complete script reference
• TypeDialog Platform Config Guide - Advanced configuration topics
• Deployment Guide - Production deployment procedures

Version: 1.0.0 Last Updated: 2026-01-05 Difficulty: Beginner to Intermediate

AI Integration - Intelligent Infrastructure Provisioning

The provisioning platform integrates AI capabilities to provide intelligent assistance for infrastructure configuration, deployment, and troubleshooting. This section documents the AI system architecture, features, and usage patterns.

Overview

The AI integration consists of multiple components working together to provide intelligent infrastructure provisioning:

• typdialog-ai: AI-assisted form filling and configuration
• typdialog-ag: Autonomous AI agents for complex workflows
• typdialog-prov-gen: Natural language to Nickel configuration generation
• ai-service: Core AI service backend with multi-provider support
• mcp-server: Model Context Protocol server for LLM integration
• rag: Retrieval-Augmented Generation for contextual knowledge

Key Features

Natural Language Configuration

Generate infrastructure configurations from plain English descriptions:

provisioning ai generate "Create a production PostgreSQL cluster with encryption and daily backups"

AI-Assisted Forms

Real-time suggestions and explanations as you fill out configuration forms via typdialog web UI.

Intelligent Troubleshooting

AI analyzes deployment failures and suggests fixes:

provisioning ai troubleshoot deployment-12345

Configuration Optimization AI reviews configurations and suggests performance and security improvements:

provisioning ai optimize workspaces/prod/config.ncl

Autonomous Agents

AI agents execute multi-step workflows with minimal human intervention:

provisioning ai agent --goal "Set up complete dev environment for Python app"

Documentation Structure

• Architecture - AI system architecture and components
• Natural Language Config - NL to Nickel generation
• AI-Assisted Forms - typdialog-ai integration
• AI Agents - typdialog-ag autonomous agents
• Config Generation - typdialog-prov-gen details
• RAG System - Retrieval-Augmented Generation
• MCP Integration - Model Context Protocol
• Security Policies - Cedar policies for AI
• Troubleshooting with AI - AI debugging workflows
• API Reference - AI service API documentation
• Configuration - AI system configuration guide
• Cost Management - Managing LLM API costs

Quick Start

Enable AI Features

# Edit provisioning config
vim provisioning/config/ai.toml

# Set provider and enable features
[ai]
enabled = true
provider = "anthropic"  # or "openai" or "local"
model = "claude-sonnet-4"

[ai.features]
form_assistance = true
config_generation = true
troubleshooting = true

Generate Configuration from Natural Language

# Simple generation
provisioning ai generate "PostgreSQL database with encryption"

# With specific schema
provisioning ai generate \
  --schema database \
  --output workspaces/dev/db.ncl \
  "Production PostgreSQL with 100GB storage and daily backups"

Use AI-Assisted Forms

# Open typdialog web UI with AI assistance
provisioning workspace init --interactive --ai-assist

# AI provides real-time suggestions as you type
# AI explains validation errors in plain English
# AI fills multiple fields from natural language description

Troubleshoot with AI

# Analyze failed deployment
provisioning ai troubleshoot deployment-12345

# AI analyzes logs and suggests fixes
# AI generates corrected configuration
# AI explains root cause in plain language

Security and Privacy

The AI system implements strict security controls:

• ✅ Cedar Policies: AI access controlled by Cedar authorization
• ✅ Secret Isolation: AI cannot access secrets directly
• ✅ Human Approval: Critical operations require human approval
• ✅ Audit Trail: All AI operations logged
• ✅ Data Sanitization: Secrets/PII sanitized before sending to LLM
• ✅ Local Models: Support for air-gapped deployments

See Security Policies for complete details.

Supported LLM Providers

| | Provider | Models | Best For | | | | ––––– | –––– | ––––– | | | | Anthropic | Claude Sonnet 4, Claude Opus 4 | Complex configs, long context | | | | OpenAI | GPT-4 Turbo, GPT-4 | Fast suggestions, tool calling | | | | Local | Llama 3, Mistral | Air-gapped, privacy-critical | |

Cost Considerations

AI features incur LLM API costs. The system implements cost controls:

• Caching: Reduces API calls by 50-80%
• Rate Limiting: Prevents runaway costs
• Budget Limits: Daily/monthly cost caps
• Local Models: Zero marginal cost for air-gapped deployments

See Cost Management for optimization strategies.

Architecture Decision Record

The AI integration is documented in:

• ADR-015: AI Integration Architecture

Next Steps

1. Read Architecture to understand AI system design
2. Configure AI features in Configuration
3. Try Natural Language Config for your first AI-generated config
4. Explore AI Agents for automation workflows
5. Review Security Policies to understand access controls

Version: 1.0 Last Updated: 2025-01-08 Status: Active

AI Integration Architecture

Overview

The provisioning platform’s AI system provides intelligent capabilities for configuration generation, troubleshooting, and automation. The architecture consists of multiple layers designed for reliability, security, and performance.

Core Components - Production-Ready

1. AI Service (provisioning/platform/ai-service)

Status: ✅ Production-Ready (2,500+ lines Rust code)

The core AI service provides:

• Multi-provider LLM support (Anthropic Claude, OpenAI GPT-4, local models)
• Streaming response support for real-time feedback
• Request caching with LRU and semantic similarity
• Rate limiting and cost control
• Comprehensive error handling
• HTTP REST API on port 8083

Supported Models:

• Claude Sonnet 4, Claude Opus 4 (Anthropic)
• GPT-4 Turbo, GPT-4 (OpenAI)
• Llama 3, Mistral (local/on-premise)

2. RAG System (Retrieval-Augmented Generation)

Status: ✅ Production-Ready (22/22 tests passing)

The RAG system enables AI to access and reason over platform documentation:

• Vector embeddings via SurrealDB vector store
• Hybrid search: vector similarity + BM25 keyword search
• Document chunking (code and markdown aware)
• Relevance ranking and context selection
• Semantic caching for repeated queries

Capabilities:

provisioning ai query "How do I set up Kubernetes?"
provisioning ai template "Describe my infrastructure"

3. MCP Server (Model Context Protocol)

Status: ✅ Production-Ready

Provides Model Context Protocol integration:

• Standardized tool interface for LLMs
• Complex workflow composition
• Integration with external AI systems (Claude, other LLMs)
• Tool calling for provisioning operations

4. CLI Integration

Status: ✅ Production-Ready

Interactive commands:

provisioning ai template --prompt "Describe infrastructure"
provisioning ai query --prompt "Configuration question"
provisioning ai chat    # Interactive mode

Configuration:

[ai]
enabled = true
provider = "anthropic"  # or "openai" or "local"
model = "claude-sonnet-4"

[ai.cache]
enabled = true
semantic_similarity = true
ttl_seconds = 3600

[ai.limits]
max_tokens = 4096
temperature = 0.7

Planned Components - Q2 2025

Autonomous Agents (typdialog-ag)

Status: 🔴 Planned

Self-directed agents for complex tasks:

• Multi-step workflow execution
• Decision making and adaptation
• Monitoring and self-healing recommendations

AI-Assisted Forms (typdialog-ai)

Status: 🔴 Planned

Real-time AI suggestions in configuration forms:

• Context-aware field recommendations
• Validation error explanations
• Auto-completion for infrastructure patterns

Advanced Features

• Fine-tuning capabilities for custom models
• Autonomous workflow execution with human approval
• Cedar authorization policies for AI actions
• Custom knowledge bases per workspace

Architecture Diagram

┌─────────────────────────────────────────────────┐
│  User Interface                                 │
│  ├── CLI (provisioning ai ...)                  │
│  ├── Web UI (typdialog)                         │
│  └── MCP Client (Claude, etc.)                  │
└──────────────┬──────────────────────────────────┘
               ↓
┌──────────────────────────────────────────────────┐
│  AI Service (Port 8083)                          │
│  ├── Request Router                             │
│  ├── Cache Layer (LRU + Semantic)              │
│  ├── Prompt Engineering                         │
│  └── Response Streaming                         │
└──────┬─────────────────┬─────────────────────────┘
       ↓                 ↓
┌─────────────┐  ┌──────────────────┐
│ RAG System  │  │ LLM Provider     │
│ SurrealDB   │  │ ├── Anthropic    │
│ Vector DB   │  │ ├── OpenAI       │
│ + BM25      │  │ └── Local Model  │
└─────────────┘  └──────────────────┘
       ↓                 ↓
┌──────────────────────────────────────┐
│  Cached Responses + Real Responses   │
│  Streamed to User                    │
└──────────────────────────────────────┘

Performance Characteristics

| | Metric | Value | | | | –––– | —–– | | | | Cold response (cache miss) | 2-5 seconds | | | | Cached response | <500ms | | | | Streaming start time | <1 second | | | | AI service memory usage | ~200MB at rest | | | | Cache size (configurable) | Up to 500MB | | | | Vector DB (SurrealDB) | Included, auto-managed | |

Security Model

Cedar Authorization

All AI operations controlled by Cedar policies:

• User role-based access control
• Operation-specific permissions
• Complete audit logging

Secret Protection

• Secrets never sent to external LLMs
• PII/sensitive data sanitized before API calls
• Encryption at rest in local cache
• HSM support for key storage

Local Model Support

Air-gapped deployments:

• On-premise LLM models (Llama 3, Mistral)
• Zero external API calls
• Full data privacy compliance
• Ideal for classified environments

Configuration

See Configuration Guide for:

• LLM provider setup
• Cache configuration
• Cost limits and budgets
• Security policies

Related Documentation

• RAG System - Retrieval implementation details
• Security Policies - Authorization and safety controls
• Configuration Guide - Setup instructions
• ADR-015 - Design decisions

Last Updated: 2025-01-13 Status: ✅ Production-Ready (core system) Test Coverage: 22/22 tests passing

Retrieval-Augmented Generation (RAG) System

Status: ✅ Production-Ready (SurrealDB 1.5.0+, 22/22 tests passing)

The RAG system enables the AI service to access, retrieve, and reason over infrastructure documentation, schemas, and past configurations. This allows the AI to generate contextually accurate infrastructure configurations and provide intelligent troubleshooting advice grounded in actual platform knowledge.

Architecture Overview

The RAG system consists of:

1. Document Store: SurrealDB vector store with semantic indexing
2. Hybrid Search: Vector similarity + BM25 keyword search
3. Chunk Management: Intelligent document chunking for code and markdown
4. Context Ranking: Relevance scoring for retrieved documents
5. Semantic Cache: Deduplication of repeated queries

Core Components

1. Vector Embeddings

The system uses embedding models to convert documents into vector representations:

┌─────────────────────┐
│ Document Source     │
│ (Markdown, Code)    │
└──────────┬──────────┘
           │
           ▼
┌──────────────────────────────────┐
│ Chunking & Tokenization          │
│ - Code-aware splits              │
│ - Markdown aware                 │
│ - Preserves context              │
└──────────┬───────────────────────┘
           │
           ▼
┌──────────────────────────────────┐
│ Embedding Model                  │
│ (OpenAI Ada, Anthropic, Local)   │
└──────────┬───────────────────────┘
           │
           ▼
┌──────────────────────────────────┐
│ Vector Storage (SurrealDB)       │
│ - Vector index                   │
│ - Metadata indexed               │
│ - BM25 index for keywords        │
└──────────────────────────────────┘

2. SurrealDB Integration

SurrealDB serves as the vector database and knowledge store:

# Configuration in provisioning/schemas/ai.ncl
let {
  rag = {
    enabled = true,
    db_url = "surreal://localhost:8000",
    namespace = "provisioning",
    database = "ai_rag",
    
    # Collections for different document types
    collections = {
      documentation = {
        chunking_strategy = "markdown",
        chunk_size = 1024,
        overlap = 256,
      },
      schemas = {
        chunking_strategy = "code",
        chunk_size = 512,
        overlap = 128,
      },
      deployments = {
        chunking_strategy = "json",
        chunk_size = 2048,
        overlap = 512,
      },
    },
    
    # Embedding configuration
    embedding = {
      provider = "openai",  # or "anthropic", "local"
      model = "text-embedding-3-small",
      cache_vectors = true,
    },
    
    # Search configuration
    search = {
      hybrid_enabled = true,
      vector_weight = 0.7,
      keyword_weight = 0.3,
      top_k = 5,  # Number of results to return
      semantic_cache = true,
    },
  }
}

3. Document Chunking

Intelligent chunking preserves context while managing token limits:

Markdown Chunking Strategy

Input Document: provisioning/docs/src/guides/from-scratch.md

Chunks:
  [1] Header + first section (up to 1024 tokens)
  [2] Next logical section + overlap with [1]
  [3] Code examples preserve as atomic units
  [4] Continue with overlap...

Each chunk includes:
  - Original section heading (for context)
  - Content
  - Source file and line numbers
  - Metadata (doctype, category, version)

Code Chunking Strategy

Input Document: provisioning/schemas/main.ncl

Chunks:
  [1] Top-level let binding + comments
  [2] Function definition (atomic, preserves signature)
  [3] Type definition (atomic, preserves interface)
  [4] Implementation blocks with context overlap

Each chunk preserves:
  - Type signatures
  - Function signatures
  - Import statements needed for context
  - Comments and docstrings

Hybrid Search

The system implements dual search strategy for optimal results:

Vector Similarity Search

// Find semantically similar documents
async fn vector_search(query: &str, top_k: usize) -> Vec<Document> {
    let embedding = embed(query).await?;
    
    // L2 distance in SurrealDB
    db.query("
        SELECT *, vector::similarity::cosine(embedding, $embedding) AS score
        FROM documents
        WHERE embedding <~> $embedding
        ORDER BY score DESC
        LIMIT $top_k
    ")
    .bind(("embedding", embedding))
    .bind(("top_k", top_k))
    .await
}

Use case: Semantic understanding of intent

• Query: “How to configure PostgreSQL”
• Finds: Documents about database configuration, examples, schemas

BM25 Keyword Search

// Find documents with matching keywords
async fn keyword_search(query: &str, top_k: usize) -> Vec<Document> {
    // BM25 full-text search in SurrealDB
    db.query("
        SELECT *, search::bm25(.) AS score
        FROM documents
        WHERE text @@ $query
        ORDER BY score DESC
        LIMIT $top_k
    ")
    .bind(("query", query))
    .bind(("top_k", top_k))
    .await
}

Use case: Exact term matching

• Query: “SurrealDB configuration”
• Finds: Documents mentioning SurrealDB specifically

Hybrid Results

async fn hybrid_search(
    query: &str,
    vector_weight: f32,
    keyword_weight: f32,
    top_k: usize,
) -> Vec<Document> {
    let vector_results = vector_search(query, top_k * 2).await?;
    let keyword_results = keyword_search(query, top_k * 2).await?;
    
    let mut scored = HashMap::new();
    
    // Score from vector search
    for (i, doc) in vector_results.iter().enumerate() {
        *scored.entry(doc.id).or_insert(0.0) +=
            vector_weight * (1.0 - (i as f32 / top_k as f32));
    }
    
    // Score from keyword search
    for (i, doc) in keyword_results.iter().enumerate() {
        *scored.entry(doc.id).or_insert(0.0) +=
            keyword_weight * (1.0 - (i as f32 / top_k as f32));
    }
    
    // Return top-k by combined score
    let mut results: Vec<_> = scored.into_iter().collect();
| results.sort_by( | a, b | b.1.partial_cmp(&a.1).unwrap()); |
| Ok(results.into_iter().take(top_k).map( | (id, _) | ...).collect()) |
}

Semantic Caching

Reduces API calls by caching embeddings of repeated queries:

struct SemanticCache {
    queries: Arc<DashMap<Vec<f32>, CachedResult>>,
    similarity_threshold: f32,
}

impl SemanticCache {
    async fn get(&self, query: &str) -> Option<CachedResult> {
        let embedding = embed(query).await?;
        
        // Find cached query with similar embedding
        // (cosine distance < threshold)
        for entry in self.queries.iter() {
            let distance = cosine_distance(&embedding, entry.key());
            if distance < self.similarity_threshold {
                return Some(entry.value().clone());
            }
        }
        None
    }
    
    async fn insert(&self, query: &str, result: CachedResult) {
        let embedding = embed(query).await?;
        self.queries.insert(embedding, result);
    }
}

Benefits:

• 50-80% reduction in embedding API calls
• Identical queries return in <10ms
• Similar queries reuse cached context

Ingestion Workflow

Document Indexing

# Index all documentation
provisioning ai index-docs provisioning/docs/src

# Index schemas
provisioning ai index-schemas provisioning/schemas

# Index past deployments
provisioning ai index-deployments workspaces/*/deployments

# Watch directory for changes (development mode)
provisioning ai watch docs provisioning/docs/src

Programmatic Indexing

// In ai-service on startup
async fn initialize_rag() -> Result<()> {
    let rag = RAGSystem::new(&config.rag).await?;
    
    // Index documentation
    let docs = load_markdown_docs("provisioning/docs/src")?;
    for doc in docs {
        rag.ingest_document(&doc).await?;
    }
    
    // Index schemas
    let schemas = load_nickel_schemas("provisioning/schemas")?;
    for schema in schemas {
        rag.ingest_schema(&schema).await?;
    }
    
    Ok(())
}

Usage Examples

Query the RAG System

# Search for context-aware information
provisioning ai query "How do I configure PostgreSQL with encryption?"

# Get configuration template
provisioning ai template "Describe production Kubernetes on AWS"

# Interactive mode
provisioning ai chat
> What are the best practices for database backup?

AI Service Integration

// AI service uses RAG to enhance generation
async fn generate_config(user_request: &str) -> Result<String> {
    // Retrieve relevant context
    let context = rag.search(user_request, top_k=5).await?;
    
    // Build prompt with context
    let prompt = build_prompt_with_context(user_request, &context);
    
    // Generate configuration
    let config = llm.generate(&prompt).await?;
    
    // Validate against schemas
    validate_nickel_config(&config)?;
    
    Ok(config)
}

Form Assistance Integration

// In typdialog-ai (JavaScript/TypeScript)
async function suggestFieldValue(fieldName, currentInput) {
    // Query RAG for similar configurations
    const context = await rag.search(
        `Field: ${fieldName}, Input: ${currentInput}`,
        { topK: 3, semantic: true }
    );
    
    // Generate suggestion using context
    const suggestion = await ai.suggest({
        field: fieldName,
        input: currentInput,
        context: context,
    });
    
    return suggestion;
}

Performance Characteristics

| | Operation | Time | Cache Hit | | | | ———– | —— | ———– | | | | Vector embedding | 200-500ms | N/A | | | | Vector search (cold) | 300-800ms | N/A | | | | Keyword search | 50-200ms | N/A | | | | Hybrid search | 500-1200ms | <100ms cached | | | | Semantic cache hit | 10-50ms | Always | |

Typical query flow:

1. Embedding: 300ms
2. Vector search: 400ms
3. Keyword search: 100ms
4. Ranking: 50ms
5. Total: ~850ms (first call), <100ms (cached)

Configuration

See Configuration Guide for detailed RAG setup:

• LLM provider for embeddings
• SurrealDB connection
• Chunking strategies
• Search weights and limits
• Cache settings and TTLs

Limitations and Considerations

Document Freshness

• RAG indexes static snapshots
• Changes to documentation require re-indexing
• Use watch mode during development

Token Limits

• Large documents chunked to fit LLM context
• Some context may be lost in chunking
• Adjustable chunk size vs. context trade-off

Embedding Quality

• Quality depends on embedding model
• Domain-specific models perform better
• Fine-tuning possible for specialized vocabularies

Monitoring and Debugging

Query Metrics

# View RAG search metrics
provisioning ai metrics show rag

# Analysis of search quality
provisioning ai eval-rag --sample-queries 100

Debug Mode

# In provisioning/config/ai.toml
[ai.rag.debug]
enabled = true
log_embeddings = true      # Log embedding vectors
log_search_scores = true   # Log relevance scores
log_context_used = true    # Log context retrieved

Related Documentation

• Architecture - AI system overview
• MCP Integration - RAG access via MCP
• Configuration - RAG setup guide
• API Reference - RAG API endpoints
• ADR-015 - Design decisions

Last Updated: 2025-01-13 Status: ✅ Production-Ready Test Coverage: 22/22 tests passing Database: SurrealDB 1.5.0+

Model Context Protocol (MCP) Integration

Status: ✅ Production-Ready (MCP 0.6.0+, integrated with Claude, compatible with all LLMs)

The MCP server provides standardized Model Context Protocol integration, allowing external LLMs (Claude, GPT-4, local models) to access provisioning platform capabilities as tools. This enables complex multi-step workflows, tool composition, and integration with existing LLM applications.

Architecture Overview

The MCP integration follows the Model Context Protocol specification:

┌──────────────────────────────────────────────────────────────┐
│ External LLM (Claude, GPT-4, etc.)                           │
└────────────────────┬─────────────────────────────────────────┘
                     │
                     │ Tool Calls (JSON-RPC)
                     ▼
┌──────────────────────────────────────────────────────────────┐
│ MCP Server (provisioning/platform/crates/mcp-server)         │
│                                                              │
│ ┌───────────────────────────────────────────────────────┐    │
│ │ Tool Registry                                         │    │
│ │ - generate_config(description, schema)                │    │
│ │ - validate_config(config)                             │    │
│ │ - search_docs(query)                                  │    │
│ │ - troubleshoot_deployment(logs)                       │    │
│ │ - get_schema(name)                                    │    │
│ │ - check_compliance(config, policy)                    │    │
│ └───────────────────────────────────────────────────────┘    │
│                         │                                    │
│                         ▼                                    │
│ ┌───────────────────────────────────────────────────────┐    │
│ │ Implementation Layer                                  │    │
│ │ - AI Service client (ai-service port 8083)            │    │
│ │ - Validator client                                    │    │
│ │ - RAG client (SurrealDB)                              │    │
│ │ - Schema loader                                       │    │
│ └───────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────┘

MCP Server Launch

The MCP server is started as a stdio-based service:

# Start MCP server (stdio transport)
provisioning-mcp-server --config /etc/provisioning/ai.toml

# With debug logging
RUST_LOG=debug provisioning-mcp-server --config /etc/provisioning/ai.toml

# In Claude Desktop configuration
~/.claude/claude_desktop_config.json:
{
  "mcpServers": {
    "provisioning": {
      "command": "provisioning-mcp-server",
      "args": ["--config", "/etc/provisioning/ai.toml"],
      "env": {
        "PROVISIONING_TOKEN": "your-auth-token"
      }
    }
  }
}

Available Tools

1. Config Generation

Tool: generate_config

Generate infrastructure configuration from natural language description.

{
  "name": "generate_config",
  "description": "Generate a Nickel infrastructure configuration from a natural language description",
  "inputSchema": {
    "type": "object",
    "properties": {
      "description": {
        "type": "string",
        "description": "Natural language description of desired infrastructure"
      },
      "schema": {
        "type": "string",
        "description": "Target schema name (e.g., 'database', 'kubernetes', 'network'). Optional."
      },
      "format": {
        "type": "string",
        "enum": ["nickel", "toml"],
        "description": "Output format (default: nickel)"
      }
    },
    "required": ["description"]
  }
}

Example Usage:

# Via MCP client
mcp-client provisioning generate_config \
  --description "Production PostgreSQL cluster with encryption and daily backups" \
  --schema database

# Claude desktop prompt:
# @provisioning: Generate a production PostgreSQL setup with automated backups

Response:

{
  database = {
    engine = "postgresql",
    version = "15.0",
    
    instance = {
      instance_class = "db.r6g.xlarge",
      allocated_storage_gb = 100,
      iops = 3000,
    },
    
    security = {
      encryption_enabled = true,
      encryption_key_id = "kms://prod-db-key",
      tls_enabled = true,
      tls_version = "1.3",
    },
    
    backup = {
      enabled = true,
      retention_days = 30,
      preferred_window = "03:00-04:00",
      copy_to_region = "us-west-2",
    },
    
    monitoring = {
      enhanced_monitoring_enabled = true,
      monitoring_interval_seconds = 60,
      log_exports = ["postgresql"],
    },
  }
}

2. Config Validation

Tool: validate_config

Validate a Nickel configuration against schemas and policies.

{
  "name": "validate_config",
  "description": "Validate a Nickel configuration file",
  "inputSchema": {
    "type": "object",
    "properties": {
      "config": {
        "type": "string",
        "description": "Nickel configuration content or file path"
      },
      "schema": {
        "type": "string",
        "description": "Schema name to validate against (optional)"
      },
      "strict": {
        "type": "boolean",
        "description": "Enable strict validation (default: true)"
      }
    },
    "required": ["config"]
  }
}

Example Usage:

# Validate configuration
mcp-client provisioning validate_config \
  --config "$(cat workspaces/prod/database.ncl)"

# With specific schema
mcp-client provisioning validate_config \
  --config "workspaces/prod/kubernetes.ncl" \
  --schema kubernetes

Response:

{
  "valid": true,
  "errors": [],
  "warnings": [
    "Consider enabling automated backups for production use"
  ],
  "metadata": {
    "schema": "kubernetes",
    "version": "1.28",
    "validated_at": "2025-01-13T10:45:30Z"
  }
}

3. Documentation Search

Tool: search_docs

Search infrastructure documentation using RAG system.

{
  "name": "search_docs",
  "description": "Search provisioning documentation for information",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query": {
        "type": "string",
        "description": "Search query (natural language)"
      },
      "top_k": {
        "type": "integer",
        "description": "Number of results (default: 5)"
      },
      "doc_type": {
        "type": "string",
        "enum": ["guide", "schema", "example", "troubleshooting"],
        "description": "Filter by document type (optional)"
      }
    },
    "required": ["query"]
  }
}

Example Usage:

# Search documentation
mcp-client provisioning search_docs \
  --query "How do I configure PostgreSQL with replication?"

# Get examples
mcp-client provisioning search_docs \
  --query "Kubernetes networking" \
  --doc_type example \
  --top_k 3

Response:

{
  "results": [
    {
      "source": "provisioning/docs/src/guides/database-replication.md",
      "excerpt": "PostgreSQL logical replication enables streaming of changes...",
      "relevance": 0.94,
      "section": "Setup Logical Replication"
    },
    {
      "source": "provisioning/schemas/database.ncl",
      "excerpt": "replication = { enabled = true, mode = \"logical\", ... }",
      "relevance": 0.87,
      "section": "Replication Configuration"
    }
  ]
}

4. Deployment Troubleshooting

Tool: troubleshoot_deployment

Analyze deployment failures and suggest fixes.

{
  "name": "troubleshoot_deployment",
  "description": "Analyze deployment logs and suggest fixes",
  "inputSchema": {
    "type": "object",
    "properties": {
      "deployment_id": {
        "type": "string",
        "description": "Deployment ID (e.g., 'deploy-2025-01-13-001')"
      },
      "logs": {
        "type": "string",
        "description": "Deployment logs (optional, if deployment_id not provided)"
      },
      "error_analysis_depth": {
        "type": "string",
        "enum": ["shallow", "deep"],
        "description": "Analysis depth (default: deep)"
      }
    }
  }
}

Example Usage:

# Troubleshoot recent deployment
mcp-client provisioning troubleshoot_deployment \
  --deployment_id "deploy-2025-01-13-001"

# With custom logs
mcp-client provisioning troubleshoot_deployment \
| --logs "$(journalctl -u provisioning --no-pager | tail -100)" |

Response:

{
  "status": "failure",
  "root_cause": "Database connection timeout during migration phase",
  "analysis": {
    "phase": "database_migration",
    "error_type": "connectivity",
    "confidence": 0.95
  },
  "suggestions": [
    "Verify database security group allows inbound on port 5432",
    "Check database instance status (may be rebooting)",
    "Increase connection timeout in configuration"
  ],
  "corrected_config": "...generated Nickel config with fixes...",
  "similar_issues": [
    "[https://docs/troubleshooting/database-connectivity.md"](https://docs/troubleshooting/database-connectivity.md")
  ]
}

5. Get Schema

Tool: get_schema

Retrieve schema definition with examples.

{
  "name": "get_schema",
  "description": "Get a provisioning schema definition",
  "inputSchema": {
    "type": "object",
    "properties": {
      "schema_name": {
        "type": "string",
        "description": "Schema name (e.g., 'database', 'kubernetes')"
      },
      "format": {
        "type": "string",
        "enum": ["schema", "example", "documentation"],
        "description": "Response format (default: schema)"
      }
    },
    "required": ["schema_name"]
  }
}

Example Usage:

# Get schema definition
mcp-client provisioning get_schema --schema_name database

# Get example configuration
mcp-client provisioning get_schema \
  --schema_name kubernetes \
  --format example

6. Compliance Check

Tool: check_compliance

Verify configuration against compliance policies (Cedar).

{
  "name": "check_compliance",
  "description": "Check configuration against compliance policies",
  "inputSchema": {
    "type": "object",
    "properties": {
      "config": {
        "type": "string",
        "description": "Configuration to check"
      },
      "policy_set": {
        "type": "string",
        "description": "Policy set to check against (e.g., 'pci-dss', 'hipaa', 'sox')"
      }
    },
    "required": ["config", "policy_set"]
  }
}

Example Usage:

# Check against PCI-DSS
mcp-client provisioning check_compliance \
  --config "$(cat workspaces/prod/database.ncl)" \
  --policy_set pci-dss

Integration Examples

Claude Desktop (Most Common)

~/.claude/claude_desktop_config.json:
{
  "mcpServers": {
    "provisioning": {
      "command": "provisioning-mcp-server",
      "args": ["--config", "/etc/provisioning/ai.toml"],
      "env": {
        "PROVISIONING_API_KEY": "sk-...",
        "PROVISIONING_BASE_URL": "[http://localhost:8083"](http://localhost:8083")
      }
    }
  }
}

Usage in Claude:

User: I need a production Kubernetes cluster in AWS with automatic scaling

Claude can now use provisioning tools:
I'll help you create a production Kubernetes cluster. Let me:
1. Search the documentation for best practices
2. Generate a configuration template
3. Validate it against your policies
4. Provide the final configuration

OpenAI Function Calling

import openai

tools = [
    {
        "type": "function",
        "function": {
            "name": "generate_config",
            "description": "Generate infrastructure configuration",
            "parameters": {
                "type": "object",
                "properties": {
                    "description": {
                        "type": "string",
                        "description": "Infrastructure description"
                    }
                },
                "required": ["description"]
            }
        }
    }
]

response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Create a PostgreSQL database"}],
    tools=tools
)

Local LLM Integration (Ollama)

# Start Ollama with provisioning MCP
OLLAMA_MCP_SERVERS=provisioning://localhost:3000 \
  ollama serve

# Use with llama2 or mistral
curl [http://localhost:11434/api/generate](http://localhost:11434/api/generate) \
  -d '{
    "model": "mistral",
    "prompt": "Create a Kubernetes cluster",
    "tools": [{"type": "mcp", "server": "provisioning"}]
  }'

Error Handling

Tools return consistent error responses:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Configuration has 3 validation errors",
    "details": [
      {
        "field": "database.version",
        "message": "PostgreSQL version 9.6 is deprecated",
        "severity": "error"
      },
      {
        "field": "backup.retention_days",
        "message": "Recommended minimum is 30 days for production",
        "severity": "warning"
      }
    ]
  }
}

Performance

| | Operation | Latency | Notes | | | | ———– | ——— | —–– | | | | generate_config | 2-5s | Depends on LLM and config complexity | | | | validate_config | 500-1000ms | Parallel schema validation | | | | search_docs | 300-800ms | RAG hybrid search | | | | troubleshoot | 3-8s | Depends on log size and analysis depth | | | | get_schema | 100-300ms | Cached schema retrieval | | | | check_compliance | 500-2000ms | Policy evaluation | |

Configuration

See Configuration Guide for MCP-specific settings:

• MCP server port and binding
• Tool registry customization
• Rate limiting for tool calls
• Access control (Cedar policies)

Security

Authentication

• Tools require valid provisioning API token
• Token scoped to user’s workspace
• All tool calls authenticated and logged

Authorization

• Cedar policies control which tools user can call
• Example: allow(principal, action, resource) when role == "admin"
• Detailed audit trail of all tool invocations

Data Protection

• Secrets never passed through MCP
• Configuration sanitized before analysis
• PII removed from logs sent to external LLMs

Monitoring and Debugging

# Monitor MCP server
provisioning admin mcp status

# View MCP tool calls
provisioning admin logs --filter "mcp_tools" --tail 100

# Debug tool response
RUST_LOG=provisioning::mcp=debug provisioning-mcp-server

Related Documentation

• Architecture - AI system overview
• RAG System - Documentation search
• Configuration - MCP setup
• API Reference - Detailed API endpoints
• ADR-015 - Design decisions

Last Updated: 2025-01-13 Status: ✅ Production-Ready MCP Version: 0.6.0+ Supported LLMs: Claude, GPT-4, Llama, Mistral, all MCP-compatible models

AI System Configuration Guide

Status: ✅ Production-Ready (Configuration system)

Complete setup guide for AI features in the provisioning platform. This guide covers LLM provider configuration, feature enablement, cache setup, cost controls, and security settings.

Quick Start

Minimal Configuration

# provisioning/config/ai.toml
[ai]
enabled = true
provider = "anthropic"  # or "openai" or "local"
model = "claude-sonnet-4"
api_key = "sk-ant-..."  # Set via PROVISIONING_AI_API_KEY env var

[ai.cache]
enabled = true

[ai.limits]
max_tokens = 4096
temperature = 0.7

Initialize Configuration

# Generate default configuration
provisioning config init ai

# Edit configuration
provisioning config edit ai

# Validate configuration
provisioning config validate ai

# Show current configuration
provisioning config show ai

Provider Configuration

Anthropic Claude

[ai]
enabled = true
provider = "anthropic"
model = "claude-sonnet-4"  # or "claude-opus-4", "claude-haiku-4"
api_key = "${PROVISIONING_AI_API_KEY}"
api_base = "[https://api.anthropic.com"](https://api.anthropic.com")

# Request parameters
[ai.request]
max_tokens = 4096
temperature = 0.7
top_p = 0.95
top_k = 40

# Supported models
# - claude-opus-4: Most capable, for complex reasoning ($15/MTok input, $45/MTok output)
# - claude-sonnet-4: Balanced (recommended), ($3/MTok input, $15/MTok output)
# - claude-haiku-4: Fast, for simple tasks ($0.80/MTok input, $4/MTok output)

OpenAI GPT-4

[ai]
enabled = true
provider = "openai"
model = "gpt-4-turbo"  # or "gpt-4", "gpt-4o"
api_key = "${OPENAI_API_KEY}"
api_base = "[https://api.openai.com/v1"](https://api.openai.com/v1")

[ai.request]
max_tokens = 4096
temperature = 0.7
top_p = 0.95

# Supported models
# - gpt-4: Most capable ($0.03/1K input, $0.06/1K output)
# - gpt-4-turbo: Better at code ($0.01/1K input, $0.03/1K output)
# - gpt-4o: Latest, multi-modal ($5/MTok input, $15/MTok output)

Local Models

[ai]
enabled = true
provider = "local"
model = "llama2-70b"  # or "mistral", "neural-chat"
api_base = "[http://localhost:8000"](http://localhost:8000")  # Local Ollama or LM Studio

# Local model support
# - Ollama: docker run -d -v ollama:/root/.ollama -p 11434:11434 ollama/ollama
# - LM Studio: GUI app with API
# - vLLM: High-throughput serving
# - llama.cpp: CPU inference

[ai.local]
gpu_enabled = true
gpu_memory_gb = 24
max_batch_size = 4

Feature Configuration

Enable Specific Features

[ai.features]
# Core features (production-ready)
rag_search = true           # Retrieve-Augmented Generation
config_generation = true    # Generate Nickel from natural language
mcp_server = true           # Model Context Protocol server
troubleshooting = true      # AI-assisted debugging

# Form assistance (planned Q2 2025)
form_assistance = false     # AI suggestions in forms
form_explanations = false   # AI explains validation errors

# Agents (planned Q2 2025)
autonomous_agents = false   # AI agents for workflows
agent_learning = false      # Agents learn from deployments

# Advanced features
fine_tuning = false        # Fine-tune models for domain
knowledge_base = false     # Custom knowledge base per workspace

Cache Configuration

Cache Strategy

[ai.cache]
enabled = true
cache_type = "memory"  # or "redis", "disk"
ttl_seconds = 3600     # Cache entry lifetime

# Memory cache (recommended for single server)
[ai.cache.memory]
max_size_mb = 500
eviction_policy = "lru"  # Least Recently Used

# Redis cache (recommended for distributed)
[ai.cache.redis]
url = "redis://localhost:6379"
db = 0
password = "${REDIS_PASSWORD}"
ttl_seconds = 3600

# Disk cache (recommended for persistent caching)
[ai.cache.disk]
path = "/var/cache/provisioning/ai"
max_size_mb = 5000

# Semantic caching (for RAG)
[ai.cache.semantic]
enabled = true
similarity_threshold = 0.95  # Cache hit if query similarity > 0.95
cache_embeddings = true       # Cache embedding vectors

Cache Metrics

# Monitor cache performance
provisioning admin cache stats ai

# Clear cache
provisioning admin cache clear ai

# Analyze cache efficiency
provisioning admin cache analyze ai --hours 24

Rate Limiting and Cost Control

Rate Limits

[ai.limits]
# Tokens per request
max_tokens = 4096
max_input_tokens = 8192
max_output_tokens = 4096

# Requests per minute/hour
rpm_limit = 60              # Requests per minute
rpm_burst = 100             # Allow bursts up to 100 RPM

# Daily cost limit
daily_cost_limit_usd = 100
warn_at_percent = 80        # Warn when at 80% of daily limit
stop_at_percent = 95        # Stop accepting requests at 95%

# Token usage tracking
track_token_usage = true
track_cost_per_request = true

Cost Budgeting

[ai.budget]
enabled = true
monthly_limit_usd = 1000

# Budget alerts
alert_at_percent = [50, 75, 90]
alert_email = "ops@company.com"
alert_slack = "[https://hooks.slack.com/services/..."](https://hooks.slack.com/services/...")

# Cost by provider
[ai.budget.providers]
anthropic_limit = 500
openai_limit = 300
local_limit = 0  # Free (run locally)

Track Costs

# View cost metrics
provisioning admin costs show ai --period month

# Forecast cost
provisioning admin costs forecast ai --days 30

# Analyze cost by feature
provisioning admin costs analyze ai --by feature

# Export cost report
provisioning admin costs export ai --format csv --output costs.csv

Security Configuration

Authentication

[ai.auth]
# API key from environment variable
api_key = "${PROVISIONING_AI_API_KEY}"

# Or from secure store
api_key_vault = "secrets/ai-api-key"

# Token rotation
rotate_key_days = 90
rotation_alert_days = 7

# Request signing (for cloud providers)
sign_requests = true
signing_method = "hmac-sha256"

Authorization (Cedar)

[ai.authorization]
enabled = true
policy_file = "provisioning/policies/ai-policies.cedar"

# Example policies:
# allow(principal, action, resource) when principal.role == "admin"
# allow(principal == ?principal, action == "ai_generate_config", resource)
#   when principal.workspace == resource.workspace

Data Protection

[ai.security]
# Sanitize data before sending to external LLM
sanitize_pii = true
sanitize_secrets = true
redact_patterns = [
  "(?i)password\\s*[:=]\\s*[^\\s]+",  # Passwords
  "(?i)api[_-]?key\\s*[:=]\\s*[^\\s]+", # API keys
  "(?i)secret\\s*[:=]\\s*[^\\s]+",     # Secrets
]

# Encryption
encryption_enabled = true
encryption_algorithm = "aes-256-gcm"
key_derivation = "argon2id"

# Local-only mode (never send to external LLM)
local_only = false  # Set true for air-gapped deployments

RAG Configuration

Vector Store Setup

[ai.rag]
enabled = true

# SurrealDB backend
[ai.rag.database]
url = "surreal://localhost:8000"
username = "root"
password = "${SURREALDB_PASSWORD}"
namespace = "provisioning"
database = "ai_rag"

# Embedding model
[ai.rag.embedding]
provider = "openai"  # or "anthropic", "local"
model = "text-embedding-3-small"
batch_size = 100
cache_embeddings = true

# Search configuration
[ai.rag.search]
hybrid_enabled = true
vector_weight = 0.7      # Weight for vector search
keyword_weight = 0.3     # Weight for BM25 search
top_k = 5                # Number of results to return
rerank_enabled = false   # Use cross-encoder to rerank results

# Chunking strategy
[ai.rag.chunking]
markdown_chunk_size = 1024
markdown_overlap = 256
code_chunk_size = 512
code_overlap = 128

Index Management

# Create indexes
provisioning ai index create rag

# Rebuild indexes
provisioning ai index rebuild rag

# Show index status
provisioning ai index status rag

# Remove old indexes
provisioning ai index cleanup rag --older-than 30days

MCP Server Configuration

MCP Server Setup

[ai.mcp]
enabled = true
port = 3000
host = "127.0.0.1"  # Change to 0.0.0.0 for network access

# Tool registry
[ai.mcp.tools]
generate_config = true
validate_config = true
search_docs = true
troubleshoot_deployment = true
get_schema = true
check_compliance = true

# Rate limiting for tool calls
rpm_limit = 30
burst_limit = 50

# Tool request timeout
timeout_seconds = 30

MCP Client Configuration

~/.claude/claude_desktop_config.json:
{
  "mcpServers": {
    "provisioning": {
      "command": "provisioning-mcp-server",
      "args": ["--config", "/etc/provisioning/ai.toml"],
      "env": {
        "PROVISIONING_API_KEY": "sk-ant-...",
        "RUST_LOG": "info"
      }
    }
  }
}

Logging and Observability

Logging Configuration

[ai.logging]
level = "info"  # or "debug", "warn", "error"
format = "json"  # or "text"
output = "stdout"  # or "file"

# Log file
[ai.logging.file]
path = "/var/log/provisioning/ai.log"
max_size_mb = 100
max_backups = 10
retention_days = 30

# Log filters
[ai.logging.filters]
log_requests = true
log_responses = false  # Don't log full responses (verbose)
log_token_usage = true
log_costs = true

Metrics and Monitoring

# View AI service metrics
provisioning admin metrics show ai

# Prometheus metrics endpoint
curl [http://localhost:8083/metrics](http://localhost:8083/metrics)

# Key metrics:
# - ai_requests_total: Total requests by provider/model
# - ai_request_duration_seconds: Request latency
# - ai_token_usage_total: Token consumption by provider
# - ai_cost_total: Cumulative cost by provider
# - ai_cache_hits: Cache hit rate
# - ai_errors_total: Errors by type

Health Checks

Configuration Validation

# Validate configuration syntax
provisioning config validate ai

# Test provider connectivity
provisioning ai test provider anthropic

# Test RAG system
provisioning ai test rag

# Test MCP server
provisioning ai test mcp

# Full health check
provisioning ai health-check

Environment Variables

Common Settings

# Provider configuration
export PROVISIONING_AI_PROVIDER="anthropic"
export PROVISIONING_AI_MODEL="claude-sonnet-4"
export PROVISIONING_AI_API_KEY="sk-ant-..."

# Feature flags
export PROVISIONING_AI_ENABLED="true"
export PROVISIONING_AI_CACHE_ENABLED="true"
export PROVISIONING_AI_RAG_ENABLED="true"

# Cost control
export PROVISIONING_AI_DAILY_LIMIT_USD="100"
export PROVISIONING_AI_RPM_LIMIT="60"

# Security
export PROVISIONING_AI_SANITIZE_PII="true"
export PROVISIONING_AI_LOCAL_ONLY="false"

# Logging
export RUST_LOG="provisioning::ai=info"

Troubleshooting Configuration

Common Issues

Issue: API key not recognized

# Check environment variable is set
echo $PROVISIONING_AI_API_KEY

# Test connectivity
provisioning ai test provider anthropic

# Verify key format (should start with sk-ant- or sk-)
| provisioning config show ai | grep api_key |

Issue: Cache not working

# Check cache status
provisioning admin cache stats ai

# Clear cache and restart
provisioning admin cache clear ai
provisioning service restart ai-service

# Enable cache debugging
RUST_LOG=provisioning::cache=debug provisioning-ai-service

Issue: RAG search not finding results

# Rebuild RAG indexes
provisioning ai index rebuild rag

# Test search
provisioning ai query "test query"

# Check index status
provisioning ai index status rag

Upgrading Configuration

Backward Compatibility

New AI versions automatically migrate old configurations:

# Check configuration version
provisioning config version ai

# Migrate configuration to latest version
provisioning config migrate ai --auto

# Backup before migration
provisioning config backup ai

Production Deployment

Recommended Production Settings

[ai]
enabled = true
provider = "anthropic"
model = "claude-sonnet-4"
api_key = "${PROVISIONING_AI_API_KEY}"

[ai.features]
rag_search = true
config_generation = true
mcp_server = true
troubleshooting = true

[ai.cache]
enabled = true
cache_type = "redis"
ttl_seconds = 3600

[ai.limits]
rpm_limit = 60
daily_cost_limit_usd = 1000
max_tokens = 4096

[ai.security]
sanitize_pii = true
sanitize_secrets = true
encryption_enabled = true

[ai.logging]
level = "warn"  # Less verbose in production
format = "json"
output = "file"

[ai.rag.database]
url = "surreal://surrealdb-cluster:8000"

Related Documentation

• Architecture - System overview
• RAG System - Vector database setup
• MCP Integration - MCP configuration
• Security Policies - Authorization policies
• Cost Management - Budget tracking

Last Updated: 2025-01-13 Status: ✅ Production-Ready Versions Supported: v1.0+

AI Security Policies and Cedar Authorization

Status: ✅ Production-Ready (Cedar integration, policy enforcement)

Comprehensive documentation of security controls, authorization policies, and data protection mechanisms for the AI system. All AI operations are controlled through Cedar policies and include strict secret isolation.

Security Model Overview

Defense in Depth

┌─────────────────────────────────────────┐
│ User Request to AI                      │
└──────────────┬──────────────────────────┘
               ↓
┌─────────────────────────────────────────┐
│ Layer 1: Authentication                 │
│ - Verify user identity                  │
│ - Validate API token/credentials        │
└──────────────┬──────────────────────────┘
               ↓
┌─────────────────────────────────────────┐
│ Layer 2: Authorization (Cedar)          │
│ - Check if user can access AI features  │
│ - Verify workspace permissions          │
│ - Check role-based access               │
└──────────────┬──────────────────────────┘
               ↓
┌─────────────────────────────────────────┐
│ Layer 3: Data Sanitization              │
│ - Remove secrets from data              │
│ - Redact PII                            │
│ - Filter sensitive information          │
└──────────────┬──────────────────────────┘
               ↓
┌─────────────────────────────────────────┐
│ Layer 4: Request Validation             │
│ - Check request parameters              │
│ - Verify resource constraints           │
│ - Apply rate limits                     │
└──────────────┬──────────────────────────┘
               ↓
┌─────────────────────────────────────────┐
│ Layer 5: External API Call              │
│ - Only if all previous checks pass      │
│ - Encrypted TLS connection              │
│ - No secrets in request                 │
└──────────────┬──────────────────────────┘
               ↓
┌─────────────────────────────────────────┐
│ Layer 6: Audit Logging                  │
│ - Log all AI operations                 │
│ - Capture user, time, action            │
│ - Store in tamper-proof log             │
└─────────────────────────────────────────┘

Cedar Policies

Policy Engine Setup

// File: provisioning/policies/ai-policies.cedar

// Core principle: Least privilege
// All actions denied by default unless explicitly allowed

// Admin users can access all AI features
permit(
  principal == ?principal,
  action == Action::"ai_generate_config",
  resource == ?resource
)
when {
  principal.role == "admin"
};

// Developers can use AI within their workspace
permit(
  principal == ?principal,
  action in [
    Action::"ai_query",
    Action::"ai_generate_config",
    Action::"ai_troubleshoot"
  ],
  resource == ?resource
)
when {
  principal.role in ["developer", "senior_engineer"]
  && principal.workspace == resource.workspace
};

// Operators can access troubleshooting and queries
permit(
  principal == ?principal,
  action in [
    Action::"ai_query",
    Action::"ai_troubleshoot"
  ],
  resource == ?resource
)
when {
  principal.role in ["operator", "devops"]
};

// Form assistance enabled for all authenticated users
permit(
  principal == ?principal,
  action == Action::"ai_form_assistance",
  resource == ?resource
)
when {
  principal.authenticated == true
};

// Agents (when available) require explicit approval
permit(
  principal == ?principal,
  action == Action::"ai_agent_execute",
  resource == ?resource
)
when {
  principal.role == "automation_admin"
  && resource.requires_approval == true
};

// MCP tool access - restrictive by default
permit(
  principal == ?principal,
  action == Action::"mcp_tool_call",
  resource == ?resource
)
when {
  principal.role == "admin"
|  |  | (principal.role == "developer" && resource.tool in ["generate_config", "validate_config"]) |
};

// Cost control policies
permit(
  principal == ?principal,
  action == Action::"ai_generate_config",
  resource == ?resource
)
when {
  // User must have remaining budget
  principal.ai_budget_remaining_usd > resource.estimated_cost_usd
  // Workspace must be under budget
  && resource.workspace.ai_budget_remaining_usd > resource.estimated_cost_usd
};

Policy Best Practices

1. Explicit Allow: Only allow specific actions, deny by default
2. Workspace Isolation: Users can’t access AI in other workspaces
3. Role-Based: Use consistent role definitions
4. Cost-Aware: Check budgets before operations
5. Audit Trail: Log all policy decisions

Data Sanitization

Automatic PII Removal

Before sending data to external LLMs, the system removes:

Patterns Removed:
├─ Passwords: password="...", pwd=..., etc.
├─ API Keys: api_key=..., api-key=..., etc.
├─ Tokens: token=..., bearer=..., etc.
├─ Email addresses: user@example.com (unless necessary for context)
├─ Phone numbers: +1-555-0123 patterns
├─ Credit cards: 4111-1111-1111-1111 patterns
├─ SSH keys: -----BEGIN RSA PRIVATE KEY-----...
└─ AWS/GCP/Azure: AKIA2..., AIza..., etc.

Configuration

[ai.security]
sanitize_pii = true
sanitize_secrets = true

# Custom redaction patterns
redact_patterns = [
  # Database passwords
  "(?i)db[_-]?password\\s*[:=]\\s*'?[^'\\n]+'?",
  # Generic secrets
  "(?i)secret\\s*[:=]\\s*'?[^'\\n]+'?",
  # API endpoints that shouldn't be logged
  "https?://api[.-]secret\\..+",
]

# Exceptions (patterns NOT to redact)
preserve_patterns = [
  # Preserve example.com domain for docs
  "example\\.com",
  # Preserve placeholder emails
  "user@example\\.com",
]

Example Sanitization

Before:

Error configuring database:
connection_string: postgresql://dbadmin:MySecurePassword123@prod-db.us-east-1.rds.amazonaws.com:5432/app
api_key: sk-ant-abc123def456
vault_token: hvs.CAESIyg7...

After Sanitization:

Error configuring database:
connection_string: postgresql://dbadmin:[REDACTED]@prod-db.us-east-1.rds.amazonaws.com:5432/app
api_key: [REDACTED]
vault_token: [REDACTED]

Secret Isolation

Never Access Secrets Directly

AI cannot directly access secrets. Instead:

User wants: "Configure PostgreSQL with encrypted backups"
  ↓
AI generates: Configuration schema with placeholders
  ↓
User inserts: Actual secret values (connection strings, passwords)
  ↓
System encrypts: Secrets remain encrypted at rest
  ↓
Deployment: Uses secrets from secure store (Vault, AWS Secrets Manager)

Secret Protection Rules

1. No Direct Access: AI never reads from Vault/Secrets Manager
2. Never in Logs: Secrets never logged or stored in cache
3. Sanitization: All secrets redacted before sending to LLM
4. Encryption: Secrets encrypted at rest and in transit
5. Audit Trail: All access to secrets logged
6. TTL: Temporary secrets auto-expire

Local Models Support

Air-Gapped Deployments

For environments requiring zero external API calls:

# Deploy local Ollama with provisioning support
docker run -d \
  --name provisioning-ai \
  -p 11434:11434 \
  -v ollama:/root/.ollama \
  -e OLLAMA_HOST=0.0.0.0:11434 \
  ollama/ollama

# Pull model
ollama pull mistral
ollama pull llama2-70b

# Configure provisioning to use local model
provisioning config edit ai

[ai]
provider = "local"
model = "mistral"
api_base = "[http://localhost:11434"](http://localhost:11434")

Benefits

• ✅ Zero external API calls
• ✅ Full data privacy (no LLM vendor access)
• ✅ Compliance with classified/regulated data
• ✅ No API key exposure
• ✅ Deterministic (same results each run)

Performance Trade-offs

| | Factor | Local | Cloud | | | | –––– | —–– | —–– | | | | Privacy | Excellent | Requires trust | | | | Cost | Free (hardware) | Per token | | | | Speed | 5-30s/response | 2-5s/response | | | | Quality | Good (70B models) | Excellent (Opus) | | | | Hardware | Requires GPU | None | |

HSM Integration

Hardware Security Module Support

For highly sensitive environments:

[ai.security.hsm]
enabled = true
provider = "aws-cloudhsm"  # or "thales", "yubihsm"

[ai.security.hsm.aws]
cluster_id = "cluster-123"
customer_ca_cert = "/etc/provisioning/certs/customerCA.crt"
server_cert = "/etc/provisioning/certs/server.crt"
server_key = "/etc/provisioning/certs/server.key"

Encryption

Data at Rest

[ai.security.encryption]
enabled = true
algorithm = "aes-256-gcm"
key_derivation = "argon2id"

# Key rotation
key_rotation_enabled = true
key_rotation_days = 90
rotation_alert_days = 7

# Encrypted storage
cache_encryption = true
log_encryption = true

Data in Transit

All external LLM API calls:
├─ TLS 1.3 (minimum)
├─ Certificate pinning (optional)
├─ Mutual TLS (with cloud providers)
└─ No plaintext transmission

Audit Logging

What Gets Logged

{
  "timestamp": "2025-01-13T10:30:45Z",
  "event_type": "ai_action",
  "action": "generate_config",
  "principal": {
    "user_id": "user-123",
    "role": "developer",
    "workspace": "prod"
  },
  "resource": {
    "type": "database",
    "name": "prod-postgres"
  },
  "authorization": {
    "decision": "permit",
    "policy": "ai-policies.cedar",
    "reason": "developer role in workspace"
  },
  "cost": {
    "tokens_used": 1250,
    "estimated_cost_usd": 0.037
  },
  "sanitization": {
    "items_redacted": 3,
    "patterns_matched": ["db_password", "api_key", "token"]
  },
  "status": "success"
}

Audit Trail Access

# View recent AI actions
provisioning audit log ai --tail 100

# Filter by user
provisioning audit log ai --user alice@company.com

# Filter by action
provisioning audit log ai --action generate_config

# Filter by time range
provisioning audit log ai --from "2025-01-01" --to "2025-01-13"

# Export for analysis
provisioning audit export ai --format csv --output audit.csv

# Full-text search
provisioning audit search ai "error in database configuration"

Compliance Frameworks

Built-in Compliance Checks

[ai.compliance]
frameworks = ["pci-dss", "hipaa", "sox", "gdpr"]

[ai.compliance.pci-dss]
enabled = true
# Requires encryption, audit logs, access controls

[ai.compliance.hipaa]
enabled = true
# Requires local models, encrypted storage, audit logs

[ai.compliance.gdpr]
enabled = true
# Requires data deletion, consent tracking, privacy by design

Compliance Reports

# Generate compliance report
provisioning audit compliance-report \
  --framework pci-dss \
  --period month \
  --output report.pdf

# Verify compliance
provisioning audit verify-compliance \
  --framework hipaa \
  --verbose

Security Best Practices

For Administrators

1. Rotate API Keys: Every 90 days minimum
2. Monitor Budget: Set up alerts at 80% and 90%
3. Review Policies: Quarterly policy audit
4. Audit Logs: Weekly review of AI operations
5. Update Models: Use latest stable models
6. Test Recovery: Monthly rollback drills

For Developers

1. Use Workspace Isolation: Never share workspace access
2. Don’t Log Secrets: Use sanitization, never bypass it
3. Validate Outputs: Always review AI-generated configs
4. Report Issues: Security issues to security-ai@company.com
5. Stay Updated: Follow security bulletins

For Operators

1. Monitor Costs: Alert if exceeding 110% of budget
2. Watch Errors: Unusual error patterns may indicate attacks
3. Check Audit Logs: Unauthorized access attempts
4. Test Policies: Periodically verify Cedar policies work
5. Backup Configs: Secure backup of policy files

Incident Response

Compromised API Key

# 1. Immediately revoke key
provisioning admin revoke-key ai-api-key-123

# 2. Rotate key
provisioning admin rotate-key ai \
  --notify ops-team@company.com

# 3. Audit usage since compromise
provisioning audit log ai \
  --since "2025-01-13T09:00:00Z" \
  --api-key-id ai-api-key-123

# 4. Review any generated configs from this period
# Configs generated while key was compromised may need review

Unauthorized Access

# Review Cedar policy logs
provisioning audit log ai \
  --decision deny \
  --last-hour

# Check for pattern
provisioning audit search ai "authorization.*deny" \
  --trend-analysis

# Update policies if needed
provisioning policy update ai-policies.cedar

Security Checklist

Pre-Production

• ✅ Cedar policies reviewed and tested
• ✅ API keys rotated and secured
• ✅ Data sanitization tested with real secrets
• ✅ Encryption enabled for cache
• ✅ Audit logging configured
• ✅ Cost limits set appropriately
• ✅ Local-only mode tested (if needed)
• ✅ HSM configured (if required)

Ongoing

• ✅ Monthly policy review
• ✅ Weekly audit log review
• ✅ Quarterly key rotation
• ✅ Annual compliance assessment
• ✅ Continuous budget monitoring
• ✅ Error pattern analysis

Related Documentation

• Architecture - System overview
• Configuration - Security settings
• Cost Management - Budget controls
• ADR-015 - Design decisions

Last Updated: 2025-01-13 Status: ✅ Production-Ready Compliance: PCI-DSS, HIPAA, SOX, GDPR Cedar Version: 3.0+

AI-Assisted Troubleshooting and Debugging

Status: ✅ Production-Ready (AI troubleshooting analysis, log parsing)

The AI troubleshooting system provides intelligent debugging assistance for infrastructure failures. The system analyzes deployment logs, identifies root causes, suggests fixes, and generates corrected configurations based on failure patterns.

Feature Overview

What It Does

Transform deployment failures into actionable insights:

Deployment Fails with Error
        ↓
AI analyzes logs:
  - Identifies failure phase (networking, database, k8s, etc.)
  - Detects root cause (resource limits, configuration, timeout)
  - Correlates with similar past failures
  - Reviews deployment configuration
        ↓
AI generates report:
  - Root cause explanation in plain English
  - Configuration issues identified
  - Suggested fixes with rationale
  - Alternative solutions
  - Links to relevant documentation
        ↓
Developer reviews and accepts:
  - Understands what went wrong
  - Knows how to fix it
  - Can implement fix with confidence

Troubleshooting Workflow

Automatic Detection and Analysis

┌──────────────────────────────────────────┐
│ Deployment Monitoring                    │
│ - Watches deployment for failures        │
│ - Captures logs in real-time             │
│ - Detects failure events                 │
└──────────────┬───────────────────────────┘
               ↓
┌──────────────────────────────────────────┐
│ Log Collection                           │
│ - Gather all relevant logs               │
│ - Include stack traces                   │
│ - Capture metrics at failure time        │
│ - Get resource usage data                │
└──────────────┬───────────────────────────┘
               ↓
┌──────────────────────────────────────────┐
│ Context Retrieval (RAG)                  │
│ - Find similar past failures             │
│ - Retrieve troubleshooting guides        │
│ - Get schema constraints                 │
│ - Find best practices                    │
└──────────────┬───────────────────────────┘
               ↓
┌──────────────────────────────────────────┐
│ AI Analysis                              │
│ - Identify failure pattern               │
│ - Determine root cause                   │
│ - Generate hypotheses                    │
│ - Score likely causes                    │
└──────────────┬───────────────────────────┘
               ↓
┌──────────────────────────────────────────┐
│ Solution Generation                      │
│ - Create fixed configuration             │
│ - Generate step-by-step fix guide        │
│ - Suggest preventative measures          │
│ - Provide alternative approaches         │
└──────────────┬───────────────────────────┘
               ↓
┌──────────────────────────────────────────┐
│ Report and Recommendations               │
│ - Explain what went wrong                │
│ - Show how to fix it                     │
│ - Provide corrected configuration        │
│ - Link to prevention strategies          │
└──────────────────────────────────────────┘

Usage Examples

Example 1: Database Connection Timeout

Failure:

Deployment: deploy-2025-01-13-001
Status: FAILED at phase database_migration
Error: connection timeout after 30s connecting to postgres://...

Run Troubleshooting:

$ provisioning ai troubleshoot deploy-2025-01-13-001

Analyzing deployment failure...

╔════════════════════════════════════════════════════════════════╗
║ Root Cause Analysis: Database Connection Timeout              ║
╠════════════════════════════════════════════════════════════════╣
║                                                                ║
║ Phase: database_migration (occurred during migration job)     ║
║ Error: Timeout after 30 seconds connecting to database        ║
║                                                                ║
║ Most Likely Causes (confidence):                              ║
║   1. Database security group blocks migration job (85%)       ║
║   2. Database instance not fully initialized yet (60%)        ║
║   3. Network connectivity issue (40%)                         ║
║                                                                ║
║ Analysis:                                                     ║
║   - Database was created only 2 seconds before connection    ║
║   - Migration job started immediately (no wait time)         ║
║   - Security group: allows 5432 only from default SG         ║
║   - Migration pod uses different security group              ║
║                                                                ║
╠════════════════════════════════════════════════════════════════╣
║ Recommended Fix                                                ║
╠════════════════════════════════════════════════════════════════╣
║                                                                ║
║ Issue: Migration security group not in database's inbound    ║
║                                                                ║
║ Solution: Add migration pod security group to DB inbound     ║
║                                                                ║
║   database.security_group.ingress = [                         ║
║     {                                                          ║
║       from_port = 5432,                                       ║
║       to_port = 5432,                                         ║
║       source_security_group = "migration-pods-sg"             ║
║     }                                                          ║
║   ]                                                            ║
║                                                                ║
║ Alternative: Add 30-second wait after database creation      ║
║                                                                ║
║   deployment.phases.database.post_actions = [                 ║
║     {action = "wait_for_database", timeout_seconds = 30}     ║
║   ]                                                            ║
║                                                                ║
╠════════════════════════════════════════════════════════════════╣
║ Prevention                                                     ║
╠════════════════════════════════════════════════════════════════╣
║                                                                ║
║ To prevent this in future deployments:                        ║
║                                                                ║
║ 1. Always verify security group rules before migration       ║
║ 2. Add health check: `SELECT 1` before starting migration    ║
║ 3. Increase initial timeout: database can be slow to start   ║
║ 4. Use RDS wait condition instead of time-based wait         ║
║                                                                ║
║ See: docs/troubleshooting/database-connectivity.md            ║
║      docs/guides/database-migrations.md                       ║
║                                                                ║
╚════════════════════════════════════════════════════════════════╝

Generate corrected configuration? [yes/no]: yes

Configuration generated and saved to:
  workspaces/prod/database.ncl.fixed

Changes made:
  ✓ Added migration security group to database inbound
  ✓ Added health check before migration
  ✓ Increased connection timeout to 60s

Ready to redeploy with corrected configuration? [yes/no]: yes

Example 2: Kubernetes Deployment Error

Failure:

Deployment: deploy-2025-01-13-002
Status: FAILED at phase kubernetes_workload
Error: failed to create deployment app: Pod exceeded capacity

Troubleshooting:

$ provisioning ai troubleshoot deploy-2025-01-13-002 --detailed

╔════════════════════════════════════════════════════════════════╗
║ Root Cause: Pod Exceeded Node Capacity                        ║
╠════════════════════════════════════════════════════════════════╣
║                                                                ║
║ Failure Analysis:                                             ║
║                                                                ║
║ Error: Pod requests 4CPU/8GB, but largest node has 2CPU/4GB  ║
║ Cluster: 3 nodes, each t3.medium (2CPU/4GB)                  ║
║ Pod requirements:                                             ║
║   - CPU: 4 (requested) + 2 (reserved system) = 6 needed      ║
║   - Memory: 8Gi (requested) + 1Gi (system) = 9Gi needed      ║
║                                                                ║
║ Why this happened:                                            ║
║   Pod spec updated to 4CPU/8GB but node group wasn't        ║
║   Node group still has t3.medium (too small)                 ║
║   No autoscaling configured (won't scale up automatically)   ║
║                                                                ║
║ Solution Options:                                             ║
║   1. Reduce pod resource requests to 2CPU/4GB (simpler)      ║
║   2. Scale up node group to t3.large (2x cost, safer)        ║
║   3. Use both: t3.large nodes + reduce pod requests          ║
║                                                                ║
╠════════════════════════════════════════════════════════════════╣
║ Recommended: Option 2 (Scale up nodes)                        ║
╠════════════════════════════════════════════════════════════════╣
║                                                                ║
║ Reason: Pod requests are reasonable for production app       ║
║         Better to scale infrastructure than reduce resources  ║
║                                                                ║
║ Changes needed:                                               ║
║                                                                ║
║   kubernetes.node_group = {                                   ║
║     instance_type = "t3.large"  # was t3.medium              ║
║     min_size = 3                                              ║
║     max_size = 10                                             ║
║                                                                ║
║     auto_scaling = {                                          ║
║       enabled = true                                          ║
║       target_cpu_percent = 70                                 ║
║     }                                                          ║
║   }                                                            ║
║                                                                ║
║ Cost Impact:                                                  ║
║   Current: 3 × t3.medium = ~$90/month                        ║
║   Proposed: 3 × t3.large = ~$180/month                       ║
║   With autoscaling, average: ~$150/month (some scale-down)   ║
║                                                                ║
╚════════════════════════════════════════════════════════════════╝

CLI Commands

Basic Troubleshooting

# Troubleshoot recent deployment
provisioning ai troubleshoot deploy-2025-01-13-001

# Get detailed analysis
provisioning ai troubleshoot deploy-2025-01-13-001 --detailed

# Analyze with specific focus
provisioning ai troubleshoot deploy-2025-01-13-001 --focus networking

# Get alternative solutions
provisioning ai troubleshoot deploy-2025-01-13-001 --alternatives

Working with Logs

# Troubleshoot from custom logs
provisioning ai troubleshoot \
| --logs "$(journalctl -u provisioning --no-pager | tail -100)" |

# Troubleshoot from file
provisioning ai troubleshoot --log-file /var/log/deployment.log

# Troubleshoot from cloud provider
provisioning ai troubleshoot \
  --cloud-logs aws-deployment-123 \
  --region us-east-1

Generate Reports

# Generate detailed troubleshooting report
provisioning ai troubleshoot deploy-123 \
  --report \
  --output troubleshooting-report.md

# Generate with suggestions
provisioning ai troubleshoot deploy-123 \
  --report \
  --include-suggestions \
  --output report-with-fixes.md

# Generate compliance report (PCI-DSS, HIPAA)
provisioning ai troubleshoot deploy-123 \
  --report \
  --compliance pci-dss \
  --output compliance-report.pdf

Analysis Depth

Shallow Analysis (Fast)

provisioning ai troubleshoot deploy-123 --depth shallow

Analyzes:
- First error message
- Last few log lines
- Basic pattern matching
- Returns in 30-60 seconds

Deep Analysis (Thorough)

provisioning ai troubleshoot deploy-123 --depth deep

Analyzes:
- Full log context
- Correlates multiple errors
- Checks resource metrics
- Compares to past failures
- Generates alternative hypotheses
- Returns in 5-10 seconds

Integration with Monitoring

Automatic Troubleshooting

# Enable auto-troubleshoot on failures
provisioning config set ai.troubleshooting.auto_analyze true

# Deployments that fail automatically get analyzed
# Reports available in provisioning dashboard
# Alerts sent to on-call engineer with analysis

WebUI Integration

Deployment Dashboard
  ├─ deployment-123 [FAILED]
  │   └─ AI Analysis
  │       ├─ Root Cause: Database timeout
  │       ├─ Suggested Fix: ✓ View
  │       ├─ Corrected Config: ✓ Download
  │       └─ Alternative Solutions: 3 options

Learning from Failures

Pattern Recognition

The system learns common failure patterns:

Collected Patterns:
├─ Database Timeouts (25% of failures)
│  └─ Usually: Security group, connection pool, slow startup
├─ Kubernetes Pod Failures (20%)
│  └─ Usually: Insufficient resources, bad config
├─ Network Connectivity (15%)
│  └─ Usually: Security groups, routing, DNS
└─ Other (40%)
   └─ Various causes, each analyzed individually

Improvement Tracking

# See patterns in your deployments
provisioning ai analytics failures --period month

Month Summary:
  Total deployments: 50
  Failed: 5 (10% failure rate)
  
  Common causes:
  1. Security group rules (3 failures, 60%)
  2. Resource limits (1 failure, 20%)
  3. Configuration error (1 failure, 20%)
  
  Improvement opportunities:
  - Pre-check security groups before deployment
  - Add health checks for resource sizing
  - Add configuration validation

Configuration

Troubleshooting Settings

[ai.troubleshooting]
enabled = true

# Analysis depth
default_depth = "deep"  # or "shallow" for speed
max_analysis_time_seconds = 30

# Features
auto_analyze_failed_deployments = true
generate_corrected_config = true
suggest_prevention = true

# Learning
track_failure_patterns = true
learn_from_similar_failures = true
improve_suggestions_over_time = true

# Reporting
auto_send_report = false  # Email report to user
report_format = "markdown"  # or "json", "pdf"
include_alternatives = true

# Cost impact analysis
estimate_fix_cost = true
estimate_alternative_costs = true

Failure Detection

[ai.troubleshooting.detection]
# Monitor logs for these patterns
watch_patterns = [
  "error",
  "timeout",
  "failed",
  "unable to",
  "refused",
  "denied",
  "exceeded",
  "quota",
]

# Minimum log lines before analyzing
min_log_lines = 10

# Time window for log collection
log_window_seconds = 300

Best Practices

For Effective Troubleshooting

1. Keep Detailed Logs: Enable verbose logging in deployments
2. Include Context: Share full logs, not just error snippet
3. Check Suggestions: Review AI suggestions even if obvious
4. Learn Patterns: Track recurring failures and address root cause
5. Update Configs: Use corrected configs from AI, validate them

For Prevention

1. Use Health Checks: Add database/service health checks
2. Test Before Deploy: Use dry-run to catch issues early
3. Monitor Metrics: Watch CPU/memory before failures occur
4. Review Policies: Ensure security groups are correct
5. Document Changes: When updating configs, note the change

Limitations

What AI Can Troubleshoot

✅ Configuration errors ✅ Resource limit problems ✅ Networking/security group issues ✅ Database connectivity problems ✅ Deployment ordering issues ✅ Common application errors ✅ Performance problems

What Requires Human Review

⚠️ Data corruption scenarios ⚠️ Multi-failure cascades ⚠️ Unclear error messages ⚠️ Custom application code failures ⚠️ Third-party service issues ⚠️ Physical infrastructure failures

Examples and Guides

Common Issues - Quick Links

• Database Connectivity
• Kubernetes Pod Failures
• Network Configuration
• Performance Issues
• Resource Limits

Related Documentation

• Architecture - AI system overview
• RAG System - Context retrieval for troubleshooting
• Configuration - Setup guide
• Security Policies - Safe log handling
• ADR-015 - Design decisions

Last Updated: 2025-01-13 Status: ✅ Production-Ready Success Rate: 85-95% accuracy in root cause identification Supported: All deployment types (infrastructure, Kubernetes, database)

AI Cost Management and Optimization

Status: ✅ Production-Ready (cost tracking, budgets, caching benefits)

Comprehensive guide to managing LLM API costs, optimizing usage through caching and rate limiting, and tracking spending. The provisioning platform includes built-in cost controls to prevent runaway spending while maximizing value.

Cost Overview

API Provider Pricing

| | Provider | Model | Input | Output | Per MTok | | | | ––––– | —–– | —–– | –––– | ––––– | | | | Anthropic | Claude Sonnet 4 | $3 | $15 | $0.003 input / $0.015 output | | | | | Claude Opus 4 | $15 | $45 | Higher accuracy, longer context | | | | | Claude Haiku 4 | $0.80 | $4 | Fast, for simple queries | | | | OpenAI | GPT-4 Turbo | $0.01 | $0.03 | Per 1K tokens | | | | | GPT-4 | $0.03 | $0.06 | Legacy, avoid | | | | | GPT-4o | $5 | $15 | Per MTok | | | | Local | Llama 2, Mistral | Free | Free | Hardware cost only | |

Cost Examples

Scenario 1: Generate simple database configuration
  - Input: 500 tokens (description + schema)
  - Output: 200 tokens (generated config)
  - Cost: (500 × $3 + 200 × $15) / 1,000,000 = $0.0045
  - With caching (hit rate 50%): $0.0023

Scenario 2: Deep troubleshooting analysis
  - Input: 5000 tokens (logs + context)
  - Output: 2000 tokens (analysis + recommendations)
  - Cost: (5000 × $3 + 2000 × $15) / 1,000,000 = $0.045
  - With caching (hit rate 70%): $0.0135

Scenario 3: Monthly usage (typical organization)
  - ~1000 config generations @ $0.005 = $5
  - ~500 troubleshooting calls @ $0.045 = $22.50
  - ~2000 form assists @ $0.002 = $4
  - ~200 agent executions @ $0.10 = $20
  - **Total: ~$50-100/month for small org**
  - **Total: ~$500-1000/month for large org**

Cost Control Mechanisms

Request Caching

Caching is the primary cost reduction strategy, cutting costs by 50-80%:

Without Caching:
  User 1: "Generate PostgreSQL config" → API call → $0.005
  User 2: "Generate PostgreSQL config" → API call → $0.005
  Total: $0.010 (2 identical requests)

With LRU Cache:
  User 1: "Generate PostgreSQL config" → API call → $0.005
  User 2: "Generate PostgreSQL config" → Cache hit → $0.00001
  Total: $0.00501 (500x cost reduction for identical)

With Semantic Cache:
  User 1: "Generate PostgreSQL database config" → API call → $0.005
  User 2: "Create a PostgreSQL database" → Semantic hit → $0.00001
  (Slightly different wording, but same intent)
  Total: $0.00501 (near 500x reduction for similar)

Cache Configuration

[ai.cache]
enabled = true
cache_type = "redis"  # Distributed cache across instances
ttl_seconds = 3600    # 1-hour cache lifetime

# Cache size limits
max_size_mb = 500
eviction_policy = "lru"  # Least Recently Used

# Semantic caching - cache similar queries
[ai.cache.semantic]
enabled = true
similarity_threshold = 0.95  # Cache if 95%+ similar to previous query
cache_embeddings = true      # Cache embedding vectors themselves

# Cache metrics
[ai.cache.metrics]
track_hit_rate = true
track_space_usage = true
alert_on_low_hit_rate = true

Rate Limiting

Prevent usage spikes from unexpected costs:

[ai.limits]
# Per-request limits
max_tokens = 4096
max_input_tokens = 8192
max_output_tokens = 4096

# Throughput limits
rpm_limit = 60                    # 60 requests per minute
rpm_burst = 100                   # Allow burst to 100
daily_request_limit = 5000        # Max 5000 requests/day

# Cost limits
daily_cost_limit_usd = 100        # Stop at $100/day
monthly_cost_limit_usd = 2000     # Stop at $2000/month

# Budget alerts
warn_at_percent = 80              # Warn when at 80% of daily budget
stop_at_percent = 95              # Stop when at 95% of budget

Workspace-Level Budgets

[ai.workspace_budgets]
# Per-workspace cost limits
dev.daily_limit_usd = 10
staging.daily_limit_usd = 50
prod.daily_limit_usd = 100

# Can override globally for specific workspaces
teams.team-a.monthly_limit = 500
teams.team-b.monthly_limit = 300

Cost Tracking

Track Spending

# View current month spending
provisioning admin costs show ai

# Forecast monthly spend
provisioning admin costs forecast ai --days-remaining 15

# Analyze by feature
provisioning admin costs analyze ai --by feature

# Analyze by user
provisioning admin costs analyze ai --by user

# Export for billing
provisioning admin costs export ai --format csv --output costs.csv

Cost Breakdown

Month: January 2025

Total Spending: $285.42

By Feature:
  Config Generation:    $150.00 (52%) [300 requests × avg $0.50]
  Troubleshooting:      $95.00  (33%) [80 requests × avg $1.19]
  Form Assistance:      $30.00  (11%) [5000 requests × avg $0.006]
  Agents:               $10.42  (4%)  [20 runs × avg $0.52]

By Provider:
  Anthropic (Claude):   $200.00 (70%)
  OpenAI (GPT-4):       $85.42  (30%)
  Local:                $0      (0%)

By User:
  alice@company.com:    $50.00  (18%)
  bob@company.com:      $45.00  (16%)
  ...
  other (20 users):     $190.42 (67%)

By Workspace:
  production:           $150.00 (53%)
  staging:              $85.00  (30%)
  development:          $50.42  (18%)

Cache Performance:
  Requests: 50,000
  Cache hits: 35,000 (70%)
  Cache misses: 15,000 (30%)
  Cost savings from cache: ~$175 (38% reduction)

Optimization Strategies

Strategy 1: Increase Cache Hit Rate

# Longer TTL = more cache hits
[ai.cache]
ttl_seconds = 7200  # 2 hours instead of 1 hour

# Semantic caching helps with slight variations
[ai.cache.semantic]
enabled = true
similarity_threshold = 0.90  # Lower threshold = more hits

# Result: Increase hit rate from 65% → 80%
# Cost reduction: 15% → 23%

Strategy 2: Use Local Models

[ai]
provider = "local"
model = "mistral-7b"  # Free, runs on GPU

# Cost: Hardware ($5-20/month) instead of API calls
# Savings: 50-100 config generations/month × $0.005 = $0.25-0.50
# Hardware amortized cost: <$0.50/month on existing GPU

# Tradeoff: Slightly lower quality, 2x slower

Strategy 3: Use Haiku for Simple Tasks

Task Complexity vs Model:

Simple (form assist): Claude Haiku 4 ($0.80/$4)
Medium (config gen): Claude Sonnet 4 ($3/$15)
Complex (agents): Claude Opus 4 ($15/$45)

Example optimization:
  Before: All tasks use Sonnet 4
  - 5000 form assists/month: 5000 × $0.006 = $30
  
  After: Route by complexity
  - 5000 form assists → Haiku: 5000 × $0.001 = $5 (83% savings)
  - 200 config gen → Sonnet: 200 × $0.005 = $1
  - 10 agent runs → Opus: 10 × $0.10 = $1

Strategy 4: Batch Operations

# Instead of individual requests, batch similar operations:

# Before: 100 configs, 100 separate API calls
provisioning ai generate "PostgreSQL config" --output db1.ncl
provisioning ai generate "PostgreSQL config" --output db2.ncl
# ... 100 calls = $0.50

# After: Batch similar requests
provisioning ai batch --input configs-list.yaml
# Groups similar requests, reuses cache
# ... 3-5 API calls = $0.02 (90% savings)

Strategy 5: Smart Feature Enablement

[ai.features]
# Enable high-ROI features
config_generation = true    # High value, moderate cost
troubleshooting = true      # High value, higher cost
rag_search = true           # Low cost, high value

# Disable low-ROI features if cost-constrained
form_assistance = false     # Low value, non-zero cost (if budget tight)
agents = false              # Complex, requires multiple calls

Budget Management Workflow

1. Set Budget

# Set monthly budget
provisioning config set ai.budget.monthly_limit_usd 500

# Set daily limit
provisioning config set ai.limits.daily_cost_limit_usd 50

# Set workspace limits
provisioning config set ai.workspace_budgets.prod.monthly_limit 300
provisioning config set ai.workspace_budgets.dev.monthly_limit 100

2. Monitor Spending

# Daily check
provisioning admin costs show ai

# Weekly analysis
provisioning admin costs analyze ai --period week

# Monthly review
provisioning admin costs analyze ai --period month

3. Adjust If Needed

# If overspending:
# - Increase cache TTL
# - Enable local models for simple tasks
# - Reduce form assistance (high volume, low cost but adds up)
# - Route complex tasks to Haiku instead of Opus

# If underspending:
# - Enable new features (agents, form assistance)
# - Increase rate limits
# - Lower cache hit requirements (broader semantic matching)

4. Forecast and Plan

# Current monthly run rate
provisioning admin costs forecast ai

# If trending over budget, recommend actions:
# - Reduce daily limit
# - Switch to local model for 50% of tasks
# - Increase batch processing

# If trending under budget:
# - Enable agents for automation workflows
# - Enable form assistance across all workspaces

Cost Allocation

Chargeback Models

Per-Workspace Model:

Development workspace: $50/month
Staging workspace:     $100/month
Production workspace:  $300/month
------
Total:                 $450/month

Per-User Model:

Each user charged based on their usage
Encourages efficiency
Difficult to track/allocate

Shared Pool Model:

All teams share $1000/month budget
Budget splits by consumption rate
Encourages optimization
Most flexible

Cost Reporting

Generate Reports

# Monthly cost report
provisioning admin costs report ai \
  --format pdf \
  --period month \
  --output cost-report-2025-01.pdf

# Detailed analysis for finance
provisioning admin costs report ai \
  --format xlsx \
  --include-forecasts \
  --include-optimization-suggestions

# Executive summary
provisioning admin costs report ai \
  --format markdown \
  --summary-only

Cost-Benefit Analysis

ROI Examples

Scenario 1: Developer Time Savings
  Problem: Manual config creation takes 2 hours
  Solution: AI config generation, 10 minutes (12x faster)
  Time saved: 1.83 hours/config
  Hourly rate: $100
  Value: $183/config
  
  AI cost: $0.005/config
  ROI: 36,600x (far exceeds cost)

Scenario 2: Troubleshooting Efficiency
  Problem: Manual debugging takes 4 hours
  Solution: AI troubleshooting analysis, 2 minutes
  Time saved: 3.97 hours
  Value: $397/incident
  
  AI cost: $0.045/incident
  ROI: 8,822x

Scenario 3: Reduction in Failed Deployments
  Before: 5% of 1000 deployments fail (50 failures)
  Failure cost: $500 each (lost time, data cleanup)
  Total: $25,000/month
  
  After: With AI analysis, 2% fail (20 failures)
  Total: $10,000/month
  Savings: $15,000/month
  
  AI cost: $200/month
  Net savings: $14,800/month
  ROI: 74:1

Advanced Cost Optimization

Hybrid Strategy (Recommended)

✓ Local models for:
  - Form assistance (high volume, low complexity)
  - Simple validation checks
  - Document retrieval (RAG)
  Cost: Hardware only (~$500 setup)

✓ Cloud API for:
  - Complex generation (requires latest model capability)
  - Troubleshooting (needs high accuracy)
  - Agents (complex reasoning)
  Cost: $50-200/month per organization

Result:
  - 70% of requests → Local (free after hardware amortization)
  - 30% of requests → Cloud ($50/month)
  - 80% overall cost reduction vs cloud-only

Monitoring and Alerts

Cost Anomaly Detection

# Enable anomaly detection
provisioning config set ai.monitoring.anomaly_detection true

# Set thresholds
provisioning config set ai.monitoring.cost_spike_percent 150
# Alert if daily cost is 150% of average

# System alerts:
# - Daily cost exceeded by 10x normal
# - New expensive operation (agent run)
# - Cache hit rate dropped below 40%
# - Rate limit nearly exhausted

Alert Configuration

[ai.monitoring.alerts]
enabled = true
spike_threshold_percent = 150
check_interval_minutes = 5

[ai.monitoring.alerts.channels]
email = "ops@company.com"
slack = "[https://hooks.slack.com/..."](https://hooks.slack.com/...")
pagerduty = "integration-key"

# Alert thresholds
[ai.monitoring.alerts.thresholds]
daily_budget_warning_percent = 80
daily_budget_critical_percent = 95
monthly_budget_warning_percent = 70

Related Documentation

• Architecture - AI system overview
• Configuration - Cost control settings
• Security Policies - Cost-aware policies
• RAG System - Caching details
• ADR-015 - Design decisions

Last Updated: 2025-01-13 Status: ✅ Production-Ready Average Savings: 50-80% through caching Typical Cost: $50-500/month per organization ROI: 100:1 to 10,000:1 depending on use case

Natural Language Configuration Generation

Status: 🔴 Planned (Q2 2025 target)

Natural Language Configuration (NLC) is a planned feature that enables users to describe infrastructure requirements in plain English and have the system automatically generate validated Nickel configurations. This feature combines natural language understanding with schema-aware generation and validation.

Feature Overview

What It Does

Transform infrastructure descriptions into production-ready Nickel configurations:

User Input:
  "Create a production PostgreSQL cluster with 100GB storage,
   daily backups, encryption enabled, and cross-region replication
   to us-west-2"

System Output:
  provisioning/schemas/database.ncl (validated, production-ready)

Primary Use Cases

1. Rapid Prototyping: From description to working config in seconds
2. Infrastructure Documentation: Describe infrastructure as code
3. Configuration Templates: Generate reusable patterns
4. Non-Expert Operations: Enable junior developers to provision infrastructure
5. Configuration Migration: Describe existing infrastructure to generate Nickel

Architecture

Generation Pipeline

Input Description (Natural Language)
        ↓
┌─────────────────────────────────────┐
│ Understanding & Analysis             │
│ - Intent extraction                  │
│ - Entity recognition                 │
│ - Constraint identification          │
│ - Best practice inference            │
└─────────────────────┬───────────────┘
                      ↓
┌─────────────────────────────────────┐
│ RAG Context Retrieval                │
│ - Find similar configs               │
│ - Retrieve best practices            │
│ - Get schema examples                │
│ - Identify constraints               │
└─────────────────────┬───────────────┘
                      ↓
┌─────────────────────────────────────┐
│ Schema-Aware Generation              │
│ - Map entities to schema fields      │
│ - Apply type constraints             │
│ - Include required fields            │
│ - Generate valid Nickel              │
└─────────────────────┬───────────────┘
                      ↓
┌─────────────────────────────────────┐
│ Validation & Refinement              │
│ - Type checking                      │
│ - Schema validation                  │
│ - Policy compliance                  │
│ - Security checks                    │
└─────────────────────┬───────────────┘
                      ↓
┌─────────────────────────────────────┐
│ Output & Explanation                 │
│ - Generated Nickel config            │
│ - Decision rationale                 │
│ - Alternative suggestions            │
│ - Warnings if any                    │
└─────────────────────────────────────┘

Planned Implementation Details

1. Intent Extraction

Extract structured intent from natural language:

Input: "Create a production PostgreSQL cluster with encryption and backups"

Extracted Intent:
{
  resource_type: "database",
  engine: "postgresql",
  environment: "production",
  requirements: [
    {constraint: "encryption", type: "boolean", value: true},
    {constraint: "backups", type: "enabled", frequency: "daily"},
  ],
  modifiers: ["production"],
}

2. Entity Mapping

Map natural language entities to schema fields:

Description Terms → Schema Fields:
  "100GB storage" → database.instance.allocated_storage_gb = 100
  "daily backups" → backup.enabled = true, backup.frequency = "daily"
  "encryption" → security.encryption_enabled = true
  "cross-region" → backup.copy_to_region = "us-west-2"
  "PostgreSQL 15" → database.engine_version = "15.0"

3. Prompt Engineering

Sophisticated prompting for schema-aware generation:

System Prompt:
You are generating Nickel infrastructure configurations.
Generate ONLY valid Nickel syntax.
Follow these rules:
- Use record syntax: `field = value`
- Type annotations must be valid
- All required fields must be present
- Apply best practices for [ENVIRONMENT]

Schema Context:
[Database schema from provisioning/schemas/database.ncl]

Examples:
[3 relevant examples from RAG]

User Request:
[User natural language description]

Generate the complete Nickel configuration.
Start with: let { database = {

4. Iterative Refinement

Handle generation errors through iteration:

Attempt 1: Generate initial config
  ↓ Validate
  ✗ Error: field `version` type mismatch (string vs number)
  ↓ Re-prompt with error
Attempt 2: Fix with context from error
  ↓ Validate
  ✓ Success: Config is valid

Command Interface

CLI Usage

# Simple generation
provisioning ai generate "PostgreSQL database for production"

# With schema specification
provisioning ai generate \
  --schema database \
  "Create PostgreSQL 15 with encryption and daily backups"

# Interactive generation (refine output)
provisioning ai generate --interactive \
  "Kubernetes cluster on AWS"

# Generate and validate
provisioning ai generate \
  --validate \
  "Production Redis cluster with sentinel"

# Generate and save directly
provisioning ai generate \
  --schema database \
  --output workspaces/prod/database.ncl \
  "PostgreSQL production setup"

# Batch generation from file
provisioning ai generate --batch descriptions.yaml

Interactive Refinement

$ provisioning ai generate --interactive
> Describe infrastructure: Create production PostgreSQL cluster

Generated configuration shown.

> Refine: Add cross-region backup to us-west-2
Configuration updated.

> Refine: Use larger instance class for performance
Configuration updated.

> Accept? [y/n]: y
Configuration saved to: workspaces/prod/database.ncl

Example Generations

Example 1: Simple Database

Input:

"PostgreSQL database with 50GB storage and encryption"

Output:

let {
  database = {
    engine = "postgresql",
    version = "15.0",
    
    instance = {
      instance_class = "db.t3.medium",
      allocated_storage_gb = 50,
      iops = 1000,
    },
    
    security = {
      encryption_enabled = true,
      tls_enabled = true,
      tls_version = "1.3",
    },
    
    backup = {
      enabled = true,
      retention_days = 7,
    },
  }
}

Rationale:

• PostgreSQL 15 is current stable version
• db.t3.medium suitable for 50GB with general workload
• Encryption enabled per requirement
• Automatic backups with 7-day retention (default)

Example 2: Complex Kubernetes Setup

Input:

"Production Kubernetes cluster in AWS with 3 availability zones,
auto-scaling from 3 to 10 nodes, managed PostgreSQL, and monitoring"

Output:

let {
  kubernetes = {
    version = "1.28.0",
    
    cluster = {
      name = "prod-cluster",
      region = "us-east-1",
      availability_zones = ["us-east-1a", "us-east-1b", "us-east-1c"],
    },
    
    node_group = {
      min_size = 3,
      max_size = 10,
      desired_size = 3,
      instance_type = "t3.large",
      
      auto_scaling = {
        enabled = true,
        target_cpu = 70,
        scale_down_delay = 300,
      },
    },
    
    managed_services = {
      postgres = {
        enabled = true,
        engine = "postgresql",
        version = "15.0",
        storage_gb = 100,
      },
    },
    
    monitoring = {
      prometheus = {enabled = true},
      grafana = {enabled = true},
      cloudwatch_integration = true,
    },
    
    networking = {
      vpc_cidr = "10.0.0.0/16",
      enable_nat_gateway = true,
      enable_dns_hostnames = true,
    },
  }
}

Rationale:

• 3 AZs for high availability
• t3.large balances cost and performance for general workload
• Auto-scaling target 70% CPU (best practice)
• Managed PostgreSQL reduces operational overhead
• Full observability with Prometheus + Grafana

Configuration and Constraints

Configurable Generation Parameters

# In provisioning/config/ai.toml
[ai.generation]
# Which schema to use by default
default_schema = "database"

# Whether to require explicit environment specification
require_environment = false

# Optimization targets
optimization_target = "balanced"  # or "cost", "performance"

# Best practices to always apply
best_practices = [
  "encryption",
  "high_availability",
  "monitoring",
  "backup",
]

# Constraints that limit generation
[ai.generation.constraints]
min_storage_gb = 10
max_instances = 100
allowed_engines = ["postgresql", "mysql", "mongodb"]

# Validation before accepting generated config
[ai.generation.validation]
strict_mode = true
require_security_review = false
require_compliance_check = true

Safety Guardrails

1. Required Fields: All schema required fields must be present
2. Type Validation: Generated values must match schema types
3. Security Checks: Encryption/backups enabled for production
4. Cost Estimation: Warn if projected cost exceeds threshold
5. Resource Limits: Enforce organizational constraints
6. Policy Compliance: Check against Cedar policies

User Workflow

Typical Usage Session

# 1. Describe infrastructure need
$ provisioning ai generate "I need a database for my web app"

# System generates basic config, suggests refinements
# Generated config shown with explanations

# 2. Refine if needed
$ provisioning ai generate --interactive

# 3. Review and validate
$ provisioning ai validate workspaces/dev/database.ncl

# 4. Deploy
$ provisioning workspace apply workspaces/dev

# 5. Monitor
$ provisioning workspace logs database

Integration with Other Systems

RAG Integration

NLC uses RAG to find similar configurations:

User: "Create Kubernetes cluster"
  ↓
RAG searches for:
  - Existing Kubernetes configs in workspaces
  - Kubernetes documentation and examples
  - Best practices from provisioning/docs/guides/kubernetes.md
  ↓
Context fed to LLM for generation

Form Assistance

NLC and form assistance share components:

• Intent extraction for pre-filling forms
• Constraint validation for form field values
• Explanation generation for validation errors

CLI Integration

# Generate then preview
| provisioning ai generate "PostgreSQL prod" | \ |
  provisioning config preview

# Generate and apply
provisioning ai generate \
  --apply \
  --environment prod \
  "PostgreSQL cluster"

Testing and Validation

Test Cases (Planned)

1. Simple Descriptions: Single resource, few requirements

   • “PostgreSQL database”
   • “Redis cache”

2. Complex Descriptions: Multiple resources, constraints

   • “Kubernetes with managed database and monitoring”
   • “Multi-region deployment with failover”

3. Edge Cases:

   • Conflicting requirements
   • Ambiguous specifications
   • Deprecated technologies

4. Refinement Cycles:

   • Interactive generation with multiple refines
   • Error recovery and re-prompting
   • User feedback incorporation

Success Criteria (Q2 2025)

• ✅ Generates valid Nickel for 90% of user descriptions
• ✅ Generated configs pass all schema validation
• ✅ Supports top 10 infrastructure patterns
• ✅ Interactive refinement works smoothly
• ✅ Error messages explain issues clearly
• ✅ User testing with non-experts succeeds
• ✅ Documentation complete with examples
• ✅ Integration with form assistance operational

Related Documentation

• Architecture - AI system overview
• AI-Assisted Forms - Related form feature
• RAG System - Context retrieval
• Configuration - Setup guide
• ADR-015 - Design decisions

Status: 🔴 Planned Target Release: Q2 2025 Last Updated: 2025-01-13 Architecture: Complete Implementation: In Design Phase

Configuration Generation (typdialog-prov-gen)

Status: 🔴 Planned for Q2 2025

Overview

The Configuration Generator (typdialog-prov-gen) will provide template-based Nickel configuration generation with AI-powered customization.

Planned Features

Template Selection

• Library of production-ready infrastructure templates
• AI recommends templates based on requirements
• Preview before generation

Customization via Natural Language

provisioning ai config-gen \
  --template "kubernetes-cluster" \
  --customize "Add Prometheus monitoring, increase replicas to 5, use us-east-1"

Multi-Provider Support

• AWS, Hetzner, UpCloud, local infrastructure
• Automatic provider-specific optimizations
• Cost estimation across providers

Validation and Testing

• Type-checking via Nickel before deployment
• Dry-run execution for safety
• Test data fixtures for verification

Architecture

Template Library
      ↓
Template Selection (AI + User)
      ↓
Customization Layer (NL → Nickel)
      ↓
Validation (Type + Runtime)
      ↓
Generated Configuration

Integration Points

• typdialog web UI for template browsing
• CLI for batch generation
• AI service for customization suggestions
• Nickel for type-safe validation

Related Documentation

• Natural Language Configuration - NL to config generation
• Architecture - AI system overview
• Configuration Guide - Setup instructions

Status: 🔴 Planned Expected Release: Q2 2025 Priority: High (enables non-technical users to generate configs)

AI-Assisted Forms (typdialog-ai)

Status: 🔴 Planned (Q2 2025 target)

AI-Assisted Forms is a planned feature that integrates intelligent suggestions, context-aware assistance, and natural language understanding into the typdialog web UI. This enables users to configure infrastructure through interactive forms with real-time AI guidance.

Feature Overview

What It Does

Enhance configuration forms with AI-powered assistance:

User typing in form field: "storage"
  ↓
AI analyzes context:
  - Current form (database configuration)
  - Field type (storage capacity)
  - Similar past configurations
  - Best practices for this workload
  ↓
Suggestions appear:
  ✓ "100 GB (standard production size)"
  ✓ "50 GB (development environment)"
  ✓ "500 GB (large-scale analytics)"

Primary Use Cases

1. Guided Configuration: Step-by-step assistance filling complex forms
2. Error Explanation: AI explains validation failures in plain English
3. Smart Autocomplete: Suggestions based on context, not just keywords
4. Learning: New users learn patterns from AI explanations
5. Efficiency: Experienced users get quick suggestions

Architecture

User Interface Integration

┌────────────────────────────────────────┐
│ Typdialog Web UI (React/TypeScript)    │
│                                        │
│ ┌──────────────────────────────────┐  │
│ │ Form Fields                      │  │
│ │                                  │  │
│ │ Database Engine: [postgresql  ▼] │  │
│ │ Storage (GB):    [100 GB    ↓ ?] │  │
│ │                   AI suggestions  │  │
│ │ Encryption:      [✓ enabled  ]   │  │
│ │                   "Required for  │  │
│ │                    production"   │  │
│ │                                  │  │
│ │ [← Back] [Next →]                │  │
│ └──────────────────────────────────┘  │
│                  ↓                     │
│         AI Assistance Panel            │
│      (suggestions & explanations)      │
└────────────────────────────────────────┘
        ↓                    ↑
   User Input           AI Service
                      (port 8083)

Suggestion Pipeline

User Event (typing, focusing field, validation error)
        ↓
┌─────────────────────────────────────┐
│ Context Extraction                   │
│ - Current field and value            │
│ - Form schema and constraints        │
│ - Other filled fields                │
│ - User role and workspace            │
└─────────────────────┬───────────────┘
                      ↓
┌─────────────────────────────────────┐
│ RAG Retrieval                        │
│ - Find similar configs               │
│ - Get examples for field type        │
│ - Retrieve relevant documentation    │
│ - Find validation rules              │
└─────────────────────┬───────────────┘
                      ↓
┌─────────────────────────────────────┐
│ Suggestion Generation                │
│ - AI generates suggestions           │
│ - Rank by relevance                  │
│ - Format for display                 │
│ - Generate explanation               │
└─────────────────────┬───────────────┘
                      ↓
┌─────────────────────────────────────┐
│ Response Formatting                  │
│ - Debounce (don't update too fast)   │
│ - Cache identical results            │
│ - Stream if long response            │
│ - Display to user                    │
└─────────────────────────────────────┘

Planned Features

1. Smart Field Suggestions

Intelligent suggestions based on context:

Scenario: User filling database configuration form

1. Engine selection
   User types: "post" 
   Suggestion: "postgresql" (99% match)
   Explanation: "PostgreSQL is the most popular open-source relational database"

2. Storage size
   User has selected: "postgresql", "production", "web-application"
   Suggestions appear:
   • "100 GB" (standard production web app database)
   • "500 GB" (if expected growth > 1000 connections)
   • "1 TB" (high-traffic SaaS platform)
   Explanation: "For typical web applications with 1000s of concurrent users, 100 GB is recommended"

3. Backup frequency
   User has selected: "production", "critical-data"
   Suggestions appear:
   • "Daily" (standard for critical databases)
   • "Hourly" (for data warehouses with frequent updates)
   Explanation: "Critical production data requires daily or more frequent backups"

2. Validation Error Explanation

Human-readable error messages with fixes:

User enters: "storage = -100"

Current behavior:
  ✗ Error: Expected positive integer

Planned AI behavior:
  ✗ Storage must be positive (1-65535 GB)
  
  Why: Negative storage doesn't make sense.
       Storage capacity must be at least 1 GB.
  
  Fix suggestions:
  • Use 100 GB (typical production size)
  • Use 50 GB (development environment)
  • Use your required size in GB

3. Field-to-Field Context Awareness

Suggestions change based on other fields:

Scenario: Multi-step configuration form

Step 1: Select environment
User: "production"
  → Form shows constraints: (min storage 50GB, encryption required, backup required)

Step 2: Select database engine
User: "postgresql"
  → Suggestions adapted:
    - PostgreSQL 15 recommended for production
    - Point-in-time recovery available
    - Replication options highlighted

Step 3: Storage size
  → Suggestions show:
    - Minimum 50 GB for production
    - Examples from similar production configs
    - Cost estimate updates in real-time

Step 4: Encryption
  → Suggestion appears: "Recommended: AES-256"
  → Explanation: "Required for production environments"

4. Inline Documentation

Quick access to relevant docs:

Field: "Backup Retention Days"

Suggestion popup:
  ┌─────────────────────────────────┐
  │ Suggested value: 30              │
  │                                 │
  │ Why: 30 days is industry-standard│
  │ standard for compliance (PCI-DSS)│
  │                                 │
  │ Learn more:                      │
  │ → Backup best practices guide    │
  │ → Your compliance requirements   │
  │ → Cost vs retention trade-offs   │
  └─────────────────────────────────┘

5. Multi-Field Suggestions

Suggest multiple related fields together:

User selects: environment = "production"

AI suggests completing:
  ┌─────────────────────────────────┐
  │ Complete Production Setup        │
  │                                 │
  │ Based on production environment │
  │ we recommend:                    │
  │                                 │
  │ Encryption: enabled              │ ← Auto-fill
  │ Backups: daily                   │ ← Auto-fill
  │ Monitoring: enabled              │ ← Auto-fill
  │ High availability: enabled       │ ← Auto-fill
  │ Retention: 30 days              │ ← Auto-fill
  │                                 │
  │ [Accept All] [Review] [Skip]    │
  └─────────────────────────────────┘

Implementation Components

Frontend (typdialog-ai JavaScript/TypeScript)

// React component for field with AI assistance
interface AIFieldProps {
  fieldName: string;
  fieldType: string;
  currentValue: string;
  formContext: Record<string, any>;
  schema: FieldSchema;
}

function AIAssistedField({fieldName, formContext, schema}: AIFieldProps) {
  const [suggestions, setSuggestions] = useState<Suggestion[]>([]);
  const [explanation, setExplanation] = useState<string>("");
  
  // Debounced suggestion generation
  useEffect(() => {
    const timer = setTimeout(async () => {
      const suggestions = await ai.suggestFieldValue({
        field: fieldName,
        context: formContext,
        schema: schema,
      });
      setSuggestions(suggestions);
| setExplanation(suggestions[0]?.explanation |  | ""); |
    }, 300);  // Debounce 300ms
    
    return () => clearTimeout(timer);
  }, [formContext[fieldName]]);
  
  return (
    <div className="ai-field">
      <input 
        value={formContext[fieldName]}
        onChange={(e) => handleChange(e.target.value)}
      />
      
      {suggestions.length > 0 && (
        <div className="ai-suggestions">
          {suggestions.map((s) => (
            <button key={s.value} onClick={() => accept(s.value)}>
              {s.label}
            </button>
          ))}
          {explanation && (
            <p className="ai-explanation">{explanation}</p>
          )}
        </div>
      )}
    </div>
  );
}

Backend Service Integration

// In AI Service: field suggestion endpoint
async fn suggest_field_value(
    req: SuggestFieldRequest,
) -> Result<Vec<Suggestion>> {
    // Build context for the suggestion
    let context = build_field_context(&req.form_context, &req.field_name)?;
    
    // Retrieve relevant examples from RAG
    let examples = rag.search_by_field(&req.field_name, &context)?;
    
    // Generate suggestions via LLM
    let suggestions = llm.generate_suggestions(
        &req.field_name,
        &req.field_type,
        &context,
        &examples,
    ).await?;
    
    // Rank and format suggestions
    let ranked = rank_suggestions(suggestions, &context);
    
    Ok(ranked)
}

Configuration

Form Assistant Settings

# In provisioning/config/ai.toml
[ai.forms]
enabled = true

# Suggestion delivery
suggestions_enabled = true
suggestions_debounce_ms = 300
max_suggestions_per_field = 3

# Error explanations
error_explanations_enabled = true
explain_validation_errors = true
suggest_fixes = true

# Field context awareness
field_context_enabled = true
cross_field_suggestions = true

# Inline documentation
inline_docs_enabled = true
docs_link_type = "modal"  # or "sidebar", "tooltip"

# Performance
cache_suggestions = true
cache_ttl_seconds = 3600

# Learning
track_accepted_suggestions = true
track_rejected_suggestions = true

User Experience Flow

Scenario: New User Configuring PostgreSQL

1. User opens typdialog form
   - Form title: "Create Database"
   - First field: "Database Engine"
   - AI shows: "PostgreSQL recommended for relational data"

2. User types "post"
   - Autocomplete shows: "postgresql"
   - AI explains: "PostgreSQL is the most stable open-source database"

3. User selects "postgresql"
   - Form progresses
   - Next field: "Version"
   - AI suggests: "PostgreSQL 15 (latest stable)"
   - Explanation: "Version 15 is current stable, recommended for new deployments"

4. User selects version 15
   - Next field: "Environment"
   - User selects "production"
   - AI note appears: "Production environment requires encryption and backups"

5. Next field: "Storage (GB)"
   - Form shows: Minimum 50 GB (production requirement)
   - AI suggestions:
     • 100 GB (standard production)
     • 250 GB (high-traffic site)
   - User accepts: 100 GB

6. Validation error on next field
   - Old behavior: "Invalid backup_days value"
   - New behavior: 
     "Backup retention must be 1-35 days. Recommended: 30 days.
     30-day retention meets compliance requirements for production systems."

7. User completes form
   - Summary shows all AI-assisted decisions
   - Generate button creates configuration

Integration with Natural Language Generation

NLC and form assistance share the same backend:

Natural Language Generation    AI-Assisted Forms
        ↓                              ↓
    "Create a PostgreSQL db"    Select field values
        ↓                              ↓
    Intent Extraction         Context Extraction
        ↓                              ↓
    RAG Search              RAG Search (same results)
        ↓                              ↓
    LLM Generation          LLM Suggestions
        ↓                              ↓
    Config Output           Form Field Population

Success Criteria (Q2 2025)

• ✅ Suggestions appear within 300ms of user action
• ✅ 80% suggestion acceptance rate in user testing
• ✅ Error explanations clearly explain issues and fixes
• ✅ Cross-field context awareness works for 5+ database scenarios
• ✅ Form completion time reduced by 40% with AI
• ✅ User satisfaction > 8/10 in testing
• ✅ No false suggestions (all suggestions are valid)
• ✅ Offline mode works with cached suggestions

Related Documentation

• Architecture - AI system overview
• Natural Language Config - Related generation feature
• RAG System - Suggestion retrieval
• Configuration - Setup guide
• ADR-015 - Design decisions

Status: 🔴 Planned Target Release: Q2 2025 Last Updated: 2025-01-13 Component: typdialog-ai Architecture: Complete Implementation: In Design Phase

Autonomous AI Agents (typdialog-ag)

Status: 🔴 Planned (Q2 2025 target)

Autonomous AI Agents is a planned feature that enables AI agents to execute multi-step infrastructure provisioning workflows with minimal human intervention. Agents make decisions, adapt to changing conditions, and execute complex tasks while maintaining security and requiring human approval for critical operations.

Feature Overview

What It Does

Enable AI agents to manage complex provisioning workflows:

User Goal:
  "Set up a complete development environment with:
   - PostgreSQL database
   - Redis cache
   - Kubernetes cluster
   - Monitoring stack
   - Logging infrastructure"

AI Agent executes:
1. Analyzes requirements and constraints
2. Plans multi-step deployment sequence
3. Creates configurations for all components
4. Validates configurations against policies
5. Requests human approval for critical decisions
6. Executes deployment in correct order
7. Monitors for failures and adapts
8. Reports completion and recommendations

Agent Capabilities

Multi-Step Workflow Execution

Agents coordinate complex, multi-component deployments:

Goal: "Deploy production Kubernetes cluster with managed databases"

Agent Plan:
  Phase 1: Infrastructure
    ├─ Create VPC and networking
    ├─ Set up security groups
    └─ Configure IAM roles

  Phase 2: Kubernetes
    ├─ Create EKS cluster
    ├─ Configure network plugins
    ├─ Set up autoscaling
    └─ Install cluster add-ons

  Phase 3: Managed Services
    ├─ Provision RDS PostgreSQL
    ├─ Configure backups
    └─ Set up replicas

  Phase 4: Observability
    ├─ Deploy Prometheus
    ├─ Deploy Grafana
    ├─ Configure log collection
    └─ Set up alerting

  Phase 5: Validation
    ├─ Run smoke tests
    ├─ Verify connectivity
    └─ Check compliance

Adaptive Decision Making

Agents adapt to conditions and make intelligent decisions:

Scenario: Database provisioning fails due to resource quota

Standard approach (human):
1. Detect failure
2. Investigate issue
3. Decide on fix (reduce size, change region, etc.)
4. Update config
5. Retry

Agent approach:
1. Detect failure
2. Analyze error: "Quota exceeded for db.r6g.xlarge"
3. Check available options:
   - Try smaller instance: db.r6g.large (may be insufficient)
   - Try different region: different cost, latency
   - Request quota increase (requires human approval)
4. Ask human: "Quota exceeded. Suggest: use db.r6g.large instead 
   (slightly reduced performance). Approve? [yes/no/try-other]"
5. Execute based on approval
6. Continue workflow

Dependency Management

Agents understand resource dependencies:

Knowledge graph of dependencies:

  VPC ──→ Subnets ──→ EC2 Instances
   ├─────────→ Security Groups
   └────→ NAT Gateway ──→ Route Tables

  RDS ──→ DB Subnet Group ──→ VPC
   ├─────────→ Security Group
   └────→ Parameter Group

Agent ensures:
- VPC exists before creating subnets
- Subnets exist before creating EC2
- Security groups reference correct VPC
- Deployment order respects all dependencies
- Rollback order is reverse of creation

Architecture

Agent Design Pattern

┌────────────────────────────────────────────────────────┐
│ Agent Supervisor (Orchestrator)                        │
│ - Accepts user goal                                    │
│ - Plans workflow                                       │
│ - Coordinates specialist agents                        │
│ - Requests human approvals                             │
│ - Monitors overall progress                            │
└────────────────────────────────────────────────────────┘
        ↑                    ↑                    ↑
        │                    │                    │
        ↓                    ↓                    ↓
┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│ Database     │  │ Kubernetes   │  │ Monitoring   │
│ Specialist   │  │ Specialist   │  │ Specialist   │
│              │  │              │  │              │
│ Tasks:       │  │ Tasks:       │  │ Tasks:       │
│ - Create DB  │  │ - Create K8s │  │ - Deploy     │
│ - Configure  │  │ - Configure  │  │   Prometheus │
│ - Validate   │  │ - Validate   │  │ - Deploy     │
│ - Report     │  │ - Report     │  │   Grafana    │
└──────────────┘  └──────────────┘  └──────────────┘

Agent Workflow

Start: User Goal
  ↓
┌─────────────────────────────────────────┐
│ Goal Analysis & Planning                │
│ - Parse user intent                     │
│ - Identify resources needed             │
│ - Plan dependency graph                 │
│ - Generate task list                    │
└──────────────┬──────────────────────────┘
               ↓
┌─────────────────────────────────────────┐
│ Resource Generation                     │
│ - Generate configs for each resource    │
│ - Validate against schemas              │
│ - Check compliance policies             │
│ - Identify potential issues             │
└──────────────┬──────────────────────────┘
               ↓
         Human Review Point?
         ├─ No issues: Continue
         └─ Issues found: Request approval/modification
               ↓
┌─────────────────────────────────────────┐
│ Execution Plan Verification             │
│ - Check all configs are valid           │
│ - Verify dependencies are resolvable    │
│ - Estimate costs and timeline           │
│ - Identify risks                        │
└──────────────┬──────────────────────────┘
               ↓
         Execute Workflow?
         ├─ User approves: Start execution
         └─ User modifies: Return to planning
               ↓
┌─────────────────────────────────────────┐
│ Phase-by-Phase Execution                │
│ - Execute one logical phase             │
│ - Monitor for errors                    │
│ - Report progress                       │
│ - Ask for decisions if needed           │
└──────────────┬──────────────────────────┘
               ↓
         All Phases Complete?
         ├─ No: Continue to next phase
         └─ Yes: Final validation
               ↓
┌─────────────────────────────────────────┐
│ Final Validation & Reporting            │
│ - Smoke tests                           │
│ - Connectivity tests                    │
│ - Compliance verification               │
│ - Performance checks                    │
│ - Generate final report                 │
└──────────────┬──────────────────────────┘
               ↓
Success: Deployment Complete

Planned Agent Types

1. Database Specialist Agent

Responsibilities:
- Create and configure databases
- Set up replication and backups
- Configure encryption and security
- Monitor database health
- Handle database-specific issues

Examples:
- Provision PostgreSQL cluster with replication
- Set up MySQL with read replicas
- Configure MongoDB sharding
- Create backup pipelines

2. Kubernetes Specialist Agent

Responsibilities:
- Create and configure Kubernetes clusters
- Configure networking and ingress
- Set up autoscaling policies
- Deploy cluster add-ons
- Manage workload placement

Examples:
- Create EKS/GKE/AKS cluster
- Configure Istio service mesh
- Deploy Prometheus + Grafana
- Configure auto-scaling policies

3. Infrastructure Agent

Responsibilities:
- Create networking infrastructure
- Configure security and firewalls
- Set up load balancers
- Configure DNS and CDN
- Manage identity and access

Examples:
- Create VPC with subnets
- Configure security groups
- Set up application load balancer
- Configure Route53 DNS

4. Monitoring Agent

Responsibilities:
- Deploy monitoring stack
- Configure alerting
- Set up logging infrastructure
- Create dashboards
- Configure notification channels

Examples:
- Deploy Prometheus + Grafana
- Set up CloudWatch dashboards
- Configure log aggregation
- Set up PagerDuty integration

5. Compliance Agent

Responsibilities:
- Check security policies
- Verify compliance requirements
- Audit configurations
- Generate compliance reports
- Recommend security improvements

Examples:
- Check PCI-DSS compliance
- Verify encryption settings
- Audit access controls
- Generate compliance report

Usage Examples

Example 1: Development Environment Setup

$ provisioning ai agent --goal "Set up dev environment for Python web app"

Agent Plan Generated:
┌─────────────────────────────────────────┐
│ Environment: Development                │
│ Components: PostgreSQL + Redis + Monitoring
│                                         │
│ Phase 1: Database (1-2 min)            │
│   - PostgreSQL 15                       │
│   - 10 GB storage                       │
│   - Dev security settings               │
│                                         │
│ Phase 2: Cache (1 min)                 │
│   - Redis Cluster Mode disabled         │
│   - Single node                         │
│   - 2 GB memory                         │
│                                         │
│ Phase 3: Monitoring (1-2 min)          │
│   - Prometheus (metrics)                │
│   - Grafana (dashboards)                │
│   - Log aggregation                     │
│                                         │
│ Estimated time: 5-10 minutes            │
│ Estimated cost: $15/month               │
│                                         │
│ [Approve] [Modify] [Cancel]             │
└─────────────────────────────────────────┘

Agent: Approve to proceed with setup.

User: Approve

[Agent execution starts]
Creating PostgreSQL...     [████████░░] 80%
Creating Redis...          [░░░░░░░░░░] 0%
[Waiting for PostgreSQL creation...]

PostgreSQL created successfully!
Connection string: postgresql://dev:pwd@db.internal:5432/app

Creating Redis...          [████████░░] 80%
[Waiting for Redis creation...]

Redis created successfully!
Connection string: redis://cache.internal:6379

Deploying monitoring...    [████████░░] 80%
[Waiting for Grafana startup...]

All services deployed successfully!
Grafana dashboards: [http://grafana.internal:3000](http://grafana.internal:3000)

Example 2: Production Kubernetes Deployment

$ provisioning ai agent --interactive \
    --goal "Deploy production Kubernetes cluster with managed databases"

Agent Analysis:
- Cluster size: 3-10 nodes (auto-scaling)
- Databases: RDS PostgreSQL + ElastiCache Redis
- Monitoring: Full observability stack
- Security: TLS, encryption, VPC isolation

Agent suggests modifications:
  1. Enable cross-AZ deployment for HA
  2. Add backup retention: 30 days
  3. Add network policies for security
  4. Enable cluster autoscaling
  Approve all? [yes/review]

User: Review

Agent points out:
  - Network policies may affect performance
  - Cross-AZ increases costs by ~20%
  - Backup retention meets compliance

User: Approve with modifications
  - Network policies: use audit mode first
  - Keep cross-AZ
  - Keep backups

[Agent creates configs with modifications]

Configs generated:
  ✓ infrastructure/vpc.ncl
  ✓ infrastructure/kubernetes.ncl
  ✓ databases/postgres.ncl
  ✓ databases/redis.ncl
  ✓ monitoring/prometheus.ncl
  ✓ monitoring/grafana.ncl

Estimated deployment time: 15-20 minutes
Estimated cost: $2,500/month

[Start deployment?] [Review configs]

User: Review configs

[User reviews and approves]

[Agent executes deployment in phases]

Safety and Control

Human-in-the-Loop Checkpoints

Agents stop and ask humans for approval at critical points:

Automatic Approval (Agent decides):
- Create configuration
- Validate configuration
- Check dependencies
- Generate execution plan

Human Approval Required:
- First-time resource creation
- Cost changes > 10%
- Security policy changes
- Cross-region deployment
- Data deletion operations
- Major version upgrades

Decision Logging

All decisions logged for audit trail:

Agent Decision Log:
| 2025-01-13 10:00:00 | Generate database config |
| 2025-01-13 10:00:05 | Config validation: PASS |
| 2025-01-13 10:00:07 | Requesting human approval: "Create new PostgreSQL instance" |
| 2025-01-13 10:00:45 | Human approval: APPROVED |
| 2025-01-13 10:00:47 | Cost estimate: $100/month - within budget |
| 2025-01-13 10:01:00 | Creating infrastructure... |
| 2025-01-13 10:02:15 | Database created successfully |
| 2025-01-13 10:02:16 | Running health checks... |
| 2025-01-13 10:02:45 | Health check: PASSED |

Rollback Capability

Agents can rollback on failure:

Scenario: Database creation succeeds, but Kubernetes creation fails

Agent behavior:
1. Detect failure in Kubernetes phase
2. Try recovery (retry, different configuration)
3. Recovery fails
4. Ask human: "Kubernetes creation failed. Rollback database creation? [yes/no]"
5. If yes: Delete database, clean up, report failure
6. If no: Keep database, manual cleanup needed

Full rollback capability if entire workflow fails before human approval.

Configuration

Agent Settings

# In provisioning/config/ai.toml
[ai.agents]
enabled = true

# Agent decision-making
auto_approve_threshold = 0.95  # Approve if confidence > 95%
require_approval_for = [
  "first_resource_creation",
  "cost_change_above_percent",
  "security_policy_change",
  "data_deletion",
]

cost_change_threshold_percent = 10

# Execution control
max_parallel_phases = 2
phase_timeout_minutes = 30
execution_log_retention_days = 90

# Safety
dry_run_mode = false  # Always perform dry run first
require_final_approval = true
rollback_on_failure = true

# Learning
track_agent_decisions = true
track_success_rate = true
improve_from_feedback = true

Success Criteria (Q2 2025)

• ✅ Agents complete 5 standard workflows without human intervention
• ✅ Cost estimation accuracy within 5%
• ✅ Execution time matches or beats manual setup by 30%
• ✅ Success rate > 95% for tested scenarios
• ✅ Zero unapproved critical decisions
• ✅ Full decision audit trail for all operations
• ✅ Rollback capability tested and verified
• ✅ User satisfaction > 8/10 in testing
• ✅ Documentation complete with examples
• ✅ Integration with form assistance and NLC working

Related Documentation

• Architecture - AI system overview
• Natural Language Config - Config generation
• AI-Assisted Forms - Interactive forms
• Configuration - Setup guide
• ADR-015 - Design decisions

Status: 🔴 Planned Target Release: Q2 2025 Last Updated: 2025-01-13 Component: typdialog-ag Architecture: Complete Implementation: In Design Phase

System Overview

Executive Summary

Provisioning is an Infrastructure Automation Platform built with a hybrid Rust/Nushell architecture. It enables Infrastructure as Code (IaC) with multi-provider support (AWS, UpCloud, local), sophisticated workflow orchestration, and configuration-driven operations.

The system solves fundamental technical challenges through architectural innovation and hybrid language design.

High-Level Architecture

System Diagram

┌─────────────────────────────────────────────────────────────────┐
│                        User Interface Layer                     │
├─────────────────┬─────────────────┬─────────────────────────────┤
│   CLI Tools     │   REST API      │   Control Center UI         │
│   (Nushell)     │   (Rust)        │   (Web Interface)           │
└─────────────────┴─────────────────┴─────────────────────────────┘
                           │
┌─────────────────────────────────────────────────────────────────┐
│                    Orchestration Layer                          │
├─────────────────────────────────────────────────────────────────┤
│   Rust Orchestrator: Workflow Coordination & State Management   │
│   • Task Queue & Scheduling    • Batch Processing               │
│   • State Persistence         • Error Recovery & Rollback       │
│   • REST API Server          • Real-time Monitoring             │
└─────────────────────────────────────────────────────────────────┘
                           │
┌─────────────────────────────────────────────────────────────────┐
│                    Business Logic Layer                         │
├─────────────────┬─────────────────┬─────────────────────────────┤
│   Providers     │   Task Services │   Workflows                 │
│   (Nushell)     │   (Nushell)     │   (Nushell)                 │
│   • AWS         │   • Kubernetes  │   • Server Creation         │
│   • UpCloud     │   • Storage     │   • Cluster Deployment      │
│   • Local       │   • Networking  │   • Batch Operations        │
└─────────────────┴─────────────────┴─────────────────────────────┘
                           │
┌─────────────────────────────────────────────────────────────────┐
│                    Configuration Layer                          │
├─────────────────┬─────────────────┬─────────────────────────────┤
│   Nickel Schemas│   TOML Config   │   Templates                 │
│   • Type Safety │   • Hierarchy   │   • Infrastructure          │
│   • Validation  │   • Environment │   • Service Configs         │
│   • Extensible  │   • User Prefs  │   • Code Generation         │
└─────────────────┴─────────────────┴─────────────────────────────┘
                           │
┌─────────────────────────────────────────────────────────────────┐
│                      Infrastructure Layer                       │
├─────────────────┬─────────────────┬─────────────────────────────┤
│   Cloud APIs    │   Kubernetes    │   Local Systems             │
│   • AWS EC2     │   • Clusters    │   • Docker                  │
│   • UpCloud     │   • Services    │   • Containers              │
│   • Others      │   • Storage     │   • Host Services           │
└─────────────────┴─────────────────┴─────────────────────────────┘

Core Components

1. Hybrid Architecture Foundation

Coordination Layer (Rust)

Purpose: High-performance workflow orchestration and system coordination

Components:

• Orchestrator Engine: Task scheduling and execution coordination
• REST API Server: HTTP endpoints for external integration
• State Management: Persistent state tracking with checkpoint recovery
• Batch Processor: Parallel execution of complex multi-provider workflows
• File-based Queue: Lightweight, reliable task persistence
• Error Recovery: Sophisticated rollback and cleanup capabilities

Key Features:

• Solves Nushell deep call stack limitations
• Handles 1000+ concurrent operations
• Checkpoint-based recovery from any failure point
• Real-time workflow monitoring and status tracking

Business Logic Layer (Nushell)

Purpose: Domain-specific operations and configuration management

Components:

• Provider Implementations: Cloud-specific operations (AWS, UpCloud, local)
• Task Service Management: Infrastructure component lifecycle
• Configuration Processing: Nickel-based configuration validation and templating
• CLI Interface: User-facing command-line tools
• Workflow Definitions: Business process implementations

Key Features:

• 65+ domain-specific modules preserved and enhanced
• Configuration-driven operations with zero hardcoded values
• Type-safe Nickel integration for Infrastructure as Code
• Extensible provider and service architecture

2. Configuration System (v2.0.0)

Hierarchical Configuration Management

Migration Achievement: 65+ files migrated, 200+ ENV variables → 476 config accessors

Configuration Hierarchy (precedence order):

1. Runtime Parameters (command line, environment variables)
2. Environment Configuration (dev/test/prod specific)
3. Infrastructure Configuration (project-specific settings)
4. User Configuration (personal preferences)
5. System Defaults (system-wide defaults)

Configuration Files:

• config.defaults.toml - System-wide defaults
• config.user.toml - User-specific preferences
• config.{dev,test,prod}.toml - Environment-specific configurations
• Infrastructure-specific configuration files

Features:

• Variable Interpolation: {{paths.base}}, {{env.HOME}}, {{now.date}}, {{git.branch}}
• Environment Switching: PROVISIONING_ENV=prod for environment-specific configs
• Validation Framework: Comprehensive configuration validation and error reporting
• Migration Tools: Automated migration from ENV-based to config-driven architecture

3. Workflow System (v3.1.0)

Batch Workflow Engine

Batch Capabilities:

• Provider-Agnostic Workflows: Mix UpCloud, AWS, and local providers in single workflow
• Dependency Resolution: Topological sorting with soft/hard dependency support
• Parallel Execution: Configurable parallelism limits with resource management
• State Recovery: Checkpoint-based recovery with rollback capabilities
• Real-time Monitoring: Live progress tracking and health monitoring

Workflow Types:

• Server Workflows: Multi-provider server provisioning and management
• Task Service Workflows: Infrastructure component installation and configuration
• Cluster Workflows: Complete Kubernetes cluster deployment and management
• Batch Workflows: Complex multi-step operations with dependency management

Nickel Workflow Definitions:

{
  batch_workflow = {
    name = "multi_cloud_deployment",
    version = "1.0.0",
    parallel_limit = 5,
    rollback_enabled = true,

    operations = [
      {
        id = "servers",
        type = "server_batch",
        provider = "upcloud",
        dependencies = [],
      },
      {
        id = "services",
        type = "taskserv_batch",
        provider = "aws",
        dependencies = ["servers"],
      }
    ]
  }
}

4. Provider Ecosystem

Multi-Provider Architecture

Supported Providers:

• AWS: Amazon Web Services integration
• UpCloud: UpCloud provider with full feature support
• Local: Local development and testing provider

Provider Features:

• Standardized Interfaces: Consistent API across all providers
• Configuration Templates: Provider-specific configuration generation
• Resource Management: Complete lifecycle management for cloud resources
• Cost Optimization: Pricing information and cost optimization recommendations
• Regional Support: Multi-region deployment capabilities

Task Services Ecosystem

Infrastructure Components (40+ services):

• Container Orchestration: Kubernetes, container runtimes (containerd, cri-o, crun, runc, youki)
• Networking: Cilium, CoreDNS, HAProxy, service mesh integration
• Storage: Rook-Ceph, external-NFS, Mayastor, persistent volumes
• Security: Policy engines, secrets management, RBAC
• Observability: Monitoring, logging, tracing, metrics collection
• Development Tools: Gitea, databases, build systems

Service Features:

• Version Management: Real-time version checking against GitHub releases
• Configuration Generation: Automated service configuration from templates
• Dependency Management: Automatic dependency resolution and installation order
• Health Monitoring: Service health checks and status reporting

Key Architectural Decisions

1. Hybrid Language Architecture (ADR-004)

Decision: Use Rust for coordination, Nushell for business logic Rationale: Solves Nushell’s deep call stack limitations while preserving domain expertise Impact: Eliminates technical limitations while maintaining productivity and configuration advantages

2. Configuration-Driven Architecture (ADR-002)

Decision: Complete migration from ENV variables to hierarchical configuration Rationale: True Infrastructure as Code requires configuration flexibility without hardcoded fallbacks Impact: 476 configuration accessors provide complete customization without code changes

3. Domain-Driven Structure (ADR-001)

Decision: Organize by functional domains (core, platform, provisioning) Rationale: Clear boundaries enable scalable development and maintenance Impact: Enables specialized development while maintaining system coherence

4. Workspace Isolation (ADR-003)

Decision: Isolated user workspaces with hierarchical configuration Rationale: Multi-user support and customization without system impact Impact: Complete user independence with easy backup and migration

5. Registry-Based Extensions (ADR-005)

Decision: Manifest-driven extension framework with structured discovery Rationale: Enable community contributions while maintaining system stability Impact: Extensible system supporting custom providers, services, and workflows

Data Flow Architecture

Configuration Resolution Flow

1. Workspace Discovery → 2. Configuration Loading → 3. Hierarchy Merge →
4. Variable Interpolation → 5. Schema Validation → 6. Runtime Application

Workflow Execution Flow

1. Workflow Submission → 2. Dependency Analysis → 3. Task Scheduling →
4. Parallel Execution → 5. State Tracking → 6. Result Aggregation →
7. Error Handling → 8. Cleanup/Rollback

Provider Integration Flow

1. Provider Discovery → 2. Configuration Validation → 3. Authentication →
4. Resource Planning → 5. Operation Execution → 6. State Persistence →
7. Result Reporting

Technology Stack

Core Technologies

• Nushell 0.107.1: Primary shell and scripting language
• Rust: High-performance coordination and orchestration
• Nickel 1.15.0+: Configuration language for Infrastructure as Code
• TOML: Configuration file format with human readability
• JSON: Data exchange format between components

Infrastructure Technologies

• Kubernetes: Container orchestration platform
• Docker/Containerd: Container runtime environments
• SOPS 3.10.2: Secrets management and encryption
• Age 1.2.1: Encryption tool for secrets
• HTTP/REST: API communication protocols

Development Technologies

• nu_plugin_tera: Native Nushell template rendering
• K9s 0.50.6: Kubernetes management interface
• Git: Version control and configuration management

Scalability and Performance

Performance Characteristics

• Batch Processing: 1000+ concurrent operations with configurable parallelism
• Provider Operations: Sub-second response for most cloud API operations
• Configuration Loading: Millisecond-level configuration resolution
• State Persistence: File-based persistence with minimal overhead
• Memory Usage: Efficient memory management with streaming operations

Scalability Features

• Horizontal Scaling: Multiple orchestrator instances for high availability
• Resource Management: Configurable resource limits and quotas
• Caching Strategy: Multi-level caching for performance optimization
• Streaming Operations: Large dataset processing without memory limits
• Async Processing: Non-blocking operations for improved throughput

Security Architecture

Security Layers

• Workspace Isolation: User data isolated from system installation
• Configuration Security: Encrypted secrets with SOPS/Age integration
• Extension Sandboxing: Extensions run in controlled environments
• API Authentication: Secure REST API endpoints with authentication
• Audit Logging: Comprehensive audit trails for all operations

Security Features

• Secrets Management: Encrypted configuration files with rotation support
• Permission Model: Role-based access control for operations
• Code Signing: Digital signature verification for extensions
• Network Security: Secure communication with cloud providers
• Input Validation: Comprehensive input validation and sanitization

Quality Attributes

Reliability

• Error Recovery: Sophisticated error handling and rollback capabilities
• State Consistency: Transactional operations with rollback support
• Health Monitoring: Comprehensive system health checks and monitoring
• Fault Tolerance: Graceful degradation and recovery from failures

Maintainability

• Clear Architecture: Well-defined boundaries and responsibilities
• Documentation: Comprehensive architecture and development documentation
• Testing Strategy: Multi-layer testing with integration validation
• Code Quality: Consistent patterns and quality standards

Extensibility

• Plugin Framework: Registry-based extension system
• Provider API: Standardized interfaces for new providers
• Configuration Schema: Extensible configuration with validation
• Workflow Engine: Custom workflow definitions and execution

This system architecture represents a mature, production-ready platform for Infrastructure as Code with unique architectural innovations and proven scalability.

Provisioning Platform - Architecture Overview

Version: 3.5.0 Date: 2025-10-06 Status: Production Maintainers: Architecture Team

Table of Contents

1. Executive Summary
2. System Architecture
3. Component Architecture
4. Mode Architecture
5. Network Architecture
6. Data Architecture
7. Security Architecture
8. Deployment Architecture
9. Integration Architecture
10. Performance and Scalability
11. Evolution and Roadmap

Executive Summary

What is the Provisioning Platform

The Provisioning Platform is a modern, cloud-native infrastructure automation system that combines:

• the simplicity of declarative configuration (Nickel)
• the power of shell scripting (Nushell)
• high-performance coordination (Rust).

Key Characteristics

• Hybrid Architecture: Rust for coordination, Nushell for business logic, Nickel for configuration
• Mode-Based: Adapts from solo development to enterprise production
• OCI-Native: Extends leveraging industry-standard OCI distribution
• Provider-Agnostic: Supports multiple cloud providers (AWS, UpCloud) and local infrastructure
• Extension-Driven: Core functionality enhanced through modular extensions

Architecture at a Glance

┌─────────────────────────────────────────────────────────────────────┐
│                        Provisioning Platform                        │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   ┌──────────────┐   ┌─────────────┐    ┌──────────────┐            │
│   │ User Layer   │   │  Extension  │    │   Service    │            │
│   │  (CLI/UI)    │   │  Registry   │    │   Registry   │            │
│   └──────┬───────┘   └──────┬──────┘    └──────┬───────┘            │
│          │                  │                  │                    │
│   ┌──────┴──────────────────┴──────────────────┴──--────┐           │
│   │            Core Provisioning Engine                 │           │
│   │  (Config | Dependency Resolution | Workflows)       │           │
│   └──────┬──────────────────────────────────────┬───────┘           │
│          │                                      │                   │
│   ┌──────┴─────────┐                   ┌──────-─┴─────────┐         │
│   │  Orchestrator  │                   │   Business Logic │         │
│   │    (Rust)      │ ←─ Coordination → │    (Nushell)     │         │
│   └──────┬─────────┘                   └───────┬──────────┘         │
│          │                                     │                    │
│   ┌──────┴─────────────────────────────────────┴---──────┐          │
│   │                  Extension System                    │          │
│   │      (Providers | Task Services | Clusters)          │          │
│   └──────┬───────────────────────────────────────────────┘          │
│          │                                                          │
│   ┌──────┴──────────────────────────────────────────────────-─┐     │
│   │        Infrastructure (Cloud | Local | Kubernetes)        │     │
│   └───────────────────────────────────────────────────────────┘     │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Key Metrics

System Architecture

High-Level Architecture

┌────────────────────────────────────────────────────────────────────────────┐
│                         PRESENTATION LAYER                                 │
├────────────────────────────────────────────────────────────────────────────┤
│                                                                            │
│    ┌─────────────┐  ┌──────────────┐  ┌──────────────┐  ┌────────────┐     │
│    │  CLI (Nu)   │  │ Control      │  │  REST API    │  │  MCP       │     │
│    │             │  │ Center (Yew) │  │  Gateway     │  │  Server    │     │
│    └─────────────┘  └──────────────┘  └──────────────┘  └────────────┘     │
│                                                                            │
└──────────────────────────────────┬─────────────────────────────────────────┘
                                   │
┌──────────────────────────────────┴─────────────────────────────────────────┐
│                         CORE LAYER                                         │
├────────────────────────────────────────────────────────────────────────────┤
│                                                                            │
│   ┌─────────────────────────────────────────────────────────────────┐      │
│   │               Configuration Management                          │      │
│   │   (Nickel Schemas | TOML Config | Hierarchical Loading)         │      │
│   └─────────────────────────────────────────────────────────────────┘      │
│                                                                            │
│   ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐         │
│   │   Dependency     │  │   Module/Layer   │  │   Workspace      │         │
│   │   Resolution     │  │     System       │  │   Management     │         │
│   └──────────────────┘  └──────────────────┘  └──────────────────┘         │
│                                                                            │
│  ┌──────────────────────────────────────────────────────────────────┐      │
│  │                  Workflow Engine                                 │      │
│  │   (Batch Operations | Checkpoints | Rollback)                    │      │
│  └──────────────────────────────────────────────────────────────────┘      │
│                                                                            │
└──────────────────────────────────┬─────────────────────────────────────────┘
                                   │
┌──────────────────────────────────┴─────────────────────────────────────────┐
│                      ORCHESTRATION LAYER                                   │
├────────────────────────────────────────────────────────────────────────────┤
│                                                                            │
│  ┌──────────────────────────────────────────────────────────────────┐      │
│  │                Orchestrator (Rust)                               │      │
│  │   • Task Queue (File-based persistence)                          │      │
│  │   • State Management (Checkpoints)                               │      │
│  │   • Health Monitoring                                            │      │
│  │   • REST API (HTTP/WS)                                           │      │
│  └──────────────────────────────────────────────────────────────────┘      │
│                                                                            │
│  ┌──────────────────────────────────────────────────────────────────┐      │
│  │           Business Logic (Nushell)                               │      │
│  │   • Provider operations (AWS, UpCloud, Local)                    │      │
│  │   • Server lifecycle (create, delete, configure)                 │      │
│  │   • Taskserv installation (50+ services)                         │      │
│  │   • Cluster deployment                                           │      │
│  └──────────────────────────────────────────────────────────────────┘      │
│                                                                            │
└──────────────────────────────────┬─────────────────────────────────────────┘
                                   │
┌──────────────────────────────────┴─────────────────────────────────────────┐
│                      EXTENSION LAYER                                       │
├────────────────────────────────────────────────────────────────────────────┤
│                                                                            │
│   ┌────────────────┐  ┌──────────────────┐  ┌───────────────────┐          │
│   │   Providers    │  │   Task Services  │  │    Clusters       │          │
│   │   (3 types)    │  │   (50+ types)    │  │   (10+ types)     │          │
│   │                │  │                  │  │                   │          │
│   │  • AWS         │  │  • Kubernetes    │  │  • Buildkit       │          │
│   │  • UpCloud     │  │  • Containerd    │  │  • Web cluster    │          │
│   │  • Local       │  │  • Databases     │  │  • CI/CD          │          │
│   │                │  │  • Monitoring    │  │                   │          │
│   └────────────────┘  └──────────────────┘  └───────────────────┘          │
│                                                                            │
│  ┌──────────────────────────────────────────────────────────────────┐      │
│  │            Extension Distribution (OCI Registry)                 │      │
│  │   • Zot (local development)                                      │      │
│  │   • Harbor (multi-user/enterprise)                               │      │
│  └──────────────────────────────────────────────────────────────────┘      │
│                                                                            │
└──────────────────────────────────┬─────────────────────────────────────────┘
                                   │
┌──────────────────────────────────┴─────────────────────────────────────────┐
│                      INFRASTRUCTURE LAYER                                  │
├────────────────────────────────────────────────────────────────────────────┤
│                                                                            │
│   ┌────────────────┐  ┌──────────────────┐  ┌───────────────────┐          │
│   │  Cloud (AWS)   │  │ Cloud (UpCloud)  │  │  Local (Docker)   │          │
│   │                │  │                  │  │                   │          │
│   │  • EC2         │  │  • Servers       │  │  • Containers     │          │
│   │  • EKS         │  │  • LoadBalancer  │  │  • Local K8s      │          │
│   │  • RDS         │  │  • Networking    │  │  • Processes      │          │
│   └────────────────┘  └──────────────────┘  └───────────────────┘          │
│                                                                            │
└────────────────────────────────────────────────────────────────────────────┘

Multi-Repository Architecture

The system is organized into three separate repositories:

provisioning-core

Core system functionality
├── CLI interface (Nushell entry point)
├── Core libraries (lib_provisioning)
├── Base Nickel schemas
├── Configuration system
├── Workflow engine
└── Build/distribution tools

Distribution: oci://registry/provisioning-core:v3.5.0

provisioning-extensions

All provider, taskserv, cluster extensions
├── providers/
│   ├── aws/
│   ├── upcloud/
│   └── local/
├── taskservs/
│   ├── kubernetes/
│   ├── containerd/
│   ├── postgres/
│   └── (50+ more)
└── clusters/
    ├── buildkit/
    ├── web/
    └── (10+ more)

Distribution: Each extension as separate OCI artifact

• oci://registry/provisioning-extensions/kubernetes:1.28.0
• oci://registry/provisioning-extensions/aws:2.0.0

provisioning-platform

Platform services
├── orchestrator/      (Rust)
├── control-center/    (Rust/Yew)
├── mcp-server/        (Rust)
└── api-gateway/       (Rust)

Distribution: Docker images in OCI registry

• oci://registry/provisioning-platform/orchestrator:v1.2.0

Component Architecture

Core Components

1. CLI Interface (Nushell)

Location: provisioning/core/cli/provisioning

Purpose: Primary user interface for all provisioning operations

Architecture:

Main CLI (211 lines)
    ↓
Command Dispatcher (264 lines)
    ↓
Domain Handlers (7 modules)
    ├── infrastructure.nu (117 lines)
    ├── orchestration.nu (64 lines)
    ├── development.nu (72 lines)
    ├── workspace.nu (56 lines)
    ├── generation.nu (78 lines)
    ├── utilities.nu (157 lines)
    └── configuration.nu (316 lines)

Key Features:

• 80+ command shortcuts
• Bi-directional help system
• Centralized flag handling
• Domain-driven design

2. Configuration System (Nickel + TOML)

Hierarchical Loading:

1. System defaults     (config.defaults.toml)
2. User config         (~/.provisioning/config.user.toml)
3. Workspace config    (workspace/config/provisioning.yaml)
4. Environment config  (workspace/config/{env}-defaults.toml)
5. Infrastructure config (workspace/infra/{name}/config.toml)
6. Runtime overrides   (CLI flags, ENV variables)

Variable Interpolation:

• {{paths.base}} - Path references
• {{env.HOME}} - Environment variables
• {{now.date}} - Dynamic values
• {{git.branch}} - Git context

3. Orchestrator (Rust)

Location: provisioning/platform/orchestrator/

Architecture:

src/
├── main.rs              // Entry point
├── api/
│   ├── routes.rs        // HTTP routes
│   ├── workflows.rs     // Workflow endpoints
│   └── batch.rs         // Batch endpoints
├── workflow/
│   ├── engine.rs        // Workflow execution
│   ├── state.rs         // State management
│   └── checkpoint.rs    // Checkpoint/recovery
├── task_queue/
│   ├── queue.rs         // File-based queue
│   ├── priority.rs      // Priority scheduling
│   └── retry.rs         // Retry logic
├── health/
│   └── monitor.rs       // Health checks
├── nushell/
│   └── bridge.rs        // Nu execution bridge
└── test_environment/    // Test env management
    ├── container_manager.rs
    ├── test_orchestrator.rs
    └── topologies.rs

Key Features:

• File-based task queue (reliable, simple)
• Checkpoint-based recovery
• Priority scheduling
• REST API (HTTP/WebSocket)
• Nushell script execution bridge

4. Workflow Engine (Nushell)

Location: provisioning/core/nulib/workflows/

Workflow Types:

workflows/
├── server_create.nu     // Server provisioning
├── taskserv.nu          // Task service management
├── cluster.nu           // Cluster deployment
├── batch.nu             // Batch operations
└── management.nu        // Workflow monitoring

Batch Workflow Features:

• Provider-agnostic (mix AWS, UpCloud, local)
• Dependency resolution (hard/soft dependencies)
• Parallel execution (configurable limits)
• Rollback support
• Real-time monitoring

5. Extension System

Extension Types:

Extension Structure:

extension-name/
├── schemas/
│   ├── main.ncl             // Main schema
│   ├── contracts.ncl        // Contract definitions
│   ├── defaults.ncl         // Default values
│   └── version.ncl          // Version management
├── scripts/
│   ├── install.nu           // Installation logic
│   ├── check.nu             // Health check
│   └── uninstall.nu         // Cleanup
├── templates/               // Config templates
├── docs/                    // Documentation
├── tests/                   // Extension tests
└── manifest.yaml            // Extension metadata

OCI Distribution: Each extension packaged as OCI artifact:

• Nickel schemas
• Nushell scripts
• Templates
• Documentation
• Manifest

6. Module and Layer System

Module System:

# Discover available extensions
provisioning module discover taskservs

# Load into workspace
provisioning module load taskserv my-workspace kubernetes containerd

# List loaded modules
provisioning module list taskserv my-workspace

Layer System (Configuration Inheritance):

Layer 1: Core     (provisioning/extensions/{type}/{name})
    ↓
Layer 2: Workspace (workspace/extensions/{type}/{name})
    ↓
Layer 3: Infrastructure (workspace/infra/{infra}/extensions/{type}/{name})

Resolution Priority: Infrastructure → Workspace → Core

7. Dependency Resolution

Algorithm: Topological sort with cycle detection

Features:

• Hard dependencies (must exist)
• Soft dependencies (optional enhancement)
• Conflict detection
• Circular dependency prevention
• Version compatibility checking

Example:

let { TaskservDependencies } = import "provisioning/dependencies.ncl" in
{
  kubernetes = TaskservDependencies {
    name = "kubernetes",
    version = "1.28.0",
    requires = ["containerd", "etcd", "os"],
    optional = ["cilium", "helm"],
    conflicts = ["docker", "podman"],
  }
}

8. Service Management

Supported Services:

Lifecycle Management:

# Start all auto-start services
provisioning platform start

# Start specific service (with dependencies)
provisioning platform start orchestrator

# Check health
provisioning platform health

# View logs
provisioning platform logs orchestrator --follow

9. Test Environment Service

Architecture:

User Command (CLI)
    ↓
Test Orchestrator (Rust)
    ↓
Container Manager (bollard)
    ↓
Docker API
    ↓
Isolated Test Containers

Test Types:

• Single taskserv testing
• Server simulation (multiple taskservs)
• Multi-node cluster topologies

Topology Templates:

• kubernetes_3node - 3-node HA cluster
• kubernetes_single - All-in-one K8s
• etcd_cluster - 3-node etcd
• postgres_redis - Database stack

Mode Architecture

Mode-Based System Overview

The platform supports four operational modes that adapt the system from individual development to enterprise production.

Mode Comparison

┌───────────────────────────────────────────────────────────────────────┐
│                        MODE ARCHITECTURE                              │
├───────────────┬───────────────┬───────────────┬───────────────────────┤
│    SOLO       │  MULTI-USER   │    CI/CD      │    ENTERPRISE         │
├───────────────┼───────────────┼───────────────┼───────────────────────┤
│               │               │               │                       │
│  Single Dev   │  Team (5-20)  │  Pipelines    │  Production           │
│               │               │               │                       │
│  ┌─────────┐  │ ┌──────────┐  │ ┌──────────┐  │ ┌──────────────────┐  │
│  │ No Auth │  │ │Token(JWT)│  │ │Token(1h) │  │ │  mTLS (TLS 1.3)  │  │
│  └─────────┘  │ └──────────┘  │ └──────────┘  │ └──────────────────┘  │
│               │               │               │                       │
│  ┌─────────┐  │ ┌──────────┐  │ ┌──────────┐  │ ┌──────────────────┐  │
│  │ Local   │  │ │ Remote   │  │ │ Remote   │  │ │ Kubernetes (HA)  │  │
│  │ Binary  │  │ │ Docker   │  │ │ K8s      │  │ │ Multi-AZ         │  │
│  └─────────┘  │ └──────────┘  │ └──────────┘  │ └──────────────────┘  │
│               │               │               │                       │
│  ┌─────────┐  │ ┌──────────┐  │ ┌──────────┐  │ ┌──────────────────┐  │
│  │ Local   │  │ │ OCI (Zot)│  │ │OCI(Harbor│  │ │ OCI (Harbor HA)  │  │
│  │ Files   │  │ │ or Harbor│  │ │ required)│  │ │ + Replication    │  │
│  └─────────┘  │ └──────────┘  │ └──────────┘  │ └──────────────────┘  │
│               │               │               │                       │
│  ┌─────────┐  │ ┌──────────┐  │ ┌──────────-┐ │ ┌──────────────────┐  │
│  │ None    │  │ │ Gitea    │  │ │ Disabled  │ │ │ etcd (mandatory) │  │
│  │         │  │ │(optional)│  │ │(stateless)| │ │                  │  │
│  └─────────┘  │ └──────────┘  │ └─────────-─┘ │ └──────────────────┘  │
│               │               │               │                       │
│  Unlimited    │  10 srv, 32   │  5 srv, 16    │ 20 srv, 64 cores      │
│               │ cores, 128 GB  │ cores, 64 GB   │ 256 GB per user        │
│               │               │               │                       │
└───────────────┴───────────────┴───────────────┴───────────────────────┘

Mode Configuration

Mode Templates: workspace/config/modes/{mode}.yaml

Active Mode: ~/.provisioning/config/active-mode.yaml

Switching Modes:

# Check current mode
provisioning mode current

# Switch to another mode
provisioning mode switch multi-user

# Validate mode requirements
provisioning mode validate enterprise

Mode-Specific Workflows

Solo Mode

# 1. Default mode, no setup needed
provisioning workspace init

# 2. Start local orchestrator
provisioning platform start orchestrator

# 3. Create infrastructure
provisioning server create

Multi-User Mode

# 1. Switch mode and authenticate
provisioning mode switch multi-user
provisioning auth login

# 2. Lock workspace
provisioning workspace lock my-infra

# 3. Pull extensions from OCI
provisioning extension pull upcloud kubernetes

# 4. Work...

# 5. Unlock workspace
provisioning workspace unlock my-infra

CI/CD Mode

# GitLab CI
deploy:
  stage: deploy
  script:
    - export PROVISIONING_MODE=cicd
    - echo "$TOKEN" > /var/run/secrets/provisioning/token
    - provisioning validate --all
    - provisioning test quick kubernetes
    - provisioning server create --check
    - provisioning server create
  after_script:
    - provisioning workspace cleanup

Enterprise Mode

# 1. Switch to enterprise, verify K8s
provisioning mode switch enterprise
kubectl get pods -n provisioning-system

# 2. Request workspace (approval required)
provisioning workspace request prod-deployment

# 3. After approval, lock with etcd
provisioning workspace lock prod-deployment --provider etcd

# 4. Pull verified extensions
provisioning extension pull upcloud --verify-signature

# 5. Deploy
provisioning infra create --check
provisioning infra create

# 6. Release
provisioning workspace unlock prod-deployment

Network Architecture

Service Communication

┌──────────────────────────────────────────────────────────────────────┐
│                         NETWORK LAYER                                 │
├──────────────────────────────────────────────────────────────────────┤
│                                                                        │
│  ┌───────────────────────┐          ┌──────────────────────────┐     │
│  │   Ingress/Load        │          │    API Gateway           │     │
│  │   Balancer            │──────────│   (Optional)             │     │
│  └───────────────────────┘          └──────────────────────────┘     │
│              │                                    │                   │
│              │                                    │                   │
│  ┌───────────┴────────────────────────────────────┴──────────┐       │
│  │                 Service Mesh (Optional)                    │       │
│  │           (mTLS, Circuit Breaking, Retries)               │       │
│  └────┬──────────┬───────────┬────────────┬──────────────┬───┘       │
│       │          │           │            │              │            │
│  ┌────┴─────┐ ┌─┴────────┐ ┌┴─────────┐ ┌┴──────────┐ ┌┴───────┐   │
│  │ Orchestr │ │ Control  │ │ CoreDNS  │ │   Gitea   │ │  OCI   │   │
│  │   ator   │ │ Center   │ │          │ │           │ │Registry│   │
│  │          │ │          │ │          │ │           │ │        │   │
│  │ :9090    │ │ :3000    │ │ :5353    │ │ :3001     │ │ :5000  │   │
│  └──────────┘ └──────────┘ └──────────┘ └───────────┘ └────────┘   │
│                                                                        │
│  ┌────────────────────────────────────────────────────────────┐       │
│  │              DNS Resolution (CoreDNS)                       │       │
│  │  • *.prov.local  →  Internal services                      │       │
│  │  • *.infra.local →  Infrastructure nodes                   │       │
│  └────────────────────────────────────────────────────────────┘       │
│                                                                        │
└──────────────────────────────────────────────────────────────────────┘

Port Allocation

Network Security

Solo Mode:

• Localhost-only bindings
• No authentication
• No encryption

Multi-User Mode:

• Token-based authentication (JWT)
• TLS for external access
• Firewall rules

CI/CD Mode:

• Token authentication (short-lived)
• Full TLS encryption
• Network isolation

Enterprise Mode:

• mTLS for all connections
• Network policies (Kubernetes)
• Zero-trust networking
• Audit logging

Data Architecture

Data Storage

┌────────────────────────────────────────────────────────────────┐
│                     DATA LAYER                                  │
├────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │            Configuration Data (Hierarchical)             │   │
│  │                                                           │   │
│  │  ~/.provisioning/                                        │   │
│  │  ├── config.user.toml       (User preferences)          │   │
│  │  └── config/                                             │   │
│  │      ├── active-mode.yaml   (Active mode)               │   │
│  │      └── user_config.yaml   (Workspaces, preferences)   │   │
│  │                                                           │   │
│  │  workspace/                                              │   │
│  │  ├── config/                                             │   │
│  │  │   ├── provisioning.yaml  (Workspace config)          │   │
│  │  │   └── modes/*.yaml       (Mode templates)            │   │
│  │  └── infra/{name}/                                       │   │
│  │      ├── main.ncl           (Infrastructure Nickel)     │   │
│  │      └── config.toml        (Infra-specific)            │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                  │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │            State Data (Runtime)                          │   │
│  │                                                           │   │
│  │  ~/.provisioning/orchestrator/data/                      │   │
│  │  ├── tasks/                  (Task queue)                │   │
│  │  ├── workflows/              (Workflow state)            │   │
│  │  └── checkpoints/            (Recovery points)           │   │
│  │                                                           │   │
│  │  ~/.provisioning/services/                               │   │
│  │  ├── pids/                   (Process IDs)               │   │
│  │  ├── logs/                   (Service logs)              │   │
│  │  └── state/                  (Service state)             │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                  │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │            Cache Data (Performance)                      │   │
│  │                                                           │   │
│  │  ~/.provisioning/cache/                                  │   │
│  │  ├── oci/                    (OCI artifacts)             │   │
│  │  ├── schemas/                (Nickel compiled)           │   │
│  │  └── modules/                (Module cache)              │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                  │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │            Extension Data (OCI Artifacts)                │   │
│  │                                                           │   │
│  │  OCI Registry (localhost:5000 or harbor.company.com)    │   │
│  │  ├── provisioning-core:v3.5.0                           │   │
│  │  ├── provisioning-extensions/                           │   │
│  │  │   ├── kubernetes:1.28.0                              │   │
│  │  │   ├── aws:2.0.0                                      │   │
│  │  │   └── (100+ artifacts)                               │   │
│  │  └── provisioning-platform/                             │   │
│  │      ├── orchestrator:v1.2.0                            │   │
│  │      └── (4 service images)                             │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                  │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │            Secrets (Encrypted)                           │   │
│  │                                                           │   │
│  │  workspace/secrets/                                      │   │
│  │  ├── keys.yaml.enc           (SOPS-encrypted)           │   │
│  │  ├── ssh-keys/               (SSH keys)                 │   │
│  │  └── tokens/                 (API tokens)               │   │
│  │                                                           │   │
│  │  KMS Integration (Enterprise):                          │   │
│  │  • AWS KMS                                               │   │
│  │  • HashiCorp Vault                                       │   │
│  │  • Age encryption (local)                                │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                  │
└────────────────────────────────────────────────────────────────┘

Data Flow

Configuration Loading:

1. Load system defaults (config.defaults.toml)
2. Merge user config (~/.provisioning/config.user.toml)
3. Load workspace config (workspace/config/provisioning.yaml)
4. Load environment config (workspace/config/{env}-defaults.toml)
5. Load infrastructure config (workspace/infra/{name}/config.toml)
6. Apply runtime overrides (ENV variables, CLI flags)

State Persistence:

Workflow execution
    ↓
Create checkpoint (JSON)
    ↓
Save to ~/.provisioning/orchestrator/data/checkpoints/
    ↓
On failure, load checkpoint and resume

OCI Artifact Flow:

1. Package extension (oci-package.nu)
2. Push to OCI registry (provisioning oci push)
3. Extension stored as OCI artifact
4. Pull when needed (provisioning oci pull)
5. Cache locally (~/.provisioning/cache/oci/)

Security Architecture

Security Layers

┌─────────────────────────────────────────────────────────────────┐
│                     SECURITY ARCHITECTURE                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                   │
│  ┌────────────────────────────────────────────────────────┐     │
│  │  Layer 1: Authentication & Authorization               │     │
│  │                                                          │     │
│  │  Solo:       None (local development)                  │     │
│  │  Multi-user: JWT tokens (24h expiry)                   │     │
│  │  CI/CD:      CI-injected tokens (1h expiry)            │     │
│  │  Enterprise: mTLS (TLS 1.3, mutual auth)               │     │
│  └────────────────────────────────────────────────────────┘     │
│                                                                   │
│  ┌────────────────────────────────────────────────────────┐     │
│  │  Layer 2: Encryption                                    │     │
│  │                                                          │     │
│  │  In Transit:                                            │     │
│  │  • TLS 1.3 (multi-user, CI/CD, enterprise)             │     │
│  │  • mTLS (enterprise)                                    │     │
│  │                                                          │     │
│  │  At Rest:                                               │     │
│  │  • SOPS + Age (secrets encryption)                      │     │
│  │  • KMS integration (CI/CD, enterprise)                  │     │
│  │  • Encrypted filesystems (enterprise)                   │     │
│  └────────────────────────────────────────────────────────┘     │
│                                                                   │
│  ┌────────────────────────────────────────────────────────┐     │
│  │  Layer 3: Secret Management                             │     │
│  │                                                          │     │
│  │  • SOPS for file encryption                             │     │
│  │  • Age for key management                               │     │
│  │  • KMS integration (AWS KMS, Vault)                     │     │
│  │  • SSH key storage (KMS-backed)                         │     │
│  │  • API token management                                 │     │
│  └────────────────────────────────────────────────────────┘     │
│                                                                   │
│  ┌────────────────────────────────────────────────────────┐     │
│  │  Layer 4: Access Control                                │     │
│  │                                                          │     │
│  │  • RBAC (Role-Based Access Control)                     │     │
│  │  • Workspace isolation                                   │     │
│  │  • Workspace locking (Gitea, etcd)                      │     │
│  │  • Resource quotas (per-user limits)                    │     │
│  └────────────────────────────────────────────────────────┘     │
│                                                                   │
│  ┌────────────────────────────────────────────────────────┐     │
│  │  Layer 5: Network Security                              │     │
│  │                                                          │     │
│  │  • Network policies (Kubernetes)                        │     │
│  │  • Firewall rules                                       │     │
│  │  • Zero-trust networking (enterprise)                   │     │
│  │  • Service mesh (optional, mTLS)                        │     │
│  └────────────────────────────────────────────────────────┘     │
│                                                                   │
│  ┌────────────────────────────────────────────────────────┐     │
│  │  Layer 6: Audit & Compliance                            │     │
│  │                                                          │     │
│  │  • Audit logs (all operations)                          │     │
│  │  • Compliance policies (SOC2, ISO27001)                 │     │
│  │  • Image signing (cosign, notation)                     │     │
│  │  • Vulnerability scanning (Harbor)                      │     │
│  └────────────────────────────────────────────────────────┘     │
│                                                                   │
└─────────────────────────────────────────────────────────────────┘

Secret Management

SOPS Integration:

# Edit encrypted file
provisioning sops workspace/secrets/keys.yaml.enc

# Encryption happens automatically on save
# Decryption happens automatically on load

KMS Integration (Enterprise):

# workspace/config/provisioning.yaml
secrets:
  provider: "kms"
  kms:
    type: "aws"  # or "vault"
    region: "us-east-1"
    key_id: "arn:aws:kms:..."

Image Signing and Verification

CI/CD Mode (Required):

# Sign OCI artifact
cosign sign oci://registry/kubernetes:1.28.0

# Verify signature
cosign verify oci://registry/kubernetes:1.28.0

Enterprise Mode (Mandatory):

# Pull with verification
provisioning extension pull kubernetes --verify-signature

# System blocks unsigned artifacts

Deployment Architecture

Deployment Modes

1. Binary Deployment (Solo, Multi-user)

User Machine
├── ~/.provisioning/bin/
│   ├── provisioning-orchestrator
│   ├── provisioning-control-center
│   └── ...
├── ~/.provisioning/orchestrator/data/
├── ~/.provisioning/services/
└── Process Management (PID files, logs)

Pros: Simple, fast startup, no Docker dependency Cons: Platform-specific binaries, manual updates

2. Docker Deployment (Multi-user, CI/CD)

Docker Daemon
├── Container: provisioning-orchestrator
├── Container: provisioning-control-center
├── Container: provisioning-coredns
├── Container: provisioning-gitea
├── Container: provisioning-oci-registry
└── Volumes: ~/.provisioning/data/

Pros: Consistent environment, easy updates Cons: Requires Docker, resource overhead

3. Docker Compose Deployment (Multi-user)

# provisioning/platform/docker-compose.yaml
services:
  orchestrator:
    image: provisioning-platform/orchestrator:v1.2.0
    ports:
      - "8080:9090"
    volumes:
      - orchestrator-data:/data

  control-center:
    image: provisioning-platform/control-center:v1.2.0
    ports:
      - "3000:3000"
    depends_on:
      - orchestrator

  coredns:
    image: coredns/coredns:1.11.1
    ports:
      - "5353:53/udp"

  gitea:
    image: gitea/gitea:1.20
    ports:
      - "3001:3000"

  oci-registry:
    image: ghcr.io/project-zot/zot:latest
    ports:
      - "5000:5000"

Pros: Easy multi-service orchestration, declarative Cons: Local only, no HA

4. Kubernetes Deployment (CI/CD, Enterprise)

# Namespace: provisioning-system
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orchestrator
spec:
  replicas: 3  # HA
  selector:
    matchLabels:
      app: orchestrator
  template:
    metadata:
      labels:
        app: orchestrator
    spec:
      containers:
      - name: orchestrator
        image: harbor.company.com/provisioning-platform/orchestrator:v1.2.0
        ports:
        - containerPort: 8080
        env:
        - name: RUST_LOG
          value: "info"
        volumeMounts:
        - name: data
          mountPath: /data
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: orchestrator-data

Pros: HA, scalability, production-ready Cons: Complex setup, Kubernetes required

5. Remote Deployment (All modes)

# Connect to remotely-running services
services:
  orchestrator:
    deployment:
      mode: "remote"
      remote:
        endpoint: "https://orchestrator.company.com"
        tls_enabled: true
        auth_token_path: "~/.provisioning/tokens/orchestrator.token"

Pros: No local resources, centralized Cons: Network dependency, latency

Integration Architecture

Integration Patterns

1. Hybrid Language Integration (Rust ↔ Nushell)

Rust Orchestrator
    ↓ (HTTP API)
Nushell CLI
    ↓ (exec via bridge)
Nushell Business Logic
    ↓ (returns JSON)
Rust Orchestrator
    ↓ (updates state)
File-based Task Queue

Communication: HTTP API + stdin/stdout JSON

2. Provider Abstraction

Unified Provider Interface
├── create_server(config) -> Server
├── delete_server(id) -> bool
├── list_servers() -> [Server]
└── get_server_status(id) -> Status

Provider Implementations:
├── AWS Provider (aws-sdk-rust, aws cli)
├── UpCloud Provider (upcloud API)
└── Local Provider (Docker, libvirt)

3. OCI Registry Integration

Extension Development
    ↓
Package (oci-package.nu)
    ↓
Push (provisioning oci push)
    ↓
OCI Registry (Zot/Harbor)
    ↓
Pull (provisioning oci pull)
    ↓
Cache (~/.provisioning/cache/oci/)
    ↓
Load into Workspace

4. Gitea Integration (Multi-user, Enterprise)

Workspace Operations
    ↓
Check Lock Status (Gitea API)
    ↓
Acquire Lock (Create lock file in Git)
    ↓
Perform Changes
    ↓
Commit + Push
    ↓
Release Lock (Delete lock file)

Benefits:

• Distributed locking
• Change tracking via Git history
• Collaboration features

5. CoreDNS Integration

Service Registration
    ↓
Update CoreDNS Corefile
    ↓
Reload CoreDNS
    ↓
DNS Resolution Available

Zones:
├── *.prov.local     (Internal services)
├── *.infra.local    (Infrastructure nodes)
└── *.test.local     (Test environments)

Performance and Scalability

Performance Characteristics

Scalability Limits

Solo Mode:

• Unlimited local resources
• Limited by machine capacity

Multi-User Mode:

• 10 servers per user
• 32 cores, 128 GB RAM per user
• 5-20 concurrent users

CI/CD Mode:

• 5 servers per pipeline
• 16 cores, 64 GB RAM per pipeline
• 100+ concurrent pipelines

Enterprise Mode:

• 20 servers per user
• 64 cores, 256 GB RAM per user
• 1000+ concurrent users
• Horizontal scaling via Kubernetes

Optimization Strategies

Caching:

• OCI artifacts cached locally
• Nickel compilation cached
• Module resolution cached

Parallel Execution:

• Batch operations with configurable limits
• Dependency-aware parallel starts
• Workflow DAG execution

Incremental Operations:

• Only update changed resources
• Checkpoint-based recovery
• Delta synchronization

Evolution and Roadmap

Version History

Roadmap (Future Versions)

v3.6.0 (Q1 2026):

• GraphQL API
• Advanced RBAC
• Multi-tenancy
• Observability enhancements (OpenTelemetry)

v4.0.0 (Q2 2026):

• Multi-repository split complete
• Extension marketplace
• Advanced workflow features (conditional execution, loops)
• Cost optimization engine

v4.1.0 (Q3 2026):

• AI-assisted infrastructure generation
• Policy-as-code (OPA integration)
• Advanced compliance features

Long-term Vision:

• Serverless workflow execution
• Edge computing support
• Multi-cloud failover
• Self-healing infrastructure

Related Documentation

Architecture

• Multi-Repo Architecture - Repository organization
• Design Principles - Architectural philosophy
• Integration Patterns - Integration details
• Orchestrator Model - Hybrid orchestration

ADRs

• ADR-001 - Project structure
• ADR-002 - Distribution strategy
• ADR-003 - Workspace isolation
• ADR-004 - Hybrid architecture
• ADR-005 - Extension framework
• ADR-006 - CLI refactoring

User Guides

• Getting Started - First steps
• Mode System - Modes overview
• Service Management - Services
• OCI Registry - OCI operations

Maintained By: Architecture Team Review Cycle: Quarterly Next Review: 2026-01-06

Design Principles

Overview

Provisioning is built on a foundation of architectural principles that guide design decisions, ensure system quality, and maintain consistency across the codebase. These principles have evolved from real-world experience and represent lessons learned from complex infrastructure automation challenges.

Core Architectural Principles

1. Project Architecture Principles (PAP) Compliance

Principle: Fully agnostic and configuration-driven, not hardcoded. Use abstraction layers dynamically loaded from configurations.

Rationale: Infrastructure as Code (IaC) systems must be flexible enough to adapt to any environment without code changes. Hardcoded values defeat the purpose of IaC and create maintenance burdens.

Implementation Guidelines:

• Never patch the system with hardcoded fallbacks when configuration parsing fails
• All behavior must be configurable through the hierarchical configuration system
• Use abstraction layers that are dynamically loaded from configuration
• Validate configuration fully before execution, fail fast on invalid config

Anti-Patterns (Anti-PAP):

• Hardcoded provider endpoints or credentials
• Environment-specific logic in code
• Fallback to default values when configuration is missing
• Mixed configuration and implementation logic

Example:

# ✅ PAP Compliant - Configuration-driven
[providers.aws]
regions = ["us-west-2", "us-east-1"]
instance_types = ["t3.micro", "t3.small"]
api_endpoint = "https://ec2.amazonaws.com"

# ❌ Anti-PAP - Hardcoded fallback in code
if config.providers.aws.regions.is_empty() {
    regions = vec!["us-west-2"]; // Hardcoded fallback
}

2. Hybrid Architecture Optimization

Principle: Use each language for what it does best - Rust for coordination, Nushell for business logic.

Rationale: Different languages have different strengths. Rust excels at performance-critical coordination tasks, while Nushell excels at configuration management and domain-specific operations.

Implementation Guidelines:

• Rust handles orchestration, state management, and performance-critical paths
• Nushell handles provider operations, configuration processing, and CLI interfaces
• Clear boundaries between language responsibilities
• Structured data exchange (JSON) between languages
• Preserve existing domain expertise in Nushell

Language Responsibility Matrix:

Rust Layer:
├── Workflow orchestration and coordination
├── REST API servers and HTTP endpoints
├── State persistence and checkpoint management
├── Parallel processing and batch operations
├── Error recovery and rollback logic
└── Performance-critical data processing

Nushell Layer:
├── Provider implementations (AWS, UpCloud, local)
├── Task service management and configuration
├── Nickel configuration processing and validation
├── Template generation and Infrastructure as Code
├── CLI user interfaces and interactive tools
└── Domain-specific business logic

3. Configuration-First Architecture

Principle: All system behavior is determined by configuration, with clear hierarchical precedence and validation.

Rationale: True Infrastructure as Code requires that all behavior be configurable without code changes. Configuration hierarchy provides flexibility while maintaining predictability.

Configuration Hierarchy (precedence order):

1. Runtime Parameters (highest precedence)
2. Environment Configuration
3. Infrastructure Configuration
4. User Configuration
5. System Defaults (lowest precedence)

Implementation Guidelines:

• Complete configuration validation before execution
• Variable interpolation for dynamic values
• Schema-based validation using Nickel
• Configuration immutability during execution
• Comprehensive error reporting for configuration issues

4. Domain-Driven Structure

Principle: Organize code by business domains and functional boundaries, not by technical concerns.

Rationale: Domain-driven organization scales better, reduces coupling, and enables focused development by domain experts.

Domain Organization:

├── core/           # Core system and library functions
├── platform/       # High-performance coordination layer
├── provisioning/   # Main business logic with providers and services
├── control-center/ # Web-based management interface
├── tools/          # Development and utility tools
└── extensions/     # Plugin and extension framework

Domain Responsibilities:

• Each domain has clear ownership and boundaries
• Cross-domain communication through well-defined interfaces
• Domain-specific testing and validation strategies
• Independent evolution and versioning within architectural guidelines

5. Isolation and Modularity

Principle: Components are isolated, modular, and independently deployable with clear interface contracts.

Rationale: Isolation enables independent development, testing, and deployment. Clear interfaces prevent tight coupling and enable system evolution.

Implementation Guidelines:

• User workspace isolation from system installation
• Extension sandboxing and security boundaries
• Provider abstraction with standardized interfaces
• Service modularity with dependency management
• Clear API contracts between components

Quality Attribute Principles

6. Reliability Through Recovery

Principle: Build comprehensive error recovery and rollback capabilities into every operation.

Rationale: Infrastructure operations can fail at any point. Systems must be able to recover gracefully and maintain consistent state.

Implementation Guidelines:

• Checkpoint-based recovery for long-running workflows
• Comprehensive rollback capabilities for all operations
• Transactional semantics where possible
• State validation and consistency checks
• Detailed audit trails for debugging and recovery

Recovery Strategies:

Operation Level:
├── Atomic operations with rollback
├── Retry logic with exponential backoff
├── Circuit breakers for external dependencies
└── Graceful degradation on partial failures

Workflow Level:
├── Checkpoint-based recovery
├── Dependency-aware rollback
├── State consistency validation
└── Resume from failure points

System Level:
├── Health monitoring and alerting
├── Automatic recovery procedures
├── Data backup and restoration
└── Disaster recovery capabilities

7. Performance Through Parallelism

Principle: Design for parallel execution and efficient resource utilization while maintaining correctness.

Rationale: Infrastructure operations often involve multiple independent resources that can be processed in parallel for significant performance gains.

Implementation Guidelines:

• Configurable parallelism limits to prevent resource exhaustion
• Dependency-aware parallel execution
• Resource pooling and connection management
• Efficient data structures and algorithms
• Memory-conscious processing for large datasets

8. Security Through Isolation

Principle: Implement security through isolation boundaries, least privilege, and comprehensive validation.

Rationale: Infrastructure systems handle sensitive data and powerful operations. Security must be built in at the architectural level.

Security Implementation:

Authentication & Authorization:
├── API authentication for external access
├── Role-based access control for operations
├── Permission validation before execution
└── Audit logging for all security events

Data Protection:
├── Encrypted secrets management (SOPS/Age)
├── Secure configuration file handling
├── Network communication encryption
└── Sensitive data sanitization in logs

Isolation Boundaries:
├── User workspace isolation
├── Extension sandboxing
├── Provider credential isolation
└── Process and network isolation

Development Methodology Principles

9. Configuration-Driven Testing

Principle: Tests should be configuration-driven and validate both happy path and error conditions.

Rationale: Infrastructure systems must work across diverse environments and configurations. Tests must validate the configuration-driven nature of the system.

Testing Strategy:

Unit Testing:
├── Configuration validation tests
├── Individual component tests
├── Error condition tests
└── Performance benchmark tests

Integration Testing:
├── Multi-provider workflow tests
├── Configuration hierarchy tests
├── Error recovery tests
└── End-to-end scenario tests

System Testing:
├── Full deployment tests
├── Upgrade and migration tests
├── Performance and scalability tests
└── Security and isolation tests

Error Handling Principles

11. Fail Fast, Recover Gracefully

Principle: Validate early and fail fast on errors, but provide comprehensive recovery mechanisms.

Rationale: Early validation prevents complex error states, while graceful recovery maintains system reliability.

Implementation Guidelines:

• Complete configuration validation before execution
• Input validation at system boundaries
• Clear error messages without internal stack traces (except in DEBUG mode)
• Comprehensive error categorization and handling
• Recovery procedures for all error categories

Error Categories:

Configuration Errors:
├── Invalid configuration syntax
├── Missing required configuration
├── Configuration conflicts
└── Schema validation failures

Runtime Errors:
├── Provider API failures
├── Network connectivity issues
├── Resource availability problems
└── Permission and authentication errors

System Errors:
├── File system access problems
├── Memory and resource exhaustion
├── Process communication failures
└── External dependency failures

12. Observable Operations

Principle: All operations must be observable through comprehensive logging, metrics, and monitoring.

Rationale: Infrastructure operations must be debuggable and monitorable in production environments.

Observability Implementation:

Logging:
├── Structured JSON logging
├── Configurable log levels
├── Context-aware log messages
└── Audit trail for all operations

Metrics:
├── Operation performance metrics
├── Resource utilization metrics
├── Error rate and type metrics
└── Business logic metrics

Monitoring:
├── Health check endpoints
├── Real-time status reporting
├── Workflow progress tracking
└── Alert integration capabilities

Evolution and Maintenance Principles

13. Backward Compatibility

Principle: Maintain backward compatibility for configuration, APIs, and user interfaces.

Rationale: Infrastructure systems are long-lived and must support existing configurations and workflows during evolution.

Compatibility Guidelines:

• Semantic versioning for all interfaces
• Configuration migration tools and procedures
• Deprecation warnings and migration guides
• API versioning for external interfaces
• Comprehensive upgrade testing

14. Documentation-Driven Development

Principle: Architecture decisions, APIs, and operational procedures must be thoroughly documented.

Rationale: Infrastructure systems are complex and require clear documentation for operation, maintenance, and evolution.

Documentation Requirements:

• Architecture Decision Records (ADRs) for major decisions
• API documentation with examples
• Operational runbooks and procedures
• Configuration guides and examples
• Troubleshooting guides and common issues

15. Technical Debt Management

Principle: Actively manage technical debt through regular assessment and systematic improvement.

Rationale: Infrastructure systems accumulate complexity over time. Proactive debt management prevents system degradation.

Debt Management Strategy:

Assessment:
├── Regular code quality reviews
├── Performance profiling and optimization
├── Security audit and updates
└── Dependency management and updates

Improvement:
├── Refactoring for clarity and maintainability
├── Performance optimization based on metrics
├── Security enhancement and hardening
└── Test coverage improvement and validation

Trade-off Management

16. Explicit Trade-off Documentation

Principle: All architectural trade-offs must be explicitly documented with rationale and alternatives considered.

Rationale: Understanding trade-offs enables informed decision making and future evolution of the system.

Trade-off Categories:

Performance vs. Maintainability:
├── Rust coordination layer for performance
├── Nushell business logic for maintainability
├── Caching strategies for speed vs. consistency
└── Parallel processing vs. resource usage

Flexibility vs. Complexity:
├── Configuration-driven architecture vs. simplicity
├── Extension framework vs. core system complexity
├── Multi-provider support vs. specialization
└── Hierarchical configuration vs. simple key-value

Security vs. Usability:
├── Workspace isolation vs. convenience
├── Extension sandboxing vs. functionality
├── Authentication requirements vs. ease of use
└── Audit logging vs. performance overhead

Conclusion

These design principles form the foundation of provisioning’s architecture. They guide decision making, ensure quality, and provide a framework for system evolution. Adherence to these principles has enabled the development of a sophisticated, reliable, and maintainable infrastructure automation platform.

The principles are living guidelines that evolve with the system while maintaining core architectural integrity. They serve as both implementation guidance and evaluation criteria for new features and modifications.

Success in applying these principles is measured by:

• System reliability and error recovery capabilities
• Development efficiency and maintainability
• Configuration flexibility and user experience
• Performance and scalability characteristics
• Security and isolation effectiveness

These principles represent the distilled wisdom from building and operating complex infrastructure automation systems at scale.

Integration Patterns

Overview

Provisioning implements sophisticated integration patterns to coordinate between its hybrid Rust/Nushell architecture, manage multi-provider workflows, and enable extensible functionality. This document outlines the key integration patterns, their implementations, and best practices.

Core Integration Patterns

1. Hybrid Language Integration

Rust-to-Nushell Communication Pattern

Use Case: Orchestrator invoking business logic operations

Implementation:

use tokio::process::Command;
use serde_json;

pub async fn execute_nushell_workflow(
    workflow: &str,
    args: &[String]
) -> Result<WorkflowResult, Error> {
    let mut cmd = Command::new("nu");
    cmd.arg("-c")
       .arg(format!("use core/nulib/workflows/{}.nu *; {}", workflow, args.join(" ")));

    let output = cmd.output().await?;
    let result: WorkflowResult = serde_json::from_slice(&output.stdout)?;
    Ok(result)
}

Data Exchange Format:

{
    "status": "success" | "error" | "partial",
    "result": {
        "operation": "server_create",
        "resources": ["server-001", "server-002"],
        "metadata": { ... }
    },
    "error": null | { "code": "ERR001", "message": "..." },
    "context": { "workflow_id": "wf-123", "step": 2 }
}

Nushell-to-Rust Communication Pattern

Use Case: Business logic submitting workflows to orchestrator

Implementation:

def submit-workflow [workflow: record] -> record {
    let payload = $workflow | to json

    http post "http://localhost:9090/workflows/submit" {
        headers: { "Content-Type": "application/json" }
        body: $payload
    }
    | from json
}

API Contract:

{
    "workflow_id": "wf-456",
    "name": "multi_cloud_deployment",
    "operations": [...],
    "dependencies": { ... },
    "configuration": { ... }
}

2. Provider Abstraction Pattern

Standard Provider Interface

Purpose: Uniform API across different cloud providers

Interface Definition:

# Standard provider interface that all providers must implement
export def list-servers [] -> table {
    # Provider-specific implementation
}

export def create-server [config: record] -> record {
    # Provider-specific implementation
}

export def delete-server [id: string] -> nothing {
    # Provider-specific implementation
}

export def get-server [id: string] -> record {
    # Provider-specific implementation
}

Configuration Integration:

[providers.aws]
region = "us-west-2"
credentials_profile = "default"
timeout = 300

[providers.upcloud]
zone = "de-fra1"
api_endpoint = "https://api.upcloud.com"
timeout = 180

[providers.local]
docker_socket = "/var/run/docker.sock"
network_mode = "bridge"

Provider Discovery and Loading

def load-providers [] -> table {
    let provider_dirs = glob "providers/*/nulib"

    $provider_dirs
    | each { |dir|
        let provider_name = $dir | path basename | path dirname | path basename
        let provider_config = get-provider-config $provider_name

        {
            name: $provider_name,
            path: $dir,
            config: $provider_config,
            available: (test-provider-connectivity $provider_name)
        }
    }
}

3. Configuration Resolution Pattern

Hierarchical Configuration Loading

Implementation:

def resolve-configuration [context: record] -> record {
    let base_config = open config.defaults.toml
    let user_config = if ("config.user.toml" | path exists) {
        open config.user.toml
    } else { {} }

    let env_config = if ($env.PROVISIONING_ENV? | is-not-empty) {
        let env_file = $"config.($env.PROVISIONING_ENV).toml"
        if ($env_file | path exists) { open $env_file } else { {} }
    } else { {} }

    let merged_config = $base_config
    | merge $user_config
    | merge $env_config
    | merge ($context.runtime_config? | default {})

    interpolate-variables $merged_config
}

Variable Interpolation Pattern

def interpolate-variables [config: record] -> record {
    let interpolations = {
        "{{paths.base}}": ($env.PWD),
        "{{env.HOME}}": ($env.HOME),
        "{{now.date}}": (date now | format date "%Y-%m-%d"),
        "{{git.branch}}": (git branch --show-current | str trim)
    }

    $config
    | to json
    | str replace --all "{{paths.base}}" $interpolations."{{paths.base}}"
    | str replace --all "{{env.HOME}}" $interpolations."{{env.HOME}}"
    | str replace --all "{{now.date}}" $interpolations."{{now.date}}"
    | str replace --all "{{git.branch}}" $interpolations."{{git.branch}}"
    | from json
}

4. Workflow Orchestration Patterns

Dependency Resolution Pattern

Use Case: Managing complex workflow dependencies

Implementation (Rust):

use petgraph::{Graph, Direction};
use std::collections::HashMap;

pub struct DependencyResolver {
    graph: Graph<String, ()>,
    node_map: HashMap<String, petgraph::graph::NodeIndex>,
}

impl DependencyResolver {
    pub fn resolve_execution_order(&self) -> Result<Vec<String>, Error> {
        let mut topo = petgraph::algo::toposort(&self.graph, None)
            .map_err(|_| Error::CyclicDependency)?;

        Ok(topo.into_iter()
            .map(|idx| self.graph[idx].clone())
            .collect())
    }

    pub fn add_dependency(&mut self, from: &str, to: &str) {
        let from_idx = self.get_or_create_node(from);
        let to_idx = self.get_or_create_node(to);
        self.graph.add_edge(from_idx, to_idx, ());
    }
}

Parallel Execution Pattern

use tokio::task::JoinSet;
use futures::stream::{FuturesUnordered, StreamExt};

pub async fn execute_parallel_batch(
    operations: Vec<Operation>,
    parallelism_limit: usize
) -> Result<Vec<OperationResult>, Error> {
    let semaphore = tokio::sync::Semaphore::new(parallelism_limit);
    let mut join_set = JoinSet::new();

    for operation in operations {
        let permit = semaphore.clone();
        join_set.spawn(async move {
            let _permit = permit.acquire().await?;
            execute_operation(operation).await
        });
    }

    let mut results = Vec::new();
    while let Some(result) = join_set.join_next().await {
        results.push(result??);
    }

    Ok(results)
}

5. State Management Patterns

Checkpoint-Based Recovery Pattern

Use Case: Reliable state persistence and recovery

Implementation:

#[derive(Serialize, Deserialize)]
pub struct WorkflowCheckpoint {
    pub workflow_id: String,
    pub step: usize,
    pub completed_operations: Vec<String>,
    pub current_state: serde_json::Value,
    pub metadata: HashMap<String, String>,
    pub timestamp: chrono::DateTime<chrono::Utc>,
}

pub struct CheckpointManager {
    checkpoint_dir: PathBuf,
}

impl CheckpointManager {
    pub fn save_checkpoint(&self, checkpoint: &WorkflowCheckpoint) -> Result<(), Error> {
        let checkpoint_file = self.checkpoint_dir
            .join(&checkpoint.workflow_id)
            .with_extension("json");

        let checkpoint_data = serde_json::to_string_pretty(checkpoint)?;
        std::fs::write(checkpoint_file, checkpoint_data)?;
        Ok(())
    }

    pub fn restore_checkpoint(&self, workflow_id: &str) -> Result<Option<WorkflowCheckpoint>, Error> {
        let checkpoint_file = self.checkpoint_dir
            .join(workflow_id)
            .with_extension("json");

        if checkpoint_file.exists() {
            let checkpoint_data = std::fs::read_to_string(checkpoint_file)?;
            let checkpoint = serde_json::from_str(&checkpoint_data)?;
            Ok(Some(checkpoint))
        } else {
            Ok(None)
        }
    }
}

Rollback Pattern

pub struct RollbackManager {
    rollback_stack: Vec<RollbackAction>,
}

#[derive(Clone, Debug)]
pub enum RollbackAction {
    DeleteResource { provider: String, resource_id: String },
    RestoreFile { path: PathBuf, content: String },
    RevertConfiguration { key: String, value: serde_json::Value },
    CustomAction { command: String, args: Vec<String> },
}

impl RollbackManager {
    pub async fn execute_rollback(&self) -> Result<(), Error> {
        // Execute rollback actions in reverse order
        for action in self.rollback_stack.iter().rev() {
            match action {
                RollbackAction::DeleteResource { provider, resource_id } => {
                    self.delete_resource(provider, resource_id).await?;
                }
                RollbackAction::RestoreFile { path, content } => {
                    tokio::fs::write(path, content).await?;
                }
                // ... handle other rollback actions
            }
        }
        Ok(())
    }
}

6. Event and Messaging Patterns

Event-Driven Architecture Pattern

Use Case: Decoupled communication between components

Event Definition:

#[derive(Serialize, Deserialize, Clone, Debug)]
pub enum SystemEvent {
    WorkflowStarted { workflow_id: String, name: String },
    WorkflowCompleted { workflow_id: String, result: WorkflowResult },
    WorkflowFailed { workflow_id: String, error: String },
    ResourceCreated { provider: String, resource_type: String, resource_id: String },
    ResourceDeleted { provider: String, resource_type: String, resource_id: String },
    ConfigurationChanged { key: String, old_value: serde_json::Value, new_value: serde_json::Value },
}

Event Bus Implementation:

use tokio::sync::broadcast;

pub struct EventBus {
    sender: broadcast::Sender<SystemEvent>,
}

impl EventBus {
    pub fn new(capacity: usize) -> Self {
        let (sender, _) = broadcast::channel(capacity);
        Self { sender }
    }

    pub fn publish(&self, event: SystemEvent) -> Result<(), Error> {
        self.sender.send(event)
            .map_err(|_| Error::EventPublishFailed)?;
        Ok(())
    }

    pub fn subscribe(&self) -> broadcast::Receiver<SystemEvent> {
        self.sender.subscribe()
    }
}

7. Extension Integration Patterns

Extension Discovery and Loading

def discover-extensions [] -> table {
    let extension_dirs = glob "extensions/*/extension.toml"

    $extension_dirs
    | each { |manifest_path|
        let extension_dir = $manifest_path | path dirname
        let manifest = open $manifest_path

        {
            name: $manifest.extension.name,
            version: $manifest.extension.version,
            type: $manifest.extension.type,
            path: $extension_dir,
            manifest: $manifest,
            valid: (validate-extension $manifest),
            compatible: (check-compatibility $manifest.compatibility)
        }
    }
    | where valid and compatible
}

Extension Interface Pattern

# Standard extension interface
export def extension-info [] -> record {
    {
        name: "custom-provider",
        version: "1.0.0",
        type: "provider",
        description: "Custom cloud provider integration",
        entry_points: {
            cli: "nulib/cli.nu",
            provider: "nulib/provider.nu"
        }
    }
}

export def extension-validate [] -> bool {
    # Validate extension configuration and dependencies
    true
}

export def extension-activate [] -> nothing {
    # Perform extension activation tasks
}

export def extension-deactivate [] -> nothing {
    # Perform extension cleanup tasks
}

8. API Design Patterns

REST API Standardization

Base API Structure:

use axum::{
    extract::{Path, State},
    response::Json,
    routing::{get, post, delete},
    Router,
};

pub fn create_api_router(state: AppState) -> Router {
    Router::new()
        .route("/health", get(health_check))
        .route("/workflows", get(list_workflows).post(create_workflow))
        .route("/workflows/:id", get(get_workflow).delete(delete_workflow))
        .route("/workflows/:id/status", get(workflow_status))
        .route("/workflows/:id/logs", get(workflow_logs))
        .with_state(state)
}

Standard Response Format:

{
    "status": "success" | "error" | "pending",
    "data": { ... },
    "metadata": {
        "timestamp": "2025-09-26T12:00:00Z",
        "request_id": "req-123",
        "version": "3.1.0"
    },
    "error": null | {
        "code": "ERR001",
        "message": "Human readable error",
        "details": { ... }
    }
}

Error Handling Patterns

Structured Error Pattern

#[derive(thiserror::Error, Debug)]
pub enum ProvisioningError {
    #[error("Configuration error: {message}")]
    Configuration { message: String },

    #[error("Provider error [{provider}]: {message}")]
    Provider { provider: String, message: String },

    #[error("Workflow error [{workflow_id}]: {message}")]
    Workflow { workflow_id: String, message: String },

    #[error("Resource error [{resource_type}/{resource_id}]: {message}")]
    Resource { resource_type: String, resource_id: String, message: String },
}

Error Recovery Pattern

def with-retry [operation: closure, max_attempts: int = 3] {
    mut attempts = 0
    mut last_error = null

    while $attempts < $max_attempts {
        try {
            return (do $operation)
        } catch { |error|
            $attempts = $attempts + 1
            $last_error = $error

            if $attempts < $max_attempts {
                let delay = (2 ** ($attempts - 1)) * 1000  # Exponential backoff
                sleep $"($delay)ms"
            }
        }
    }

    error make { msg: $"Operation failed after ($max_attempts) attempts: ($last_error)" }
}

Performance Optimization Patterns

Caching Strategy Pattern

use std::sync::Arc;
use tokio::sync::RwLock;
use std::collections::HashMap;
use chrono::{DateTime, Utc, Duration};

#[derive(Clone)]
pub struct CacheEntry<T> {
    pub value: T,
    pub expires_at: DateTime<Utc>,
}

pub struct Cache<T> {
    store: Arc<RwLock<HashMap<String, CacheEntry<T>>>>,
    default_ttl: Duration,
}

impl<T: Clone> Cache<T> {
    pub async fn get(&self, key: &str) -> Option<T> {
        let store = self.store.read().await;
        if let Some(entry) = store.get(key) {
            if entry.expires_at > Utc::now() {
                Some(entry.value.clone())
            } else {
                None
            }
        } else {
            None
        }
    }

    pub async fn set(&self, key: String, value: T) {
        let expires_at = Utc::now() + self.default_ttl;
        let entry = CacheEntry { value, expires_at };

        let mut store = self.store.write().await;
        store.insert(key, entry);
    }
}

Streaming Pattern for Large Data

def process-large-dataset [source: string] -> nothing {
    # Stream processing instead of loading entire dataset
    open $source
    | lines
    | each { |line|
        # Process line individually
        $line | process-record
    }
    | save output.json
}

Testing Integration Patterns

Integration Test Pattern

#[cfg(test)]
mod integration_tests {
    use super::*;
    use tokio_test;

    #[tokio::test]
    async fn test_workflow_execution() {
        let orchestrator = setup_test_orchestrator().await;
        let workflow = create_test_workflow();

        let result = orchestrator.execute_workflow(workflow).await;

        assert!(result.is_ok());
        assert_eq!(result.unwrap().status, WorkflowStatus::Completed);
    }
}

These integration patterns provide the foundation for the system’s sophisticated multi-component architecture, enabling reliable, scalable, and maintainable infrastructure automation.

Orchestrator Integration Model - Deep Dive

Date: 2025-10-01 Status: Clarification Document Related: Multi-Repo Strategy, Hybrid Orchestrator v3.0

Executive Summary

This document clarifies how the Rust orchestrator integrates with Nushell core in both monorepo and multi-repo architectures. The orchestrator is a critical performance layer that coordinates Nushell business logic execution, solving deep call stack limitations while preserving all existing functionality.

Current Architecture (Hybrid Orchestrator v3.0)

The Problem Being Solved

Original Issue:

Deep call stack in Nushell (template.nu:71)
→ "Type not supported" errors
→ Cannot handle complex nested workflows
→ Performance bottlenecks with recursive calls

Solution: Rust orchestrator provides:

1. Task queue management (file-based, reliable)
2. Priority scheduling (intelligent task ordering)
3. Deep call stack elimination (Rust handles recursion)
4. Performance optimization (async/await, parallel execution)
5. State management (workflow checkpointing)

How It Works Today (Monorepo)

┌─────────────────────────────────────────────────────────────┐
│                        User                                  │
└───────────────────────────┬─────────────────────────────────┘
                            │ calls
                            ↓
                    ┌───────────────┐
                    │ provisioning  │ (Nushell CLI)
                    │      CLI      │
                    └───────┬───────┘
                            │
        ┌───────────────────┼───────────────────┐
        │                   │                   │
        ↓                   ↓                   ↓
┌───────────────┐   ┌───────────────┐   ┌──────────────┐
│ Direct Mode   │   │Orchestrated   │   │ Workflow     │
│ (Simple ops)  │   │ Mode          │   │ Mode         │
└───────────────┘   └───────┬───────┘   └──────┬───────┘
                            │                   │
                            ↓                   ↓
                    ┌────────────────────────────────┐
                    │   Rust Orchestrator Service    │
                    │   (Background daemon)           │
                    │                                 │
                    │ • Task Queue (file-based)      │
                    │ • Priority Scheduler           │
                    │ • Workflow Engine              │
                    │ • REST API Server              │
                    └────────┬───────────────────────┘
                            │ spawns
                            ↓
                    ┌────────────────┐
                    │ Nushell        │
                    │ Business Logic │
                    │                │
                    │ • servers.nu   │
                    │ • taskservs.nu │
                    │ • clusters.nu  │
                    └────────────────┘

Three Execution Modes

Mode 1: Direct Mode (Simple Operations)

# No orchestrator needed
provisioning server list
provisioning env
provisioning help

# Direct Nushell execution
provisioning (CLI) → Nushell scripts → Result

Mode 2: Orchestrated Mode (Complex Operations)

# Uses orchestrator for coordination
provisioning server create --orchestrated

# Flow:
provisioning CLI → Orchestrator API → Task Queue → Nushell executor
                                                 ↓
                                            Result back to user

Mode 3: Workflow Mode (Batch Operations)

# Complex workflows with dependencies
provisioning workflow submit server-cluster.ncl

# Flow:
provisioning CLI → Orchestrator Workflow Engine → Dependency Graph
                                                 ↓
                                            Parallel task execution
                                                 ↓
                                            Nushell scripts for each task
                                                 ↓
                                            Checkpoint state

Integration Patterns

Pattern 1: CLI Submits Tasks to Orchestrator

Current Implementation:

Nushell CLI (core/nulib/workflows/server_create.nu):

# Submit server creation workflow to orchestrator
export def server_create_workflow [
    infra_name: string
    --orchestrated
] {
    if $orchestrated {
        # Submit task to orchestrator
        let task = {
            type: "server_create"
            infra: $infra_name
            params: { ... }
        }

        # POST to orchestrator REST API
        http post http://localhost:9090/workflows/servers/create $task
    } else {
        # Direct execution (old way)
        do-server-create $infra_name
    }
}

Rust Orchestrator (platform/orchestrator/src/api/workflows.rs):

// Receive workflow submission from Nushell CLI
#[axum::debug_handler]
async fn create_server_workflow(
    State(state): State<Arc<AppState>>,
    Json(request): Json<ServerCreateRequest>,
) -> Result<Json<WorkflowResponse>, ApiError> {
    // Create task
    let task = Task {
        id: Uuid::new_v4(),
        task_type: TaskType::ServerCreate,
        payload: serde_json::to_value(&request)?,
        priority: Priority::Normal,
        status: TaskStatus::Pending,
        created_at: Utc::now(),
    };

    // Queue task
    state.task_queue.enqueue(task).await?;

    // Return immediately (async execution)
    Ok(Json(WorkflowResponse {
        workflow_id: task.id,
        status: "queued",
    }))
}

Flow:

User → provisioning server create --orchestrated
     ↓
Nushell CLI prepares task
     ↓
HTTP POST to orchestrator (localhost:9090)
     ↓
Orchestrator queues task
     ↓
Returns workflow ID immediately
     ↓
User can monitor: provisioning workflow monitor <id>

Pattern 2: Orchestrator Executes Nushell Scripts

Orchestrator Task Executor (platform/orchestrator/src/executor.rs):

// Orchestrator spawns Nushell to execute business logic
pub async fn execute_task(task: Task) -> Result<TaskResult> {
    match task.task_type {
        TaskType::ServerCreate => {
            // Orchestrator calls Nushell script via subprocess
            let output = Command::new("nu")
                .arg("-c")
                .arg(format!(
                    "use {}/servers/create.nu; create-server '{}'",
                    PROVISIONING_LIB_PATH,
                    task.payload.infra_name
                ))
                .output()
                .await?;

            // Parse Nushell output
            let result = parse_nushell_output(&output)?;

            Ok(TaskResult {
                task_id: task.id,
                status: if result.success { "completed" } else { "failed" },
                output: result.data,
            })
        }
        // Other task types...
    }
}

Flow:

Orchestrator task queue has pending task
     ↓
Executor picks up task
     ↓
Spawns Nushell subprocess: nu -c "use servers/create.nu; create-server 'wuji'"
     ↓
Nushell executes business logic
     ↓
Returns result to orchestrator
     ↓
Orchestrator updates task status
     ↓
User monitors via: provisioning workflow status <id>

Pattern 3: Bidirectional Communication

Nushell Calls Orchestrator API:

# Nushell script checks orchestrator status during execution
export def check-orchestrator-health [] {
    let response = (http get http://localhost:9090/health)

    if $response.status != "healthy" {
        error make { msg: "Orchestrator not available" }
    }

    $response
}

# Nushell script reports progress to orchestrator
export def report-progress [task_id: string, progress: int] {
    http post http://localhost:9090/tasks/$task_id/progress {
        progress: $progress
        status: "in_progress"
    }
}

Orchestrator Monitors Nushell Execution:

// Orchestrator tracks Nushell subprocess
pub async fn execute_with_monitoring(task: Task) -> Result<TaskResult> {
    let mut child = Command::new("nu")
        .arg("-c")
        .arg(&task.script)
        .stdout(Stdio::piped())
        .stderr(Stdio::piped())
        .spawn()?;

    // Monitor stdout/stderr in real-time
    let stdout = child.stdout.take().unwrap();
    tokio::spawn(async move {
        let reader = BufReader::new(stdout);
        let mut lines = reader.lines();

        while let Some(line) = lines.next_line().await.unwrap() {
            // Parse progress updates from Nushell
            if line.contains("PROGRESS:") {
                update_task_progress(&line);
            }
        }
    });

    // Wait for completion with timeout
    let result = tokio::time::timeout(
        Duration::from_secs(3600),
        child.wait()
    ).await??;

    Ok(TaskResult::from_exit_status(result))
}

Multi-Repo Architecture Impact

Repository Split Doesn’t Change Integration Model

In Multi-Repo Setup:

Repository: provisioning-core

• Contains: Nushell business logic
• Installs to: /usr/local/lib/provisioning/
• Package: provisioning-core-3.2.1.tar.gz

Repository: provisioning-platform

• Contains: Rust orchestrator
• Installs to: /usr/local/bin/provisioning-orchestrator
• Package: provisioning-platform-2.5.3.tar.gz

Runtime Integration (Same as Monorepo):

User installs both packages:
  provisioning-core-3.2.1     → /usr/local/lib/provisioning/
  provisioning-platform-2.5.3 → /usr/local/bin/provisioning-orchestrator

Orchestrator expects core at:  /usr/local/lib/provisioning/
Core expects orchestrator at:  http://localhost:9090/

No code dependencies, just runtime coordination!

Configuration-Based Integration

Core Package (provisioning-core) config:

# /usr/local/share/provisioning/config/config.defaults.toml

[orchestrator]
enabled = true
endpoint = "http://localhost:9090"
timeout = 60
auto_start = true  # Start orchestrator if not running

[execution]
default_mode = "orchestrated"  # Use orchestrator by default
fallback_to_direct = true      # Fall back if orchestrator down

Platform Package (provisioning-platform) config:

# /usr/local/share/provisioning/platform/config.toml

[orchestrator]
host = "127.0.0.1"
port = 8080
data_dir = "/var/lib/provisioning/orchestrator"

[executor]
nushell_binary = "nu"  # Expects nu in PATH
provisioning_lib = "/usr/local/lib/provisioning"
max_concurrent_tasks = 10
task_timeout_seconds = 3600

Version Compatibility

Compatibility Matrix (provisioning-distribution/versions.toml):

[compatibility.platform."2.5.3"]
core = "^3.2"  # Platform 2.5.3 compatible with core 3.2.x
min-core = "3.2.0"
api-version = "v1"

[compatibility.core."3.2.1"]
platform = "^2.5"  # Core 3.2.1 compatible with platform 2.5.x
min-platform = "2.5.0"
orchestrator-api = "v1"

Execution Flow Examples

Example 1: Simple Server Creation (Direct Mode)

No Orchestrator Needed:

provisioning server list

# Flow:
CLI → servers/list.nu → Query state → Return results
(Orchestrator not involved)

Example 2: Server Creation with Orchestrator

Using Orchestrator:

provisioning server create --orchestrated --infra wuji

# Detailed Flow:
1. User executes command
   ↓
2. Nushell CLI (provisioning binary)
   ↓
3. Reads config: orchestrator.enabled = true
   ↓
4. Prepares task payload:
   {
     type: "server_create",
     infra: "wuji",
     params: { ... }
   }
   ↓
5. HTTP POST → http://localhost:9090/workflows/servers/create
   ↓
6. Orchestrator receives request
   ↓
7. Creates task with UUID
   ↓
8. Enqueues to task queue (file-based: /var/lib/provisioning/queue/)
   ↓
9. Returns immediately: { workflow_id: "abc-123", status: "queued" }
   ↓
10. User sees: "Workflow submitted: abc-123"
   ↓
11. Orchestrator executor picks up task
   ↓
12. Spawns Nushell subprocess:
    nu -c "use /usr/local/lib/provisioning/servers/create.nu; create-server 'wuji'"
   ↓
13. Nushell executes business logic:
    - Reads Nickel config
    - Calls provider API (UpCloud/AWS)
    - Creates server
    - Returns result
   ↓
14. Orchestrator captures output
   ↓
15. Updates task status: "completed"
   ↓
16. User monitors: provisioning workflow status abc-123
    → Shows: "Server wuji created successfully"

Example 3: Batch Workflow with Dependencies

Complex Workflow:

provisioning batch submit multi-cloud-deployment.ncl

# Workflow contains:
- Create 5 servers (parallel)
- Install Kubernetes on servers (depends on server creation)
- Deploy applications (depends on Kubernetes)

# Detailed Flow:
1. CLI submits Nickel workflow to orchestrator
   ↓
2. Orchestrator parses workflow
   ↓
3. Builds dependency graph using petgraph (Rust)
   ↓
4. Topological sort determines execution order
   ↓
5. Creates tasks for each operation
   ↓
6. Executes in parallel where possible:

   [Server 1] [Server 2] [Server 3] [Server 4] [Server 5]
       ↓          ↓          ↓          ↓          ↓
   (All execute in parallel via Nushell subprocesses)
       ↓          ↓          ↓          ↓          ↓
       └──────────┴──────────┴──────────┴──────────┘
                           │
                           ↓
                    [All servers ready]
                           ↓
                  [Install Kubernetes]
                  (Nushell subprocess)
                           ↓
                  [Kubernetes ready]
                           ↓
                  [Deploy applications]
                  (Nushell subprocess)
                           ↓
                       [Complete]

7. Orchestrator checkpoints state at each step
   ↓
8. If failure occurs, can retry from checkpoint
   ↓
9. User monitors real-time: provisioning batch monitor <id>

Why This Architecture

Orchestrator Benefits

1. Eliminates Deep Call Stack Issues

   
   Without Orchestrator:
   template.nu → calls → cluster.nu → calls → taskserv.nu → calls → provider.nu
   (Deep nesting causes "Type not supported" errors)
   
   With Orchestrator:
   Orchestrator → spawns → Nushell subprocess (flat execution)
   (No deep nesting, fresh Nushell context for each task)
   
   

2. Performance Optimization

   // Orchestrator executes tasks in parallel
   let tasks = vec![task1, task2, task3, task4, task5];
   
   let results = futures::future::join_all(
       tasks.iter().map(|t| execute_task(t))
   ).await;
   
   // 5 Nushell subprocesses run concurrently

3. Reliable State Management

   Orchestrator maintains:
   - Task queue (survives crashes)
   - Workflow checkpoints (resume on failure)
   - Progress tracking (real-time monitoring)
   - Retry logic (automatic recovery)

1. Clean Separation

   Orchestrator (Rust):     Performance, concurrency, state
   Business Logic (Nushell): Providers, taskservs, workflows

   Each does what it's best at!

Why NOT Pure Rust

Question: Why not implement everything in Rust?

Answer:

1. Nushell is perfect for infrastructure automation:

   • Shell-like scripting for system operations
   • Built-in structured data handling
   • Easy template rendering
   • Readable business logic

2. Rapid iteration:

   • Change Nushell scripts without recompiling
   • Community can contribute Nushell modules
   • Template-based configuration generation

3. Best of both worlds:

   • Rust: Performance, type safety, concurrency
   • Nushell: Flexibility, readability, ease of use

Multi-Repo Integration Example

Installation

User installs bundle:

curl -fsSL https://get.provisioning.io | sh

# Installs:
1. provisioning-core-3.2.1.tar.gz
   → /usr/local/bin/provisioning (Nushell CLI)
   → /usr/local/lib/provisioning/ (Nushell libraries)
   → /usr/local/share/provisioning/ (configs, templates)

2. provisioning-platform-2.5.3.tar.gz
   → /usr/local/bin/provisioning-orchestrator (Rust binary)
   → /usr/local/share/provisioning/platform/ (platform configs)

3. Sets up systemd/launchd service for orchestrator

Runtime Coordination

Core package expects orchestrator:

# core/nulib/lib_provisioning/orchestrator/client.nu

# Check if orchestrator is running
export def orchestrator-available [] {
    let config = (load-config)
    let endpoint = $config.orchestrator.endpoint

    try {
        let response = (http get $"($endpoint)/health")
        $response.status == "healthy"
    } catch {
        false
    }
}

# Auto-start orchestrator if needed
export def ensure-orchestrator [] {
    if not (orchestrator-available) {
        if (load-config).orchestrator.auto_start {
            print "Starting orchestrator..."
            ^provisioning-orchestrator --daemon
            sleep 2sec
        }
    }
}

Platform package executes core scripts:

// platform/orchestrator/src/executor/nushell.rs

pub struct NushellExecutor {
    provisioning_lib: PathBuf,  // /usr/local/lib/provisioning
    nu_binary: PathBuf,          // nu (from PATH)
}

impl NushellExecutor {
    pub async fn execute_script(&self, script: &str) -> Result<Output> {
        Command::new(&self.nu_binary)
            .env("NU_LIB_DIRS", &self.provisioning_lib)
            .arg("-c")
            .arg(script)
            .output()
            .await
    }

    pub async fn execute_module_function(
        &self,
        module: &str,
        function: &str,
        args: &[String],
    ) -> Result<Output> {
        let script = format!(
            "use {}/{}; {} {}",
            self.provisioning_lib.display(),
            module,
            function,
            args.join(" ")
        );

        self.execute_script(&script).await
    }
}

Configuration Examples

Core Package Config

/usr/local/share/provisioning/config/config.defaults.toml:

[orchestrator]
enabled = true
endpoint = "http://localhost:9090"
timeout_seconds = 60
auto_start = true
fallback_to_direct = true

[execution]
# Modes: "direct", "orchestrated", "auto"
default_mode = "auto"  # Auto-detect based on complexity

# Operations that always use orchestrator
force_orchestrated = [
    "server.create",
    "cluster.create",
    "batch.*",
    "workflow.*"
]

# Operations that always run direct
force_direct = [
    "*.list",
    "*.show",
    "help",
    "version"
]

Platform Package Config

/usr/local/share/provisioning/platform/config.toml:

[server]
host = "127.0.0.1"
port = 8080

[storage]
backend = "filesystem"  # or "surrealdb"
data_dir = "/var/lib/provisioning/orchestrator"

[executor]
max_concurrent_tasks = 10
task_timeout_seconds = 3600
checkpoint_interval_seconds = 30

[nushell]
binary = "nu"  # Expects nu in PATH
provisioning_lib = "/usr/local/lib/provisioning"
env_vars = { NU_LIB_DIRS = "/usr/local/lib/provisioning" }

Key Takeaways

1. Orchestrator is Essential

• Solves deep call stack problems
• Provides performance optimization
• Enables complex workflows
• NOT optional for production use

2. Integration is Loose but Coordinated

• No code dependencies between repos
• Runtime integration via CLI + REST API
• Configuration-driven coordination
• Works in both monorepo and multi-repo

3. Best of Both Worlds

• Rust: High-performance coordination
• Nushell: Flexible business logic
• Clean separation of concerns
• Each technology does what it’s best at

4. Multi-Repo Doesn’t Change Integration

• Same runtime model as monorepo
• Package installation sets up paths
• Configuration enables discovery
• Versioning ensures compatibility

Conclusion

The confusing example in the multi-repo doc was oversimplified. The real architecture is:

✅ Orchestrator IS USED and IS ESSENTIAL
✅ Platform (Rust) coordinates Core (Nushell) execution
✅ Loose coupling via CLI + REST API (not code dependencies)
✅ Works identically in monorepo and multi-repo
✅ Configuration-based integration (no hardcoded paths)

The orchestrator provides:

• Performance layer (async, parallel execution)
• Workflow engine (complex dependencies)
• State management (checkpoints, recovery)
• Task queue (reliable execution)

While Nushell provides:

• Business logic (providers, taskservs, clusters)
• Template rendering (Jinja2 via nu_plugin_tera)
• Configuration management (KCL integration)
• User-facing scripting

Multi-repo just splits WHERE the code lives, not HOW it works together.

Multi-Repository Architecture with OCI Registry Support

Version: 1.0.0 Date: 2025-10-06 Status: Implementation Complete

Overview

This document describes the multi-repository architecture for the provisioning system, enabling modular development, independent versioning, and distributed extension management through OCI registry integration.

Architecture Goals

1. Separation of Concerns: Core, Extensions, and Platform in separate repositories
2. Independent Versioning: Each component can be versioned and released independently
3. Distributed Development: Multiple teams can work on different repositories
4. OCI-Native Distribution: Extensions distributed as OCI artifacts
5. Dependency Management: Automated dependency resolution across repositories
6. Backward Compatibility: Support legacy monorepo structure during transition

Repository Structure

Repository 1: provisioning-core

Purpose: Core system functionality - CLI, libraries, base schemas

provisioning-core/
├── core/
│   ├── cli/                    # Command-line interface
│   │   ├── provisioning        # Main CLI entry point
│   │   └── module-loader       # Dynamic module loader
│   ├── nulib/                  # Core Nushell libraries
│   │   ├── lib_provisioning/   # Core library modules
│   │   │   ├── config/         # Configuration management
│   │   │   ├── oci/            # OCI client integration
│   │   │   ├── dependencies/   # Dependency resolution
│   │   │   ├── module/         # Module system
│   │   │   ├── layer/          # Layer system
│   │   │   └── workspace/      # Workspace management
│   │   └── workflows/          # Core workflow system
│   ├── plugins/                # System plugins
│   └── scripts/                # Utility scripts
├── schemas/                    # Base Nickel schemas
│   ├── main.ncl                # Main schema entry
│   ├── lib.ncl                 # Core library types
│   ├── settings.ncl            # Settings schema
│   ├── dependencies.ncl        # Dependency schemas (with OCI support)
│   ├── server.ncl              # Server schemas
│   ├── cluster.ncl             # Cluster schemas
│   └── workflows.ncl           # Workflow schemas
├── config/                     # Core configuration templates
├── templates/                  # Core templates
├── tools/                      # Build and distribution tools
│   ├── oci-package.nu          # OCI packaging tool
│   ├── build-core.nu           # Core build script
│   └── release-core.nu         # Core release script
├── tests/                      # Core system tests
└── docs/                       # Core documentation
    ├── api/                    # API documentation
    ├── architecture/           # Architecture docs
    └── development/            # Development guides

Distribution:

• Published as OCI artifact: oci://registry/provisioning-core:v3.5.0
• Contains all core functionality needed to run the provisioning system
• Version format: v{major}.{minor}.{patch} (for example, v3.5.0)

CI/CD:

• Build on commit to main
• Publish OCI artifact on git tag (v*)
• Run integration tests before publishing
• Update changelog automatically

Repository 2: provisioning-extensions

Purpose: All provider, taskserv, and cluster extensions

provisioning-extensions/
├── providers/
│   ├── aws/
│   │   ├── schemas/            # Nickel schemas
│   │   │   ├── manifest.toml   # Nickel dependencies
│   │   │   ├── aws.ncl         # Main provider schema
│   │   │   ├── defaults_aws.ncl # AWS defaults
│   │   │   └── server_aws.ncl  # AWS server schema
│   │   ├── scripts/            # Nushell scripts
│   │   │   └── install.nu      # Installation script
│   │   ├── templates/          # Provider templates
│   │   ├── docs/               # Provider documentation
│   │   └── manifest.yaml       # Extension manifest
│   ├── upcloud/
│   │   └── (same structure)
│   └── local/
│       └── (same structure)
├── taskservs/
│   ├── kubernetes/
│   │   ├── schemas/
│   │   │   ├── manifest.toml
│   │   │   ├── kubernetes.ncl  # Main taskserv schema
│   │   │   ├── version.ncl     # Version management
│   │   │   └── dependencies.ncl # Taskserv dependencies
│   │   ├── scripts/
│   │   │   ├── install.nu      # Installation script
│   │   │   ├── check.nu        # Health check script
│   │   │   └── uninstall.nu    # Uninstall script
│   │   ├── templates/          # Config templates
│   │   ├── docs/               # Taskserv docs
│   │   ├── tests/              # Taskserv tests
│   │   └── manifest.yaml       # Extension manifest
│   ├── containerd/
│   ├── cilium/
│   ├── postgres/
│   └── (50+ more taskservs...)
├── clusters/
│   ├── buildkit/
│   │   └── (same structure)
│   ├── web/
│   └── (other clusters...)
├── tools/
│   ├── extension-builder.nu   # Build individual extensions
│   ├── mass-publish.nu         # Publish all extensions
│   └── validate-extensions.nu # Validate all extensions
└── docs/
    ├── extension-guide.md      # Extension development guide
    └── publishing.md           # Publishing guide

Distribution: Each extension published separately as OCI artifact:

• oci://registry/provisioning-extensions/kubernetes:1.28.0
• oci://registry/provisioning-extensions/aws:2.0.0
• oci://registry/provisioning-extensions/buildkit:0.12.0

Extension Manifest (manifest.yaml):

name: kubernetes
type: taskserv
version: 1.28.0
description: Kubernetes container orchestration platform
author: Provisioning Team
license: MIT
homepage: https://kubernetes.io
repository: https://gitea.example.com/provisioning-extensions/kubernetes

dependencies:
  containerd: ">=1.7.0"
  etcd: ">=3.5.0"

tags:
  - kubernetes
  - container-orchestration
  - cncf

platforms:
  - linux/amd64
  - linux/arm64

min_provisioning_version: "3.0.0"

CI/CD:

• Build and publish each extension independently
• Git tag format: {extension-type}/{extension-name}/v{version}
  • Example: taskservs/kubernetes/v1.28.0
• Automated publishing to OCI registry on tag
• Run extension-specific tests before publishing

Repository 3: provisioning-platform

Purpose: Platform services (orchestrator, control-center, MCP server, API gateway)

provisioning-platform/
├── orchestrator/               # Rust orchestrator service
│   ├── src/
│   ├── Cargo.toml
│   ├── Dockerfile
│   └── README.md
├── control-center/             # Web control center
│   ├── src/
│   ├── package.json
│   ├── Dockerfile
│   └── README.md
├── mcp-server/                 # Model Context Protocol server
│   ├── src/
│   ├── Cargo.toml
│   ├── Dockerfile
│   └── README.md
├── api-gateway/                # REST API gateway
│   ├── src/
│   ├── Cargo.toml
│   ├── Dockerfile
│   └── README.md
├── docker-compose.yml          # Local development stack
├── kubernetes/                 # K8s deployment manifests
│   ├── orchestrator.yaml
│   ├── control-center.yaml
│   ├── mcp-server.yaml
│   └── api-gateway.yaml
└── docs/
    ├── deployment.md
    └── api-reference.md

Distribution: Standard Docker images in OCI registry:

• oci://registry/provisioning-platform/orchestrator:v1.2.0
• oci://registry/provisioning-platform/control-center:v1.2.0
• oci://registry/provisioning-platform/mcp-server:v1.0.0
• oci://registry/provisioning-platform/api-gateway:v1.0.0

CI/CD:

• Build Docker images on commit to main
• Publish images on git tag (v*)
• Multi-architecture builds (amd64, arm64)
• Security scanning before publishing

OCI Registry Integration

Registry Structure

OCI Registry (localhost:5000 or harbor.company.com)
├── provisioning-core/
│   ├── v3.5.0                  # Core system artifact
│   ├── v3.4.0
│   └── latest -> v3.5.0
├── provisioning-extensions/
│   ├── kubernetes:1.28.0       # Individual extension artifacts
│   ├── kubernetes:1.27.0
│   ├── containerd:1.7.0
│   ├── aws:2.0.0
│   ├── upcloud:1.5.0
│   └── (100+ more extensions)
└── provisioning-platform/
    ├── orchestrator:v1.2.0     # Platform service images
    ├── control-center:v1.2.0
    ├── mcp-server:v1.0.0
    └── api-gateway:v1.0.0

OCI Artifact Structure

Each extension packaged as OCI artifact:

kubernetes-1.28.0.tar.gz
├── schemas/                    # Nickel schemas
│   ├── kubernetes.ncl
│   ├── version.ncl
│   └── dependencies.ncl
├── scripts/                    # Nushell scripts
│   ├── install.nu
│   ├── check.nu
│   └── uninstall.nu
├── templates/                  # Template files
│   ├── kubeconfig.j2
│   └── kubelet-config.yaml.j2
├── docs/                       # Documentation
│   └── README.md
├── manifest.yaml               # Extension manifest
└── oci-manifest.json           # OCI manifest metadata

Dependency Management

Workspace Configuration

File: workspace/config/provisioning.yaml

# Core system dependency
dependencies:
  core:
    source: "oci://harbor.company.com/provisioning-core:v3.5.0"
    # Alternative: source: "gitea://provisioning-core"

  # Extensions repository configuration
  extensions:
    source_type: "oci"          # oci, gitea, local

    # OCI registry configuration
    oci:
      registry: "localhost:5000"
      namespace: "provisioning-extensions"
      tls_enabled: false
      auth_token_path: "~/.provisioning/tokens/oci"

    # Loaded extension modules
    modules:
      providers:
        - "oci://localhost:5000/provisioning-extensions/aws:2.0.0"
        - "oci://localhost:5000/provisioning-extensions/upcloud:1.5.0"

      taskservs:
        - "oci://localhost:5000/provisioning-extensions/kubernetes:1.28.0"
        - "oci://localhost:5000/provisioning-extensions/containerd:1.7.0"
        - "oci://localhost:5000/provisioning-extensions/cilium:1.14.0"

      clusters:
        - "oci://localhost:5000/provisioning-extensions/buildkit:0.12.0"

  # Platform services
  platform:
    source_type: "oci"

    oci:
      registry: "harbor.company.com"
      namespace: "provisioning-platform"

      images:
        orchestrator: "harbor.company.com/provisioning-platform/orchestrator:v1.2.0"
        control_center: "harbor.company.com/provisioning-platform/control-center:v1.2.0"

  # OCI registry configuration
  registry:
    type: "oci"                 # oci, gitea, http

    oci:
      endpoint: "localhost:5000"
      namespaces:
        extensions: "provisioning-extensions"
        nickel: "provisioning-nickel"
        platform: "provisioning-platform"
        test: "provisioning-test"

Dependency Resolution

The system resolves dependencies in this order:

1. Parse Configuration: Read provisioning.yaml and extract dependencies
2. Resolve Core: Ensure core system version is compatible
3. Resolve Extensions: For each extension:
   • Check if already installed and version matches
   • Pull from OCI registry if needed
   • Recursively resolve extension dependencies
4. Validate Graph: Check for dependency cycles and conflicts
5. Install: Install extensions in topological order

Dependency Resolution Commands

# Resolve and install all dependencies
provisioning dep resolve

# Check for dependency updates
provisioning dep check-updates

# Update specific extension
provisioning dep update kubernetes

# Validate dependency graph
provisioning dep validate

# Show dependency tree
provisioning dep tree kubernetes

OCI Client Operations

CLI Commands

# Pull extension from OCI registry
provisioning oci pull kubernetes:1.28.0

# Push extension to OCI registry
provisioning oci push ./extensions/kubernetes kubernetes 1.28.0

# List available extensions
provisioning oci list --namespace provisioning-extensions

# Search for extensions
provisioning oci search kubernetes

# Show extension versions
provisioning oci tags kubernetes

# Inspect extension manifest
provisioning oci inspect kubernetes:1.28.0

# Login to OCI registry
provisioning oci login localhost:5000 --username _token --password-stdin

# Delete extension
provisioning oci delete kubernetes:1.28.0

# Copy extension between registries
provisioning oci copy \
  localhost:5000/provisioning-extensions/kubernetes:1.28.0 \
  harbor.company.com/provisioning-extensions/kubernetes:1.28.0

OCI Configuration

# Show OCI configuration
provisioning oci config

# Output:
{
  tool: "oras"  # or "crane" or "skopeo"
  registry: "localhost:5000"
  namespace: {
    extensions: "provisioning-extensions"
    platform: "provisioning-platform"
  }
  cache_dir: "~/.provisioning/oci-cache"
  tls_enabled: false
}

Extension Development Workflow

1. Develop Extension

# Create new extension from template
provisioning generate extension taskserv redis

# Directory structure created:
# extensions/taskservs/redis/
# ├── schemas/
# │   ├── manifest.toml
# │   ├── redis.ncl
# │   ├── version.ncl
# │   └── dependencies.ncl
# ├── scripts/
# │   ├── install.nu
# │   ├── check.nu
# │   └── uninstall.nu
# ├── templates/
# ├── docs/
# │   └── README.md
# ├── tests/
# └── manifest.yaml

2. Test Extension Locally

# Load extension from local path
provisioning module load taskserv workspace_dev redis --source local

# Test installation
provisioning taskserv create redis --infra test-env --check

# Run extension tests
provisioning test extension redis

3. Package Extension

# Validate extension structure
provisioning oci package validate ./extensions/taskservs/redis

# Package as OCI artifact
provisioning oci package ./extensions/taskservs/redis

# Output: redis-1.0.0.tar.gz

4. Publish Extension

# Login to registry (one-time)
provisioning oci login localhost:5000

# Publish extension
provisioning oci push ./extensions/taskservs/redis redis 1.0.0

# Verify publication
provisioning oci tags redis

# Output:
# ┬───────────┬─────────┬───────────────────────────────────────────────────┐
# │ artifact  │ version │ reference                                         │
# ├───────────┼─────────┼───────────────────────────────────────────────────┤
# │ redis     │ 1.0.0   │ localhost:5000/provisioning-extensions/redis:1.0.0│
# └───────────┴─────────┴───────────────────────────────────────────────────┘

5. Use Published Extension

# Add to workspace configuration
# workspace/config/provisioning.yaml:
# dependencies:
#   extensions:
#     modules:
#       taskservs:
#         - "oci://localhost:5000/provisioning-extensions/redis:1.0.0"

# Pull and install
provisioning dep resolve

# Extension automatically downloaded and installed

Registry Deployment Options

Local Registry (Solo Development)

Using Zot (lightweight OCI registry):

# Start local OCI registry
provisioning oci-registry start

# Configuration:
# - Endpoint: localhost:5000
# - Storage: ~/.provisioning/oci-registry/
# - No authentication by default
# - TLS disabled (local only)

# Stop registry
provisioning oci-registry stop

# Check status
provisioning oci-registry status

Remote Registry (Multi-User/Enterprise)

Using Harbor:

# workspace/config/provisioning.yaml
dependencies:
  registry:
    type: "oci"
    oci:
      endpoint: "https://harbor.company.com"
      namespaces:
        extensions: "provisioning/extensions"
        platform: "provisioning/platform"
      tls_enabled: true
      auth_token_path: "~/.provisioning/tokens/harbor"

Features:

• Multi-user authentication
• Role-based access control (RBAC)
• Vulnerability scanning
• Replication across registries
• Webhook notifications
• Image signing (cosign/notation)

Migration from Monorepo

Phase 1: Parallel Structure (Current)

• Monorepo still exists and works
• OCI distribution layer added on top
• Extensions can be loaded from local or OCI
• No breaking changes

Phase 2: Gradual Migration

# Migrate extensions one by one
for ext in (ls provisioning/extensions/taskservs); do
  provisioning oci publish $ext.name
done

# Update workspace configurations to use OCI
provisioning workspace migrate-to-oci workspace_prod

Phase 3: Repository Split

1. Create provisioning-core repository

   • Extract core/ and schemas/ directories
   • Set up CI/CD for core publishing
   • Publish initial OCI artifact

2. Create provisioning-extensions repository

   • Extract extensions/ directory
   • Set up CI/CD for extension publishing
   • Publish all extensions to OCI registry

3. Create provisioning-platform repository

   • Extract platform/ directory
   • Set up Docker image builds
   • Publish platform services

4. Update workspaces

   • Reconfigure to use OCI dependencies
   • Test multi-repo setup
   • Verify all functionality works

Phase 4: Deprecate Monorepo

• Archive monorepo
• Redirect to new repositories
• Update documentation
• Announce migration complete

Benefits Summary

Modularity

✅ Independent repositories for core, extensions, and platform ✅ Extensions can be developed and versioned separately ✅ Clear ownership and responsibility boundaries

Distribution

✅ OCI-native distribution (industry standard) ✅ Built-in versioning with OCI tags ✅ Efficient caching with OCI layers ✅ Works with standard tools (skopeo, crane, oras)

Security

✅ TLS support for registries ✅ Authentication and authorization ✅ Vulnerability scanning (Harbor) ✅ Image signing (cosign, notation) ✅ RBAC for access control

Developer Experience

✅ Simple CLI commands for extension management ✅ Automatic dependency resolution ✅ Local testing before publishing ✅ Easy extension discovery and installation

Operations

✅ Air-gapped deployments (mirror OCI registry) ✅ Bandwidth efficient (only download what’s needed) ✅ Version pinning for reproducibility ✅ Rollback support (use previous versions)

Ecosystem

✅ Compatible with existing OCI tooling ✅ Can use public registries (DockerHub, GitHub, etc.) ✅ Mirror to multiple registries ✅ Replication for high availability

Implementation Status

Related Documentation

• OCI Packaging Tool - Extension packaging
• OCI Client Library - OCI operations
• Dependency Resolver - Dependency management
• Nickel Schemas - Type definitions
• Extension Development Guide - How to create extensions

Maintained By: Architecture Team Review Cycle: Quarterly Next Review: 2026-01-06

Multi-Repository Strategy Analysis

Date: 2025-10-01 Status: Strategic Analysis Related: Repository Distribution Analysis

Executive Summary

This document analyzes a multi-repository strategy as an alternative to the monorepo approach. After careful consideration of the provisioning system’s architecture, a hybrid approach with 4 core repositories is recommended, avoiding submodules in favor of a cleaner package-based dependency model.

Repository Architecture Options

Option A: Pure Monorepo (Original Recommendation)

Single repository: provisioning

Pros:

• Simplest development workflow
• Atomic cross-component changes
• Single version number
• One CI/CD pipeline

Cons:

• Large repository size
• Mixed language tooling (Rust + Nushell)
• All-or-nothing updates
• Unclear ownership boundaries

Option B: Multi-Repo with Submodules (❌ Not Recommended)

Repositories:

• provisioning-core (main, contains submodules)
• provisioning-platform (submodule)
• provisioning-extensions (submodule)
• provisioning-workspace (submodule)

Why Not Recommended:

• Submodule hell: complex, error-prone workflows
• Detached HEAD issues
• Update synchronization nightmares
• Clone complexity for users
• Difficult to maintain version compatibility
• Poor developer experience

Option C: Multi-Repo with Package Dependencies (✅ RECOMMENDED)

Independent repositories with package-based integration:

• provisioning-core - Nushell libraries and Nickel schemas
• provisioning-platform - Rust services (orchestrator, control-center, MCP)
• provisioning-extensions - Extension marketplace/catalog
• provisioning-workspace - Project templates and examples
• provisioning-distribution - Release automation and packaging

Why Recommended:

• Clean separation of concerns
• Independent versioning and release cycles
• Language-specific tooling and workflows
• Clear ownership boundaries
• Package-based dependencies (no submodules)
• Easier community contributions

Recommended Multi-Repo Architecture

Repository 1: provisioning-core

Purpose: Core Nushell infrastructure automation engine

Contents:

provisioning-core/
├── nulib/                   # Nushell libraries
│   ├── lib_provisioning/    # Core library functions
│   ├── servers/             # Server management
│   ├── taskservs/           # Task service management
│   ├── clusters/            # Cluster management
│   └── workflows/           # Workflow orchestration
├── cli/                     # CLI entry point
│   └── provisioning         # Pure Nushell CLI
├── schemas/                 # Nickel schemas
│   ├── main.ncl
│   ├── settings.ncl
│   ├── server.ncl
│   ├── cluster.ncl
│   └── workflows.ncl
├── config/                  # Default configurations
│   └── config.defaults.toml
├── templates/               # Core templates
├── tools/                   # Build and packaging tools
├── tests/                   # Core tests
├── docs/                    # Core documentation
├── LICENSE
├── README.md
├── CHANGELOG.md
└── version.toml             # Core version file

Technology: Nushell, Nickel Primary Language: Nushell Release Frequency: Monthly (stable) Ownership: Core team Dependencies: None (foundation)

Package Output:

• provisioning-core-{version}.tar.gz - Installable package
• Published to package registry

Installation Path:

/usr/local/
├── bin/provisioning
├── lib/provisioning/
└── share/provisioning/

Repository 2: provisioning-platform

Purpose: High-performance Rust platform services

Contents:

provisioning-platform/
├── orchestrator/            # Rust orchestrator
│   ├── src/
│   ├── tests/
│   ├── benches/
│   └── Cargo.toml
├── control-center/          # Web control center (Leptos)
│   ├── src/
│   ├── tests/
│   └── Cargo.toml
├── mcp-server/              # Model Context Protocol server
│   ├── src/
│   ├── tests/
│   └── Cargo.toml
├── api-gateway/             # REST API gateway
│   ├── src/
│   ├── tests/
│   └── Cargo.toml
├── shared/                  # Shared Rust libraries
│   ├── types/
│   └── utils/
├── docs/                    # Platform documentation
├── Cargo.toml               # Workspace root
├── Cargo.lock
├── LICENSE
├── README.md
└── CHANGELOG.md

Technology: Rust, WebAssembly Primary Language: Rust Release Frequency: Bi-weekly (fast iteration) Ownership: Platform team Dependencies:

• provisioning-core (runtime integration, loose coupling)

Package Output:

• provisioning-platform-{version}.tar.gz - Binaries
• Binaries for: Linux (x86_64, arm64), macOS (x86_64, arm64)

Installation Path:

/usr/local/
├── bin/
│   ├── provisioning-orchestrator
│   └── provisioning-control-center
└── share/provisioning/platform/

Integration with Core:

• Platform services call provisioning CLI via subprocess
• No direct code dependencies
• Communication via REST API and file-based queues
• Core and Platform can be deployed independently

Repository 3: provisioning-extensions

Purpose: Extension marketplace and community modules

Contents:

provisioning-extensions/
├── registry/                # Extension registry
│   ├── index.json          # Searchable index
│   └── catalog/            # Extension metadata
├── providers/               # Additional cloud providers
│   ├── azure/
│   ├── gcp/
│   ├── digitalocean/
│   └── hetzner/
├── taskservs/               # Community task services
│   ├── databases/
│   │   ├── mongodb/
│   │   ├── redis/
│   │   └── cassandra/
│   ├── development/
│   │   ├── gitlab/
│   │   ├── jenkins/
│   │   └── sonarqube/
│   └── observability/
│       ├── prometheus/
│       ├── grafana/
│       └── loki/
├── clusters/                # Cluster templates
│   ├── ml-platform/
│   ├── data-pipeline/
│   └── gaming-backend/
├── workflows/               # Workflow templates
├── tools/                   # Extension development tools
├── docs/                    # Extension development guide
├── LICENSE
└── README.md

Technology: Nushell, Nickel Primary Language: Nushell Release Frequency: Continuous (per-extension) Ownership: Community + Core team Dependencies:

• provisioning-core (extends core functionality)

Package Output:

• Individual extension packages: provisioning-ext-{name}-{version}.tar.gz
• Registry index for discovery

Installation:

# Install extension via core CLI
provisioning extension install mongodb
provisioning extension install azure-provider

Extension Structure: Each extension is self-contained:

mongodb/
├── manifest.toml           # Extension metadata
├── taskserv.nu             # Implementation
├── templates/              # Templates
├── schemas/                # Nickel schemas
├── tests/                  # Tests
└── README.md

Repository 4: provisioning-workspace

Purpose: Project templates and starter kits

Contents:

provisioning-workspace/
├── templates/               # Workspace templates
│   ├── minimal/            # Minimal starter
│   ├── kubernetes/         # Full K8s cluster
│   ├── multi-cloud/        # Multi-cloud setup
│   ├── microservices/      # Microservices platform
│   ├── data-platform/      # Data engineering
│   └── ml-ops/             # MLOps platform
├── examples/               # Complete examples
│   ├── blog-deployment/
│   ├── e-commerce/
│   └── saas-platform/
├── blueprints/             # Architecture blueprints
├── docs/                   # Template documentation
├── tools/                  # Template scaffolding
│   └── create-workspace.nu
├── LICENSE
└── README.md

Technology: Configuration files, Nickel Primary Language: TOML, Nickel, YAML Release Frequency: Quarterly (stable templates) Ownership: Community + Documentation team Dependencies:

• provisioning-core (templates use core)
• provisioning-extensions (may reference extensions)

Package Output:

• provisioning-templates-{version}.tar.gz

Usage:

# Create workspace from template
provisioning workspace init my-project --template kubernetes

# Or use separate tool
gh repo create my-project --template provisioning-workspace
cd my-project
provisioning workspace init

Repository 5: provisioning-distribution

Purpose: Release automation, packaging, and distribution infrastructure

Contents:

provisioning-distribution/
├── release-automation/      # Automated release workflows
│   ├── build-all.nu        # Build all packages
│   ├── publish.nu          # Publish to registries
│   └── validate.nu         # Validation suite
├── installers/             # Installation scripts
│   ├── install.nu          # Nushell installer
│   ├── install.sh          # Bash installer
│   └── install.ps1         # PowerShell installer
├── packaging/              # Package builders
│   ├── core/
│   ├── platform/
│   └── extensions/
├── registry/               # Package registry backend
│   ├── api/               # Registry REST API
│   └── storage/           # Package storage
├── ci-cd/                  # CI/CD configurations
│   ├── github/            # GitHub Actions
│   ├── gitlab/            # GitLab CI
│   └── jenkins/           # Jenkins pipelines
├── version-management/     # Cross-repo version coordination
│   ├── versions.toml      # Version matrix
│   └── compatibility.toml  # Compatibility matrix
├── docs/                   # Distribution documentation
│   ├── release-process.md
│   └── packaging-guide.md
├── LICENSE
└── README.md

Technology: Nushell, Bash, CI/CD Primary Language: Nushell, YAML Release Frequency: As needed Ownership: Release engineering team Dependencies: All repositories (orchestrates releases)

Responsibilities:

• Build packages from all repositories
• Coordinate multi-repo releases
• Publish to package registries
• Manage version compatibility
• Generate release notes
• Host package registry

Dependency and Integration Model

Package-Based Dependencies (Not Submodules)

┌─────────────────────────────────────────────────────────────┐
│                  provisioning-distribution                   │
│              (Release orchestration & registry)              │
└──────────────────────────┬──────────────────────────────────┘
                           │ publishes packages
                           ↓
                    ┌──────────────┐
                    │   Registry   │
                    └──────┬───────┘
                           │
        ┌──────────────────┼──────────────────┐
        ↓                  ↓                  ↓
┌───────────────┐  ┌──────────────┐  ┌──────────────┐
│  provisioning │  │ provisioning │  │ provisioning │
│     -core     │  │  -platform   │  │  -extensions │
└───────┬───────┘  └──────┬───────┘  └──────┬───────┘
        │                 │                  │
        │                 │ depends on       │ extends
        │                 └─────────┐        │
        │                           ↓        │
        └───────────────────────────────────→┘
                    runtime integration

Integration Mechanisms

1. Core ↔ Platform Integration

Method: Loose coupling via CLI + REST API

# Platform calls Core CLI (subprocess)
def create-server [name: string] {
    # Orchestrator executes Core CLI
    ^provisioning server create $name --infra production
}

# Core calls Platform API (HTTP)
def submit-workflow [workflow: record] {
    http post http://localhost:9090/workflows/submit $workflow
}

Version Compatibility:

# platform/Cargo.toml
[package.metadata.provisioning]
core-version = "^3.0"  # Compatible with core 3.x

2. Core ↔ Extensions Integration

Method: Plugin/module system

# Extension manifest
# extensions/mongodb/manifest.toml
[extension]
name = "mongodb"
version = "1.0.0"
type = "taskserv"
core-version = "^3.0"

[dependencies]
provisioning-core = "^3.0"

# Extension installation
# Core downloads and validates extension
provisioning extension install mongodb
# → Downloads from registry
# → Validates compatibility
# → Installs to ~/.provisioning/extensions/mongodb

3. Workspace Templates

Method: Git templates or package templates

# Option 1: GitHub template repository
gh repo create my-infra --template provisioning-workspace
cd my-infra
provisioning workspace init

# Option 2: Template package
provisioning workspace create my-infra --template kubernetes
# → Downloads template package
# → Scaffolds workspace
# → Initializes configuration

Version Management Strategy

Semantic Versioning Per Repository

Each repository maintains independent semantic versioning:

provisioning-core:       3.2.1
provisioning-platform:   2.5.3
provisioning-extensions: (per-extension versioning)
provisioning-workspace:  1.4.0

Compatibility Matrix

provisioning-distribution/version-management/versions.toml:

# Version compatibility matrix
[compatibility]

# Core versions and compatible platform versions
[compatibility.core]
"3.2.1" = { platform = "^2.5", extensions = "^1.0", workspace = "^1.0" }
"3.2.0" = { platform = "^2.4", extensions = "^1.0", workspace = "^1.0" }
"3.1.0" = { platform = "^2.3", extensions = "^0.9", workspace = "^1.0" }

# Platform versions and compatible core versions
[compatibility.platform]
"2.5.3" = { core = "^3.2", min-core = "3.2.0" }
"2.5.0" = { core = "^3.1", min-core = "3.1.0" }

# Release bundles (tested combinations)
[bundles]

[bundles.stable-3.2]
name = "Stable 3.2 Bundle"
release-date = "2025-10-15"
core = "3.2.1"
platform = "2.5.3"
extensions = ["mongodb@1.2.0", "redis@1.1.0", "azure@2.0.0"]
workspace = "1.4.0"

[bundles.lts-3.1]
name = "LTS 3.1 Bundle"
release-date = "2025-09-01"
lts-until = "2026-09-01"
core = "3.1.5"
platform = "2.4.8"
workspace = "1.3.0"

Release Coordination

Coordinated releases for major versions:

# Major release: All repos release together
provisioning-core:     3.0.0
provisioning-platform: 2.0.0
provisioning-workspace: 1.0.0

# Minor/patch releases: Independent
provisioning-core:     3.1.0 (adds features, platform stays 2.0.x)
provisioning-platform: 2.1.0 (improves orchestrator, core stays 3.1.x)

Development Workflow

Working on Single Repository

# Developer working on core only
git clone https://github.com/yourorg/provisioning-core
cd provisioning-core

# Install dependencies
just install-deps

# Development
just dev-check
just test

# Build package
just build

# Test installation locally
just install-dev

Working Across Repositories

# Scenario: Adding new feature requiring core + platform changes

# 1. Clone both repositories
git clone https://github.com/yourorg/provisioning-core
git clone https://github.com/yourorg/provisioning-platform

# 2. Create feature branches
cd provisioning-core
git checkout -b feat/batch-workflow-v2

cd ../provisioning-platform
git checkout -b feat/batch-workflow-v2

# 3. Develop with local linking
cd provisioning-core
just install-dev  # Installs to /usr/local/bin/provisioning

cd ../provisioning-platform
# Platform uses system provisioning CLI (local dev version)
cargo run

# 4. Test integration
cd ../provisioning-core
just test-integration

cd ../provisioning-platform
cargo test

# 5. Create PRs in both repositories
# PR #123 in provisioning-core
# PR #456 in provisioning-platform (references core PR)

# 6. Coordinate merge
# Merge core PR first, cut release 3.3.0
# Update platform dependency to core 3.3.0
# Merge platform PR, cut release 2.6.0

Testing Cross-Repo Integration

# Integration tests in provisioning-distribution
cd provisioning-distribution

# Test specific version combination
just test-integration \
    --core 3.3.0 \
    --platform 2.6.0

# Test bundle
just test-bundle stable-3.3

Distribution Strategy

Individual Repository Releases

Each repository releases independently:

# Core release
cd provisioning-core
git tag v3.2.1
git push --tags
# → GitHub Actions builds package
# → Publishes to package registry

# Platform release
cd provisioning-platform
git tag v2.5.3
git push --tags
# → GitHub Actions builds binaries
# → Publishes to package registry

Bundle Releases (Coordinated)

Distribution repository creates tested bundles:

cd provisioning-distribution

# Create bundle
just create-bundle stable-3.2 \
    --core 3.2.1 \
    --platform 2.5.3 \
    --workspace 1.4.0

# Test bundle
just test-bundle stable-3.2

# Publish bundle
just publish-bundle stable-3.2
# → Creates meta-package with all components
# → Publishes bundle to registry
# → Updates documentation

User Installation Options

Option 1: Bundle Installation (Recommended for Users)

# Install stable bundle (easiest)
curl -fsSL https://get.provisioning.io | sh

# Installs:
# - provisioning-core 3.2.1
# - provisioning-platform 2.5.3
# - provisioning-workspace 1.4.0

Option 2: Individual Component Installation

# Install only core (minimal)
curl -fsSL https://get.provisioning.io/core | sh

# Add platform later
provisioning install platform

# Add extensions
provisioning extension install mongodb

Option 3: Custom Combination

# Install specific versions
provisioning install core@3.1.0
provisioning install platform@2.4.0

Repository Ownership and Contribution Model

Core Team Ownership

Contribution Workflow

For Core:

1. Create issue in provisioning-core
2. Discuss design
3. Submit PR with tests
4. Strict code review
5. Merge to main
6. Release when ready

For Extensions:

1. Create extension in provisioning-extensions
2. Follow extension guidelines
3. Submit PR
4. Community review
5. Merge and publish to registry
6. Independent versioning

For Platform:

1. Create issue in provisioning-platform
2. Implement with benchmarks
3. Submit PR
4. Performance review
5. Merge and release

CI/CD Strategy

Per-Repository CI/CD

Core CI (provisioning-core/.github/workflows/ci.yml):

name: Core CI

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install Nushell
        run: cargo install nu
      - name: Run tests
        run: just test
      - name: Validate Nickel schemas
        run: just validate-nickel

  package:
    runs-on: ubuntu-latest
    if: startsWith(github.ref, 'refs/tags/v')
    steps:
      - uses: actions/checkout@v3
      - name: Build package
        run: just build
      - name: Publish to registry
        run: just publish
        env:
          REGISTRY_TOKEN: ${{ secrets.REGISTRY_TOKEN }}

Platform CI (provisioning-platform/.github/workflows/ci.yml):

name: Platform CI

on: [push, pull_request]

jobs:
  test:
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v3
      - name: Build
        run: cargo build --release
      - name: Test
        run: cargo test --workspace
      - name: Benchmark
        run: cargo bench

  cross-compile:
    runs-on: ubuntu-latest
    if: startsWith(github.ref, 'refs/tags/v')
    steps:
      - uses: actions/checkout@v3
      - name: Build for Linux x86_64
        run: cargo build --release --target x86_64-unknown-linux-gnu
      - name: Build for Linux arm64
        run: cargo build --release --target aarch64-unknown-linux-gnu
      - name: Publish binaries
        run: just publish-binaries

Integration Testing (Distribution Repo)

Distribution CI (provisioning-distribution/.github/workflows/integration.yml):

name: Integration Tests

on:
  schedule:
    - cron: '0 0 * * *'  # Daily
  workflow_dispatch:

jobs:
  test-bundle:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install bundle
        run: |
          nu release-automation/install-bundle.nu stable-3.2

      - name: Run integration tests
        run: |
          nu tests/integration/test-all.nu

      - name: Test upgrade path
        run: |
          nu tests/integration/test-upgrade.nu 3.1.0 3.2.1

File and Directory Structure Comparison

Monorepo Structure

provisioning/                          (One repo, ~500 MB)
├── core/                             (Nushell)
├── platform/                         (Rust)
├── extensions/                       (Community)
├── workspace/                        (Templates)
└── distribution/                     (Build)

Multi-Repo Structure

provisioning-core/                     (Repo 1, ~50 MB)
├── nulib/
├── cli/
├── schemas/
└── tools/

provisioning-platform/                 (Repo 2, ~150 MB with target/)
├── orchestrator/
├── control-center/
├── mcp-server/
└── Cargo.toml

provisioning-extensions/               (Repo 3, ~100 MB)
├── registry/
├── providers/
├── taskservs/
└── clusters/

provisioning-workspace/                (Repo 4, ~20 MB)
├── templates/
├── examples/
└── blueprints/

provisioning-distribution/             (Repo 5, ~30 MB)
├── release-automation/
├── installers/
├── packaging/
└── registry/

Decision Matrix