provisioning/docs/src/development/build-system.md

# Build System Documentation

This document provides comprehensive documentation for the provisioning project's build system, including the complete Makefile reference with 40+
targets, build tools, compilation instructions, and troubleshooting.

## Table of Contents

1. [Overview](#overview)
2. [Quick Start](#quick-start)
3. [Makefile Reference](#makefile-reference)
4. [Build Tools](#build-tools)
5. [Cross-Platform Compilation](#cross-platform-compilation)
6. [Dependency Management](#dependency-management)
7. [Troubleshooting](#troubleshooting)
8. [CI/CD Integration](#cicd-integration)

## Overview

The build system is a comprehensive, Makefile-based solution that orchestrates:

- **Rust compilation**: Platform binaries (orchestrator, control-center, etc.)
- **Nushell bundling**: Core libraries and CLI tools
- **Nickel validation**: Configuration schema validation
- **Distribution generation**: Multi-platform packages
- **Release management**: Automated release pipelines
- **Documentation generation**: API and user documentation

**Location**: `/src/tools/`
**Main entry point**: `/src/tools/Makefile`

## Quick Start

```bash
# Navigate to build system
cd src/tools

# View all available targets
make help

# Complete build and package
make all

# Development build (quick)
make dev-build

# Build for specific platform
make linux
make macos
make windows

# Clean everything
make clean

# Check build system status
make status
```

## Makefile Reference

### Build Configuration

**Variables**:

```bash
# Project metadata
PROJECT_NAME := provisioning
VERSION := $(git describe --tags --always --dirty)
BUILD_TIME := $(date -u +"%Y-%m-%dT%H:%M:%SZ")

# Build configuration
RUST_TARGET := x86_64-unknown-linux-gnu
BUILD_MODE := release
PLATFORMS := linux-amd64,macos-amd64,windows-amd64
VARIANTS := complete,minimal

# Flags
VERBOSE := false
DRY_RUN := false
PARALLEL := true
```

### Build Targets

#### Primary Build Targets

**`make all`** - Complete build, package, and test

- Runs: `clean build-all package-all test-dist`
- Use for: Production releases, complete validation

**`make build-all`** - Build all components

- Runs: `build-platform build-core validate-nickel`
- Use for: Complete system compilation

**`make build-platform`** - Build platform binaries for all targets

```bash
make build-platform
# Equivalent to:
nu tools/build/compile-platform.nu 
    --target x86_64-unknown-linux-gnu 
    --release 
    --output-dir dist/platform 
    --verbose=false
```

**`make build-core`** - Bundle core Nushell libraries

```nushell
make build-core
# Equivalent to:
nu tools/build/bundle-core.nu 
    --output-dir dist/core 
    --config-dir dist/config 
    --validate 
    --exclude-dev
```

**`make validate-nickel`** - Validate and compile Nickel schemas

```nickel
make validate-nickel
# Equivalent to:
nu tools/build/validate-nickel.nu 
    --output-dir dist/schemas 
    --format-code 
    --check-dependencies
```

**`make build-cross`** - Cross-compile for multiple platforms

- Builds for all platforms in `PLATFORMS` variable
- Parallel execution support
- Failure handling for each platform

#### Package Targets

**`make package-all`** - Create all distribution packages

- Runs: `dist-generate package-binaries package-containers`

**`make dist-generate`** - Generate complete distributions

```bash
make dist-generate
# Advanced usage:
make dist-generate PLATFORMS=linux-amd64,macos-amd64 VARIANTS=complete
```

**`make package-binaries`** - Package binaries for distribution

- Creates platform-specific archives
- Strips debug symbols
- Generates checksums

**`make package-containers`** - Build container images

- Multi-platform container builds
- Optimized layers and caching
- Version tagging

**`make create-archives`** - Create distribution archives

- TAR and ZIP formats
- Platform-specific and universal archives
- Compression and checksums

**`make create-installers`** - Create installation packages

- Shell script installers
- Platform-specific packages (DEB, RPM, MSI)
- Uninstaller creation

#### Release Targets

**`make release`** - Create a complete release (requires VERSION)

```bash
make release VERSION=2.1.0
```

Features:

- Automated changelog generation
- Git tag creation and push
- Artifact upload
- Comprehensive validation

**`make release-draft`** - Create a draft release

- Create without publishing
- Review artifacts before release
- Manual approval workflow

**`make upload-artifacts`** - Upload release artifacts

- GitHub Releases
- Container registries
- Package repositories
- Verification and validation

**`make notify-release`** - Send release notifications

- Slack notifications
- Discord announcements
- Email notifications
- Custom webhook support

**`make update-registry`** - Update package manager registries

- Homebrew formula updates
- APT repository updates
- Custom registry support

#### Development and Testing Targets

**`make dev-build`** - Quick development build

```bash
make dev-build
# Fast build with minimal validation
```

**`make test-build`** - Test build system

- Validates build process
- Runs with test configuration
- Comprehensive logging

**`make test-dist`** - Test generated distributions

- Validates distribution integrity
- Tests installation process
- Platform compatibility checks

**`make validate-all`** - Validate all components

- Nickel schema validation
- Package validation
- Configuration validation

**`make benchmark`** - Run build benchmarks

- Times build process
- Performance analysis
- Resource usage monitoring

#### Documentation Targets

**`make docs`** - Generate documentation

```bash
make docs
# Generates API docs, user guides, and examples
```

**`make docs-serve`** - Generate and serve documentation locally

- Starts local HTTP server on port 8000
- Live documentation browsing
- Development documentation workflow

#### Utility Targets

**`make clean`** - Clean all build artifacts

```bash
make clean
# Removes all build, distribution, and package directories
```

**`make clean-dist`** - Clean only distribution artifacts

- Preserves build cache
- Removes distribution packages
- Faster cleanup option

**`make install`** - Install the built system locally

- Requires distribution to be built
- Installs to system directories
- Creates uninstaller

**`make uninstall`** - Uninstall the system

- Removes system installation
- Cleans configuration
- Removes service files

**`make status`** - Show build system status

```bash
make status
# Output:
# Build System Status
# ===================
# Project: provisioning
# Version: v2.1.0-5-g1234567
# Git Commit: 1234567890abcdef
# Build Time: 2025-09-25T14:30:22Z
#
# Directories:
#   Source: /Users/user/repo-cnz/src
#   Tools: /Users/user/repo-cnz/src/tools
#   Build: /Users/user/repo-cnz/src/target
#   Distribution: /Users/user/repo-cnz/src/dist
#   Packages: /Users/user/repo-cnz/src/packages
```

**`make info`** - Show detailed system information

- OS and architecture details
- Tool versions (Nushell, Rust, Docker, Git)
- Environment information
- Build prerequisites

#### CI/CD Integration Targets

**`make ci-build`** - CI build pipeline

- Complete validation build
- Suitable for automated CI systems
- Comprehensive testing

**`make ci-test`** - CI test pipeline

- Validation and testing only
- Fast feedback for pull requests
- Quality assurance

**`make ci-release`** - CI release pipeline

- Build and packaging for releases
- Artifact preparation
- Release candidate creation

**`make cd-deploy`** - CD deployment pipeline

- Complete release and deployment
- Artifact upload and distribution
- User notifications

#### Platform-Specific Targets

**`make linux`** - Build for Linux only

```bash
make linux
# Sets PLATFORMS=linux-amd64
```

**`make macos`** - Build for macOS only

```bash
make macos
# Sets PLATFORMS=macos-amd64
```

**`make windows`** - Build for Windows only

```bash
make windows
# Sets PLATFORMS=windows-amd64
```

#### Debugging Targets

**`make debug`** - Build with debug information

```bash
make debug
# Sets BUILD_MODE=debug VERBOSE=true
```

**`make debug-info`** - Show debug information

- Make variables and environment
- Build system diagnostics
- Troubleshooting information

## Build Tools

### Core Build Scripts

All build tools are implemented as Nushell scripts with comprehensive parameter validation and error handling.

#### `/src/tools/build/compile-platform.nu`

**Purpose**: Compiles all Rust components for distribution

**Components Compiled**:

- `orchestrator` → `provisioning-orchestrator` binary
- `control-center` → `control-center` binary
- `control-center-ui` → Web UI assets
- `mcp-server-rust` → MCP integration binary

**Usage**:

```nushell
nu compile-platform.nu [options]

Options:
  --target STRING          Target platform (default: x86_64-unknown-linux-gnu)
  --release                Build in release mode
  --features STRING        Comma-separated features to enable
  --output-dir STRING      Output directory (default: dist/platform)
  --verbose                Enable verbose logging
  --clean                  Clean before building
```

**Example**:

```nushell
nu compile-platform.nu 
    --target x86_64-apple-darwin 
    --release 
    --features "surrealdb,telemetry" 
    --output-dir dist/macos 
    --verbose
```

#### `/src/tools/build/bundle-core.nu`

**Purpose**: Bundles Nushell core libraries and CLI for distribution

**Components Bundled**:

- Nushell provisioning CLI wrapper
- Core Nushell libraries (`lib_provisioning`)
- Configuration system
- Template system
- Extensions and plugins

**Usage**:

```nushell
nu bundle-core.nu [options]

Options:
  --output-dir STRING      Output directory (default: dist/core)
  --config-dir STRING      Configuration directory (default: dist/config)
  --validate               Validate Nushell syntax
  --compress               Compress bundle with gzip
  --exclude-dev            Exclude development files (default: true)
  --verbose                Enable verbose logging
```

**Validation Features**:

- Syntax validation of all Nushell files
- Import dependency checking
- Function signature validation
- Test execution (if tests present)

#### `/src/tools/build/validate-nickel.nu`

**Purpose**: Validates and compiles Nickel schemas

**Validation Process**:

1. Syntax validation of all `.ncl` files
2. Schema dependency checking
3. Type constraint validation
4. Example validation against schemas
5. Documentation generation

**Usage**:

```nushell
nu validate-nickel.nu [options]

Options:
  --output-dir STRING      Output directory (default: dist/schemas)
  --format-code            Format Nickel code during validation
  --check-dependencies     Validate schema dependencies
  --verbose                Enable verbose logging
```

#### `/src/tools/build/test-distribution.nu`

**Purpose**: Tests generated distributions for correctness

**Test Types**:

- **Basic**: Installation test, CLI help, version check
- **Integration**: Server creation, configuration validation
- **Complete**: Full workflow testing including cluster operations

**Usage**:

```nushell
nu test-distribution.nu [options]

Options:
  --dist-dir STRING        Distribution directory (default: dist)
  --test-types STRING      Test types: basic,integration,complete
  --platform STRING        Target platform for testing
  --cleanup                Remove test files after completion
  --verbose                Enable verbose logging
```

#### `/src/tools/build/clean-build.nu`

**Purpose**: Intelligent build artifact cleanup

**Cleanup Scopes**:

- **all**: Complete cleanup (build, dist, packages, cache)
- **dist**: Distribution artifacts only
- **cache**: Build cache and temporary files
- **old**: Files older than specified age

**Usage**:

```nushell
nu clean-build.nu [options]

Options:
  --scope STRING           Cleanup scope: all,dist,cache,old
  --age DURATION          Age threshold for 'old' scope (default: 7d)
  --force                  Force cleanup without confirmation
  --dry-run               Show what would be cleaned without doing it
  --verbose               Enable verbose logging
```

### Distribution Tools

#### `/src/tools/distribution/generate-distribution.nu`

**Purpose**: Main distribution generator orchestrating the complete process

**Generation Process**:

1. Platform binary compilation
2. Core library bundling
3. Nickel schema validation and packaging
4. Configuration system preparation
5. Documentation generation
6. Archive creation and compression
7. Installer generation
8. Validation and testing

**Usage**:

```nushell
nu generate-distribution.nu [command] [options]

Commands:
  <default>                Generate complete distribution
  quick                    Quick development distribution
  status                   Show generation status

Options:
  --version STRING         Version to build (default: auto-detect)
  --platforms STRING       Comma-separated platforms
  --variants STRING        Variants: complete,minimal
  --output-dir STRING      Output directory (default: dist)
  --compress               Enable compression
  --generate-docs          Generate documentation
  --parallel-builds        Enable parallel builds
  --validate-output        Validate generated output
  --verbose                Enable verbose logging
```

**Advanced Examples**:

```bash
# Complete multi-platform release
nu generate-distribution.nu 
    --version 2.1.0 
    --platforms linux-amd64,macos-amd64,windows-amd64 
    --variants complete,minimal 
    --compress 
    --generate-docs 
    --parallel-builds 
    --validate-output

# Quick development build
nu generate-distribution.nu quick 
    --platform linux 
    --variant minimal

# Status check
nu generate-distribution.nu status
```

#### `/src/tools/distribution/create-installer.nu`

**Purpose**: Creates platform-specific installers

**Installer Types**:

- **shell**: Shell script installer (cross-platform)
- **package**: Platform packages (DEB, RPM, MSI, PKG)
- **container**: Container image with provisioning
- **source**: Source distribution with build instructions

**Usage**:

```nushell
nu create-installer.nu DISTRIBUTION_DIR [options]

Options:
  --output-dir STRING      Installer output directory
  --installer-types STRING Installer types: shell,package,container,source
  --platforms STRING       Target platforms
  --include-services       Include systemd/launchd service files
  --create-uninstaller     Generate uninstaller
  --validate-installer     Test installer functionality
  --verbose                Enable verbose logging
```

### Package Tools

#### `/src/tools/package/package-binaries.nu`

**Purpose**: Packages compiled binaries for distribution

**Package Formats**:

- **archive**: TAR.GZ and ZIP archives
- **standalone**: Single binary with embedded resources
- **installer**: Platform-specific installer packages

**Features**:

- Binary stripping for size reduction
- Compression optimization
- Checksum generation (SHA256, MD5)
- Digital signing (if configured)

#### `/src/tools/package/build-containers.nu`

**Purpose**: Builds optimized container images

**Container Features**:

- Multi-stage builds for minimal image size
- Security scanning integration
- Multi-platform image generation
- Layer caching optimization
- Runtime environment configuration

### Release Tools

#### `/src/tools/release/create-release.nu`

**Purpose**: Automated release creation and management

**Release Process**:

1. Version validation and tagging
2. Changelog generation from git history
3. Asset building and validation
4. Release creation (GitHub, GitLab, etc.)
5. Asset upload and verification
6. Release announcement preparation

**Usage**:

```nushell
nu create-release.nu [options]

Options:
  --version STRING         Release version (required)
  --asset-dir STRING       Directory containing release assets
  --draft                  Create draft release
  --prerelease             Mark as pre-release
  --generate-changelog     Auto-generate changelog
  --push-tag               Push git tag
  --auto-upload            Upload assets automatically
  --verbose                Enable verbose logging
```

## Cross-Platform Compilation

### Supported Platforms

**Primary Platforms**:

- `linux-amd64` (x86_64-unknown-linux-gnu)
- `macos-amd64` (x86_64-apple-darwin)
- `windows-amd64` (x86_64-pc-windows-gnu)

**Additional Platforms**:

- `linux-arm64` (aarch64-unknown-linux-gnu)
- `macos-arm64` (aarch64-apple-darwin)
- `freebsd-amd64` (x86_64-unknown-freebsd)

### Cross-Compilation Setup

**Install Rust Targets**:

```rust
# Install additional targets
rustup target add x86_64-apple-darwin
rustup target add x86_64-pc-windows-gnu
rustup target add aarch64-unknown-linux-gnu
rustup target add aarch64-apple-darwin
```

**Platform-Specific Dependencies**:

**macOS Cross-Compilation**:

```bash
# Install osxcross toolchain
brew install FiloSottile/musl-cross/musl-cross
brew install mingw-w64
```

**Windows Cross-Compilation**:

```bash
# Install Windows dependencies
brew install mingw-w64
# or on Linux:
sudo apt-get install gcc-mingw-w64
```

### Cross-Compilation Usage

**Single Platform**:

```bash
# Build for macOS from Linux
make build-platform RUST_TARGET=x86_64-apple-darwin

# Build for Windows
make build-platform RUST_TARGET=x86_64-pc-windows-gnu
```

**Multiple Platforms**:

```bash
# Build for all configured platforms
make build-cross

# Specify platforms
make build-cross PLATFORMS=linux-amd64,macos-amd64,windows-amd64
```

**Platform-Specific Targets**:

```bash
# Quick platform builds
make linux      # Linux AMD64
make macos      # macOS AMD64
make windows    # Windows AMD64
```

## Dependency Management

### Build Dependencies

**Required Tools**:

- **Nushell 0.107.1+**: Core shell and scripting
- **Rust 1.70+**: Platform binary compilation
- **Cargo**: Rust package management
- **KCL 0.11.2+**: Configuration language
- **Git**: Version control and tagging

**Optional Tools**:

- **Docker**: Container image building
- **Cross**: Simplified cross-compilation
- **SOPS**: Secrets management
- **Age**: Encryption for secrets

### Dependency Validation

**Check Dependencies**:

```bash
make info
# Shows versions of all required tools

# Output example:
# Tool Versions:
#   Nushell: 0.107.1
#   Rust: rustc 1.75.0
#   Docker: Docker version 24.0.6
#   Git: git version 2.42.0
```

**Install Missing Dependencies**:

```bash
# Install Nushell
cargo install nu

# Install Nickel
cargo install nickel

# Install Cross (for cross-compilation)
cargo install cross
```

### Dependency Caching

**Rust Dependencies**:

- Cargo cache: `~/.cargo/registry`
- Target cache: `target/` directory
- Cross-compilation cache: `~/.cache/cross`

**Build Cache Management**:

```bash
# Clean Cargo cache
cargo clean

# Clean cross-compilation cache
cross clean

# Clean all caches
make clean SCOPE=cache
```

## Troubleshooting

### Common Build Issues

#### Rust Compilation Errors

**Error**: `linker 'cc' not found`

```bash
# Solution: Install build essentials
sudo apt-get install build-essential  # Linux
xcode-select --install                 # macOS
```

**Error**: `target not found`

```bash
# Solution: Install target
rustup target add x86_64-unknown-linux-gnu
```

**Error**: Cross-compilation linking errors

```bash
# Solution: Use cross instead of cargo
cargo install cross
make build-platform CROSS=true
```

#### Nushell Script Errors

**Error**: `command not found`

```bash
# Solution: Ensure Nushell is in PATH
which nu
export PATH="$HOME/.cargo/bin:$PATH"
```

**Error**: Permission denied

```bash
# Solution: Make scripts executable
chmod +x src/tools/build/*.nu
```

**Error**: Module not found

```bash
# Solution: Check working directory
cd src/tools
nu build/compile-platform.nu --help
```

#### Nickel Validation Errors

**Error**: `nickel command not found`

```nickel
# Solution: Install Nickel
cargo install nickel
# or
brew install nickel
```

**Error**: Schema validation failed

```bash
# Solution: Check Nickel syntax
nickel fmt schemas/
nickel check schemas/
```

### Build Performance Issues

#### Slow Compilation

**Optimizations**:

```bash
# Enable parallel builds
make build-all PARALLEL=true

# Use faster linker
export RUSTFLAGS="-C link-arg=-fuse-ld=lld"

# Increase build jobs
export CARGO_BUILD_JOBS=8
```

**Cargo Configuration** (`~/.cargo/config.toml`):

```toml
[build]
jobs = 8

[target.x86_64-unknown-linux-gnu]
linker = "lld"
```

#### Memory Issues

**Solutions**:

```bash
# Reduce parallel jobs
export CARGO_BUILD_JOBS=2

# Use debug build for development
make dev-build BUILD_MODE=debug

# Clean up between builds
make clean-dist
```

### Distribution Issues

#### Missing Assets

**Validation**:

```bash
# Test distribution
make test-dist

# Detailed validation
nu src/tools/package/validate-package.nu dist/
```

#### Size Optimization

**Optimizations**:

```bash
# Strip binaries
make package-binaries STRIP=true

# Enable compression
make dist-generate COMPRESS=true

# Use minimal variant
make dist-generate VARIANTS=minimal
```

### Debug Mode

**Enable Debug Logging**:

```bash
# Set environment
export PROVISIONING_DEBUG=true
export RUST_LOG=debug

# Run with debug
make debug

# Verbose make output
make build-all VERBOSE=true
```

**Debug Information**:

```bash
# Show debug information
make debug-info

# Build system status
make status

# Tool information
make info
```

## CI/CD Integration

### GitHub Actions

**Example Workflow** (`.github/workflows/build.yml`):

```yaml
name: Build and Test
on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Nushell
        uses: hustcer/setup-nu@v3.5

      - name: Setup Rust
        uses: actions-rs/toolchain@v1
        with:
          toolchain: stable

      - name: CI Build
        run: |
          cd src/tools
          make ci-build

      - name: Upload Artifacts
        uses: actions/upload-artifact@v4
        with:
          name: build-artifacts
          path: src/dist/
```

### Release Automation

**Release Workflow**:

```bash
name: Release
on:
  push:
    tags: ['v*']

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build Release
        run: |
          cd src/tools
          make ci-release VERSION=${{ github.ref_name }}

      - name: Create Release
        run: |
          cd src/tools
          make release VERSION=${{ github.ref_name }}
```

### Local CI Testing

**Test CI Pipeline Locally**:

```bash
# Run CI build pipeline
make ci-build

# Run CI test pipeline
make ci-test

# Full CI/CD pipeline
make ci-release
```

This build system provides a comprehensive, maintainable foundation for the provisioning project's development lifecycle, from local development to
production releases.