syntaxis/scripts/provisioning/README.md

# syntaxis Provisioning System

A modular, offline-first provisioning system for building, packaging, and distributing syntaxis binaries and configurations across multiple platforms.

**This system focuses on LOCAL and SINGLE-MACHINE provisioning. For remote/multi-host deployment, see integration with [project-provisioning](https://github.com/Akasha/project-provisioning).**

## Overview

The provisioning system is built on **modular NuShell scripts** that operate **independently** while supporting orchestrated workflows. Each component can be used standalone or combined through the `provctl` orchestrator.

### Key Features

- ✅ **Service Catalog**: Central registry of all available services with metadata
- ✅ **Service Discovery**: Tools to explore available services and presets
- ✅ **Modular Design**: Each script can be used independently
- ✅ **Offline-First**: Create self-contained bundles with all artifacts
- ✅ **Multi-Platform**: Support for Linux, macOS, Windows (x86_64, ARM64)
- ✅ **Configuration-Driven**: TOML-based configuration (no hardcoded values)
- ✅ **Integrity Verification**: SHA256 checksums for all artifacts
- ✅ **Backup & Safety**: Automatic backups before overwriting
- ✅ **Manifest Tracking**: Installation tracking via TOML manifests
- ✅ **Local Service Management**: Support for local service startup and monitoring

## Architecture

```
syntaxis/
├── configs/
│   ├── services-catalog.toml      ← Central service registry (NEW)
│   ├── installation.toml           ← Presets and local config
│   └── provisioning/services/      ← Service definitions
│
└── scripts/provisioning/
    ├── service-catalog.nu          ← Service discovery tool (NEW)
    ├── pack.nu                      ← Create distribution bundles
    ├── unpack.nu                    ← Extract bundles
    ├── install.nu                   ← Install binaries locally
    ├── deploy.nu                    ← Deploy configurations
    ├── install-with-presets.nu      ← Local installation with presets
    ├── provctl.nu                   ← Orchestrator for local workflows
    ├── common/
    │   ├── platform.nu              ← OS/arch detection
    │   ├── manifest.nu              ← Manifest I/O
    │   └── validate.nu              ← Integrity checking
    └── README.md                    ← This file
```

## What's New: Service Catalog System

**The service catalog is the bridge between syntaxis and project-provisioning.**

### Service Registry (`configs/services-catalog.toml`)

Centralized specification of all syntaxis services:

```toml
[service.syntaxis-api]
name = "syntaxis-api"
display_name = "syntaxis REST API"
description = "REST API server"
type = "server"

[service.syntaxis-api.server]
default_host = "127.0.0.1"
default_port = 3000
scheme = "http"

[service.syntaxis-api.usage]
health_check_endpoint = "/health"
examples = [
    "syntaxis-api --bind 127.0.0.1:3000",
    "curl http://127.0.0.1:3000/health",
]
```

### Service Discovery Tool (`scripts/provisioning/service-catalog.nu`)

Explore what's available:

```bash
# List all services
nu scripts/provisioning/service-catalog.nu list

# Get details about a service
nu scripts/provisioning/service-catalog.nu info syntaxis-api

# See how to use a service
nu scripts/provisioning/service-catalog.nu usage syntaxis-api

# Show what's in a preset
nu scripts/provisioning/service-catalog.nu preset dev

# View port assignments
nu scripts/provisioning/service-catalog.nu ports

# Export for project-provisioning
nu scripts/provisioning/service-catalog.nu export staging
```

### Why This Matters

The service catalog allows:

1. **Users to discover** what services are available and how to use them
2. **Developers to understand** service dependencies and configurations
3. **project-provisioning to automatically** deploy services with correct:
   - Port assignments
   - Health checks
   - Resource limits
   - Dependencies
   - Startup ordering

**Without hardcoding anything in project-provisioning!**

## Configuration

The system is configured via `configs/provisioning.toml`. Key sections:

### Targets
Define supported platforms and their properties:
```toml
[targets.x86_64-unknown-linux-gnu]
enabled = true
os = "linux"
arch = "x86_64"
libc = "glibc"
formats = ["tar.gz", "deb"]
default_prefix = "/usr/local"
```

### Artifacts
Specify what gets included in bundles:
```toml
[artifacts]
binaries = [
    { name = "syntaxis-cli", crate = "cli" },
    { name = "syntaxis-tui", crate = "tui" },
    { name = "syntaxis-api", crate = "api" },
]
configs = ["configs/provisioning.toml", "configs/database-default.toml", ...]
docs = ["README.md", "docs/installation.md"]
```

### Packaging Strategies
Configure how bundles are created:
```toml
[packaging.tar_gz]
enabled = true
format = "tar.gz"
compression = "gzip"
platforms = ["linux", "macos"]
```

## Commands

### 1. pack.nu - Create Bundles

Creates distribution bundles from built binaries.

```bash
# Create bundle for current platform
nu scripts/provisioning/pack.nu

# Create for specific target
nu scripts/provisioning/pack.nu --target x86_64-unknown-linux-gnu

# Create with specific format
nu scripts/provisioning/pack.nu --target aarch64-apple-darwin --format zip

# List available targets
nu scripts/provisioning/pack.nu --list-targets

# Dry run
nu scripts/provisioning/pack.nu --dry-run
```

**Options:**
- `--target <TARGET>` - Target triple (default: current platform)
- `--format <FORMAT>` - Package format: tar.gz, zip, deb, rpm (default: tar.gz)
- `--output <DIR>` - Output directory (default: dist/)
- `--verify` - Verify bundle after creation
- `--list-targets` - Show available targets
- `--dry-run` - Preview without creating

**Output:**
```
dist/
├── syntaxis-v0.1.0-x86_64-linux.tar.gz    # The bundle
├── syntaxis-v0.1.0-x86_64-linux.checksums.toml  # SHA256 checksums
```

### 2. unpack.nu - Extract Bundles

Extracts and validates bundle contents.

```bash
# Extract to current directory
nu scripts/provisioning/unpack.nu bundle.tar.gz

# Extract to specific location with verification
nu scripts/provisioning/unpack.nu bundle.tar.gz --dest /tmp/syntaxis --verify

# Dry run (preview)
nu scripts/provisioning/unpack.nu bundle.tar.gz --dry-run
```

**Options:**
- `--dest <DIR>` - Destination directory (default: current)
- `--verify` - Verify checksums after extraction
- `--dry-run` - Preview without extracting

**Output:**
```
destination/
└── syntaxis-v0.1.0-x86_64-linux/
    ├── manifest.toml
    ├── bin/
    │   ├── syntaxis-cli
    │   ├── syntaxis-tui
    │   └── syntaxis-api
    ├── configs/
    │   ├── provisioning.toml
    │   └── ...
    └── docs/
        ├── README.md
        └── ...
```

### 3. install.nu - Install Binaries

Installs binaries to system from extracted bundles.

```bash
# Install from extracted bundle
nu scripts/provisioning/install.nu --source ./syntaxis-v0.1.0

# Install to custom prefix
nu scripts/provisioning/install.nu --prefix /opt/syntaxis

# Install to home directory with symlinks
nu scripts/provisioning/install.nu --prefix ~/.local --create-links

# Install with backup of existing binaries
nu scripts/provisioning/install.nu --backup

# Force overwrite without prompting
nu scripts/provisioning/install.nu --force
```

**Options:**
- `--source <DIR>` - Source directory containing binaries (default: bin/)
- `--prefix <PATH>` - Installation prefix (default: /usr/local or ~/.local)
- `--create-links` - Create symlinks in ~/.local/bin
- `--backup` - Backup existing binaries before overwriting
- `--force` - Force overwrite without prompting
- `--dry-run` - Preview installation

**Creates:**
- Binaries in `<prefix>/bin/`
- Installation manifest in `~/.syntaxis/manifest.toml`

### 4. deploy.nu - Deploy Configurations

Deploys configuration files to system.

```bash
# Deploy to default location
nu scripts/provisioning/deploy.nu --source ./syntaxis-bundle/configs

# Deploy to custom directory
nu scripts/provisioning/deploy.nu --source ./configs --config-dir /etc/syntaxis

# Deploy with backup
nu scripts/provisioning/deploy.nu --backup

# Dry run
nu scripts/provisioning/deploy.nu --dry-run
```

**Options:**
- `--source <DIR>` - Source configs directory (default: configs/)
- `--config-dir <PATH>` - Target location (default: ~/.config/syntaxis)
- `--backup` - Backup existing configs before overwriting
- `--force` - Force overwrite without prompting
- `--dry-run` - Preview deployment

**Creates:**
- Config files in `<config-dir>/`
- Deployment manifest in `~/.syntaxis/deployment.toml`

### 5. provctl.nu - Orchestrator

Coordinates multiple provisioning operations.

```bash
# Show help
nu scripts/provisioning/provctl.nu help

# Build a bundle
nu scripts/provisioning/provctl.nu build-bundle --target x86_64-linux

# Extract and install a bundle
nu scripts/provisioning/provctl.nu install-bundle --bundle bundle.tar.gz

# Complete setup: extract + install + deploy
nu scripts/provisioning/provctl.nu full-install --bundle bundle.tar.gz --verify

# Check installation status
nu scripts/provisioning/provctl.nu status

# List available targets
nu scripts/provisioning/provctl.nu list-targets

# Verify bundle integrity
nu scripts/provisioning/provctl.nu verify-bundle --bundle bundle.tar.gz

# Clean staging directories
nu scripts/provisioning/provctl.nu clean
```

## Workflow Examples

### Scenario 1: Build and Distribute Bundle (Developer)

```bash
# 1. Create bundle for all platforms
nu scripts/provisioning/pack.nu --target x86_64-unknown-linux-gnu --output dist/
nu scripts/provisioning/pack.nu --target aarch64-apple-darwin --output dist/
nu scripts/provisioning/pack.nu --target x86_64-pc-windows-msvc --output dist/

# 2. Bundles are ready in dist/
ls dist/
# → syntaxis-v0.1.0-x86_64-linux.tar.gz
# → syntaxis-v0.1.0-aarch64-darwin.tar.gz
# → syntaxis-v0.1.0-windows.zip
```

### Scenario 2: Install from Bundle (User - Offline)

```bash
# 1. User receives bundle: syntaxis-v0.1.0-x86_64-linux.tar.gz

# 2. Option A: Step-by-step (see what's happening)
nu scripts/provisioning/unpack.nu bundle.tar.gz --dest /tmp
nu scripts/provisioning/install.nu --source /tmp/syntaxis-v0.1.0/bin --prefix ~/.local
nu scripts/provisioning/deploy.nu --source /tmp/syntaxis-v0.1.0/configs

# 3. Option B: Orchestrated (one command)
nu scripts/provisioning/provctl.nu full-install --bundle bundle.tar.gz --prefix ~/.local

# 4. Verify installation
nu scripts/provisioning/provctl.nu status
syntaxis-cli --version
```

### Scenario 3: Deploy to Team Infrastructure

```bash
# On CI/CD machine (build server)
nu scripts/provisioning/pack.nu --target x86_64-linux --output releases/

# On target machines (multiple VMs/servers)
for machine in server1 server2 server3; do
    scp releases/syntaxis-v0.1.0-x86_64-linux.tar.gz $machine:/tmp/
    ssh $machine "nu provisioning/provctl.nu full-install --bundle /tmp/bundle.tar.gz"
done
```

### Scenario 4: Upgrade Existing Installation

```bash
# User already has syntaxis installed

# 1. Get new bundle
curl -O https://releases.example.com/syntaxis-v0.2.0-x86_64-linux.tar.gz

# 2. Install with backup
nu scripts/provisioning/unpack.nu bundle.tar.gz --dest /tmp --verify
nu scripts/provisioning/install.nu --source /tmp/syntaxis-v0.2.0/bin --backup --force
nu scripts/provisioning/deploy.nu --source /tmp/syntaxis-v0.2.0/configs --backup

# 3. Verify new version
syntaxis-cli --version
```

## Common Tasks

### List Available Build Targets

```bash
nu scripts/provisioning/pack.nu --list-targets
```

### Verify Bundle Integrity

```bash
nu scripts/provisioning/provctl.nu verify-bundle --bundle bundle.tar.gz
```

### Check Installation Status

```bash
nu scripts/provisioning/provctl.nu status
```

### Create Dry Run

```bash
# See what would be packed
nu scripts/provisioning/pack.nu --dry-run

# See what would be installed
nu scripts/provisioning/install.nu --dry-run

# See what would be deployed
nu scripts/provisioning/deploy.nu --dry-run
```

### Find Backup Files

```bash
ls -la ~/.local/bin/syntaxis-*.backup.*
ls -la ~/.config/syntaxis/*.backup.*
```

### Clean Staging Directories

```bash
nu scripts/provisioning/provctl.nu clean
```

## Manifest Files

Installation tracking is done via TOML manifest files in `~/.syntaxis/`:

### Installation Manifest

`~/.syntaxis/manifest.toml` - Created by `install.nu`:

```toml
[installation]
timestamp = "2025-11-17T12:00:00Z"
prefix = "~/.local"
version = "0.1.0"

[[binaries]]
name = "syntaxis-cli"
installed_at = "2025-11-17T12:00:00Z"

[[binaries]]
name = "syntaxis-tui"
installed_at = "2025-11-17T12:00:00Z"

[[binaries]]
name = "syntaxis-api"
installed_at = "2025-11-17T12:00:00Z"
```

### Deployment Manifest

`~/.syntaxis/deployment.toml` - Created by `deploy.nu`:

```toml
[deployment]
timestamp = "2025-11-17T12:00:05Z"
config_dir = "~/.config/syntaxis"
version = "0.1.0"

[[configs]]
name = "provisioning.toml"
deployed_at = "2025-11-17T12:00:05Z"
```

## Bundle Structure

### Bundle Layout

```
syntaxis-v0.1.0-x86_64-linux/
├── manifest.toml                    # Bundle metadata
├── bin/                             # Compiled binaries
│   ├── syntaxis-cli
│   ├── syntaxis-tui
│   └── syntaxis-api
├── configs/                         # Configuration files
│   ├── provisioning.toml
│   ├── database-default.toml
│   ├── cli/
│   │   └── syntaxis-cli.toml
│   ├── tui/
│   │   └── syntaxis-tui.toml
│   └── api/
│       ├── syntaxis-api.toml
│       └── features/
├── docs/                            # Documentation (offline)
│   ├── README.md
│   ├── docs/installation.md
│   └── CLAUDE.md
└── scripts/                         # Helper scripts (optional)
    └── install.sh                   # Bash installer
```

### Bundle Manifest

`manifest.toml` inside bundle:

```toml
[bundle]
version = "0.1.0"
target = "x86_64-unknown-linux-gnu"
created_at = "2025-11-17T12:00:00Z"
format = "tar.gz"

[artifacts]
binaries = ["syntaxis-cli", "syntaxis-tui", "syntaxis-api"]
configs = ["provisioning.toml", "database-default.toml", ...]
docs = ["README.md", "docs/installation.md", "CLAUDE.md"]

[checksums]
"bin/syntaxis-cli" = "sha256:abcd1234..."
"bin/syntaxis-tui" = "sha256:efgh5678..."
"bin/syntaxis-api" = "sha256:ijkl9012..."
```

## Utility Modules (common/)

### platform.nu

Functions for platform detection:

```nushell
# Detect current OS
detect-os               # → "linux", "macos", "windows"

# Detect current architecture
detect-arch             # → "x86_64", "aarch64"

# Map to cargo target triple
current-target          # → "x86_64-unknown-linux-gnu"

# Check if target is available
is-target-enabled target config_file

# List all available targets
list-targets config_file

# Get supported formats for a target
target-formats target config_file
```

### manifest.nu

Functions for manifest handling:

```nushell
# Load manifest from file
load manifest_file      # → record

# Create new manifest
create version target format

# Add checksums
add-checksums manifest checksums_map

# Verify checksums
verify-checksums manifest bundle_dir

# Validate manifest structure
validate manifest

# Save manifest to file
save manifest manifest_file

# Show manifest information
info manifest_file
```

### validate.nu

Functions for integrity checking:

```nushell
# Calculate SHA256 checksum
sha256 file_path

# Generate checksums for directory
generate-checksums source_dir

# Verify single file
verify-file-checksum file_path expected_checksum

# Verify all checksums
verify-checksums checksums_file bundle_dir

# Validate bundle structure
validate-bundle-structure bundle_file

# Validate directory permissions
validate-permissions target_dir --write true
```

## Environment Variables

Scripts respect these environment variables:

- `SYNTAXIS_CONFIG_DIR` - Configuration directory (default: ~/.config/syntaxis)
- `SYNTAXIS_DATA_DIR` - Data directory (default: ~/.local/share/syntaxis)
- `PROVISIONING_CONFIG` - Path to provisioning.toml (default: configs/provisioning.toml)

## Error Handling

All scripts provide clear error messages:

```bash
# Invalid target
Error: Unsupported combination: os=linux, arch=arm64, libc=custom

# Missing files
Error: Bundle file not found: ./bundle.tar.gz

# Permission denied
Error: Cannot write to /usr/local/bin: insufficient permissions

# Checksum mismatch
Error: Checksum verification failed:
  - bin/syntaxis-cli: mismatch
```

## Future Enhancements

- [ ] Convert scripts to Rust binaries for performance
- [ ] Homebrew formula support
- [ ] Debian/RPM package generation
- [ ] Docker image distribution
- [ ] Auto-update mechanism
- [ ] Telemetry and analytics
- [ ] Signed releases (GPG)
- [ ] Incremental/delta updates

## Troubleshooting

### Bundle creation fails

```bash
# Check that binaries are built
cargo build --release

# Verify provisioning.toml
nu scripts/provisioning/pack.nu --dry-run

# Check available targets
nu scripts/provisioning/pack.nu --list-targets
```

### Installation fails

```bash
# Check permissions
ls -la ~/.local/bin/

# Try with sudo (not recommended)
sudo nu scripts/provisioning/install.nu --prefix /usr/local

# Check manifests
cat ~/.syntaxis/manifest.toml
```

### Configuration deployment fails

```bash
# Check source directory
ls -la configs/

# Try dry run
nu scripts/provisioning/deploy.nu --dry-run

# Check target permissions
ls -la ~/.config/
```

## Contributing

To extend the provisioning system:

1. **Add new command**: Create `new-command.nu` in `scripts/provisioning/`
2. **Add utilities**: Add functions to `common/` modules
3. **Update config**: Add sections to `configs/provisioning.toml`
4. **Update provctl**: Add action to `provctl.nu`
5. **Document**: Update this README with examples

## License

Part of the syntaxis project. See LICENSE for details.

---

**Last Updated**: 2025-11-17
**Status**: MVP - Core functionality complete
**Next**: Additional package format support (deb, rpm, brew)