provisioning/docs/src/getting-started/installation-validation-guide.md

# Installation Validation & Bootstrap Guide

**Objective**: Validate your provisioning installation, run bootstrap to initialize the workspace, and verify all components are working correctly.

**Expected Duration**: 30-45 minutes

**Prerequisites**: Fresh clone of provisioning repository at `/Users/Akasha/project-provisioning`

---

## Section 1: Prerequisites Verification

Before running the bootstrap script, verify that your system has all required dependencies.

### Step 1.1: Check System Requirements

Run these commands to verify your system meets minimum requirements:

```text
# Check OS
uname -s
# Expected: Darwin (macOS), Linux, or WSL2

# Check CPU cores
sysctl -n hw.physicalcpu  # macOS
# OR
nproc  # Linux
# Expected: 2 or more cores

# Check RAM
sysctl -n hw.memsize | awk '{print int($1 / 1024 / 1024 / 1024) " GB"}' # macOS
# OR
grep MemTotal /proc/meminfo | awk '{print int($2 / 1024 / 1024) " GB"}'  # Linux
# Expected: 2 GB or more (4 GB+ recommended)

# Check free disk space
df -h | grep -E '^/dev|^Filesystem'
# Expected: At least 2 GB free (10 GB+ recommended)
```

**Success Criteria**:
- OS is macOS, Linux, or WSL2
- CPU: 2+ cores available
- RAM: 2 GB minimum, 4+ GB recommended
- Disk: 2 GB free minimum

### Step 1.2: Verify Nushell Installation

Nushell is required for bootstrap and CLI operations:

```text
command -v nu
# Expected output: /path/to/nu

nu --version
# Expected output: 0.109.0 or higher
```

**If Nushell is not installed:**

```text
# macOS (using Homebrew)
brew install nushell

# Linux (Debian/Ubuntu)
sudo apt-get update && sudo apt-get install nushell

# Linux (RHEL/CentOS)
sudo yum install nushell

# Or install from source: https://nushell.sh/book/installation.html
```

### Step 1.3: Verify Nickel Installation

Nickel is required for configuration validation:

```text
command -v nickel
# Expected output: /path/to/nickel

nickel --version
# Expected output: nickel 1.x.x or higher
```

**If Nickel is not installed:**

```text
# Install via Cargo (requires Rust)
cargo install nickel-lang-cli

# Or: https://nickel-lang.org/
```

### Step 1.4: Verify Docker Installation

Docker is required for running containerized services:

```text
command -v docker
# Expected output: /path/to/docker

docker --version
# Expected output: Docker version 20.10 or higher
```

**If Docker is not installed:**

Visit [Docker installation guide](https://docs.docker.com/get-docker/) and install for your OS.

### Step 1.5: Check Provisioning Binary

Verify the provisioning CLI binary exists:

```text
ls -la /Users/Akasha/project-provisioning/provisioning/core/cli/provisioning
# Expected: -rwxr-xr-x (executable)

file /Users/Akasha/project-provisioning/provisioning/core/cli/provisioning
# Expected: ELF 64-bit or similar binary format
```

**If binary is not executable:**

```text
chmod +x /Users/Akasha/project-provisioning/provisioning/core/cli/provisioning
```

### Prerequisites Checklist

```text
[ ] OS is macOS, Linux, or WSL2
[ ] CPU: 2+ cores available
[ ] RAM: 2 GB minimum installed
[ ] Disk: 2+ GB free space
[ ] Nushell 0.109.0+ installed
[ ] Nickel 1.x.x installed
[ ] Docker 20.10+ installed
[ ] Provisioning binary exists and is executable
```

---

## Section 2: Bootstrap Installation

The bootstrap script automates 7 stages of installation and initialization. Run it from the project root directory.

### Step 2.1: Navigate to Project Root

```text
cd /Users/Akasha/project-provisioning
```

### Step 2.2: Run Bootstrap Script

```text
./provisioning/bootstrap/install.sh
```

### Bootstrap Output

You should see output similar to this:

```text
╔════════════════════════════════════════════════════════════════╗
║              PROVISIONING BOOTSTRAP (Bash)                     ║
╚════════════════════════════════════════════════════════════════╝

📊 Stage 1: System Detection
─────────────────────────────────────────────────────────────────
  OS: Darwin
  Architecture: arm64 (or x86_64)
  CPU Cores: 8
  Memory: 16 GB
  ✅ System requirements met

📦 Stage 2: Checking Dependencies
─────────────────────────────────────────────────────────────────
  Versions:
    Docker: Docker version 28.5.2
    Rust: rustc 1.75.0
    Nushell: 0.109.1
  ✅ All dependencies found

📁 Stage 3: Creating Directory Structure
─────────────────────────────────────────────────────────────────
  ✅ Directory structure created

⚙️  Stage 4: Validating Configuration
─────────────────────────────────────────────────────────────────
  ✅ Configuration syntax valid

📤 Stage 5: Exporting Configuration to TOML
─────────────────────────────────────────────────────────────────
  ✅ Configuration exported

🚀 Stage 6: Initializing Orchestrator Service
─────────────────────────────────────────────────────────────────
  ✅ Orchestrator started

✅ Stage 7: Verification
─────────────────────────────────────────────────────────────────
  ✅ All configuration files generated
  ✅ All required directories created

╔════════════════════════════════════════════════════════════════╗
║                   BOOTSTRAP COMPLETE ✅                        ║
╚════════════════════════════════════════════════════════════════╝

📍 Next Steps:

1. Verify configuration:
   cat /Users/Akasha/project-provisioning/workspaces/workspace_librecloud/config/config.ncl

2. Check orchestrator is running:
   curl http://localhost:9090/health

3. Start provisioning:
   provisioning server create --infra sgoyol --name web-01
```

### What Bootstrap Does

The bootstrap script automatically:

1. **Detects your system** (OS, CPU, RAM, architecture)
2. **Verifies dependencies** (Docker, Rust, Nushell)
3. **Creates workspace directories** (config, state, cache)
4. **Validates Nickel configuration** (syntax checking)
5. **Exports configuration** (Nickel → TOML files)
6. **Initializes orchestrator** (starts service in background)
7. **Verifies installation** (checks all files created)

---

## Section 3: Installation Validation

After bootstrap completes, verify that all components are working correctly.

### Step 3.1: Verify Workspace Directories

Bootstrap should have created workspace directories. Verify they exist:

```text
cd /Users/Akasha/project-provisioning

# Check all required directories
ls -la workspaces/workspace_librecloud/.orchestrator/data/queue/
ls -la workspaces/workspace_librecloud/.kms/
ls -la workspaces/workspace_librecloud/.providers/
ls -la workspaces/workspace_librecloud/.taskservs/
ls -la workspaces/workspace_librecloud/.clusters/
```

**Expected Output**:
```text
total 0
drwxr-xr-x  2 user  group  64 Jan  7 10:30 .

(directories exist and are accessible)
```

### Step 3.2: Verify Generated Configuration Files

Bootstrap should have exported Nickel configuration to TOML format:

```text
# Check generated files exist
ls -la workspaces/workspace_librecloud/config/generated/

# View workspace configuration
cat workspaces/workspace_librecloud/config/generated/workspace.toml

# View provider configuration
cat workspaces/workspace_librecloud/config/generated/providers/upcloud.toml

# View orchestrator configuration
cat workspaces/workspace_librecloud/config/generated/platform/orchestrator.toml
```

**Expected Output**:
```text
config/
├── generated/
│   ├── workspace.toml
│   ├── providers/
│   │   └── upcloud.toml
│   └── platform/
│       └── orchestrator.toml
```

### Step 3.3: Type-Check Nickel Configuration

Verify Nickel configuration files have valid syntax:

```text
cd /Users/Akasha/project-provisioning/workspaces/workspace_librecloud

# Type-check main workspace config
nickel typecheck config/config.ncl
# Expected: No output (success) or clear error messages

# Type-check infrastructure configs
nickel typecheck infra/wuji/main.ncl
nickel typecheck infra/sgoyol/main.ncl

# Use workspace utility for comprehensive validation
nu workspace.nu validate
# Expected: ✓ All files validated successfully

# Type-check all Nickel files
nu workspace.nu typecheck
```

**Expected Output**:
```text
✓ All files validated successfully
✓ infra/wuji/main.ncl
✓ infra/sgoyol/main.ncl
```

### Step 3.4: Verify Orchestrator Service

The orchestrator service manages workflows and deployments:

```text
# Check if orchestrator is running (health check)
curl http://localhost:9090/health
# Expected: {"status": "healthy"} or similar response

# If health check fails, check orchestrator logs
tail -f /Users/Akasha/project-provisioning/provisioning/platform/orchestrator/data/orchestrator.log

# Alternative: Check if orchestrator process is running
ps aux | grep orchestrator
# Expected: Running orchestrator process visible
```

**Expected Output**:
```text
{
  "status": "healthy",
  "uptime": "0:05:23"
}
```

**If Orchestrator Failed to Start:**

Check logs and restart manually:

```text
cd /Users/Akasha/project-provisioning/provisioning/platform/orchestrator

# Check log file
cat data/orchestrator.log

# Or start orchestrator manually
./scripts/start-orchestrator.nu --background

# Verify it's running
curl http://localhost:9090/health
```

### Step 3.5: Install Provisioning CLI (Optional)

You can install the provisioning CLI globally for easier access:

```text
# Option A: System-wide installation (requires sudo)
cd /Users/Akasha/project-provisioning
sudo ./scripts/install-provisioning.sh

# Verify installation
provisioning --version
provisioning help

# Option B: Add to PATH temporarily (current session only)
export PATH="$PATH:/Users/Akasha/project-provisioning/provisioning/core/cli"

# Verify
provisioning --version
```

**Expected Output**:
```text
provisioning version 1.0.0

Usage: provisioning [OPTIONS] COMMAND

Commands:
  server     - Server management
  workspace  - Workspace management
  config     - Configuration management
  help       - Show help information
```

### Installation Validation Checklist

```text
[ ] Workspace directories created (.orchestrator, .kms, .providers, .taskservs, .clusters)
[ ] Generated TOML files exist in config/generated/
[ ] Nickel type-checking passes (no errors)
[ ] Workspace utility validation passes
[ ] Orchestrator responding to health check
[ ] Orchestrator process running
[ ] Provisioning CLI accessible and working
```

---

## Section 4: Troubleshooting

This section covers common issues and solutions.

### Issue: "Nushell not found"

**Symptoms**:
```text
./provisioning/bootstrap/install.sh: line X: nu: command not found
```

**Solution**:
1. Install Nushell (see Step 1.2)
2. Verify installation: `nu --version`
3. Retry bootstrap script

### Issue: "Nickel configuration validation failed"

**Symptoms**:
```text
⚙️  Stage 4: Validating Configuration
Error: Nickel configuration validation failed
```

**Solution**:
1. Check Nickel syntax: `nickel typecheck config/config.ncl`
2. Review error message for specific issue
3. Edit config file: `vim config/config.ncl`
4. Run bootstrap again

### Issue: "Docker not installed"

**Symptoms**:
```text
❌ Docker is required but not installed
```

**Solution**:
1. Install Docker: [Docker installation guide](https://docs.docker.com/get-docker/)
2. Verify: `docker --version`
3. Retry bootstrap script

### Issue: "Configuration export failed"

**Symptoms**:
```text
⚠️ Configuration export encountered issues (may continue)
```

**Solution**:
1. Check Nushell library paths: `nu -c "use provisioning/core/nulib/lib_provisioning/config/export.nu *"`
2. Verify export library exists: `ls provisioning/core/nulib/lib_provisioning/config/export.nu`
3. Re-export manually:
   ```bash
   cd /Users/Akasha/project-provisioning
   nu -c "
     use provisioning/core/nulib/lib_provisioning/config/export.nu *
     export-all-configs 'workspaces/workspace_librecloud'
   "
   ```

### Issue: "Orchestrator didn't start"

**Symptoms**:
```text
🚀 Stage 6: Initializing Orchestrator Service
⚠️ Orchestrator may not have started (check logs)

curl http://localhost:9090/health
# Connection refused
```

**Solution**:
1. Check for port conflicts: `lsof -i :9090`
2. If port 9090 is in use, either:
   - Stop the conflicting service
   - Change orchestrator port in configuration
3. Check logs: `tail -f provisioning/platform/orchestrator/data/orchestrator.log`
4. Start manually: `cd provisioning/platform/orchestrator && ./scripts/start-orchestrator.nu --background`
5. Verify: `curl http://localhost:9090/health`

### Issue: "Sudo password prompt during bootstrap"

**Symptoms**:
```text
Stage 3: Creating Directory Structure
[sudo] password for user:
```

**Solution**:
- This is normal if creating directories in system locations
- Enter your sudo password when prompted
- Or: Run bootstrap from home directory instead

### Issue: "Permission denied" on binary

**Symptoms**:
```text
bash: ./provisioning/bootstrap/install.sh: Permission denied
```

**Solution**:
```text
# Make script executable
chmod +x /Users/Akasha/project-provisioning/provisioning/bootstrap/install.sh

# Retry
./provisioning/bootstrap/install.sh
```

---

## Section 5: Next Steps

After successful installation validation, you can:

### Option 1: Deploy workspace_librecloud

To deploy infrastructure to UpCloud:

```text
# Read workspace deployment guide
cat workspaces/workspace_librecloud/docs/deployment-guide.md

# Or: From workspace directory
cd workspaces/workspace_librecloud
cat docs/deployment-guide.md
```

### Option 2: Create a New Workspace

To create a new workspace for different infrastructure:

```text
provisioning workspace init my_workspace --template minimal
```

### Option 3: Explore Available Modules

Discover what's available to deploy:

```text
# List available task services
provisioning mod discover taskservs

# List available providers
provisioning mod discover providers

# List available clusters
provisioning mod discover clusters
```

---

## Section 6: Verification Checklist

After completing all steps, verify with this final checklist:

```text
Prerequisites Verified:
  [ ] OS is macOS, Linux, or WSL2
  [ ] CPU: 2+ cores
  [ ] RAM: 2+ GB available
  [ ] Disk: 2+ GB free
  [ ] Nushell 0.109.0+ installed
  [ ] Nickel 1.x.x installed
  [ ] Docker 20.10+ installed
  [ ] Provisioning binary executable

Bootstrap Completed:
  [ ] All 7 stages completed successfully
  [ ] No error messages in output
  [ ] Installation log shows success

Installation Validated:
  [ ] Workspace directories exist
  [ ] Generated TOML files exist
  [ ] Nickel type-checking passes
  [ ] Workspace validation passes
  [ ] Orchestrator health check passes
  [ ] Provisioning CLI works (if installed)

Ready to Deploy:
  [ ] No errors in validation steps
  [ ] All services responding correctly
  [ ] Configuration properly exported
```

---

## Getting Help

If you encounter issues not covered here:

1. **Check logs**: `tail -f provisioning/platform/orchestrator/data/orchestrator.log`
2. **Enable debug mode**: `provisioning --debug <command>`
3. **Review bootstrap output**: Scroll up to see detailed error messages
4. **Check documentation**: `provisioning help` or `provisioning guide <topic>`
5. **Workspace guide**: `cat workspaces/workspace_librecloud/docs/deployment-guide.md`

---

## Summary

This guide covers:
- ✅ Prerequisites verification (Nushell, Nickel, Docker)
- ✅ Bootstrap installation (7-stage automated process)
- ✅ Installation validation (directories, configs, services)
- ✅ Troubleshooting common issues
- ✅ Next steps for deployment

You now have a fully installed and validated provisioning system ready for workspace deployment.