provisioning/docs/src/architecture/package-and-loader-system.md

# Nickel Package and Module Loader System

This document describes the package-based architecture implemented for the provisioning system, replacing hardcoded extension paths with a
flexible module discovery and loading system using Nickel for type-safe configuration.

## Architecture Overview

The system consists of two main components:

1. **Core Nickel Package**: Distributable core provisioning schemas with type safety
2. **Module Loader System**: Dynamic discovery and loading of extensions

### Benefits

- **Type-Safe Configuration**: Nickel ensures configuration validity at evaluation time
- **Clean Separation**: Core package is self-contained and distributable
- **Plug-and-Play Extensions**: Taskservs, providers, and clusters can be loaded dynamically
- **Version Management**: Core package and extensions can be versioned independently
- **Developer Friendly**: Easy workspace setup and module management with lazy evaluation

## Components

### 1. Core Nickel Package (`/provisioning/schemas/`)

Contains fundamental schemas for provisioning:

- `main.ncl` - Primary provisioning configuration
- `server.ncl` - Server definitions and schemas
- `defaults.ncl` - Default configurations
- `lib.ncl` - Common library schemas
- `dependencies.ncl` - Dependency management schemas

**Key Features:**

- No hardcoded extension paths
- Self-contained and distributable
- Type-safe package-based imports
- Lazy evaluation of expensive computations

### 2. Module Discovery System

#### Discovery Commands

```bash
# Discover available modules
module-loader discover taskservs              # List all taskservs
module-loader discover providers --format yaml # List providers as YAML
module-loader discover clusters redis          # Search for redis clusters
```

#### Supported Module Types

- **Taskservs**: Infrastructure services (kubernetes, redis, postgres, etc.)
- **Providers**: Cloud providers (upcloud, aws, local)
- **Clusters**: Complete configurations (buildkit, web, oci-reg)

### 3. Module Loading System

#### Loading Commands

```bash
# Load modules into workspace
module-loader load taskservs . [kubernetes, cilium, containerd]
module-loader load providers . [upcloud]
module-loader load clusters . [buildkit]

# Initialize workspace with modules
module-loader init workspace/infra/production 
    --taskservs [kubernetes, cilium] 
    --providers [upcloud]
```

#### Generated Files

- `taskservs.ncl` - Auto-generated taskserv imports
- `providers.ncl` - Auto-generated provider imports
- `clusters.ncl` - Auto-generated cluster imports
- `.manifest/*.yaml` - Module loading manifests

## Workspace Structure

### New Workspace Layout

```bash
workspace/infra/my-project/
├── kcl.mod                    # Package dependencies
├── servers.ncl                  # Main server configuration
├── taskservs.ncl               # Auto-generated taskserv imports
├── providers.ncl               # Auto-generated provider imports
├── clusters.ncl                # Auto-generated cluster imports
├── .taskservs/               # Loaded taskserv modules
│   ├── kubernetes/
│   ├── cilium/
│   └── containerd/
├── .providers/               # Loaded provider modules
│   └── upcloud/
├── .clusters/                # Loaded cluster modules
│   └── buildkit/
├── .manifest/                # Module manifests
│   ├── taskservs.yaml
│   ├── providers.yaml
│   └── clusters.yaml
├── data/                     # Runtime data
├── tmp/                      # Temporary files
├── resources/                # Resource definitions
└── clusters/                 # Cluster configurations
```

### Import Patterns

#### Before (Old System)

```bash
# Hardcoded relative paths
import ../../../kcl/server as server
import ../../../extensions/taskservs/kubernetes/kcl/kubernetes as k8s
```

#### After (New System)

```bash
# Package-based imports
import provisioning.server as server

# Auto-generated module imports (after loading)
import .taskservs.nclubernetes.kubernetes as k8s
```

## Package Distribution

### Building Core Package

```bash
# Build distributable package
./provisioning/tools/kcl-packager.nu build --version 1.0.0

# Install locally
./provisioning/tools/kcl-packager.nu install dist/provisioning-1.0.0.tar.gz

# Create release
./provisioning/tools/kcl-packager.nu build --format tar.gz --include-docs
```

### Package Installation Methods

#### Method 1: Local Installation (Recommended for development)

```toml
[dependencies]
provisioning = { path = "~/.kcl/packages/provisioning", version = "0.0.1" }
```

#### Method 2: Git Repository (For distributed teams)

```toml
[dependencies]
provisioning = { git = "https://github.com/your-org/provisioning-kcl", version = "v0.0.1" }
```

#### Method 3: KCL Registry (When available)

```toml
[dependencies]
provisioning = { version = "0.0.1" }
```

## Developer Workflows

### 1. New Project Setup

```bash
# Create workspace from template
cp -r provisioning/templates/workspaces/kubernetes ./my-k8s-cluster
cd my-k8s-cluster

# Initialize with modules
workspace-init.nu . init

# Load required modules
module-loader load taskservs . [kubernetes, cilium, containerd]
module-loader load providers . [upcloud]

# Validate and deploy
kcl run servers.ncl
provisioning server create --infra . --check
```

### 2. Extension Development

```bash
# Create new taskserv
mkdir -p extensions/taskservs/my-service/kcl
cd extensions/taskservs/my-service/kcl

# Initialize KCL module
kcl mod init my-service
echo 'provisioning = { path = "~/.kcl/packages/provisioning", version = "0.0.1" }' >> kcl.mod

# Develop and test
module-loader discover taskservs   # Should find your service
```

### 3. Workspace Migration

```bash
# Analyze existing workspace
workspace-migrate.nu workspace/infra/old-project dry-run

# Perform migration
workspace-migrate.nu workspace/infra/old-project

# Verify migration
module-loader validate workspace/infra/old-project
```

### 4. Multi-Environment Management

```bash
# Development environment
cd workspace/infra/dev
module-loader load taskservs . [redis, postgres]
module-loader load providers . [local]

# Production environment
cd workspace/infra/prod
module-loader load taskservs . [redis, postgres, kubernetes, monitoring]
module-loader load providers . [upcloud, aws]  # Multi-cloud
```

## Module Management

### Listing and Validation

```bash
# List loaded modules
module-loader list taskservs .
module-loader list providers .
module-loader list clusters .

# Validate workspace
module-loader validate .

# Show workspace info
workspace-init.nu . info
```

### Unloading Modules

```bash
# Remove specific modules
module-loader unload taskservs . redis
module-loader unload providers . aws

# This regenerates import files automatically
```

### Module Information

```bash
# Get detailed module info
module-loader info taskservs kubernetes
module-loader info providers upcloud
module-loader info clusters buildkit
```

## CI/CD Integration

### Pipeline Example

```nushell
#!/usr/bin/env nu
# deploy-pipeline.nu

# Install specific versions
kcl-packager.nu install --version $env.PROVISIONING_VERSION

# Load production modules
module-loader init $env.WORKSPACE_PATH 
    --taskservs $env.REQUIRED_TASKSERVS 
    --providers [$env.CLOUD_PROVIDER]

# Validate configuration
module-loader validate $env.WORKSPACE_PATH

# Deploy infrastructure
provisioning server create --infra $env.WORKSPACE_PATH
```

## Troubleshooting

### Common Issues

#### Module Import Errors

```bash
Error: module not found
```

**Solution**: Verify modules are loaded and regenerate imports

```bash
module-loader list taskservs .
module-loader load taskservs . [kubernetes, cilium, containerd]
```

#### Provider Configuration Issues

**Solution**: Check provider-specific configuration in `.providers/` directory

#### KCL Compilation Errors

**Solution**: Verify core package installation and kcl.mod configuration

```toml
kcl-packager.nu install --version latest
kcl run --dry-run servers.ncl
```

### Debug Commands

```bash
# Show workspace structure
tree -a workspace/infra/my-project

# Check generated imports
cat workspace/infra/my-project/taskservs.ncl

# Validate KCL files
nickel typecheck workspace/infra/my-project/*.ncl

# Show module manifests
cat workspace/infra/my-project/.manifest/taskservs.yaml
```

## Best Practices

### 1. Version Management

- Pin core package versions in production
- Use semantic versioning for extensions
- Test compatibility before upgrading

### 2. Module Organization

- Load only required modules to keep workspaces clean
- Use meaningful workspace names
- Document required modules in README

### 3. Security

- Exclude `.manifest/` and `data/` from version control
- Use secrets management for sensitive configuration
- Validate modules before loading in production

### 4. Performance

- Load modules at workspace initialization, not runtime
- Cache discovery results when possible
- Use parallel loading for multiple modules

## Migration Guide

For existing workspaces, follow these steps:

### 1. Backup Current Workspace

```bash
cp -r workspace/infra/existing workspace/infra/existing-backup
```

### 2. Analyze Migration Requirements

```bash
workspace-migrate.nu workspace/infra/existing dry-run
```

### 3. Perform Migration

```bash
workspace-migrate.nu workspace/infra/existing
```

### 4. Load Required Modules

```bash
cd workspace/infra/existing
module-loader load taskservs . [kubernetes, cilium]
module-loader load providers . [upcloud]
```

### 5. Test and Validate

```bash
kcl run servers.ncl
module-loader validate .
```

### 6. Deploy

```bash
provisioning server create --infra . --check
```

## Future Enhancements

- Registry-based module distribution
- Module dependency resolution
- Automatic version updates
- Module templates and scaffolding
- Integration with external package managers