provisioning/docs/package-and-loader-system.md

# KCL Package and Module Loader System

This document describes the new package-based architecture implemented for the provisioning system, replacing hardcoded extension paths with a flexible module discovery and loading system.

## Architecture Overview

The new system consists of two main components:

1. **Core KCL Package**: Distributable core provisioning schemas
2. **Module Loader System**: Dynamic discovery and loading of extensions

### Benefits

- **Clean Separation**: Core package is self-contained and distributable
- **Plug-and-Play Extensions**: Taskservs, providers, and clusters can be loaded dynamically
- **Version Management**: Core package and extensions can be versioned independently
- **Developer Friendly**: Easy workspace setup and module management

## Components

### 1. Core KCL Package (`/provisioning/kcl/`)

Contains fundamental schemas for provisioning:
- `settings.k` - System settings and configuration
- `server.k` - Server definitions and schemas
- `defaults.k` - Default configurations
- `lib.k` - Common library schemas
- `dependencies.k` - Dependency management schemas

**Key Features:**
- No hardcoded extension paths
- Self-contained and distributable
- Package-based imports only

### 2. Module Discovery System

#### Discovery Commands
```bash
# Discover available modules
module-loader discover taskservs              # List all taskservs
module-loader discover providers --format yaml # List providers as YAML
module-loader discover clusters redis          # Search for redis clusters
```

#### Supported Module Types
- **Taskservs**: Infrastructure services (kubernetes, redis, postgres, etc.)
- **Providers**: Cloud providers (upcloud, aws, local)
- **Clusters**: Complete configurations (buildkit, web, oci-reg)

### 3. Module Loading System

#### Loading Commands
```bash
# Load modules into workspace
module-loader load taskservs . [kubernetes, cilium, containerd]
module-loader load providers . [upcloud]
module-loader load clusters . [buildkit]

# Initialize workspace with modules
module-loader init workspace/infra/production \
    --taskservs [kubernetes, cilium] \
    --providers [upcloud]
```

#### Generated Files
- `taskservs.k` - Auto-generated taskserv imports
- `providers.k` - Auto-generated provider imports
- `clusters.k` - Auto-generated cluster imports
- `.manifest/*.yaml` - Module loading manifests

## Workspace Structure

### New Workspace Layout
```
workspace/infra/my-project/
├── kcl.mod                    # Package dependencies
├── servers.k                  # Main server configuration
├── taskservs.k               # Auto-generated taskserv imports
├── providers.k               # Auto-generated provider imports
├── clusters.k                # Auto-generated cluster imports
├── .taskservs/               # Loaded taskserv modules
│   ├── kubernetes/
│   ├── cilium/
│   └── containerd/
├── .providers/               # Loaded provider modules
│   └── upcloud/
├── .clusters/                # Loaded cluster modules
│   └── buildkit/
├── .manifest/                # Module manifests
│   ├── taskservs.yaml
│   ├── providers.yaml
│   └── clusters.yaml
├── data/                     # Runtime data
├── tmp/                      # Temporary files
├── resources/                # Resource definitions
└── clusters/                 # Cluster configurations
```

### Import Patterns

#### Before (Old System)
```kcl
# Hardcoded relative paths
import ../../../kcl/server as server
import ../../../extensions/taskservs/kubernetes/kcl/kubernetes as k8s
```

#### After (New System)
```kcl
# Package-based imports
import provisioning.server as server

# Auto-generated module imports (after loading)
import .taskservs.kubernetes.kubernetes as k8s
```

## Package Distribution

### Building Core Package
```bash
# Build distributable package
./provisioning/tools/kcl-packager.nu build --version 1.0.0

# Install locally
./provisioning/tools/kcl-packager.nu install dist/provisioning-1.0.0.tar.gz

# Create release
./provisioning/tools/kcl-packager.nu build --format tar.gz --include-docs
```

### Package Installation Methods

#### Method 1: Local Installation (Recommended for development)
```toml
[dependencies]
provisioning = { path = "~/.kcl/packages/provisioning", version = "0.0.1" }
```

#### Method 2: Git Repository (For distributed teams)
```toml
[dependencies]
provisioning = { git = "https://github.com/your-org/provisioning-kcl", version = "v0.0.1" }
```

#### Method 3: KCL Registry (When available)
```toml
[dependencies]
provisioning = { version = "0.0.1" }
```

## Developer Workflows

### 1. New Project Setup
```bash
# Create workspace from template
cp -r provisioning/templates/workspaces/kubernetes ./my-k8s-cluster
cd my-k8s-cluster

# Initialize with modules
workspace-init.nu . init

# Load required modules
module-loader load taskservs . [kubernetes, cilium, containerd]
module-loader load providers . [upcloud]

# Validate and deploy
kcl run servers.k
provisioning server create --infra . --check
```

### 2. Extension Development
```bash
# Create new taskserv
mkdir -p extensions/taskservs/my-service/kcl
cd extensions/taskservs/my-service/kcl

# Initialize KCL module
kcl mod init my-service
echo 'provisioning = { path = "~/.kcl/packages/provisioning", version = "0.0.1" }' >> kcl.mod

# Develop and test
module-loader discover taskservs   # Should find your service
```

### 3. Workspace Migration
```bash
# Analyze existing workspace
workspace-migrate.nu workspace/infra/old-project dry-run

# Perform migration
workspace-migrate.nu workspace/infra/old-project

# Verify migration
module-loader validate workspace/infra/old-project
```

### 4. Multi-Environment Management
```bash
# Development environment
cd workspace/infra/dev
module-loader load taskservs . [redis, postgres]
module-loader load providers . [local]

# Production environment
cd workspace/infra/prod
module-loader load taskservs . [redis, postgres, kubernetes, monitoring]
module-loader load providers . [upcloud, aws]  # Multi-cloud
```

## Module Management

### Listing and Validation
```bash
# List loaded modules
module-loader list taskservs .
module-loader list providers .
module-loader list clusters .

# Validate workspace
module-loader validate .

# Show workspace info
workspace-init.nu . info
```

### Unloading Modules
```bash
# Remove specific modules
module-loader unload taskservs . redis
module-loader unload providers . aws

# This regenerates import files automatically
```

### Module Information
```bash
# Get detailed module info
module-loader info taskservs kubernetes
module-loader info providers upcloud
module-loader info clusters buildkit
```

## CI/CD Integration

### Pipeline Example
```bash
#!/usr/bin/env nu
# deploy-pipeline.nu

# Install specific versions
kcl-packager.nu install --version $env.PROVISIONING_VERSION

# Load production modules
module-loader init $env.WORKSPACE_PATH \
    --taskservs $env.REQUIRED_TASKSERVS \
    --providers [$env.CLOUD_PROVIDER]

# Validate configuration
module-loader validate $env.WORKSPACE_PATH

# Deploy infrastructure
provisioning server create --infra $env.WORKSPACE_PATH
```

## Troubleshooting

### Common Issues

#### Module Import Errors
```
Error: module not found
```
**Solution**: Verify modules are loaded and regenerate imports
```bash
module-loader list taskservs .
module-loader load taskservs . [kubernetes, cilium, containerd]
```

#### Provider Configuration Issues
**Solution**: Check provider-specific configuration in `.providers/` directory

#### KCL Compilation Errors
**Solution**: Verify core package installation and kcl.mod configuration
```bash
kcl-packager.nu install --version latest
kcl run --dry-run servers.k
```

### Debug Commands
```bash
# Show workspace structure
tree -a workspace/infra/my-project

# Check generated imports
cat workspace/infra/my-project/taskservs.k

# Validate KCL files
kcl check workspace/infra/my-project/*.k

# Show module manifests
cat workspace/infra/my-project/.manifest/taskservs.yaml
```

## Best Practices

### 1. Version Management
- Pin core package versions in production
- Use semantic versioning for extensions
- Test compatibility before upgrading

### 2. Module Organization
- Load only required modules to keep workspaces clean
- Use meaningful workspace names
- Document required modules in README

### 3. Security
- Exclude `.manifest/` and `data/` from version control
- Use secrets management for sensitive configuration
- Validate modules before loading in production

### 4. Performance
- Load modules at workspace initialization, not runtime
- Cache discovery results when possible
- Use parallel loading for multiple modules

## Migration Guide

For existing workspaces, follow these steps:

### 1. Backup Current Workspace
```bash
cp -r workspace/infra/existing workspace/infra/existing-backup
```

### 2. Analyze Migration Requirements
```bash
workspace-migrate.nu workspace/infra/existing dry-run
```

### 3. Perform Migration
```bash
workspace-migrate.nu workspace/infra/existing
```

### 4. Load Required Modules
```bash
cd workspace/infra/existing
module-loader load taskservs . [kubernetes, cilium]
module-loader load providers . [upcloud]
```

### 5. Test and Validate
```bash
kcl run servers.k
module-loader validate .
```

### 6. Deploy
```bash
provisioning server create --infra . --check
```

## Future Enhancements

- Registry-based module distribution
- Module dependency resolution
- Automatic version updates
- Module templates and scaffolding
- Integration with external package managers