# KCL Package and Module Loader System

This document describes the new package-based architecture implemented for the provisioning system, replacing hardcoded extension paths with a flexible module discovery and loading system.

## Architecture Overview

The new system consists of two main components:

1. **Core KCL Package**: Distributable core provisioning schemas
2. **Module Loader System**: Dynamic discovery and loading of extensions

### Benefits

- **Clean Separation**: Core package is self-contained and distributable
- **Plug-and-Play Extensions**: Taskservs, providers, and clusters can be loaded dynamically
- **Version Management**: Core package and extensions can be versioned independently
- **Developer Friendly**: Easy workspace setup and module management

## Components

### 1. Core KCL Package (`/provisioning/kcl/`)

Contains fundamental schemas for provisioning:

- `settings.ncl` - System settings and configuration
- `server.ncl` - Server definitions and schemas
- `defaults.ncl` - Default configurations
- `lib.ncl` - Common library schemas
- `dependencies.ncl` - Dependency management schemas

**Key Features:**

- No hardcoded extension paths
- Self-contained and distributable
- Package-based imports only

### 2. Module Discovery System

#### Discovery Commands

```bash
# Discover available modules
module-loader discover taskservs              # List all taskservs
module-loader discover providers --format yaml # List providers as YAML
module-loader discover clusters redis          # Search for redis clusters
```plaintext

#### Supported Module Types

- **Taskservs**: Infrastructure services (kubernetes, redis, postgres, etc.)
- **Providers**: Cloud providers (upcloud, aws, local)
- **Clusters**: Complete configurations (buildkit, web, oci-reg)

### 3. Module Loading System

#### Loading Commands

```bash
# Load modules into workspace
module-loader load taskservs . [kubernetes, cilium, containerd]
module-loader load providers . [upcloud]
module-loader load clusters . [buildkit]

# Initialize workspace with modules
module-loader init workspace/infra/production \
    --taskservs [kubernetes, cilium] \
    --providers [upcloud]
```plaintext

#### Generated Files

- `taskservs.ncl` - Auto-generated taskserv imports
- `providers.ncl` - Auto-generated provider imports
- `clusters.ncl` - Auto-generated cluster imports
- `.manifest/*.yaml` - Module loading manifests

## Workspace Structure

### New Workspace Layout

```plaintext
workspace/infra/my-project/
├── kcl.mod                    # Package dependencies
├── servers.ncl                  # Main server configuration
├── taskservs.ncl               # Auto-generated taskserv imports
├── providers.ncl               # Auto-generated provider imports
├── clusters.ncl                # Auto-generated cluster imports
├── .taskservs/               # Loaded taskserv modules
│   ├── kubernetes/
│   ├── cilium/
│   └── containerd/
├── .providers/               # Loaded provider modules
│   └── upcloud/
├── .clusters/                # Loaded cluster modules
│   └── buildkit/
├── .manifest/                # Module manifests
│   ├── taskservs.yaml
│   ├── providers.yaml
│   └── clusters.yaml
├── data/                     # Runtime data
├── tmp/                      # Temporary files
├── resources/                # Resource definitions
└── clusters/                 # Cluster configurations
```plaintext

### Import Patterns

#### Before (Old System)

```kcl
# Hardcoded relative paths
import ../../../kcl/server as server
import ../../../extensions/taskservs/kubernetes/kcl/kubernetes as k8s
```plaintext

#### After (New System)

```kcl
# Package-based imports
import provisioning.server as server

# Auto-generated module imports (after loading)
import .taskservs.nclubernetes.kubernetes as k8s
```plaintext

## Package Distribution

### Building Core Package

```bash
# Build distributable package
./provisioning/tools/kcl-packager.nu build --version 1.0.0

# Install locally
./provisioning/tools/kcl-packager.nu install dist/provisioning-1.0.0.tar.gz

# Create release
./provisioning/tools/kcl-packager.nu build --format tar.gz --include-docs
```plaintext

### Package Installation Methods

#### Method 1: Local Installation (Recommended for development)

```toml
[dependencies]
provisioning = { path = "~/.kcl/packages/provisioning", version = "0.0.1" }
```plaintext

#### Method 2: Git Repository (For distributed teams)

```toml
[dependencies]
provisioning = { git = "https://github.com/your-org/provisioning-kcl", version = "v0.0.1" }
```plaintext

#### Method 3: KCL Registry (When available)

```toml
[dependencies]
provisioning = { version = "0.0.1" }
```plaintext

## Developer Workflows

### 1. New Project Setup

```bash
# Create workspace from template
cp -r provisioning/templates/workspaces/kubernetes ./my-k8s-cluster
cd my-k8s-cluster

# Initialize with modules
workspace-init.nu . init

# Load required modules
module-loader load taskservs . [kubernetes, cilium, containerd]
module-loader load providers . [upcloud]

# Validate and deploy
kcl run servers.ncl
provisioning server create --infra . --check
```plaintext

### 2. Extension Development

```bash
# Create new taskserv
mkdir -p extensions/taskservs/my-service/kcl
cd extensions/taskservs/my-service/kcl

# Initialize KCL module
kcl mod init my-service
echo 'provisioning = { path = "~/.kcl/packages/provisioning", version = "0.0.1" }' >> kcl.mod

# Develop and test
module-loader discover taskservs   # Should find your service
```plaintext

### 3. Workspace Migration

```bash
# Analyze existing workspace
workspace-migrate.nu workspace/infra/old-project dry-run

# Perform migration
workspace-migrate.nu workspace/infra/old-project

# Verify migration
module-loader validate workspace/infra/old-project
```plaintext

### 4. Multi-Environment Management

```bash
# Development environment
cd workspace/infra/dev
module-loader load taskservs . [redis, postgres]
module-loader load providers . [local]

# Production environment
cd workspace/infra/prod
module-loader load taskservs . [redis, postgres, kubernetes, monitoring]
module-loader load providers . [upcloud, aws]  # Multi-cloud
```plaintext

## Module Management

### Listing and Validation

```bash
# List loaded modules
module-loader list taskservs .
module-loader list providers .
module-loader list clusters .

# Validate workspace
module-loader validate .

# Show workspace info
workspace-init.nu . info
```plaintext

### Unloading Modules

```bash
# Remove specific modules
module-loader unload taskservs . redis
module-loader unload providers . aws

# This regenerates import files automatically
```plaintext

### Module Information

```bash
# Get detailed module info
module-loader info taskservs kubernetes
module-loader info providers upcloud
module-loader info clusters buildkit
```plaintext

## CI/CD Integration

### Pipeline Example

```bash
#!/usr/bin/env nu
# deploy-pipeline.nu

# Install specific versions
kcl-packager.nu install --version $env.PROVISIONING_VERSION

# Load production modules
module-loader init $env.WORKSPACE_PATH \
    --taskservs $env.REQUIRED_TASKSERVS \
    --providers [$env.CLOUD_PROVIDER]

# Validate configuration
module-loader validate $env.WORKSPACE_PATH

# Deploy infrastructure
provisioning server create --infra $env.WORKSPACE_PATH
```plaintext

## Troubleshooting

### Common Issues

#### Module Import Errors

```plaintext
Error: module not found
```plaintext

**Solution**: Verify modules are loaded and regenerate imports

```bash
module-loader list taskservs .
module-loader load taskservs . [kubernetes, cilium, containerd]
```plaintext

#### Provider Configuration Issues

**Solution**: Check provider-specific configuration in `.providers/` directory

#### KCL Compilation Errors

**Solution**: Verify core package installation and kcl.mod configuration

```bash
kcl-packager.nu install --version latest
kcl run --dry-run servers.ncl
```plaintext

### Debug Commands

```bash
# Show workspace structure
tree -a workspace/infra/my-project

# Check generated imports
cat workspace/infra/my-project/taskservs.ncl

# Validate KCL files
nickel typecheck workspace/infra/my-project/*.ncl

# Show module manifests
cat workspace/infra/my-project/.manifest/taskservs.yaml
```plaintext

## Best Practices

### 1. Version Management

- Pin core package versions in production
- Use semantic versioning for extensions
- Test compatibility before upgrading

### 2. Module Organization

- Load only required modules to keep workspaces clean
- Use meaningful workspace names
- Document required modules in README

### 3. Security

- Exclude `.manifest/` and `data/` from version control
- Use secrets management for sensitive configuration
- Validate modules before loading in production

### 4. Performance

- Load modules at workspace initialization, not runtime
- Cache discovery results when possible
- Use parallel loading for multiple modules

## Migration Guide

For existing workspaces, follow these steps:

### 1. Backup Current Workspace

```bash
cp -r workspace/infra/existing workspace/infra/existing-backup
```plaintext

### 2. Analyze Migration Requirements

```bash
workspace-migrate.nu workspace/infra/existing dry-run
```plaintext

### 3. Perform Migration

```bash
workspace-migrate.nu workspace/infra/existing
```plaintext

### 4. Load Required Modules

```bash
cd workspace/infra/existing
module-loader load taskservs . [kubernetes, cilium]
module-loader load providers . [upcloud]
```plaintext

### 5. Test and Validate

```bash
kcl run servers.ncl
module-loader validate .
```plaintext

### 6. Deploy

```bash
provisioning server create --infra . --check
```plaintext

## Future Enhancements

- Registry-based module distribution
- Module dependency resolution
- Automatic version updates
- Module templates and scaffolding
- Integration with external package managers
