provisioning/docs/package-and-loader-system.md

KCL Package and Module Loader System

This document describes the new package-based architecture implemented for the provisioning system, replacing hardcoded extension paths with a flexible module discovery and loading system.

Architecture Overview

The new system consists of two main components:

1. Core KCL Package: Distributable core provisioning schemas
2. Module Loader System: Dynamic discovery and loading of extensions

Benefits

• Clean Separation: Core package is self-contained and distributable
• Plug-and-Play Extensions: Taskservs, providers, and clusters can be loaded dynamically
• Version Management: Core package and extensions can be versioned independently
• Developer Friendly: Easy workspace setup and module management

Components

1. Core KCL Package (/provisioning/kcl/)

Contains fundamental schemas for provisioning:

• settings.k - System settings and configuration
• server.k - Server definitions and schemas
• defaults.k - Default configurations
• lib.k - Common library schemas
• dependencies.k - Dependency management schemas

Key Features:

• No hardcoded extension paths
• Self-contained and distributable
• Package-based imports only

2. Module Discovery System

Discovery Commands

# Discover available modules
module-loader discover taskservs              # List all taskservs
module-loader discover providers --format yaml # List providers as YAML
module-loader discover clusters redis          # Search for redis clusters

Supported Module Types

• Taskservs: Infrastructure services (kubernetes, redis, postgres, etc.)
• Providers: Cloud providers (upcloud, aws, local)
• Clusters: Complete configurations (buildkit, web, oci-reg)

3. Module Loading System

Loading Commands

# Load modules into workspace
module-loader load taskservs . [kubernetes, cilium, containerd]
module-loader load providers . [upcloud]
module-loader load clusters . [buildkit]

# Initialize workspace with modules
module-loader init workspace/infra/production \
    --taskservs [kubernetes, cilium] \
    --providers [upcloud]

Generated Files

• taskservs.k - Auto-generated taskserv imports
• providers.k - Auto-generated provider imports
• clusters.k - Auto-generated cluster imports
• .manifest/*.yaml - Module loading manifests

Workspace Structure

New Workspace Layout

workspace/infra/my-project/
├── kcl.mod                    # Package dependencies
├── servers.k                  # Main server configuration
├── taskservs.k               # Auto-generated taskserv imports
├── providers.k               # Auto-generated provider imports
├── clusters.k                # Auto-generated cluster imports
├── .taskservs/               # Loaded taskserv modules
│   ├── kubernetes/
│   ├── cilium/
│   └── containerd/
├── .providers/               # Loaded provider modules
│   └── upcloud/
├── .clusters/                # Loaded cluster modules
│   └── buildkit/
├── .manifest/                # Module manifests
│   ├── taskservs.yaml
│   ├── providers.yaml
│   └── clusters.yaml
├── data/                     # Runtime data
├── tmp/                      # Temporary files
├── resources/                # Resource definitions
└── clusters/                 # Cluster configurations

Import Patterns

Before (Old System)

# Hardcoded relative paths
import ../../../kcl/server as server
import ../../../extensions/taskservs/kubernetes/kcl/kubernetes as k8s

After (New System)

# Package-based imports
import provisioning.server as server

# Auto-generated module imports (after loading)
import .taskservs.kubernetes.kubernetes as k8s

Package Distribution

Building Core Package

# Build distributable package
./provisioning/tools/kcl-packager.nu build --version 1.0.0

# Install locally
./provisioning/tools/kcl-packager.nu install dist/provisioning-1.0.0.tar.gz

# Create release
./provisioning/tools/kcl-packager.nu build --format tar.gz --include-docs

Package Installation Methods

Method 1: Local Installation (Recommended for development)

[dependencies]
provisioning = { path = "~/.kcl/packages/provisioning", version = "0.0.1" }

Method 2: Git Repository (For distributed teams)

[dependencies]
provisioning = { git = "https://github.com/your-org/provisioning-kcl", version = "v0.0.1" }

Method 3: KCL Registry (When available)

[dependencies]
provisioning = { version = "0.0.1" }

Developer Workflows

1. New Project Setup

# Create workspace from template
cp -r provisioning/templates/workspaces/kubernetes ./my-k8s-cluster
cd my-k8s-cluster

# Initialize with modules
workspace-init.nu . init

# Load required modules
module-loader load taskservs . [kubernetes, cilium, containerd]
module-loader load providers . [upcloud]

# Validate and deploy
kcl run servers.k
provisioning server create --infra . --check

2. Extension Development

# Create new taskserv
mkdir -p extensions/taskservs/my-service/kcl
cd extensions/taskservs/my-service/kcl

# Initialize KCL module
kcl mod init my-service
echo 'provisioning = { path = "~/.kcl/packages/provisioning", version = "0.0.1" }' >> kcl.mod

# Develop and test
module-loader discover taskservs   # Should find your service

3. Workspace Migration

# Analyze existing workspace
workspace-migrate.nu workspace/infra/old-project dry-run

# Perform migration
workspace-migrate.nu workspace/infra/old-project

# Verify migration
module-loader validate workspace/infra/old-project

4. Multi-Environment Management

# Development environment
cd workspace/infra/dev
module-loader load taskservs . [redis, postgres]
module-loader load providers . [local]

# Production environment
cd workspace/infra/prod
module-loader load taskservs . [redis, postgres, kubernetes, monitoring]
module-loader load providers . [upcloud, aws]  # Multi-cloud

Module Management

Listing and Validation

# List loaded modules
module-loader list taskservs .
module-loader list providers .
module-loader list clusters .

# Validate workspace
module-loader validate .

# Show workspace info
workspace-init.nu . info

Unloading Modules

# Remove specific modules
module-loader unload taskservs . redis
module-loader unload providers . aws

# This regenerates import files automatically

Module Information

# Get detailed module info
module-loader info taskservs kubernetes
module-loader info providers upcloud
module-loader info clusters buildkit

CI/CD Integration

Pipeline Example

#!/usr/bin/env nu
# deploy-pipeline.nu

# Install specific versions
kcl-packager.nu install --version $env.PROVISIONING_VERSION

# Load production modules
module-loader init $env.WORKSPACE_PATH \
    --taskservs $env.REQUIRED_TASKSERVS \
    --providers [$env.CLOUD_PROVIDER]

# Validate configuration
module-loader validate $env.WORKSPACE_PATH

# Deploy infrastructure
provisioning server create --infra $env.WORKSPACE_PATH

Troubleshooting

Common Issues

Module Import Errors

Error: module not found

Solution: Verify modules are loaded and regenerate imports

module-loader list taskservs .
module-loader load taskservs . [kubernetes, cilium, containerd]

Provider Configuration Issues

Solution: Check provider-specific configuration in .providers/ directory

KCL Compilation Errors

Solution: Verify core package installation and kcl.mod configuration

kcl-packager.nu install --version latest
kcl run --dry-run servers.k

Debug Commands

# Show workspace structure
tree -a workspace/infra/my-project

# Check generated imports
cat workspace/infra/my-project/taskservs.k

# Validate KCL files
kcl check workspace/infra/my-project/*.k

# Show module manifests
cat workspace/infra/my-project/.manifest/taskservs.yaml

Best Practices

1. Version Management

• Pin core package versions in production
• Use semantic versioning for extensions
• Test compatibility before upgrading

2. Module Organization

• Load only required modules to keep workspaces clean
• Use meaningful workspace names
• Document required modules in README

3. Security

• Exclude .manifest/ and data/ from version control
• Use secrets management for sensitive configuration
• Validate modules before loading in production

4. Performance

• Load modules at workspace initialization, not runtime
• Cache discovery results when possible
• Use parallel loading for multiple modules

Migration Guide

For existing workspaces, follow these steps:

1. Backup Current Workspace

cp -r workspace/infra/existing workspace/infra/existing-backup

2. Analyze Migration Requirements

workspace-migrate.nu workspace/infra/existing dry-run

3. Perform Migration

workspace-migrate.nu workspace/infra/existing

4. Load Required Modules

cd workspace/infra/existing
module-loader load taskservs . [kubernetes, cilium]
module-loader load providers . [upcloud]

5. Test and Validate

kcl run servers.k
module-loader validate .

6. Deploy

provisioning server create --infra . --check

Future Enhancements

• Registry-based module distribution
• Module dependency resolution
• Automatic version updates
• Module templates and scaffolding
• Integration with external package managers