provisioning/docs/kcl-packaging-guide.md

KCL Packaging and Extension Development Guide

This guide covers how to package the core KCL modules and develop custom extensions for the provisioning system.

Table of Contents

1. Core KCL Package Management
2. Extension Development
3. Infrastructure-Specific Extensions
4. Publishing and Distribution
5. Best Practices

Core KCL Package Management

Building the Core Package

The core provisioning package contains fundamental schemas and should be built and distributed independently from extensions.

# Navigate to provisioning directory
cd /Users/Akasha/project-provisioning/provisioning

# Build core package
./tools/kcl-packager.nu build --version 1.0.0 --output dist

# Build with documentation
./tools/kcl-packager.nu build --version 1.0.0 --include-docs --output dist

# Build different formats
./tools/kcl-packager.nu build --version 1.0.0 --format zip
./tools/kcl-packager.nu build --version 1.0.0 --format tar.gz

Installing Core Package

# Install locally built package
./tools/kcl-packager.nu install dist/provisioning-1.0.0.tar.gz

# Install to custom location
./tools/kcl-packager.nu install dist/provisioning-1.0.0.tar.gz --target ~/.local/kcl/packages

# Verify installation
kcl list packages | grep provisioning

Core Package Structure

provisioning-1.0.0/
├── kcl.mod                 # Package metadata
├── settings.k              # System settings schemas
├── server.k                # Server definition schemas
├── defaults.k              # Default configuration schemas
├── lib.k                   # Common library schemas
├── dependencies.k          # Dependency management schemas
├── cluster.k               # Cluster schemas
├── batch.k                 # Batch workflow schemas
├── README.md               # Package documentation
└── docs/                   # Additional documentation

Version Management

# Check current version
./tools/kcl-packager.nu version

# Build with semantic versioning
./tools/kcl-packager.nu build --version 1.2.0
./tools/kcl-packager.nu build --version 1.2.0-beta.1
./tools/kcl-packager.nu build --version 1.2.0-rc.1

# Clean build artifacts
./tools/kcl-packager.nu clean

Extension Development

Types of Extensions

The system supports three types of extensions:

1. Taskservs: Infrastructure services (kubernetes, redis, postgres, etc.)
2. Providers: Cloud providers (upcloud, aws, local, etc.)
3. Clusters: Complete configurations (buildkit, web, monitoring, etc.)

Creating a New Taskserv Extension

1. Basic Structure

# Create taskserv directory structure
mkdir -p extensions/taskservs/my-service/kcl
cd extensions/taskservs/my-service/kcl

# Initialize KCL module
kcl mod init my-service

2. Configure Dependencies

Edit kcl.mod:

[package]
name = "my-service"
edition = "v0.11.2"
version = "0.0.1"

[dependencies]
provisioning = { path = "~/.kcl/packages/provisioning", version = "0.0.1" }
taskservs = { path = "../..", version = "0.0.1" }

3. Create Main Schema

Create my-service.k:

"""
My Service Taskserv Schema
Custom service for specific infrastructure needs
"""

import provisioning.lib as lib
import provisioning.dependencies as deps

# Service configuration schema
schema MyServiceConfig:
    """Configuration for My Service"""
    # Service-specific settings
    enabled: bool = True
    port: int = 8080
    replicas: int = 1

    # Resource requirements
    cpu_request: str = "100m"
    memory_request: str = "128Mi"
    cpu_limit: str = "500m"
    memory_limit: str = "512Mi"

    # Storage configuration
    storage_size: str = "10Gi"
    storage_class?: str

    # Network configuration
    service_type: "ClusterIP" | "NodePort" | "LoadBalancer" = "ClusterIP"

    # Additional configuration
    environment_vars?: {str: str}
    config_map_data?: {str: str}

    check:
        port > 0 and port < 65536, "Port must be between 1 and 65535"
        replicas > 0, "Replicas must be greater than 0"
        len(cpu_request) > 0, "CPU request cannot be empty"
        len(memory_request) > 0, "Memory request cannot be empty"

# Taskserv definition using the common lib pattern
schema MyServiceTaskserv(lib.TaskServDef):
    """My Service Taskserv Definition"""
    name: str = "my-service"
    config: MyServiceConfig

    # Installation profiles
    profiles?: {str: MyServiceProfile}

# Service profiles for different environments
schema MyServiceProfile:
    """Profile-specific configuration for My Service"""
    config_overrides?: MyServiceConfig
    installation_mode?: "standalone" | "cluster" | "replicated" = "standalone"

# Dependencies definition
my_service_dependencies: deps.TaskservDependencies = {
    name = "my-service"

    # Dependencies
    requires = ["kubernetes"]  # Requires Kubernetes
    conflicts = ["old-my-service"]  # Cannot coexist with old version
    optional = ["monitoring", "logging"]  # Works better with these
    provides = ["my-service-api", "my-service-ui"]  # Services it provides

    # Resource requirements
    resources = {
        cpu = "100m"
        memory = "128Mi"
        disk = "1Gi"
        network = True
        privileged = False
    }

    # Health checks
    health_checks = [{
        command = "curl -f http://localhost:9090/health"
        interval = 30
        timeout = 10
        retries = 3
    }]

    # Installation phases
    phases = [
        {
            name = "pre-install"
            order = 1
            parallel = False
            required = True
        },
        {
            name = "install"
            order = 2
            parallel = True
            required = True
        },
        {
            name = "post-install"
            order = 3
            parallel = False
            required = False
        }
    ]

    # Compatibility
    os_support = ["linux"]
    arch_support = ["amd64", "arm64"]
    k8s_versions = ["1.25+", "1.26+", "1.27+"]
}

# Default configuration
my_service_default: MyServiceTaskserv = {
    name = "my-service"
    config = {
        enabled = True
        port = 8080
        replicas = 1
        cpu_request = "100m"
        memory_request = "128Mi"
        cpu_limit = "500m"
        memory_limit = "512Mi"
        storage_size = "10Gi"
        service_type = "ClusterIP"
    }
    profiles = {
        "default": {
            installation_mode = "standalone"
        },
        "production": {
            config_overrides = {
                replicas = 3
                cpu_request = "200m"
                memory_request = "256Mi"
                cpu_limit = "1"
                memory_limit = "1Gi"
                storage_size = "50Gi"
                service_type = "LoadBalancer"
            }
            installation_mode = "replicated"
        },
        "development": {
            config_overrides = {
                replicas = 1
                cpu_request = "50m"
                memory_request = "64Mi"
                storage_size = "5Gi"
            }
            installation_mode = "standalone"
        }
    }
}

# Export for use by provisioning system
{
    config: my_service_default,
    dependencies: my_service_dependencies,
    schema: MyServiceTaskserv
}

4. Create Version Information

Create version.k:

"""
Version information for My Service taskserv
"""

version_info = {
    name = "my-service"
    version = "0.0.1"
    description = "Custom service for specific infrastructure needs"
    author = "Your Name"
    license = "MIT"
    repository = "https://github.com/your-org/my-service-taskserv"

    # Supported versions
    min_provisioning_version = "0.0.1"
    max_provisioning_version = "1.0.0"

    # Changelog
    changelog = {
        "0.0.1" = "Initial release with basic functionality"
    }
}

version_info

5. Create Dependencies File

Create dependencies.k:

"""
Dependencies for My Service taskserv
"""

import provisioning.dependencies as deps

# Import the main dependencies from my-service.k
import .my-service as ms

# Re-export dependencies for discovery system
ms.my_service_dependencies

Creating a Provider Extension

1. Provider Structure

# Create provider directory
mkdir -p extensions/providers/my-cloud/kcl
cd extensions/providers/my-cloud/kcl

# Initialize
kcl mod init my-cloud

2. Provider Schema

Create provision_my-cloud.k:

"""
My Cloud Provider Schema
"""

import provisioning.defaults as defaults
import provisioning.server as server

# Provider-specific configuration
schema MyCloudConfig:
    """My Cloud provider configuration"""
    api_endpoint: str
    api_key: str
    region: str = "us-east-1"
    project_id?: str

    # Network settings
    vpc_id?: str
    subnet_id?: str
    security_group_id?: str

    check:
        len(api_endpoint) > 0, "API endpoint cannot be empty"
        len(api_key) > 0, "API key cannot be empty"
        len(region) > 0, "Region cannot be empty"

# Server configuration for this provider
schema MyCloudServer(server.Server):
    """Server configuration for My Cloud"""
    # Provider-specific server settings
    instance_type: str = "standard-1vcpu-1gb"
    availability_zone?: str

    # Storage configuration
    root_disk_size: int = 25
    additional_disks?: [MyCloudDisk]

    # Network configuration
    public_ip: bool = True
    private_networking: bool = False

    check:
        len(instance_type) > 0, "Instance type cannot be empty"
        root_disk_size >= 10, "Root disk must be at least 10GB"

schema MyCloudDisk:
    """Additional disk configuration"""
    size: int
    type: "ssd" | "hdd" = "ssd"
    mount_point?: str

    check:
        size > 0, "Disk size must be positive"

# Provider defaults
my_cloud_defaults: defaults.ServerDefaults = {
    lock = False
    time_zone = "UTC"
    running_wait = 15
    running_timeout = 300

    # OS configuration
    storage_os_find = "name: ubuntu-20.04 | arch: x86_64"

    # Network settings
    network_utility_ipv4 = True
    network_public_ipv4 = True

    # User settings
    user = "ubuntu"
    user_ssh_port = 22
    fix_local_hosts = True

    labels = "provider: my-cloud"
}

# Export provider configuration
{
    config: MyCloudConfig,
    server: MyCloudServer,
    defaults: my_cloud_defaults
}

Creating a Cluster Extension

1. Cluster Structure

# Create cluster directory
mkdir -p extensions/clusters/my-stack/kcl
cd extensions/clusters/my-stack/kcl

# Initialize
kcl mod init my-stack

2. Cluster Schema

Create my-stack.k:

"""
My Stack Cluster Configuration
Complete infrastructure stack with multiple services
"""

import provisioning.cluster as cluster
import provisioning.server as server

# Cluster configuration
schema MyStackCluster(cluster.ClusterDef):
    """My Stack cluster definition"""
    name: str = "my-stack"

    # Component configuration
    components: MyStackComponents

    # Infrastructure settings
    node_count: int = 3
    load_balancer: bool = True
    monitoring: bool = True
    logging: bool = True

schema MyStackComponents:
    """Components in My Stack"""
    # Web tier
    web_servers: int = 2
    web_instance_type: str = "standard-2vcpu-4gb"

    # Application tier
    app_servers: int = 3
    app_instance_type: str = "standard-4vcpu-8gb"

    # Database tier
    db_servers: int = 1
    db_instance_type: str = "standard-8vcpu-16gb"
    db_storage_size: int = 100

    # Cache tier
    cache_enabled: bool = True
    cache_instance_type: str = "standard-2vcpu-4gb"

# Generate server configurations for the cluster
my_stack_servers: [server.Server] = [
    # Web servers
    {
        hostname = "web-01"
        title = "Web Server 01"
        # ... web server configuration
        taskservs = [
            { name = "nginx", profile = "web" },
            { name = "ssl-cert", profile = "default" }
        ]
    },
    {
        hostname = "web-02"
        title = "Web Server 02"
        # ... web server configuration
        taskservs = [
            { name = "nginx", profile = "web" },
            { name = "ssl-cert", profile = "default" }
        ]
    },

    # Application servers
    {
        hostname = "app-01"
        title = "Application Server 01"
        # ... app server configuration
        taskservs = [
            { name = "kubernetes", profile = "worker" },
            { name = "containerd", profile = "default" }
        ]
    },

    # Database server
    {
        hostname = "db-01"
        title = "Database Server 01"
        # ... database configuration
        taskservs = [
            { name = "postgres", profile = "production" },
            { name = "backup-agent", profile = "default" }
        ]
    }
]

# Export cluster definition
{
    cluster: MyStackCluster,
    servers: my_stack_servers
}

Infrastructure-Specific Extensions

Creating Extensions for Specific Infrastructure

When you need extensions tailored to specific infrastructure or business requirements:

1. Organization Structure

# Create organization-specific extension directory
mkdir -p extensions/taskservs/org-specific/my-company-app/kcl
cd extensions/taskservs/org-specific/my-company-app/kcl

2. Company-Specific Taskserv

Create my-company-app.k:

"""
My Company Application Taskserv
Company-specific application with custom requirements
"""

import provisioning.lib as lib
import provisioning.dependencies as deps

schema MyCompanyAppConfig:
    """Configuration for company-specific application"""
    # Application settings
    app_version: str = "latest"
    environment: "development" | "staging" | "production" = "production"

    # Company-specific settings
    company_domain: str
    internal_api_endpoint: str

    # Integration settings
    ldap_server?: str
    sso_provider?: str
    monitoring_endpoint?: str

    # Compliance settings
    encryption_enabled: bool = True
    audit_logging: bool = True
    data_retention_days: int = 90

    # Resource settings based on environment
    resources: MyCompanyAppResources

    check:
        len(company_domain) > 0, "Company domain required"
        len(internal_api_endpoint) > 0, "Internal API endpoint required"
        data_retention_days > 0, "Data retention must be positive"

schema MyCompanyAppResources:
    """Resource configuration based on environment"""
    cpu_request: str
    memory_request: str
    cpu_limit: str
    memory_limit: str
    storage_size: str
    replicas: int

# Environment-specific resource profiles
my_company_app_resources = {
    "development": {
        cpu_request = "100m"
        memory_request = "256Mi"
        cpu_limit = "500m"
        memory_limit = "512Mi"
        storage_size = "5Gi"
        replicas = 1
    },
    "staging": {
        cpu_request = "200m"
        memory_request = "512Mi"
        cpu_limit = "1"
        memory_limit = "1Gi"
        storage_size = "20Gi"
        replicas = 2
    },
    "production": {
        cpu_request = "500m"
        memory_request = "1Gi"
        cpu_limit = "2"
        memory_limit = "4Gi"
        storage_size = "100Gi"
        replicas = 5
    }
}

# Taskserv definition
schema MyCompanyAppTaskserv(lib.TaskServDef):
    """My Company App Taskserv Definition"""
    name: str = "my-company-app"
    config: MyCompanyAppConfig

# Dependencies for company app
my_company_app_dependencies: deps.TaskservDependencies = {
    name = "my-company-app"

    # Required infrastructure
    requires = ["kubernetes", "postgres", "redis"]

    # Integrations
    optional = ["monitoring", "logging", "backup"]

    # What this app provides
    provides = ["company-api", "company-ui", "company-reports"]

    # Company-specific requirements
    resources = {
        cpu = "500m"
        memory = "1Gi"
        disk = "10Gi"
        network = True
        privileged = False
    }

    # Compliance health checks
    health_checks = [
        {
            command = "curl -f https://app.company.com/health"
            interval = 30
            timeout = 10
            retries = 3
        },
        {
            command = "check-compliance-status"
            interval = 300  # Every 5 minutes
            timeout = 30
            retries = 1
        }
    ]

    # Installation phases
    phases = [
        {
            name = "pre-install"
            order = 1
            parallel = False
            required = True
        },
        {
            name = "install"
            order = 2
            parallel = True
            required = True
        },
        {
            name = "configure-integrations"
            order = 3
            parallel = False
            required = True
        },
        {
            name = "compliance-check"
            order = 4
            parallel = False
            required = True
        }
    ]

    # Compatibility
    os_support = ["linux"]
    arch_support = ["amd64"]
    k8s_versions = ["1.25+"]
}

# Export company app configuration
{
    config: MyCompanyAppTaskserv,
    dependencies: my_company_app_dependencies
}

Multi-Environment Extensions

Environment-Specific Configurations

Create environments/production.k:

"""
Production environment configuration for My Company App
"""

import ..my-company-app as app

production_config: app.MyCompanyAppConfig = {
    app_version = "v2.1.0"
    environment = "production"
    company_domain = "company.com"
    internal_api_endpoint = "https://api.internal.company.com"

    # Production integrations
    ldap_server = "ldap.company.com"
    sso_provider = "https://sso.company.com"
    monitoring_endpoint = "https://metrics.company.com"

    # Production compliance
    encryption_enabled = True
    audit_logging = True
    data_retention_days = 365

    # Production resources
    resources = {
        cpu_request = "1"
        memory_request = "2Gi"
        cpu_limit = "4"
        memory_limit = "8Gi"
        storage_size = "500Gi"
        replicas = 10
    }
}

production_config

Publishing and Distribution

Local Development

# Develop and test locally
cd extensions/taskservs/my-service
module-loader discover taskservs | grep my-service

# Load into test workspace
cd /tmp/test-workspace
module-loader load taskservs . [my-service]

Version Control

# Create git repository for extension
cd extensions/taskservs/my-service
git init
git add .
git commit -m "Initial my-service taskserv"

# Tag versions
git tag v0.0.1
git push origin v0.0.1

Package Distribution

Creating Extension Packages

# Create extension package
tar -czf my-service-taskserv-v0.0.1.tar.gz -C extensions/taskservs my-service

# Create bundle with multiple extensions
tar -czf company-extensions-v1.0.0.tar.gz \
    -C extensions/taskservs my-company-app \
    -C extensions/providers my-cloud \
    -C extensions/clusters my-stack

Distribution Methods

1. Git Repository

   # Users can clone and load
   git clone https://github.com/company/provisioning-extensions
   module-loader load taskservs . [my-company-app]
   

2. Package Registry (Future)

   # Publish to registry
   kcl publish extensions/taskservs/my-service
   
   # Users install from registry
   module-loader install taskserv my-service@0.0.1
   

3. Internal Distribution

   # Internal package server
   curl -O https://packages.company.com/my-service-v0.0.1.tar.gz
   tar -xzf my-service-v0.0.1.tar.gz -C extensions/taskservs/
   

Best Practices

Extension Development

1. Follow Naming Conventions

   • Use kebab-case for extension names
   • Prefix company-specific extensions with organization name
   • Use semantic versioning

2. Schema Design

   • Include comprehensive validation
   • Provide sensible defaults
   • Support multiple profiles/environments
   • Document all fields

3. Dependencies

   • Declare all dependencies explicitly
   • Use optional dependencies for enhanced features
   • Specify compatibility versions

4. Testing

   # Test extension loading
   module-loader discover taskservs | grep my-service
   
   # Test in isolated workspace
   mkdir test-workspace
   cd test-workspace
   workspace-init.nu . init
   module-loader load taskservs . [my-service]
   
   # Validate KCL compilation
   kcl check taskservs.k
   

Security Considerations

1. Secrets Management

   • Never hardcode secrets in schemas
   • Use provisioning system's secrets management
   • Support external secret providers

2. Input Validation

   • Validate all user inputs
   • Use KCL check constraints
   • Sanitize external data

3. Resource Limits

   • Set reasonable resource defaults
   • Provide resource limit options
   • Include resource monitoring

Documentation

1. Extension Documentation

   • Include README.md in each extension
   • Document configuration options
   • Provide usage examples
   • Include troubleshooting guide

2. Version Documentation

   • Maintain changelog
   • Document breaking changes
   • Include migration guides

Example Directory Structure

extensions/
├── taskservs/
│   ├── my-service/
│   │   ├── kcl/
│   │   │   ├── kcl.mod
│   │   │   ├── my-service.k
│   │   │   ├── version.k
│   │   │   └── dependencies.k
│   │   ├── templates/
│   │   │   ├── deployment.yaml.j2
│   │   │   └── service.yaml.j2
│   │   ├── scripts/
│   │   │   ├── install.nu
│   │   │   └── health-check.nu
│   │   └── README.md
│   └── org-specific/
│       └── my-company-app/
│           ├── kcl/
│           │   ├── kcl.mod
│           │   ├── my-company-app.k
│           │   ├── version.k
│           │   └── environments/
│           │       ├── development.k
│           │       ├── staging.k
│           │       └── production.k
│           └── README.md
├── providers/
│   └── my-cloud/
│       ├── kcl/
│       │   ├── kcl.mod
│       │   ├── provision_my-cloud.k
│       │   ├── server_my-cloud.k
│       │   └── defaults_my-cloud.k
│       └── README.md
└── clusters/
    └── my-stack/
        ├── kcl/
        │   ├── kcl.mod
        │   └── my-stack.k
        └── README.md

This guide provides comprehensive coverage of KCL packaging and extension development for both general-purpose and infrastructure-specific use cases.