provisioning/docs/kcl-packaging-guide.md

# KCL Packaging and Extension Development Guide

This guide covers how to package the core KCL modules and develop custom extensions for the provisioning system.

## Table of Contents

1. [Core KCL Package Management](#core-kcl-package-management)
2. [Extension Development](#extension-development)
3. [Infrastructure-Specific Extensions](#infrastructure-specific-extensions)
4. [Publishing and Distribution](#publishing-and-distribution)
5. [Best Practices](#best-practices)

## Core KCL Package Management

### Building the Core Package

The core provisioning package contains fundamental schemas and should be built and distributed independently from extensions.

```bash
# Navigate to provisioning directory
cd /Users/Akasha/project-provisioning/provisioning

# Build core package
./tools/kcl-packager.nu build --version 1.0.0 --output dist

# Build with documentation
./tools/kcl-packager.nu build --version 1.0.0 --include-docs --output dist

# Build different formats
./tools/kcl-packager.nu build --version 1.0.0 --format zip
./tools/kcl-packager.nu build --version 1.0.0 --format tar.gz
```

### Installing Core Package

```bash
# Install locally built package
./tools/kcl-packager.nu install dist/provisioning-1.0.0.tar.gz

# Install to custom location
./tools/kcl-packager.nu install dist/provisioning-1.0.0.tar.gz --target ~/.local/kcl/packages

# Verify installation
kcl list packages | grep provisioning
```

### Core Package Structure

```
provisioning-1.0.0/
├── kcl.mod                 # Package metadata
├── settings.k              # System settings schemas
├── server.k                # Server definition schemas
├── defaults.k              # Default configuration schemas
├── lib.k                   # Common library schemas
├── dependencies.k          # Dependency management schemas
├── cluster.k               # Cluster schemas
├── batch.k                 # Batch workflow schemas
├── README.md               # Package documentation
└── docs/                   # Additional documentation
```

### Version Management

```bash
# Check current version
./tools/kcl-packager.nu version

# Build with semantic versioning
./tools/kcl-packager.nu build --version 1.2.0
./tools/kcl-packager.nu build --version 1.2.0-beta.1
./tools/kcl-packager.nu build --version 1.2.0-rc.1

# Clean build artifacts
./tools/kcl-packager.nu clean
```

## Extension Development

### Types of Extensions

The system supports three types of extensions:

1. **Taskservs**: Infrastructure services (kubernetes, redis, postgres, etc.)
2. **Providers**: Cloud providers (upcloud, aws, local, etc.)
3. **Clusters**: Complete configurations (buildkit, web, monitoring, etc.)

### Creating a New Taskserv Extension

#### 1. Basic Structure

```bash
# Create taskserv directory structure
mkdir -p extensions/taskservs/my-service/kcl
cd extensions/taskservs/my-service/kcl

# Initialize KCL module
kcl mod init my-service
```

#### 2. Configure Dependencies

Edit `kcl.mod`:
```toml
[package]
name = "my-service"
edition = "v0.11.2"
version = "0.0.1"

[dependencies]
provisioning = { path = "~/.kcl/packages/provisioning", version = "0.0.1" }
taskservs = { path = "../..", version = "0.0.1" }
```

#### 3. Create Main Schema

Create `my-service.k`:
```kcl
"""
My Service Taskserv Schema
Custom service for specific infrastructure needs
"""

import provisioning.lib as lib
import provisioning.dependencies as deps

# Service configuration schema
schema MyServiceConfig:
    """Configuration for My Service"""
    # Service-specific settings
    enabled: bool = True
    port: int = 8080
    replicas: int = 1

    # Resource requirements
    cpu_request: str = "100m"
    memory_request: str = "128Mi"
    cpu_limit: str = "500m"
    memory_limit: str = "512Mi"

    # Storage configuration
    storage_size: str = "10Gi"
    storage_class?: str

    # Network configuration
    service_type: "ClusterIP" | "NodePort" | "LoadBalancer" = "ClusterIP"

    # Additional configuration
    environment_vars?: {str: str}
    config_map_data?: {str: str}

    check:
        port > 0 and port < 65536, "Port must be between 1 and 65535"
        replicas > 0, "Replicas must be greater than 0"
        len(cpu_request) > 0, "CPU request cannot be empty"
        len(memory_request) > 0, "Memory request cannot be empty"

# Taskserv definition using the common lib pattern
schema MyServiceTaskserv(lib.TaskServDef):
    """My Service Taskserv Definition"""
    name: str = "my-service"
    config: MyServiceConfig

    # Installation profiles
    profiles?: {str: MyServiceProfile}

# Service profiles for different environments
schema MyServiceProfile:
    """Profile-specific configuration for My Service"""
    config_overrides?: MyServiceConfig
    installation_mode?: "standalone" | "cluster" | "replicated" = "standalone"

# Dependencies definition
my_service_dependencies: deps.TaskservDependencies = {
    name = "my-service"

    # Dependencies
    requires = ["kubernetes"]  # Requires Kubernetes
    conflicts = ["old-my-service"]  # Cannot coexist with old version
    optional = ["monitoring", "logging"]  # Works better with these
    provides = ["my-service-api", "my-service-ui"]  # Services it provides

    # Resource requirements
    resources = {
        cpu = "100m"
        memory = "128Mi"
        disk = "1Gi"
        network = True
        privileged = False
    }

    # Health checks
    health_checks = [{
        command = "curl -f http://localhost:8080/health"
        interval = 30
        timeout = 10
        retries = 3
    }]

    # Installation phases
    phases = [
        {
            name = "pre-install"
            order = 1
            parallel = False
            required = True
        },
        {
            name = "install"
            order = 2
            parallel = True
            required = True
        },
        {
            name = "post-install"
            order = 3
            parallel = False
            required = False
        }
    ]

    # Compatibility
    os_support = ["linux"]
    arch_support = ["amd64", "arm64"]
    k8s_versions = ["1.25+", "1.26+", "1.27+"]
}

# Default configuration
my_service_default: MyServiceTaskserv = {
    name = "my-service"
    config = {
        enabled = True
        port = 8080
        replicas = 1
        cpu_request = "100m"
        memory_request = "128Mi"
        cpu_limit = "500m"
        memory_limit = "512Mi"
        storage_size = "10Gi"
        service_type = "ClusterIP"
    }
    profiles = {
        "default": {
            installation_mode = "standalone"
        },
        "production": {
            config_overrides = {
                replicas = 3
                cpu_request = "200m"
                memory_request = "256Mi"
                cpu_limit = "1"
                memory_limit = "1Gi"
                storage_size = "50Gi"
                service_type = "LoadBalancer"
            }
            installation_mode = "replicated"
        },
        "development": {
            config_overrides = {
                replicas = 1
                cpu_request = "50m"
                memory_request = "64Mi"
                storage_size = "5Gi"
            }
            installation_mode = "standalone"
        }
    }
}

# Export for use by provisioning system
{
    config: my_service_default,
    dependencies: my_service_dependencies,
    schema: MyServiceTaskserv
}
```

#### 4. Create Version Information

Create `version.k`:
```kcl
"""
Version information for My Service taskserv
"""

version_info = {
    name = "my-service"
    version = "0.0.1"
    description = "Custom service for specific infrastructure needs"
    author = "Your Name"
    license = "MIT"
    repository = "https://github.com/your-org/my-service-taskserv"

    # Supported versions
    min_provisioning_version = "0.0.1"
    max_provisioning_version = "1.0.0"

    # Changelog
    changelog = {
        "0.0.1" = "Initial release with basic functionality"
    }
}

version_info
```

#### 5. Create Dependencies File

Create `dependencies.k`:
```kcl
"""
Dependencies for My Service taskserv
"""

import provisioning.dependencies as deps

# Import the main dependencies from my-service.k
import .my-service as ms

# Re-export dependencies for discovery system
ms.my_service_dependencies
```

### Creating a Provider Extension

#### 1. Provider Structure

```bash
# Create provider directory
mkdir -p extensions/providers/my-cloud/kcl
cd extensions/providers/my-cloud/kcl

# Initialize
kcl mod init my-cloud
```

#### 2. Provider Schema

Create `provision_my-cloud.k`:
```kcl
"""
My Cloud Provider Schema
"""

import provisioning.defaults as defaults
import provisioning.server as server

# Provider-specific configuration
schema MyCloudConfig:
    """My Cloud provider configuration"""
    api_endpoint: str
    api_key: str
    region: str = "us-east-1"
    project_id?: str

    # Network settings
    vpc_id?: str
    subnet_id?: str
    security_group_id?: str

    check:
        len(api_endpoint) > 0, "API endpoint cannot be empty"
        len(api_key) > 0, "API key cannot be empty"
        len(region) > 0, "Region cannot be empty"

# Server configuration for this provider
schema MyCloudServer(server.Server):
    """Server configuration for My Cloud"""
    # Provider-specific server settings
    instance_type: str = "standard-1vcpu-1gb"
    availability_zone?: str

    # Storage configuration
    root_disk_size: int = 25
    additional_disks?: [MyCloudDisk]

    # Network configuration
    public_ip: bool = True
    private_networking: bool = False

    check:
        len(instance_type) > 0, "Instance type cannot be empty"
        root_disk_size >= 10, "Root disk must be at least 10GB"

schema MyCloudDisk:
    """Additional disk configuration"""
    size: int
    type: "ssd" | "hdd" = "ssd"
    mount_point?: str

    check:
        size > 0, "Disk size must be positive"

# Provider defaults
my_cloud_defaults: defaults.ServerDefaults = {
    lock = False
    time_zone = "UTC"
    running_wait = 15
    running_timeout = 300

    # OS configuration
    storage_os_find = "name: ubuntu-20.04 | arch: x86_64"

    # Network settings
    network_utility_ipv4 = True
    network_public_ipv4 = True

    # User settings
    user = "ubuntu"
    user_ssh_port = 22
    fix_local_hosts = True

    labels = "provider: my-cloud"
}

# Export provider configuration
{
    config: MyCloudConfig,
    server: MyCloudServer,
    defaults: my_cloud_defaults
}
```

### Creating a Cluster Extension

#### 1. Cluster Structure

```bash
# Create cluster directory
mkdir -p extensions/clusters/my-stack/kcl
cd extensions/clusters/my-stack/kcl

# Initialize
kcl mod init my-stack
```

#### 2. Cluster Schema

Create `my-stack.k`:
```kcl
"""
My Stack Cluster Configuration
Complete infrastructure stack with multiple services
"""

import provisioning.cluster as cluster
import provisioning.server as server

# Cluster configuration
schema MyStackCluster(cluster.ClusterDef):
    """My Stack cluster definition"""
    name: str = "my-stack"

    # Component configuration
    components: MyStackComponents

    # Infrastructure settings
    node_count: int = 3
    load_balancer: bool = True
    monitoring: bool = True
    logging: bool = True

schema MyStackComponents:
    """Components in My Stack"""
    # Web tier
    web_servers: int = 2
    web_instance_type: str = "standard-2vcpu-4gb"

    # Application tier
    app_servers: int = 3
    app_instance_type: str = "standard-4vcpu-8gb"

    # Database tier
    db_servers: int = 1
    db_instance_type: str = "standard-8vcpu-16gb"
    db_storage_size: int = 100

    # Cache tier
    cache_enabled: bool = True
    cache_instance_type: str = "standard-2vcpu-4gb"

# Generate server configurations for the cluster
my_stack_servers: [server.Server] = [
    # Web servers
    {
        hostname = "web-01"
        title = "Web Server 01"
        # ... web server configuration
        taskservs = [
            { name = "nginx", profile = "web" },
            { name = "ssl-cert", profile = "default" }
        ]
    },
    {
        hostname = "web-02"
        title = "Web Server 02"
        # ... web server configuration
        taskservs = [
            { name = "nginx", profile = "web" },
            { name = "ssl-cert", profile = "default" }
        ]
    },

    # Application servers
    {
        hostname = "app-01"
        title = "Application Server 01"
        # ... app server configuration
        taskservs = [
            { name = "kubernetes", profile = "worker" },
            { name = "containerd", profile = "default" }
        ]
    },

    # Database server
    {
        hostname = "db-01"
        title = "Database Server 01"
        # ... database configuration
        taskservs = [
            { name = "postgres", profile = "production" },
            { name = "backup-agent", profile = "default" }
        ]
    }
]

# Export cluster definition
{
    cluster: MyStackCluster,
    servers: my_stack_servers
}
```

## Infrastructure-Specific Extensions

### Creating Extensions for Specific Infrastructure

When you need extensions tailored to specific infrastructure or business requirements:

#### 1. Organization Structure

```bash
# Create organization-specific extension directory
mkdir -p extensions/taskservs/org-specific/my-company-app/kcl
cd extensions/taskservs/org-specific/my-company-app/kcl
```

#### 2. Company-Specific Taskserv

Create `my-company-app.k`:
```kcl
"""
My Company Application Taskserv
Company-specific application with custom requirements
"""

import provisioning.lib as lib
import provisioning.dependencies as deps

schema MyCompanyAppConfig:
    """Configuration for company-specific application"""
    # Application settings
    app_version: str = "latest"
    environment: "development" | "staging" | "production" = "production"

    # Company-specific settings
    company_domain: str
    internal_api_endpoint: str

    # Integration settings
    ldap_server?: str
    sso_provider?: str
    monitoring_endpoint?: str

    # Compliance settings
    encryption_enabled: bool = True
    audit_logging: bool = True
    data_retention_days: int = 90

    # Resource settings based on environment
    resources: MyCompanyAppResources

    check:
        len(company_domain) > 0, "Company domain required"
        len(internal_api_endpoint) > 0, "Internal API endpoint required"
        data_retention_days > 0, "Data retention must be positive"

schema MyCompanyAppResources:
    """Resource configuration based on environment"""
    cpu_request: str
    memory_request: str
    cpu_limit: str
    memory_limit: str
    storage_size: str
    replicas: int

# Environment-specific resource profiles
my_company_app_resources = {
    "development": {
        cpu_request = "100m"
        memory_request = "256Mi"
        cpu_limit = "500m"
        memory_limit = "512Mi"
        storage_size = "5Gi"
        replicas = 1
    },
    "staging": {
        cpu_request = "200m"
        memory_request = "512Mi"
        cpu_limit = "1"
        memory_limit = "1Gi"
        storage_size = "20Gi"
        replicas = 2
    },
    "production": {
        cpu_request = "500m"
        memory_request = "1Gi"
        cpu_limit = "2"
        memory_limit = "4Gi"
        storage_size = "100Gi"
        replicas = 5
    }
}

# Taskserv definition
schema MyCompanyAppTaskserv(lib.TaskServDef):
    """My Company App Taskserv Definition"""
    name: str = "my-company-app"
    config: MyCompanyAppConfig

# Dependencies for company app
my_company_app_dependencies: deps.TaskservDependencies = {
    name = "my-company-app"

    # Required infrastructure
    requires = ["kubernetes", "postgres", "redis"]

    # Integrations
    optional = ["monitoring", "logging", "backup"]

    # What this app provides
    provides = ["company-api", "company-ui", "company-reports"]

    # Company-specific requirements
    resources = {
        cpu = "500m"
        memory = "1Gi"
        disk = "10Gi"
        network = True
        privileged = False
    }

    # Compliance health checks
    health_checks = [
        {
            command = "curl -f https://app.company.com/health"
            interval = 30
            timeout = 10
            retries = 3
        },
        {
            command = "check-compliance-status"
            interval = 300  # Every 5 minutes
            timeout = 30
            retries = 1
        }
    ]

    # Installation phases
    phases = [
        {
            name = "pre-install"
            order = 1
            parallel = False
            required = True
        },
        {
            name = "install"
            order = 2
            parallel = True
            required = True
        },
        {
            name = "configure-integrations"
            order = 3
            parallel = False
            required = True
        },
        {
            name = "compliance-check"
            order = 4
            parallel = False
            required = True
        }
    ]

    # Compatibility
    os_support = ["linux"]
    arch_support = ["amd64"]
    k8s_versions = ["1.25+"]
}

# Export company app configuration
{
    config: MyCompanyAppTaskserv,
    dependencies: my_company_app_dependencies
}
```

### Multi-Environment Extensions

#### Environment-Specific Configurations

Create `environments/production.k`:
```kcl
"""
Production environment configuration for My Company App
"""

import ..my-company-app as app

production_config: app.MyCompanyAppConfig = {
    app_version = "v2.1.0"
    environment = "production"
    company_domain = "company.com"
    internal_api_endpoint = "https://api.internal.company.com"

    # Production integrations
    ldap_server = "ldap.company.com"
    sso_provider = "https://sso.company.com"
    monitoring_endpoint = "https://metrics.company.com"

    # Production compliance
    encryption_enabled = True
    audit_logging = True
    data_retention_days = 365

    # Production resources
    resources = {
        cpu_request = "1"
        memory_request = "2Gi"
        cpu_limit = "4"
        memory_limit = "8Gi"
        storage_size = "500Gi"
        replicas = 10
    }
}

production_config
```

## Publishing and Distribution

### Local Development

```bash
# Develop and test locally
cd extensions/taskservs/my-service
module-loader discover taskservs | grep my-service

# Load into test workspace
cd /tmp/test-workspace
module-loader load taskservs . [my-service]
```

### Version Control

```bash
# Create git repository for extension
cd extensions/taskservs/my-service
git init
git add .
git commit -m "Initial my-service taskserv"

# Tag versions
git tag v0.0.1
git push origin v0.0.1
```

### Package Distribution

#### Creating Extension Packages

```bash
# Create extension package
tar -czf my-service-taskserv-v0.0.1.tar.gz -C extensions/taskservs my-service

# Create bundle with multiple extensions
tar -czf company-extensions-v1.0.0.tar.gz \
    -C extensions/taskservs my-company-app \
    -C extensions/providers my-cloud \
    -C extensions/clusters my-stack
```

#### Distribution Methods

1. **Git Repository**
   ```bash
   # Users can clone and load
   git clone https://github.com/company/provisioning-extensions
   module-loader load taskservs . [my-company-app]
   ```

2. **Package Registry** (Future)
   ```bash
   # Publish to registry
   kcl publish extensions/taskservs/my-service

   # Users install from registry
   module-loader install taskserv my-service@0.0.1
   ```

3. **Internal Distribution**
   ```bash
   # Internal package server
   curl -O https://packages.company.com/my-service-v0.0.1.tar.gz
   tar -xzf my-service-v0.0.1.tar.gz -C extensions/taskservs/
   ```

## Best Practices

### Extension Development

1. **Follow Naming Conventions**
   - Use kebab-case for extension names
   - Prefix company-specific extensions with organization name
   - Use semantic versioning

2. **Schema Design**
   - Include comprehensive validation
   - Provide sensible defaults
   - Support multiple profiles/environments
   - Document all fields

3. **Dependencies**
   - Declare all dependencies explicitly
   - Use optional dependencies for enhanced features
   - Specify compatibility versions

4. **Testing**
   ```bash
   # Test extension loading
   module-loader discover taskservs | grep my-service

   # Test in isolated workspace
   mkdir test-workspace
   cd test-workspace
   workspace-init.nu . init
   module-loader load taskservs . [my-service]

   # Validate KCL compilation
   kcl check taskservs.k
   ```

### Security Considerations

1. **Secrets Management**
   - Never hardcode secrets in schemas
   - Use provisioning system's secrets management
   - Support external secret providers

2. **Input Validation**
   - Validate all user inputs
   - Use KCL check constraints
   - Sanitize external data

3. **Resource Limits**
   - Set reasonable resource defaults
   - Provide resource limit options
   - Include resource monitoring

### Documentation

1. **Extension Documentation**
   - Include README.md in each extension
   - Document configuration options
   - Provide usage examples
   - Include troubleshooting guide

2. **Version Documentation**
   - Maintain changelog
   - Document breaking changes
   - Include migration guides

### Example Directory Structure

```
extensions/
├── taskservs/
│   ├── my-service/
│   │   ├── kcl/
│   │   │   ├── kcl.mod
│   │   │   ├── my-service.k
│   │   │   ├── version.k
│   │   │   └── dependencies.k
│   │   ├── templates/
│   │   │   ├── deployment.yaml.j2
│   │   │   └── service.yaml.j2
│   │   ├── scripts/
│   │   │   ├── install.nu
│   │   │   └── health-check.nu
│   │   └── README.md
│   └── org-specific/
│       └── my-company-app/
│           ├── kcl/
│           │   ├── kcl.mod
│           │   ├── my-company-app.k
│           │   ├── version.k
│           │   └── environments/
│           │       ├── development.k
│           │       ├── staging.k
│           │       └── production.k
│           └── README.md
├── providers/
│   └── my-cloud/
│       ├── kcl/
│       │   ├── kcl.mod
│       │   ├── provision_my-cloud.k
│       │   ├── server_my-cloud.k
│       │   └── defaults_my-cloud.k
│       └── README.md
└── clusters/
    └── my-stack/
        ├── kcl/
        │   ├── kcl.mod
        │   └── my-stack.k
        └── README.md
```

This guide provides comprehensive coverage of KCL packaging and extension development for both general-purpose and infrastructure-specific use cases.