provisioning/workspace/README.md

Layered Template Architecture System

This workspace provides a combined Layered Extension Architecture with Override System and Template-Based Infrastructure Pattern Library that maintains PAP principles while enabling maximum reusability of infrastructure configurations.

🏗️ Architecture Overview

Layer Hierarchy

The system resolves configurations through a three-tier layer system:

1. Core Layer (Priority 100) - provisioning/extensions/

   • Base provisioning system extensions
   • Core taskservs, providers, and clusters

2. Workspace Layer (Priority 200) - provisioning/workspace/templates/

   • Shared templates extracted from proven infrastructure patterns
   • Reusable configurations across multiple infrastructures

3. Infrastructure Layer (Priority 300) - workspace/infra/{name}/

   • Infrastructure-specific configurations and overrides
   • Custom implementations per infrastructure

Directory Structure

provisioning/workspace/
├── templates/                    # Template library
│   ├── taskservs/               # Taskserv configuration templates
│   │   ├── kubernetes/          # Kubernetes templates
│   │   │   ├── base.k          # Base configuration
│   │   │   └── variants/       # HA, single-node variants
│   │   ├── storage/            # Storage system templates
│   │   ├── networking/         # Network configuration templates
│   │   └── container-runtime/  # Container runtime templates
│   ├── providers/              # Provider templates
│   │   ├── upcloud/           # UpCloud provider templates
│   │   └── aws/               # AWS provider templates
│   ├── servers/               # Server configuration patterns
│   └── clusters/              # Complete cluster templates
├── layers/                    # Layer definitions
│   ├── core.layer.k          # Core layer definition
│   ├── workspace.layer.k     # Workspace layer definition
│   └── infra.layer.k         # Infrastructure layer definition
├── registry/                 # Extension registry
│   ├── manifest.yaml         # Template catalog and metadata
│   └── imports.k            # Central import aliases
├── templates/lib/           # Composition utilities
│   ├── compose.k           # KCL composition functions
│   └── override.k          # Override and layer utilities
└── tools/                  # Migration and management tools
    └── migrate-infra.nu    # Infrastructure migration tool

🚀 Getting Started

1. Extract Existing Patterns

Extract patterns from existing infrastructure (e.g., wuji) to create reusable templates:

# Extract all patterns from wuji infrastructure
cd provisioning/workspace/tools
./migrate-infra.nu extract wuji

# Extract specific types only
./migrate-infra.nu extract wuji --type taskservs

2. Use Enhanced Module Loader

The enhanced module loader provides template and layer management:

# List available templates
provisioning/core/cli/module-loader-enhanced template list

# Show layer resolution order
provisioning/core/cli/module-loader-enhanced layer show

# Test layer resolution for a specific module
provisioning/core/cli/module-loader-enhanced layer test kubernetes --infra wuji

3. Apply Templates to New Infrastructure

# Apply kubernetes template to new infrastructure
provisioning/core/cli/module-loader-enhanced template apply kubernetes-base new-infra --provider upcloud

# Load taskservs using templates
provisioning/core/cli/module-loader-enhanced load enhanced taskservs workspace/infra/new-infra [kubernetes, cilium] --layer workspace

📋 Usage Examples

Creating a New Infrastructure from Templates

# 1. Create directory structure
mkdir -p workspace/infra/my-new-infra/{taskservs,defs,overrides}

# 2. Apply base templates
cd provisioning
./core/cli/module-loader-enhanced template apply kubernetes-base my-new-infra

# 3. Customize for your needs
# Edit workspace/infra/my-new-infra/taskservs/kubernetes.k

# 4. Test layer resolution
./core/cli/module-loader-enhanced layer test kubernetes --infra my-new-infra

Converting Existing Infrastructure

# 1. Extract patterns to templates
cd provisioning/workspace/tools
./migrate-infra.nu extract existing-infra

# 2. Convert infrastructure to use templates
./migrate-infra.nu convert existing-infra

# 3. Validate conversion
./migrate-infra.nu validate existing-infra

Template Development

# Create a new taskserv template
# provisioning/workspace/templates/taskservs/my-service/base.k

import taskservs.my_service.kcl.my_service as service_core
import ../../../workspace/templates/lib/compose as comp

schema MyServiceBase {
    version: str = "1.0.0"
    cluster_name: str
    # ... configuration options
}

def create_my_service [cluster_name: str, overrides: any = {}] -> any {
    let base_config = MyServiceBase { cluster_name = $cluster_name }
    let final_config = comp.deep_merge $base_config $overrides

    service_core.MyService $final_config
}

🔧 Configuration Composition

Using Templates in Infrastructure

# workspace/infra/my-infra/taskservs/kubernetes.k

import provisioning.workspace.registry.imports as reg
import provisioning.workspace.templates.lib.override as ovr

# Use base template with infrastructure-specific overrides
_taskserv = ovr.infrastructure_overrides.taskserv_override(
    reg.tpl_kubernetes_base.kubernetes_base,
    "my-infra",
    "kubernetes",
    {
        cluster_name: "my-infra"
        version: "1.31.0"          # Override version
        cri: "crio"                # Override container runtime
        # Custom network configuration
        network_config: {
            pod_cidr: "10.244.0.0/16"
            service_cidr: "10.96.0.0/12"
        }
    }
)

Layer Composition

# Compose configuration through all layers
import provisioning.workspace.templates.lib.compose as comp

# Manual layer composition
final_config = comp.compose_templates(
    $core_config,      # From provisioning/extensions
    $workspace_config, # From provisioning/workspace/templates
    $infra_config      # From workspace/infra/{name}
)

🛠️ Advanced Features

Provider-Aware Composition

import provisioning.workspace.templates.lib.override as ovr

# Apply provider-specific configurations
config = ovr.override_patterns.env_override(
    $base_config,
    "upcloud",
    {
        upcloud: { zone: "es-mad1", plan: "2xCPU-4GB" },
        aws: { region: "us-west-2", instance_type: "t3.medium" },
        local: { memory: "4GB", cpus: 2 }
    }
)

Conditional Overrides

# Infrastructure-specific conditional overrides
config = ovr.layer_resolution.infra_conditional(
    $base_config,
    $infra_name,
    {
        "production": { ha: true, replicas: 3 },
        "development": { ha: false, replicas: 1 },
        "default": { ha: false, replicas: 1 }
    }
)

📚 Benefits

✅ Maintains PAP Principles

• Configuration-driven: No hardcoded values
• Modular: Clear separation of concerns
• Declarative: Infrastructure as code
• Reusable: DRY principle throughout

✅ Flexible Override System

• Layer-based resolution: Clean precedence order
• Selective overrides: Override only what's needed
• Provider-agnostic: Works across all providers
• Environment-aware: Dev/test/prod configurations

✅ Template Reusability

• Pattern extraction: Learn from existing infrastructures
• Template versioning: Track evolution over time
• Composition utilities: Rich KCL composition functions
• Migration tools: Easy conversion process

✅ No Core Changes

• Non-invasive: Core provisioning system unchanged
• Backward compatible: Existing infrastructures continue working
• Progressive adoption: Migrate at your own pace
• Extensible: Add new templates and layers easily

🔄 Migration Path

1. Extract patterns from existing infrastructures using migrate-infra.nu extract
2. Create templates in provisioning/workspace/templates/
3. Convert infrastructures to use templates with migrate-infra.nu convert
4. Validate the conversion with migrate-infra.nu validate
5. Test layer resolution with enhanced module loader
6. Iterate and improve templates based on usage

📖 Further Reading

• Layer Definitions: See layers/*.layer.k for layer configuration
• Template Examples: Browse templates/ for real-world patterns
• Composition Utilities: Check templates/lib/ for KCL utilities
• Migration Tools: Use tools/migrate-infra.nu for infrastructure conversion
• Registry System: Explore registry/ for template metadata and imports

This system provides the perfect balance of flexibility, reusability, and maintainability while preserving the core provisioning system's integrity.