provisioning/workspace/README.md

# Layered Template Architecture System

This workspace provides a combined **Layered Extension Architecture with Override System** and **Template-Based Infrastructure Pattern Library** that maintains PAP principles while enabling maximum reusability of infrastructure configurations.

## 🏗️ Architecture Overview

### Layer Hierarchy
The system resolves configurations through a three-tier layer system:

1. **Core Layer (Priority 100)** - `provisioning/extensions/`
   - Base provisioning system extensions
   - Core taskservs, providers, and clusters

2. **Workspace Layer (Priority 200)** - `provisioning/workspace/templates/`
   - Shared templates extracted from proven infrastructure patterns
   - Reusable configurations across multiple infrastructures

3. **Infrastructure Layer (Priority 300)** - `workspace/infra/{name}/`
   - Infrastructure-specific configurations and overrides
   - Custom implementations per infrastructure

### Directory Structure

```
provisioning/workspace/
├── templates/                    # Template library
│   ├── taskservs/               # Taskserv configuration templates
│   │   ├── kubernetes/          # Kubernetes templates
│   │   │   ├── base.k          # Base configuration
│   │   │   └── variants/       # HA, single-node variants
│   │   ├── storage/            # Storage system templates
│   │   ├── networking/         # Network configuration templates
│   │   └── container-runtime/  # Container runtime templates
│   ├── providers/              # Provider templates
│   │   ├── upcloud/           # UpCloud provider templates
│   │   └── aws/               # AWS provider templates
│   ├── servers/               # Server configuration patterns
│   └── clusters/              # Complete cluster templates
├── layers/                    # Layer definitions
│   ├── core.layer.k          # Core layer definition
│   ├── workspace.layer.k     # Workspace layer definition
│   └── infra.layer.k         # Infrastructure layer definition
├── registry/                 # Extension registry
│   ├── manifest.yaml         # Template catalog and metadata
│   └── imports.k            # Central import aliases
├── templates/lib/           # Composition utilities
│   ├── compose.k           # KCL composition functions
│   └── override.k          # Override and layer utilities
└── tools/                  # Migration and management tools
    └── migrate-infra.nu    # Infrastructure migration tool
```

## 🚀 Getting Started

### 1. Extract Existing Patterns

Extract patterns from existing infrastructure (e.g., wuji) to create reusable templates:

```bash
# Extract all patterns from wuji infrastructure
cd provisioning/workspace/tools
./migrate-infra.nu extract wuji

# Extract specific types only
./migrate-infra.nu extract wuji --type taskservs
```

### 2. Use Enhanced Module Loader

The enhanced module loader provides template and layer management:

```bash
# List available templates
provisioning/core/cli/module-loader-enhanced template list

# Show layer resolution order
provisioning/core/cli/module-loader-enhanced layer show

# Test layer resolution for a specific module
provisioning/core/cli/module-loader-enhanced layer test kubernetes --infra wuji
```

### 3. Apply Templates to New Infrastructure

```bash
# Apply kubernetes template to new infrastructure
provisioning/core/cli/module-loader-enhanced template apply kubernetes-base new-infra --provider upcloud

# Load taskservs using templates
provisioning/core/cli/module-loader-enhanced load enhanced taskservs workspace/infra/new-infra [kubernetes, cilium] --layer workspace
```

## 📋 Usage Examples

### Creating a New Infrastructure from Templates

```bash
# 1. Create directory structure
mkdir -p workspace/infra/my-new-infra/{taskservs,defs,overrides}

# 2. Apply base templates
cd provisioning
./core/cli/module-loader-enhanced template apply kubernetes-base my-new-infra

# 3. Customize for your needs
# Edit workspace/infra/my-new-infra/taskservs/kubernetes.k

# 4. Test layer resolution
./core/cli/module-loader-enhanced layer test kubernetes --infra my-new-infra
```

### Converting Existing Infrastructure

```bash
# 1. Extract patterns to templates
cd provisioning/workspace/tools
./migrate-infra.nu extract existing-infra

# 2. Convert infrastructure to use templates
./migrate-infra.nu convert existing-infra

# 3. Validate conversion
./migrate-infra.nu validate existing-infra
```

### Template Development

```kcl
# Create a new taskserv template
# provisioning/workspace/templates/taskservs/my-service/base.k

import taskservs.my_service.kcl.my_service as service_core
import ../../../workspace/templates/lib/compose as comp

schema MyServiceBase {
    version: str = "1.0.0"
    cluster_name: str
    # ... configuration options
}

def create_my_service [cluster_name: str, overrides: any = {}] -> any {
    let base_config = MyServiceBase { cluster_name = $cluster_name }
    let final_config = comp.deep_merge $base_config $overrides

    service_core.MyService $final_config
}
```

## 🔧 Configuration Composition

### Using Templates in Infrastructure

```kcl
# workspace/infra/my-infra/taskservs/kubernetes.k

import provisioning.workspace.registry.imports as reg
import provisioning.workspace.templates.lib.override as ovr

# Use base template with infrastructure-specific overrides
_taskserv = ovr.infrastructure_overrides.taskserv_override(
    reg.tpl_kubernetes_base.kubernetes_base,
    "my-infra",
    "kubernetes",
    {
        cluster_name: "my-infra"
        version: "1.31.0"          # Override version
        cri: "crio"                # Override container runtime
        # Custom network configuration
        network_config: {
            pod_cidr: "10.244.0.0/16"
            service_cidr: "10.96.0.0/12"
        }
    }
)
```

### Layer Composition

```kcl
# Compose configuration through all layers
import provisioning.workspace.templates.lib.compose as comp

# Manual layer composition
final_config = comp.compose_templates(
    $core_config,      # From provisioning/extensions
    $workspace_config, # From provisioning/workspace/templates
    $infra_config      # From workspace/infra/{name}
)
```

## 🛠️ Advanced Features

### Provider-Aware Composition

```kcl
import provisioning.workspace.templates.lib.override as ovr

# Apply provider-specific configurations
config = ovr.override_patterns.env_override(
    $base_config,
    "upcloud",
    {
        upcloud: { zone: "es-mad1", plan: "2xCPU-4GB" },
        aws: { region: "us-west-2", instance_type: "t3.medium" },
        local: { memory: "4GB", cpus: 2 }
    }
)
```

### Conditional Overrides

```kcl
# Infrastructure-specific conditional overrides
config = ovr.layer_resolution.infra_conditional(
    $base_config,
    $infra_name,
    {
        "production": { ha: true, replicas: 3 },
        "development": { ha: false, replicas: 1 },
        "default": { ha: false, replicas: 1 }
    }
)
```

## 📚 Benefits

### ✅ Maintains PAP Principles
- **Configuration-driven**: No hardcoded values
- **Modular**: Clear separation of concerns
- **Declarative**: Infrastructure as code
- **Reusable**: DRY principle throughout

### ✅ Flexible Override System
- **Layer-based resolution**: Clean precedence order
- **Selective overrides**: Override only what's needed
- **Provider-agnostic**: Works across all providers
- **Environment-aware**: Dev/test/prod configurations

### ✅ Template Reusability
- **Pattern extraction**: Learn from existing infrastructures
- **Template versioning**: Track evolution over time
- **Composition utilities**: Rich KCL composition functions
- **Migration tools**: Easy conversion process

### ✅ No Core Changes
- **Non-invasive**: Core provisioning system unchanged
- **Backward compatible**: Existing infrastructures continue working
- **Progressive adoption**: Migrate at your own pace
- **Extensible**: Add new templates and layers easily

## 🔄 Migration Path

1. **Extract patterns** from existing infrastructures using `migrate-infra.nu extract`
2. **Create templates** in `provisioning/workspace/templates/`
3. **Convert infrastructures** to use templates with `migrate-infra.nu convert`
4. **Validate** the conversion with `migrate-infra.nu validate`
5. **Test** layer resolution with enhanced module loader
6. **Iterate** and improve templates based on usage

## 📖 Further Reading

- **Layer Definitions**: See `layers/*.layer.k` for layer configuration
- **Template Examples**: Browse `templates/` for real-world patterns
- **Composition Utilities**: Check `templates/lib/` for KCL utilities
- **Migration Tools**: Use `tools/migrate-infra.nu` for infrastructure conversion
- **Registry System**: Explore `registry/` for template metadata and imports

This system provides the perfect balance of flexibility, reusability, and maintainability while preserving the core provisioning system's integrity.