# Customize Infrastructure - Provisioning Platform Documentation
Complete documentation for the Provisioning Platform - Infrastructure automation with Nushell, KCL, and Rust

## Keyboard shortcuts

Press `←` or `→` to navigate between chapters

Press `S` or `/` to search in the book

Press `?` to show this help

Press `Esc` to hide this help

# Customize Infrastructure

Goal: Customize infrastructure using layers, templates, and configuration patterns Time: 20-40 minutes Difficulty: Intermediate to Advanced

## Overview

This guide covers:

1. Understanding the layer system
2. Using templates
3. Creating custom modules
4. Configuration inheritance
5. Advanced customization patterns

## The Layer System

### Understanding Layers

The provisioning system uses a 3-layer architecture for configuration inheritance:

    ┌─────────────────────────────────────┐
    │  Infrastructure Layer (Priority 300)│  ← Highest priority
    │  workspace/infra/{name}/            │
    │  • Project-specific configs         │
    │  • Environment customizations       │
    │  • Local overrides                  │
    └─────────────────────────────────────┘
                  ↓ overrides
    ┌─────────────────────────────────────┐
    │  Workspace Layer (Priority 200)     │
    │  provisioning/workspace/templates/  │
    │  • Reusable patterns                │
    │  • Organization standards           │
    │  • Team conventions                 │
    └─────────────────────────────────────┘
                  ↓ overrides
    ┌─────────────────────────────────────┐
    │  Core Layer (Priority 100)          │  ← Lowest priority
    │  provisioning/extensions/           │
    │  • System defaults                  │
    │  • Provider implementations         │
    │  • Default taskserv configs         │
    └─────────────────────────────────────┘
    ```plaintext

    **Resolution Order**: Infrastructure (300) → Workspace (200) → Core (100)

    Higher numbers override lower numbers.

    ### View Layer Resolution

    ```bash
    # Explain layer concept
    provisioning lyr explain
    ```plaintext

    **Expected Output:**

    ```plaintext
    📚 LAYER SYSTEM EXPLAINED

    The layer system provides configuration inheritance across 3 levels:

    🔵 CORE LAYER (100) - System Defaults
       Location: provisioning/extensions/
       • Base taskserv configurations
       • Default provider settings
       • Standard cluster templates
       • Built-in extensions

    🟢 WORKSPACE LAYER (200) - Shared Templates
       Location: provisioning/workspace/templates/
       • Organization-wide patterns
       • Reusable configurations
       • Team standards
       • Custom extensions

    🔴 INFRASTRUCTURE LAYER (300) - Project Specific
       Location: workspace/infra/{project}/
       • Project-specific overrides
       • Environment customizations
       • Local modifications
       • Runtime settings

    Resolution: Infrastructure → Workspace → Core
    Higher priority layers override lower ones.
    ```plaintext

    ```bash
    # Show layer resolution for your project
    provisioning lyr show my-production
    ```plaintext

    **Expected Output:**

    ```plaintext
    📊 Layer Resolution for my-production:

    LAYER            PRIORITY  SOURCE                              FILES
    Infrastructure   300       workspace/infra/my-production/      4 files
                               • servers.k (overrides)
                               • taskservs.k (overrides)
                               • clusters.k (custom)
                               • providers.k (overrides)

    Workspace        200       provisioning/workspace/templates/   2 files
                               • production.k (used)
                               • kubernetes.k (used)

    Core             100       provisioning/extensions/            15 files
                               • taskservs/* (base configs)
                               • providers/* (default settings)
                               • clusters/* (templates)

    Resolution Order: Infrastructure → Workspace → Core
    Status: ✅ All layers resolved successfully
    ```plaintext

    ### Test Layer Resolution

    ```bash
    # Test how a specific module resolves
    provisioning lyr test kubernetes my-production
    ```plaintext

    **Expected Output:**

    ```plaintext
    🔍 Layer Resolution Test: kubernetes → my-production

    Resolving kubernetes configuration...

    🔴 Infrastructure Layer (300):
       ✅ Found: workspace/infra/my-production/taskservs/kubernetes.k
       Provides:
         • version = "1.30.0" (overrides)
         • control_plane_servers = ["web-01"] (overrides)
         • worker_servers = ["web-02"] (overrides)

    🟢 Workspace Layer (200):
       ✅ Found: provisioning/workspace/templates/production-kubernetes.k
       Provides:
         • security_policies (inherited)
         • network_policies (inherited)
         • resource_quotas (inherited)

    🔵 Core Layer (100):
       ✅ Found: provisioning/extensions/taskservs/kubernetes/config.k
       Provides:
         • default_version = "1.29.0" (base)
         • default_features (base)
         • default_plugins (base)

    Final Configuration (after merging all layers):
      version: "1.30.0" (from Infrastructure)
      control_plane_servers: ["web-01"] (from Infrastructure)
      worker_servers: ["web-02"] (from Infrastructure)
      security_policies: {...} (from Workspace)
      network_policies: {...} (from Workspace)
      resource_quotas: {...} (from Workspace)
      default_features: {...} (from Core)
      default_plugins: {...} (from Core)

    Resolution: ✅ Success
    ```plaintext

    ## Using Templates

    ### List Available Templates

    ```bash
    # List all templates
    provisioning tpl list
    ```plaintext

    **Expected Output:**

    ```plaintext
    📋 Available Templates:

    TASKSERVS:
      • production-kubernetes    - Production-ready Kubernetes setup
      • production-postgres      - Production PostgreSQL with replication
      • production-redis         - Redis cluster with sentinel
      • development-kubernetes   - Development Kubernetes (minimal)
      • ci-cd-pipeline          - Complete CI/CD pipeline

    PROVIDERS:
      • upcloud-production      - UpCloud production settings
      • upcloud-development     - UpCloud development settings
      • aws-production          - AWS production VPC setup
      • aws-development         - AWS development environment
      • local-docker            - Local Docker-based setup

    CLUSTERS:
      • buildkit-cluster        - BuildKit for container builds
      • monitoring-stack        - Prometheus + Grafana + Loki
      • security-stack          - Security monitoring tools

    Total: 13 templates
    ```plaintext

    ```bash
    # List templates by type
    provisioning tpl list --type taskservs
    provisioning tpl list --type providers
    provisioning tpl list --type clusters
    ```plaintext

    ### View Template Details

    ```bash
    # Show template details
    provisioning tpl show production-kubernetes
    ```plaintext

    **Expected Output:**

    ```plaintext
    📄 Template: production-kubernetes

    Description: Production-ready Kubernetes configuration with
                 security hardening, network policies, and monitoring

    Category: taskservs
    Version: 1.0.0

    Configuration Provided:
      • Kubernetes version: 1.30.0
      • Security policies: Pod Security Standards (restricted)
      • Network policies: Default deny + allow rules
      • Resource quotas: Per-namespace limits
      • Monitoring: Prometheus integration
      • Logging: Loki integration
      • Backup: Velero configuration

    Requirements:
      • Minimum 2 servers
      • 4GB RAM per server
      • Network plugin (Cilium recommended)

    Location: provisioning/workspace/templates/production-kubernetes.k

    Example Usage:
      provisioning tpl apply production-kubernetes my-production
    ```plaintext

    ### Apply Template

    ```bash
    # Apply template to your infrastructure
    provisioning tpl apply production-kubernetes my-production
    ```plaintext

    **Expected Output:**

    ```plaintext
    🚀 Applying template: production-kubernetes → my-production

    Checking compatibility... ⏳
    ✅ Infrastructure compatible with template

    Merging configuration... ⏳
    ✅ Configuration merged

    Files created/updated:
      • workspace/infra/my-production/taskservs/kubernetes.k (updated)
      • workspace/infra/my-production/policies/security.k (created)
      • workspace/infra/my-production/policies/network.k (created)
      • workspace/infra/my-production/monitoring/prometheus.k (created)

    🎉 Template applied successfully!

    Next steps:
      1. Review generated configuration
      2. Adjust as needed
      3. Deploy: provisioning t create kubernetes --infra my-production
    ```plaintext

    ### Validate Template Usage

    ```bash
    # Validate template was applied correctly
    provisioning tpl validate my-production
    ```plaintext

    **Expected Output:**

    ```plaintext
    ✅ Template Validation: my-production

    Templates Applied:
      ✅ production-kubernetes (v1.0.0)
      ✅ production-postgres (v1.0.0)

    Configuration Status:
      ✅ All required fields present
      ✅ No conflicting settings
      ✅ Dependencies satisfied

    Compliance:
      ✅ Security policies configured
      ✅ Network policies configured
      ✅ Resource quotas set
      ✅ Monitoring enabled

    Status: ✅ Valid
    ```plaintext

    ## Creating Custom Templates

    ### Step 1: Create Template Structure

    ```bash
    # Create custom template directory
    mkdir -p provisioning/workspace/templates/my-custom-template
    ```plaintext

    ### Step 2: Write Template Configuration

    **File: `provisioning/workspace/templates/my-custom-template/config.k`**

    ```kcl
    # Custom Kubernetes template with specific settings

    kubernetes_config = {
        # Version
        version = "1.30.0"

        # Custom feature gates
        feature_gates = {
            "GracefulNodeShutdown" = True
            "SeccompDefault" = True
            "StatefulSetAutoDeletePVC" = True
        }

        # Custom kubelet configuration
        kubelet_config = {
            max_pods = 110
            pod_pids_limit = 4096
            container_log_max_size = "10Mi"
            container_log_max_files = 5
        }

        # Custom API server flags
        apiserver_extra_args = {
            "enable-admission-plugins" = "NodeRestriction,PodSecurity,LimitRanger"
            "audit-log-maxage" = "30"
            "audit-log-maxbackup" = "10"
        }

        # Custom scheduler configuration
        scheduler_config = {
            profiles = [
                {
                    name = "high-availability"
                    plugins = {
                        score = {
                            enabled = [
                                {name = "NodeResourcesBalancedAllocation", weight = 2}
                                {name = "NodeResourcesLeastAllocated", weight = 1}
                            ]
                        }
                    }
                }
            ]
        }

        # Network configuration
        network = {
            service_cidr = "10.96.0.0/12"
            pod_cidr = "10.244.0.0/16"
            dns_domain = "cluster.local"
        }

        # Security configuration
        security = {
            pod_security_standard = "restricted"
            encrypt_etcd = True
            rotate_certificates = True
        }
    }
    ```plaintext

    ### Step 3: Create Template Metadata

    **File: `provisioning/workspace/templates/my-custom-template/metadata.toml`**

    ```toml
    [template]
    name = "my-custom-template"
    version = "1.0.0"
    description = "Custom Kubernetes template with enhanced security"
    category = "taskservs"
    author = "Your Name"

    [requirements]
    min_servers = 2
    min_memory_gb = 4
    required_taskservs = ["containerd", "cilium"]

    [tags]
    environment = ["production", "staging"]
    features = ["security", "monitoring", "high-availability"]
    ```plaintext

    ### Step 4: Test Custom Template

    ```bash
    # List templates (should include your custom template)
    provisioning tpl list

    # Show your template
    provisioning tpl show my-custom-template

    # Apply to test infrastructure
    provisioning tpl apply my-custom-template my-test
    ```plaintext

    ## Configuration Inheritance Examples

    ### Example 1: Override Single Value

    **Core Layer** (`provisioning/extensions/taskservs/postgres/config.k`):

    ```kcl
    postgres_config = {
        version = "15.5"
        port = 5432
        max_connections = 100
    }
    ```plaintext

    **Infrastructure Layer** (`workspace/infra/my-production/taskservs/postgres.k`):

    ```kcl
    postgres_config = {
        max_connections = 500  # Override only max_connections
    }
    ```plaintext

    **Result** (after layer resolution):

    ```kcl
    postgres_config = {
        version = "15.5"         # From Core
        port = 5432              # From Core
        max_connections = 500    # From Infrastructure (overridden)
    }
    ```plaintext

    ### Example 2: Add Custom Configuration

    **Workspace Layer** (`provisioning/workspace/templates/production-postgres.k`):

    ```kcl
    postgres_config = {
        replication = {
            enabled = True
            replicas = 2
            sync_mode = "async"
        }
    }
    ```plaintext

    **Infrastructure Layer** (`workspace/infra/my-production/taskservs/postgres.k`):

    ```kcl
    postgres_config = {
        replication = {
            sync_mode = "sync"  # Override sync mode
        }
        custom_extensions = ["pgvector", "timescaledb"]  # Add custom config
    }
    ```plaintext

    **Result**:

    ```kcl
    postgres_config = {
        version = "15.5"         # From Core
        port = 5432              # From Core
        max_connections = 100    # From Core
        replication = {
            enabled = True       # From Workspace
            replicas = 2         # From Workspace
            sync_mode = "sync"   # From Infrastructure (overridden)
        }
        custom_extensions = ["pgvector", "timescaledb"]  # From Infrastructure (added)
    }
    ```plaintext

    ### Example 3: Environment-Specific Configuration

    **Workspace Layer** (`provisioning/workspace/templates/base-kubernetes.k`):

    ```kcl
    kubernetes_config = {
        version = "1.30.0"
        control_plane_count = 3
        worker_count = 5
        resources = {
            control_plane = {cpu = "4", memory = "8Gi"}
            worker = {cpu = "8", memory = "16Gi"}
        }
    }
    ```plaintext

    **Development Infrastructure** (`workspace/infra/my-dev/taskservs/kubernetes.k`):

    ```kcl
    kubernetes_config = {
        control_plane_count = 1  # Smaller for dev
        worker_count = 2
        resources = {
            control_plane = {cpu = "2", memory = "4Gi"}
            worker = {cpu = "2", memory = "4Gi"}
        }
    }
    ```plaintext

    **Production Infrastructure** (`workspace/infra/my-prod/taskservs/kubernetes.k`):

    ```kcl
    kubernetes_config = {
        control_plane_count = 5  # Larger for prod
        worker_count = 10
        resources = {
            control_plane = {cpu = "8", memory = "16Gi"}
            worker = {cpu = "16", memory = "32Gi"}
        }
    }
    ```plaintext

    ## Advanced Customization Patterns

    ### Pattern 1: Multi-Environment Setup

    Create different configurations for each environment:

    ```bash
    # Create environments
    provisioning ws init my-app-dev
    provisioning ws init my-app-staging
    provisioning ws init my-app-prod

    # Apply environment-specific templates
    provisioning tpl apply development-kubernetes my-app-dev
    provisioning tpl apply staging-kubernetes my-app-staging
    provisioning tpl apply production-kubernetes my-app-prod

    # Customize each environment
    # Edit: workspace/infra/my-app-dev/...
    # Edit: workspace/infra/my-app-staging/...
    # Edit: workspace/infra/my-app-prod/...
    ```plaintext

    ### Pattern 2: Shared Configuration Library

    Create reusable configuration fragments:

    **File: `provisioning/workspace/templates/shared/security-policies.k`**

    ```kcl
    security_policies = {
        pod_security = {
            enforce = "restricted"
            audit = "restricted"
            warn = "restricted"
        }
        network_policies = [
            {
                name = "deny-all"
                pod_selector = {}
                policy_types = ["Ingress", "Egress"]
            },
            {
                name = "allow-dns"
                pod_selector = {}
                egress = [
                    {
                        to = [{namespace_selector = {name = "kube-system"}}]
                        ports = [{protocol = "UDP", port = 53}]
                    }
                ]
            }
        ]
    }
    ```plaintext

    Import in your infrastructure:

    ```kcl
    import "../../../provisioning/workspace/templates/shared/security-policies.k"

    kubernetes_config = {
        version = "1.30.0"
        # ... other config
        security = security_policies  # Import shared policies
    }
    ```plaintext

    ### Pattern 3: Dynamic Configuration

    Use KCL features for dynamic configuration:

    ```kcl
    # Calculate resources based on server count
    server_count = 5
    replicas_per_server = 2
    total_replicas = server_count * replicas_per_server

    postgres_config = {
        version = "16.1"
        max_connections = total_replicas * 50  # Dynamic calculation
        shared_buffers = "${total_replicas * 128}MB"
    }
    ```plaintext

    ### Pattern 4: Conditional Configuration

    ```kcl
    environment = "production"  # or "development"

    kubernetes_config = {
        version = "1.30.0"
        control_plane_count = if environment == "production" { 3 } else { 1 }
        worker_count = if environment == "production" { 5 } else { 2 }
        monitoring = {
            enabled = environment == "production"
            retention = if environment == "production" { "30d" } else { "7d" }
        }
    }
    ```plaintext

    ## Layer Statistics

    ```bash
    # Show layer system statistics
    provisioning lyr stats
    ```plaintext

    **Expected Output:**

    ```plaintext
    📊 Layer System Statistics:

    Infrastructure Layer:
      • Projects: 3
      • Total files: 15
      • Average overrides per project: 5

    Workspace Layer:
      • Templates: 13
      • Most used: production-kubernetes (5 projects)
      • Custom templates: 2

    Core Layer:
      • Taskservs: 15
      • Providers: 3
      • Clusters: 3

    Resolution Performance:
      • Average resolution time: 45ms
      • Cache hit rate: 87%
      • Total resolutions: 1,250
    ```plaintext

    ## Customization Workflow

    ### Complete Customization Example

    ```bash
    # 1. Create new infrastructure
    provisioning ws init my-custom-app

    # 2. Understand layer system
    provisioning lyr explain

    # 3. Discover templates
    provisioning tpl list --type taskservs

    # 4. Apply base template
    provisioning tpl apply production-kubernetes my-custom-app

    # 5. View applied configuration
    provisioning lyr show my-custom-app

    # 6. Customize (edit files)
    provisioning sops workspace/infra/my-custom-app/taskservs/kubernetes.k

    # 7. Test layer resolution
    provisioning lyr test kubernetes my-custom-app

    # 8. Validate configuration
    provisioning tpl validate my-custom-app
    provisioning val config --infra my-custom-app

    # 9. Deploy customized infrastructure
    provisioning s create --infra my-custom-app --check
    provisioning s create --infra my-custom-app
    provisioning t create kubernetes --infra my-custom-app
    ```plaintext

    ## Best Practices

    ### 1. Use Layers Correctly

    - **Core Layer**: Only modify for system-wide changes
    - **Workspace Layer**: Use for organization-wide templates
    - **Infrastructure Layer**: Use for project-specific customizations

    ### 2. Template Organization

    ```plaintext
    provisioning/workspace/templates/
    ├── shared/           # Shared configuration fragments
    │   ├── security-policies.k
    │   ├── network-policies.k
    │   └── monitoring.k
    ├── production/       # Production templates
    │   ├── kubernetes.k
    │   ├── postgres.k
    │   └── redis.k
    └── development/      # Development templates
        ├── kubernetes.k
        └── postgres.k
    ```plaintext

    ### 3. Documentation

    Document your customizations:

    **File: `workspace/infra/my-production/README.md`**

    ```markdown
    # My Production Infrastructure

    ## Customizations

    - Kubernetes: Using production template with 5 control plane nodes
    - PostgreSQL: Configured with streaming replication
    - Cilium: Native routing mode enabled

    ## Layer Overrides

    - `taskservs/kubernetes.k`: Control plane count (3 → 5)
    - `taskservs/postgres.k`: Replication mode (async → sync)
    - `network/cilium.k`: Routing mode (tunnel → native)
    ```plaintext

    ### 4. Version Control

    Keep templates and configurations in version control:

    ```bash
    cd provisioning/workspace/templates/
    git add .
    git commit -m "Add production Kubernetes template with enhanced security"

    cd workspace/infra/my-production/
    git add .
    git commit -m "Configure production environment for my-production"
    ```plaintext

    ## Troubleshooting Customizations

    ### Issue: Configuration not applied

    ```bash
    # Check layer resolution
    provisioning lyr show my-production

    # Verify file exists
    ls -la workspace/infra/my-production/taskservs/

    # Test specific resolution
    provisioning lyr test kubernetes my-production
    ```plaintext

    ### Issue: Conflicting configurations

    ```bash
    # Validate configuration
    provisioning val config --infra my-production

    # Show configuration merge result
    provisioning show config kubernetes --infra my-production
    ```plaintext

    ### Issue: Template not found

    ```bash
    # List available templates
    provisioning tpl list

    # Check template path
    ls -la provisioning/workspace/templates/

    # Refresh template cache
    provisioning tpl refresh
    ```plaintext

    ## Next Steps

    - **[From Scratch Guide](from-scratch.md)** - Deploy new infrastructure
    - **[Update Guide](update-infrastructure.md)** - Update existing infrastructure
    - **[Workflow Guide](../development/workflow.md)** - Automate with workflows
    - **[KCL Guide](../development/KCL_MODULE_GUIDE.md)** - Learn KCL configuration language

    ## Quick Reference

    ```bash
    # Layer system
    provisioning lyr explain              # Explain layers
    provisioning lyr show <project>       # Show layer resolution
    provisioning lyr test <module> <project>  # Test resolution
    provisioning lyr stats                # Layer statistics

    # Templates
    provisioning tpl list                 # List all templates
    provisioning tpl list --type <type>   # Filter by type
    provisioning tpl show <template>      # Show template details
    provisioning tpl apply <template> <project>  # Apply template
    provisioning tpl validate <project>   # Validate template usage
    ```plaintext

    ---

    *This guide is part of the provisioning project documentation. Last updated: 2025-09-30*