## Keyboard shortcuts

Press `←` or `→` to navigate between chapters

Press `S` or `/` to search in the book

Press `?` to show this help

Press `Esc` to hide this help

# Customize Infrastructure

Goal: Customize infrastructure using layers, templates, and configuration patterns Time: 20-40 minutes Difficulty: Intermediate to Advanced

## Overview

This guide covers:

1. Understanding the layer system
2. Using templates
3. Creating custom modules
4. Configuration inheritance
5. Advanced customization patterns

## The Layer System

### Understanding Layers

The provisioning system uses a 3-layer architecture for configuration inheritance:

    ┌─────────────────────────────────────┐
    │  Infrastructure Layer (Priority 300)│  ← Highest priority
    │  workspace/infra/{name}/            │
    │  • Project-specific configs         │
    │  • Environment customizations       │
    │  • Local overrides                  │
    └─────────────────────────────────────┘
                  ↓ overrides
    ┌─────────────────────────────────────┐
    │  Workspace Layer (Priority 200)     │
    │  provisioning/workspace/templates/  │
    │  • Reusable patterns                │
    │  • Organization standards           │
    │  • Team conventions                 │
    └─────────────────────────────────────┘
                  ↓ overrides
    ┌─────────────────────────────────────┐
    │  Core Layer (Priority 100)          │  ← Lowest priority
    │  provisioning/extensions/           │
    │  • System defaults                  │
    │  • Provider implementations         │
    │  • Default taskserv configs         │
    └─────────────────────────────────────┘

Resolution Order: Infrastructure (300) → Workspace (200) → Core (100)

Higher numbers override lower numbers.

### View Layer Resolution

    # Explain layer concept
    provisioning lyr explain

Expected Output:

    📚 LAYER SYSTEM EXPLAINED

    The layer system provides configuration inheritance across 3 levels:

    🔵 CORE LAYER (100) - System Defaults
       Location: provisioning/extensions/
       • Base taskserv configurations
       • Default provider settings
       • Standard cluster templates
       • Built-in extensions

    🟢 WORKSPACE LAYER (200) - Shared Templates
       Location: provisioning/workspace/templates/
       • Organization-wide patterns
       • Reusable configurations
       • Team standards
       • Custom extensions

    🔴 INFRASTRUCTURE LAYER (300) - Project Specific
       Location: workspace/infra/{project}/
       • Project-specific overrides
       • Environment customizations
       • Local modifications
       • Runtime settings

    Resolution: Infrastructure → Workspace → Core
    Higher priority layers override lower ones.

    # Show layer resolution for your project
    provisioning lyr show my-production

Expected Output:

    📊 Layer Resolution for my-production:

    LAYER            PRIORITY  SOURCE                              FILES
    Infrastructure   300       workspace/infra/my-production/      4 files
                               • servers.ncl (overrides)
                               • taskservs.ncl (overrides)
                               • clusters.ncl (custom)
                               • providers.ncl (overrides)

    Workspace        200       provisioning/workspace/templates/   2 files
                               • production.ncl (used)
                               • kubernetes.ncl (used)

    Core             100       provisioning/extensions/            15 files
                               • taskservs/* (base configs)
                               • providers/* (default settings)
                               • clusters/* (templates)

    Resolution Order: Infrastructure → Workspace → Core
    Status: ✅ All layers resolved successfully

### Test Layer Resolution

    # Test how a specific module resolves
    provisioning lyr test kubernetes my-production

Expected Output:

    🔍 Layer Resolution Test: kubernetes → my-production

    Resolving kubernetes configuration...

    🔴 Infrastructure Layer (300):
       ✅ Found: workspace/infra/my-production/taskservs/kubernetes.ncl
       Provides:
         • version = "1.30.0" (overrides)
         • control_plane_servers = ["web-01"] (overrides)
         • worker_servers = ["web-02"] (overrides)

    🟢 Workspace Layer (200):
       ✅ Found: provisioning/workspace/templates/production-kubernetes.ncl
       Provides:
         • security_policies (inherited)
         • network_policies (inherited)
         • resource_quotas (inherited)

    🔵 Core Layer (100):
       ✅ Found: provisioning/extensions/taskservs/kubernetes/main.ncl
       Provides:
         • default_version = "1.29.0" (base)
         • default_features (base)
         • default_plugins (base)

    Final Configuration (after merging all layers):
      version: "1.30.0" (from Infrastructure)
      control_plane_servers: ["web-01"] (from Infrastructure)
      worker_servers: ["web-02"] (from Infrastructure)
      security_policies: {...} (from Workspace)
      network_policies: {...} (from Workspace)
      resource_quotas: {...} (from Workspace)
      default_features: {...} (from Core)
      default_plugins: {...} (from Core)

    Resolution: ✅ Success

## Using Templates

### List Available Templates

    # List all templates
    provisioning tpl list

Expected Output:

    📋 Available Templates:

    TASKSERVS:
      • production-kubernetes    - Production-ready Kubernetes setup
      • production-postgres      - Production PostgreSQL with replication
      • production-redis         - Redis cluster with sentinel
      • development-kubernetes   - Development Kubernetes (minimal)
      • ci-cd-pipeline          - Complete CI/CD pipeline

    PROVIDERS:
      • upcloud-production      - UpCloud production settings
      • upcloud-development     - UpCloud development settings
      • aws-production          - AWS production VPC setup
      • aws-development         - AWS development environment
      • local-docker            - Local Docker-based setup

    CLUSTERS:
      • buildkit-cluster        - BuildKit for container builds
      • monitoring-stack        - Prometheus + Grafana + Loki
      • security-stack          - Security monitoring tools

    Total: 13 templates

    # List templates by type
    provisioning tpl list --type taskservs
    provisioning tpl list --type providers
    provisioning tpl list --type clusters

### View Template Details

    # Show template details
    provisioning tpl show production-kubernetes

Expected Output:

    📄 Template: production-kubernetes

    Description: Production-ready Kubernetes configuration with
                 security hardening, network policies, and monitoring

    Category: taskservs
    Version: 1.0.0

    Configuration Provided:
      • Kubernetes version: 1.30.0
      • Security policies: Pod Security Standards (restricted)
      • Network policies: Default deny + allow rules
      • Resource quotas: Per-namespace limits
      • Monitoring: Prometheus integration
      • Logging: Loki integration
      • Backup: Velero configuration

    Requirements:
      • Minimum 2 servers
      • 4 GB RAM per server
      • Network plugin (Cilium recommended)

    Location: provisioning/workspace/templates/production-kubernetes.ncl

    Example Usage:
      provisioning tpl apply production-kubernetes my-production

### Apply Template

    # Apply template to your infrastructure
    provisioning tpl apply production-kubernetes my-production

Expected Output:

    🚀 Applying template: production-kubernetes → my-production

    Checking compatibility... ⏳
    ✅ Infrastructure compatible with template

    Merging configuration... ⏳
    ✅ Configuration merged

    Files created/updated:
      • workspace/infra/my-production/taskservs/kubernetes.ncl (updated)
      • workspace/infra/my-production/policies/security.ncl (created)
      • workspace/infra/my-production/policies/network.ncl (created)
      • workspace/infra/my-production/monitoring/prometheus.ncl (created)

    🎉 Template applied successfully!

    Next steps:
      1. Review generated configuration
      2. Adjust as needed
      3. Deploy: provisioning t create kubernetes --infra my-production

### Validate Template Usage

    # Validate template was applied correctly
    provisioning tpl validate my-production

Expected Output:

    ✅ Template Validation: my-production

    Templates Applied:
      ✅ production-kubernetes (v1.0.0)
      ✅ production-postgres (v1.0.0)

    Configuration Status:
      ✅ All required fields present
      ✅ No conflicting settings
      ✅ Dependencies satisfied

    Compliance:
      ✅ Security policies configured
      ✅ Network policies configured
      ✅ Resource quotas set
      ✅ Monitoring enabled

    Status: ✅ Valid

## Creating Custom Templates

### Step 1: Create Template Structure

    # Create custom template directory
    mkdir -p provisioning/workspace/templates/my-custom-template

### Step 2: Write Template Configuration

File: `provisioning/workspace/templates/my-custom-template/main.ncl`

    # Custom Kubernetes template with specific settings
    let kubernetes_config = {
      # Version
      version = "1.30.0",

      # Custom feature gates
      feature_gates = {
        "GracefulNodeShutdown" = true,
        "SeccompDefault" = true,
        "StatefulSetAutoDeletePVC" = true,
      },

      # Custom kubelet configuration
      kubelet_config = {
        max_pods = 110,
        pod_pids_limit = 4096,
        container_log_max_size = "10Mi",
        container_log_max_files = 5,
      },

      # Custom API server flags
      apiserver_extra_args = {
        "enable-admission-plugins" = "NodeRestriction,PodSecurity,LimitRanger",
        "audit-log-maxage" = "30",
        "audit-log-maxbackup" = "10",
      },

      # Custom scheduler configuration
      scheduler_config = {
        profiles = [
          {
            name = "high-availability",
            plugins = {
              score = {
                enabled = [
                  {name = "NodeResourcesBalancedAllocation", weight = 2},
                  {name = "NodeResourcesLeastAllocated", weight = 1},
                ],
              },
            },
          },
        ],
      },

      # Network configuration
      network = {
        service_cidr = "10.96.0.0/12",
        pod_cidr = "10.244.0.0/16",
        dns_domain = "cluster.local",
      },

      # Security configuration
      security = {
        pod_security_standard = "restricted",
        encrypt_etcd = true,
        rotate_certificates = true,
      },
    } in
    kubernetes_config

### Step 3: Create Template Metadata

File: `provisioning/workspace/templates/my-custom-template/metadata.toml`

    [template]
    name = "my-custom-template"
    version = "1.0.0"
    description = "Custom Kubernetes template with enhanced security"
    category = "taskservs"
    author = "Your Name"

    [requirements]
    min_servers = 2
    min_memory_gb = 4
    required_taskservs = ["containerd", "cilium"]

    [tags]
    environment = ["production", "staging"]
    features = ["security", "monitoring", "high-availability"]

### Step 4: Test Custom Template

    # List templates (should include your custom template)
    provisioning tpl list

    # Show your template
    provisioning tpl show my-custom-template

    # Apply to test infrastructure
    provisioning tpl apply my-custom-template my-test

## Configuration Inheritance Examples

### Example 1: Override Single Value

Core Layer (`provisioning/extensions/taskservs/postgres/main.ncl`):

    let postgres_config = {
      version = "15.5",
      port = 5432,
      max_connections = 100,
    } in
    postgres_config

Infrastructure Layer (`workspace/infra/my-production/taskservs/postgres.ncl`):

    let postgres_config = {
      max_connections = 500,  # Override only max_connections
    } in
    postgres_config

Result (after layer resolution):

    let postgres_config = {
      version = "15.5",          # From Core
      port = 5432,               # From Core
      max_connections = 500,     # From Infrastructure (overridden)
    } in
    postgres_config

### Example 2: Add Custom Configuration

Workspace Layer (`provisioning/workspace/templates/production-postgres.ncl`):

    let postgres_config = {
      replication = {
        enabled = true,
        replicas = 2,
        sync_mode = "async",
      },
    } in
    postgres_config

Infrastructure Layer (`workspace/infra/my-production/taskservs/postgres.ncl`):

    let postgres_config = {
      replication = {
        sync_mode = "sync",  # Override sync mode
      },
      custom_extensions = ["pgvector", "timescaledb"],  # Add custom config
    } in
    postgres_config

Result:

    let postgres_config = {
      version = "15.5",          # From Core
      port = 5432,               # From Core
      max_connections = 100,     # From Core
      replication = {
        enabled = true,          # From Workspace
        replicas = 2,            # From Workspace
        sync_mode = "sync",      # From Infrastructure (overridden)
      },
      custom_extensions = ["pgvector", "timescaledb"],  # From Infrastructure (added)
    } in
    postgres_config

### Example 3: Environment-Specific Configuration

Workspace Layer (`provisioning/workspace/templates/base-kubernetes.ncl`):

    let kubernetes_config = {
      version = "1.30.0",
      control_plane_count = 3,
      worker_count = 5,
      resources = {
        control_plane = {cpu = "4", memory = "8Gi"},
        worker = {cpu = "8", memory = "16Gi"},
      },
    } in
    kubernetes_config

Development Infrastructure (`workspace/infra/my-dev/taskservs/kubernetes.ncl`):

    let kubernetes_config = {
      control_plane_count = 1,  # Smaller for dev
      worker_count = 2,
      resources = {
        control_plane = {cpu = "2", memory = "4Gi"},
        worker = {cpu = "2", memory = "4Gi"},
      },
    } in
    kubernetes_config

Production Infrastructure (`workspace/infra/my-prod/taskservs/kubernetes.ncl`):

    let kubernetes_config = {
      control_plane_count = 5,  # Larger for prod
      worker_count = 10,
      resources = {
        control_plane = {cpu = "8", memory = "16Gi"},
        worker = {cpu = "16", memory = "32Gi"},
      },
    } in
    kubernetes_config

## Advanced Customization Patterns

### Pattern 1: Multi-Environment Setup

Create different configurations for each environment:

    # Create environments
    provisioning ws init my-app-dev
    provisioning ws init my-app-staging
    provisioning ws init my-app-prod

    # Apply environment-specific templates
    provisioning tpl apply development-kubernetes my-app-dev
    provisioning tpl apply staging-kubernetes my-app-staging
    provisioning tpl apply production-kubernetes my-app-prod

    # Customize each environment
    # Edit: workspace/infra/my-app-dev/...
    # Edit: workspace/infra/my-app-staging/...
    # Edit: workspace/infra/my-app-prod/...

### Pattern 2: Shared Configuration Library

Create reusable configuration fragments:

File: `provisioning/workspace/templates/shared/security-policies.ncl`

    let security_policies = {
      pod_security = {
        enforce = "restricted",
        audit = "restricted",
        warn = "restricted",
      },
      network_policies = [
        {
          name = "deny-all",
          pod_selector = {},
          policy_types = ["Ingress", "Egress"],
        },
        {
          name = "allow-dns",
          pod_selector = {},
          egress = [
            {
              to = [{namespace_selector = {name = "kube-system"}}],
              ports = [{protocol = "UDP", port = 53}],
            },
          ],
        },
      ],
    } in
    security_policies

Import in your infrastructure:

    let security_policies = (import "../../../provisioning/workspace/templates/shared/security-policies.ncl") in

    let kubernetes_config = {
      version = "1.30.0",
      image_repo = "k8s.gcr.io",
      security = security_policies,  # Import shared policies
    } in
    kubernetes_config

### Pattern 3: Dynamic Configuration

Use Nickel features for dynamic configuration:

    # Calculate resources based on server count
    let server_count = 5 in
    let replicas_per_server = 2 in
    let total_replicas = server_count * replicas_per_server in

    let postgres_config = {
      version = "16.1",
      max_connections = total_replicas * 50,  # Dynamic calculation
      shared_buffers = "1024 MB",
    } in
    postgres_config

### Pattern 4: Conditional Configuration

    let environment = "production" in  # or "development"

    let kubernetes_config = {
      version = "1.30.0",
      control_plane_count = if environment == "production" then 3 else 1,
      worker_count = if environment == "production" then 5 else 2,
      monitoring = {
        enabled = environment == "production",
        retention = if environment == "production" then "30d" else "7d",
      },
    } in
    kubernetes_config

## Layer Statistics

    # Show layer system statistics
    provisioning lyr stats

Expected Output:

    📊 Layer System Statistics:

    Infrastructure Layer:
      • Projects: 3
      • Total files: 15
      • Average overrides per project: 5

    Workspace Layer:
      • Templates: 13
      • Most used: production-kubernetes (5 projects)
      • Custom templates: 2

    Core Layer:
      • Taskservs: 15
      • Providers: 3
      • Clusters: 3

    Resolution Performance:
      • Average resolution time: 45 ms
      • Cache hit rate: 87%
      • Total resolutions: 1,250

## Customization Workflow

### Complete Customization Example

    # 1. Create new infrastructure
    provisioning ws init my-custom-app

    # 2. Understand layer system
    provisioning lyr explain

    # 3. Discover templates
    provisioning tpl list --type taskservs

    # 4. Apply base template
    provisioning tpl apply production-kubernetes my-custom-app

    # 5. View applied configuration
    provisioning lyr show my-custom-app

    # 6. Customize (edit files)
    provisioning sops workspace/infra/my-custom-app/taskservs/kubernetes.ncl

    # 7. Test layer resolution
    provisioning lyr test kubernetes my-custom-app

    # 8. Validate configuration
    provisioning tpl validate my-custom-app
    provisioning val config --infra my-custom-app

    # 9. Deploy customized infrastructure
    provisioning s create --infra my-custom-app --check
    provisioning s create --infra my-custom-app
    provisioning t create kubernetes --infra my-custom-app

## Best Practices

### 1. Use Layers Correctly

- Core Layer: Only modify for system-wide changes
- Workspace Layer: Use for organization-wide templates
- Infrastructure Layer: Use for project-specific customizations

### 2. Template Organization

    provisioning/workspace/templates/
    ├── shared/           # Shared configuration fragments
    │   ├── security-policies.ncl
    │   ├── network-policies.ncl
    │   └── monitoring.ncl
    ├── production/       # Production templates
    │   ├── kubernetes.ncl
    │   ├── postgres.ncl
    │   └── redis.ncl
    └── development/      # Development templates
        ├── kubernetes.ncl
        └── postgres.ncl

### 3. Documentation

Document your customizations:

File: `workspace/infra/my-production/README.md`

    # My Production Infrastructure

    ## Customizations

    - Kubernetes: Using production template with 5 control plane nodes
    - PostgreSQL: Configured with streaming replication
    - Cilium: Native routing mode enabled

    ## Layer Overrides

    - `taskservs/kubernetes.ncl`: Control plane count (3 → 5)
    - `taskservs/postgres.ncl`: Replication mode (async → sync)
    - `network/cilium.ncl`: Routing mode (tunnel → native)

### 4. Version Control

Keep templates and configurations in version control:

    cd provisioning/workspace/templates/
    git add .
    git commit -m "Add production Kubernetes template with enhanced security"

    cd workspace/infra/my-production/
    git add .
    git commit -m "Configure production environment for my-production"

## Troubleshooting Customizations

### Issue: Configuration not applied

    # Check layer resolution
    provisioning lyr show my-production

    # Verify file exists
    ls -la workspace/infra/my-production/taskservs/

    # Test specific resolution
    provisioning lyr test kubernetes my-production

### Issue: Conflicting configurations

    # Validate configuration
    provisioning val config --infra my-production

    # Show configuration merge result
    provisioning show config kubernetes --infra my-production

### Issue: Template not found

    # List available templates
    provisioning tpl list

    # Check template path
    ls -la provisioning/workspace/templates/

    # Refresh template cache
    provisioning tpl refresh

## Next Steps

- From Scratch Guide - Deploy new infrastructure
- Update Guide - Update existing infrastructure
- Workflow Guide - Automate with workflows
- Nickel Guide - Learn Nickel configuration language

## Quick Reference

    # Layer system
    provisioning lyr explain              # Explain layers
    provisioning lyr show <project>       # Show layer resolution
    provisioning lyr test <module> <project>  # Test resolution
    provisioning lyr stats                # Layer statistics

    # Templates
    provisioning tpl list                 # List all templates
    provisioning tpl list --type <type>   # Filter by type
    provisioning tpl show <template>      # Show template details
    provisioning tpl apply <template> <project>  # Apply template
    provisioning tpl validate <project>   # Validate template usage

This guide is part of the provisioning project documentation. Last updated: 2025-09-30