# Customize Infrastructure

**Goal**: Customize infrastructure using layers, templates, and configuration patterns
**Time**: 20-40 minutes
**Difficulty**: Intermediate to Advanced

## Overview

This guide covers:

1. Understanding the layer system
2. Using templates
3. Creating custom modules
4. Configuration inheritance
5. Advanced customization patterns

## The Layer System

### Understanding Layers

The provisioning system uses a **3-layer architecture** for configuration inheritance:

```plaintext
┌─────────────────────────────────────┐
│  Infrastructure Layer (Priority 300)│  ← Highest priority
│  workspace/infra/{name}/            │
│  • Project-specific configs         │
│  • Environment customizations       │
│  • Local overrides                  │
└─────────────────────────────────────┘
              ↓ overrides
┌─────────────────────────────────────┐
│  Workspace Layer (Priority 200)     │
│  provisioning/workspace/templates/  │
│  • Reusable patterns                │
│  • Organization standards           │
│  • Team conventions                 │
└─────────────────────────────────────┘
              ↓ overrides
┌─────────────────────────────────────┐
│  Core Layer (Priority 100)          │  ← Lowest priority
│  provisioning/extensions/           │
│  • System defaults                  │
│  • Provider implementations         │
│  • Default taskserv configs         │
└─────────────────────────────────────┘
```plaintext

**Resolution Order**: Infrastructure (300) → Workspace (200) → Core (100)

Higher numbers override lower numbers.

### View Layer Resolution

```bash
# Explain layer concept
provisioning lyr explain
```plaintext

**Expected Output:**

```plaintext
📚 LAYER SYSTEM EXPLAINED

The layer system provides configuration inheritance across 3 levels:

🔵 CORE LAYER (100) - System Defaults
   Location: provisioning/extensions/
   • Base taskserv configurations
   • Default provider settings
   • Standard cluster templates
   • Built-in extensions

🟢 WORKSPACE LAYER (200) - Shared Templates
   Location: provisioning/workspace/templates/
   • Organization-wide patterns
   • Reusable configurations
   • Team standards
   • Custom extensions

🔴 INFRASTRUCTURE LAYER (300) - Project Specific
   Location: workspace/infra/{project}/
   • Project-specific overrides
   • Environment customizations
   • Local modifications
   • Runtime settings

Resolution: Infrastructure → Workspace → Core
Higher priority layers override lower ones.
```plaintext

```bash
# Show layer resolution for your project
provisioning lyr show my-production
```plaintext

**Expected Output:**

```plaintext
📊 Layer Resolution for my-production:

LAYER            PRIORITY  SOURCE                              FILES
Infrastructure   300       workspace/infra/my-production/      4 files
                           • servers.ncl (overrides)
                           • taskservs.ncl (overrides)
                           • clusters.ncl (custom)
                           • providers.ncl (overrides)

Workspace        200       provisioning/workspace/templates/   2 files
                           • production.ncl (used)
                           • kubernetes.ncl (used)

Core             100       provisioning/extensions/            15 files
                           • taskservs/* (base configs)
                           • providers/* (default settings)
                           • clusters/* (templates)

Resolution Order: Infrastructure → Workspace → Core
Status: ✅ All layers resolved successfully
```plaintext

### Test Layer Resolution

```bash
# Test how a specific module resolves
provisioning lyr test kubernetes my-production
```plaintext

**Expected Output:**

```plaintext
🔍 Layer Resolution Test: kubernetes → my-production

Resolving kubernetes configuration...

🔴 Infrastructure Layer (300):
   ✅ Found: workspace/infra/my-production/taskservs/kubernetes.ncl
   Provides:
     • version = "1.30.0" (overrides)
     • control_plane_servers = ["web-01"] (overrides)
     • worker_servers = ["web-02"] (overrides)

🟢 Workspace Layer (200):
   ✅ Found: provisioning/workspace/templates/production-kubernetes.ncl
   Provides:
     • security_policies (inherited)
     • network_policies (inherited)
     • resource_quotas (inherited)

🔵 Core Layer (100):
   ✅ Found: provisioning/extensions/taskservs/kubernetes/main.ncl
   Provides:
     • default_version = "1.29.0" (base)
     • default_features (base)
     • default_plugins (base)

Final Configuration (after merging all layers):
  version: "1.30.0" (from Infrastructure)
  control_plane_servers: ["web-01"] (from Infrastructure)
  worker_servers: ["web-02"] (from Infrastructure)
  security_policies: {...} (from Workspace)
  network_policies: {...} (from Workspace)
  resource_quotas: {...} (from Workspace)
  default_features: {...} (from Core)
  default_plugins: {...} (from Core)

Resolution: ✅ Success
```plaintext

## Using Templates

### List Available Templates

```bash
# List all templates
provisioning tpl list
```plaintext

**Expected Output:**

```plaintext
📋 Available Templates:

TASKSERVS:
  • production-kubernetes    - Production-ready Kubernetes setup
  • production-postgres      - Production PostgreSQL with replication
  • production-redis         - Redis cluster with sentinel
  • development-kubernetes   - Development Kubernetes (minimal)
  • ci-cd-pipeline          - Complete CI/CD pipeline

PROVIDERS:
  • upcloud-production      - UpCloud production settings
  • upcloud-development     - UpCloud development settings
  • aws-production          - AWS production VPC setup
  • aws-development         - AWS development environment
  • local-docker            - Local Docker-based setup

CLUSTERS:
  • buildkit-cluster        - BuildKit for container builds
  • monitoring-stack        - Prometheus + Grafana + Loki
  • security-stack          - Security monitoring tools

Total: 13 templates
```plaintext

```bash
# List templates by type
provisioning tpl list --type taskservs
provisioning tpl list --type providers
provisioning tpl list --type clusters
```plaintext

### View Template Details

```bash
# Show template details
provisioning tpl show production-kubernetes
```plaintext

**Expected Output:**

```plaintext
📄 Template: production-kubernetes

Description: Production-ready Kubernetes configuration with
             security hardening, network policies, and monitoring

Category: taskservs
Version: 1.0.0

Configuration Provided:
  • Kubernetes version: 1.30.0
  • Security policies: Pod Security Standards (restricted)
  • Network policies: Default deny + allow rules
  • Resource quotas: Per-namespace limits
  • Monitoring: Prometheus integration
  • Logging: Loki integration
  • Backup: Velero configuration

Requirements:
  • Minimum 2 servers
  • 4 GB RAM per server
  • Network plugin (Cilium recommended)

Location: provisioning/workspace/templates/production-kubernetes.ncl

Example Usage:
  provisioning tpl apply production-kubernetes my-production
```plaintext

### Apply Template

```bash
# Apply template to your infrastructure
provisioning tpl apply production-kubernetes my-production
```plaintext

**Expected Output:**

```plaintext
🚀 Applying template: production-kubernetes → my-production

Checking compatibility... ⏳
✅ Infrastructure compatible with template

Merging configuration... ⏳
✅ Configuration merged

Files created/updated:
  • workspace/infra/my-production/taskservs/kubernetes.ncl (updated)
  • workspace/infra/my-production/policies/security.ncl (created)
  • workspace/infra/my-production/policies/network.ncl (created)
  • workspace/infra/my-production/monitoring/prometheus.ncl (created)

🎉 Template applied successfully!

Next steps:
  1. Review generated configuration
  2. Adjust as needed
  3. Deploy: provisioning t create kubernetes --infra my-production
```plaintext

### Validate Template Usage

```bash
# Validate template was applied correctly
provisioning tpl validate my-production
```plaintext

**Expected Output:**

```plaintext
✅ Template Validation: my-production

Templates Applied:
  ✅ production-kubernetes (v1.0.0)
  ✅ production-postgres (v1.0.0)

Configuration Status:
  ✅ All required fields present
  ✅ No conflicting settings
  ✅ Dependencies satisfied

Compliance:
  ✅ Security policies configured
  ✅ Network policies configured
  ✅ Resource quotas set
  ✅ Monitoring enabled

Status: ✅ Valid
```plaintext

## Creating Custom Templates

### Step 1: Create Template Structure

```bash
# Create custom template directory
mkdir -p provisioning/workspace/templates/my-custom-template
```plaintext

### Step 2: Write Template Configuration

**File: `provisioning/workspace/templates/my-custom-template/main.ncl`**

```nickel
# Custom Kubernetes template with specific settings
let kubernetes_config = {
  # Version
  version = "1.30.0",

  # Custom feature gates
  feature_gates = {
    "GracefulNodeShutdown" = true,
    "SeccompDefault" = true,
    "StatefulSetAutoDeletePVC" = true,
  },

  # Custom kubelet configuration
  kubelet_config = {
    max_pods = 110,
    pod_pids_limit = 4096,
    container_log_max_size = "10Mi",
    container_log_max_files = 5,
  },

  # Custom API server flags
  apiserver_extra_args = {
    "enable-admission-plugins" = "NodeRestriction,PodSecurity,LimitRanger",
    "audit-log-maxage" = "30",
    "audit-log-maxbackup" = "10",
  },

  # Custom scheduler configuration
  scheduler_config = {
    profiles = [
      {
        name = "high-availability",
        plugins = {
          score = {
            enabled = [
              {name = "NodeResourcesBalancedAllocation", weight = 2},
              {name = "NodeResourcesLeastAllocated", weight = 1},
            ],
          },
        },
      },
    ],
  },

  # Network configuration
  network = {
    service_cidr = "10.96.0.0/12",
    pod_cidr = "10.244.0.0/16",
    dns_domain = "cluster.local",
  },

  # Security configuration
  security = {
    pod_security_standard = "restricted",
    encrypt_etcd = true,
    rotate_certificates = true,
  },
} in
kubernetes_config
```plaintext

### Step 3: Create Template Metadata

**File: `provisioning/workspace/templates/my-custom-template/metadata.toml`**

```toml
[template]
name = "my-custom-template"
version = "1.0.0"
description = "Custom Kubernetes template with enhanced security"
category = "taskservs"
author = "Your Name"

[requirements]
min_servers = 2
min_memory_gb = 4
required_taskservs = ["containerd", "cilium"]

[tags]
environment = ["production", "staging"]
features = ["security", "monitoring", "high-availability"]
```plaintext

### Step 4: Test Custom Template

```bash
# List templates (should include your custom template)
provisioning tpl list

# Show your template
provisioning tpl show my-custom-template

# Apply to test infrastructure
provisioning tpl apply my-custom-template my-test
```plaintext

## Configuration Inheritance Examples

### Example 1: Override Single Value

**Core Layer** (`provisioning/extensions/taskservs/postgres/main.ncl`):

```nickel
let postgres_config = {
  version = "15.5",
  port = 5432,
  max_connections = 100,
} in
postgres_config
```plaintext

**Infrastructure Layer** (`workspace/infra/my-production/taskservs/postgres.ncl`):

```nickel
let postgres_config = {
  max_connections = 500,  # Override only max_connections
} in
postgres_config
```plaintext

**Result** (after layer resolution):

```nickel
let postgres_config = {
  version = "15.5",          # From Core
  port = 5432,               # From Core
  max_connections = 500,     # From Infrastructure (overridden)
} in
postgres_config
```plaintext

### Example 2: Add Custom Configuration

**Workspace Layer** (`provisioning/workspace/templates/production-postgres.ncl`):

```nickel
let postgres_config = {
  replication = {
    enabled = true,
    replicas = 2,
    sync_mode = "async",
  },
} in
postgres_config
```plaintext

**Infrastructure Layer** (`workspace/infra/my-production/taskservs/postgres.ncl`):

```nickel
let postgres_config = {
  replication = {
    sync_mode = "sync",  # Override sync mode
  },
  custom_extensions = ["pgvector", "timescaledb"],  # Add custom config
} in
postgres_config
```plaintext

**Result**:

```nickel
let postgres_config = {
  version = "15.5",          # From Core
  port = 5432,               # From Core
  max_connections = 100,     # From Core
  replication = {
    enabled = true,          # From Workspace
    replicas = 2,            # From Workspace
    sync_mode = "sync",      # From Infrastructure (overridden)
  },
  custom_extensions = ["pgvector", "timescaledb"],  # From Infrastructure (added)
} in
postgres_config
```plaintext

### Example 3: Environment-Specific Configuration

**Workspace Layer** (`provisioning/workspace/templates/base-kubernetes.ncl`):

```nickel
let kubernetes_config = {
  version = "1.30.0",
  control_plane_count = 3,
  worker_count = 5,
  resources = {
    control_plane = {cpu = "4", memory = "8Gi"},
    worker = {cpu = "8", memory = "16Gi"},
  },
} in
kubernetes_config
```plaintext

**Development Infrastructure** (`workspace/infra/my-dev/taskservs/kubernetes.ncl`):

```nickel
let kubernetes_config = {
  control_plane_count = 1,  # Smaller for dev
  worker_count = 2,
  resources = {
    control_plane = {cpu = "2", memory = "4Gi"},
    worker = {cpu = "2", memory = "4Gi"},
  },
} in
kubernetes_config
```plaintext

**Production Infrastructure** (`workspace/infra/my-prod/taskservs/kubernetes.ncl`):

```nickel
let kubernetes_config = {
  control_plane_count = 5,  # Larger for prod
  worker_count = 10,
  resources = {
    control_plane = {cpu = "8", memory = "16Gi"},
    worker = {cpu = "16", memory = "32Gi"},
  },
} in
kubernetes_config
```plaintext

## Advanced Customization Patterns

### Pattern 1: Multi-Environment Setup

Create different configurations for each environment:

```bash
# Create environments
provisioning ws init my-app-dev
provisioning ws init my-app-staging
provisioning ws init my-app-prod

# Apply environment-specific templates
provisioning tpl apply development-kubernetes my-app-dev
provisioning tpl apply staging-kubernetes my-app-staging
provisioning tpl apply production-kubernetes my-app-prod

# Customize each environment
# Edit: workspace/infra/my-app-dev/...
# Edit: workspace/infra/my-app-staging/...
# Edit: workspace/infra/my-app-prod/...
```plaintext

### Pattern 2: Shared Configuration Library

Create reusable configuration fragments:

**File: `provisioning/workspace/templates/shared/security-policies.ncl`**

```nickel
let security_policies = {
  pod_security = {
    enforce = "restricted",
    audit = "restricted",
    warn = "restricted",
  },
  network_policies = [
    {
      name = "deny-all",
      pod_selector = {},
      policy_types = ["Ingress", "Egress"],
    },
    {
      name = "allow-dns",
      pod_selector = {},
      egress = [
        {
          to = [{namespace_selector = {name = "kube-system"}}],
          ports = [{protocol = "UDP", port = 53}],
        },
      ],
    },
  ],
} in
security_policies
```plaintext

Import in your infrastructure:

```nickel
let security_policies = (import "../../../provisioning/workspace/templates/shared/security-policies.ncl") in

let kubernetes_config = {
  version = "1.30.0",
  image_repo = "k8s.gcr.io",
  security = security_policies,  # Import shared policies
} in
kubernetes_config
```plaintext

### Pattern 3: Dynamic Configuration

Use Nickel features for dynamic configuration:

```nickel
# Calculate resources based on server count
let server_count = 5 in
let replicas_per_server = 2 in
let total_replicas = server_count * replicas_per_server in

let postgres_config = {
  version = "16.1",
  max_connections = total_replicas * 50,  # Dynamic calculation
  shared_buffers = "1024 MB",
} in
postgres_config
```plaintext

### Pattern 4: Conditional Configuration

```nickel
let environment = "production" in  # or "development"

let kubernetes_config = {
  version = "1.30.0",
  control_plane_count = if environment == "production" then 3 else 1,
  worker_count = if environment == "production" then 5 else 2,
  monitoring = {
    enabled = environment == "production",
    retention = if environment == "production" then "30d" else "7d",
  },
} in
kubernetes_config
```plaintext

## Layer Statistics

```bash
# Show layer system statistics
provisioning lyr stats
```plaintext

**Expected Output:**

```plaintext
📊 Layer System Statistics:

Infrastructure Layer:
  • Projects: 3
  • Total files: 15
  • Average overrides per project: 5

Workspace Layer:
  • Templates: 13
  • Most used: production-kubernetes (5 projects)
  • Custom templates: 2

Core Layer:
  • Taskservs: 15
  • Providers: 3
  • Clusters: 3

Resolution Performance:
  • Average resolution time: 45 ms
  • Cache hit rate: 87%
  • Total resolutions: 1,250
```plaintext

## Customization Workflow

### Complete Customization Example

```bash
# 1. Create new infrastructure
provisioning ws init my-custom-app

# 2. Understand layer system
provisioning lyr explain

# 3. Discover templates
provisioning tpl list --type taskservs

# 4. Apply base template
provisioning tpl apply production-kubernetes my-custom-app

# 5. View applied configuration
provisioning lyr show my-custom-app

# 6. Customize (edit files)
provisioning sops workspace/infra/my-custom-app/taskservs/kubernetes.ncl

# 7. Test layer resolution
provisioning lyr test kubernetes my-custom-app

# 8. Validate configuration
provisioning tpl validate my-custom-app
provisioning val config --infra my-custom-app

# 9. Deploy customized infrastructure
provisioning s create --infra my-custom-app --check
provisioning s create --infra my-custom-app
provisioning t create kubernetes --infra my-custom-app
```plaintext

## Best Practices

### 1. Use Layers Correctly

- **Core Layer**: Only modify for system-wide changes
- **Workspace Layer**: Use for organization-wide templates
- **Infrastructure Layer**: Use for project-specific customizations

### 2. Template Organization

```plaintext
provisioning/workspace/templates/
├── shared/           # Shared configuration fragments
│   ├── security-policies.ncl
│   ├── network-policies.ncl
│   └── monitoring.ncl
├── production/       # Production templates
│   ├── kubernetes.ncl
│   ├── postgres.ncl
│   └── redis.ncl
└── development/      # Development templates
    ├── kubernetes.ncl
    └── postgres.ncl
```plaintext

### 3. Documentation

Document your customizations:

**File: `workspace/infra/my-production/README.md`**

```markdown
# My Production Infrastructure

## Customizations

- Kubernetes: Using production template with 5 control plane nodes
- PostgreSQL: Configured with streaming replication
- Cilium: Native routing mode enabled

## Layer Overrides

- `taskservs/kubernetes.ncl`: Control plane count (3 → 5)
- `taskservs/postgres.ncl`: Replication mode (async → sync)
- `network/cilium.ncl`: Routing mode (tunnel → native)
```plaintext

### 4. Version Control

Keep templates and configurations in version control:

```bash
cd provisioning/workspace/templates/
git add .
git commit -m "Add production Kubernetes template with enhanced security"

cd workspace/infra/my-production/
git add .
git commit -m "Configure production environment for my-production"
```plaintext

## Troubleshooting Customizations

### Issue: Configuration not applied

```bash
# Check layer resolution
provisioning lyr show my-production

# Verify file exists
ls -la workspace/infra/my-production/taskservs/

# Test specific resolution
provisioning lyr test kubernetes my-production
```plaintext

### Issue: Conflicting configurations

```bash
# Validate configuration
provisioning val config --infra my-production

# Show configuration merge result
provisioning show config kubernetes --infra my-production
```plaintext

### Issue: Template not found

```bash
# List available templates
provisioning tpl list

# Check template path
ls -la provisioning/workspace/templates/

# Refresh template cache
provisioning tpl refresh
```plaintext

## Next Steps

- **[From Scratch Guide](from-scratch.md)** - Deploy new infrastructure
- **[Update Guide](update-infrastructure.md)** - Update existing infrastructure
- **[Workflow Guide](../development/workflow.md)** - Automate with workflows
- **[Nickel Guide](../development/nickel-module-guide.md)** - Learn Nickel configuration language

## Quick Reference

```bash
# Layer system
provisioning lyr explain              # Explain layers
provisioning lyr show <project>       # Show layer resolution
provisioning lyr test <module> <project>  # Test resolution
provisioning lyr stats                # Layer statistics

# Templates
provisioning tpl list                 # List all templates
provisioning tpl list --type <type>   # Filter by type
provisioning tpl show <template>      # Show template details
provisioning tpl apply <template> <project>  # Apply template
provisioning tpl validate <project>   # Validate template usage
```plaintext

---

*This guide is part of the provisioning project documentation. Last updated: 2025-09-30*
