# Customize Infrastructure\n\n**Goal**: Customize infrastructure using layers, templates, and configuration patterns\n**Time**: 20-40 minutes\n**Difficulty**: Intermediate to Advanced\n\n## Overview\n\nThis guide covers:\n\n1. Understanding the layer system\n2. Using templates\n3. Creating custom modules\n4. Configuration inheritance\n5. Advanced customization patterns\n\n## The Layer System\n\n### Understanding Layers\n\nThe provisioning system uses a **3-layer architecture** for configuration inheritance:\n\n```\n┌─────────────────────────────────────┐\n│  Infrastructure Layer (Priority 300)│  ← Highest priority\n│  workspace/infra/{name}/            │\n│  • Project-specific configs         │\n│  • Environment customizations       │\n│  • Local overrides                  │\n└─────────────────────────────────────┘\n              ↓ overrides\n┌─────────────────────────────────────┐\n│  Workspace Layer (Priority 200)     │\n│  provisioning/workspace/templates/  │\n│  • Reusable patterns                │\n│  • Organization standards           │\n│  • Team conventions                 │\n└─────────────────────────────────────┘\n              ↓ overrides\n┌─────────────────────────────────────┐\n│  Core Layer (Priority 100)          │  ← Lowest priority\n│  provisioning/extensions/           │\n│  • System defaults                  │\n│  • Provider implementations         │\n│  • Default taskserv configs         │\n└─────────────────────────────────────┘\n```\n\n**Resolution Order**: Infrastructure (300) → Workspace (200) → Core (100)\n\nHigher numbers override lower numbers.\n\n### View Layer Resolution\n\n```\n# Explain layer concept\nprovisioning lyr explain\n```\n\n**Expected Output:**\n\n```\n📚 LAYER SYSTEM EXPLAINED\n\nThe layer system provides configuration inheritance across 3 levels:\n\n🔵 CORE LAYER (100) - System Defaults\n   Location: provisioning/extensions/\n   • Base taskserv configurations\n   • Default provider settings\n   • Standard cluster templates\n   • Built-in extensions\n\n🟢 WORKSPACE LAYER (200) - Shared Templates\n   Location: provisioning/workspace/templates/\n   • Organization-wide patterns\n   • Reusable configurations\n   • Team standards\n   • Custom extensions\n\n🔴 INFRASTRUCTURE LAYER (300) - Project Specific\n   Location: workspace/infra/{project}/\n   • Project-specific overrides\n   • Environment customizations\n   • Local modifications\n   • Runtime settings\n\nResolution: Infrastructure → Workspace → Core\nHigher priority layers override lower ones.\n```\n\n```\n# Show layer resolution for your project\nprovisioning lyr show my-production\n```\n\n**Expected Output:**\n\n```\n📊 Layer Resolution for my-production:\n\nLAYER            PRIORITY  SOURCE                              FILES\nInfrastructure   300       workspace/infra/my-production/      4 files\n                           • servers.ncl (overrides)\n                           • taskservs.ncl (overrides)\n                           • clusters.ncl (custom)\n                           • providers.ncl (overrides)\n\nWorkspace        200       provisioning/workspace/templates/   2 files\n                           • production.ncl (used)\n                           • kubernetes.ncl (used)\n\nCore             100       provisioning/extensions/            15 files\n                           • taskservs/* (base configs)\n                           • providers/* (default settings)\n                           • clusters/* (templates)\n\nResolution Order: Infrastructure → Workspace → Core\nStatus: ✅ All layers resolved successfully\n```\n\n### Test Layer Resolution\n\n```\n# Test how a specific module resolves\nprovisioning lyr test kubernetes my-production\n```\n\n**Expected Output:**\n\n```\n🔍 Layer Resolution Test: kubernetes → my-production\n\nResolving kubernetes configuration...\n\n🔴 Infrastructure Layer (300):\n   ✅ Found: workspace/infra/my-production/taskservs/kubernetes.ncl\n   Provides:\n     • version = "1.30.0" (overrides)\n     • control_plane_servers = ["web-01"] (overrides)\n     • worker_servers = ["web-02"] (overrides)\n\n🟢 Workspace Layer (200):\n   ✅ Found: provisioning/workspace/templates/production-kubernetes.ncl\n   Provides:\n     • security_policies (inherited)\n     • network_policies (inherited)\n     • resource_quotas (inherited)\n\n🔵 Core Layer (100):\n   ✅ Found: provisioning/extensions/taskservs/kubernetes/main.ncl\n   Provides:\n     • default_version = "1.29.0" (base)\n     • default_features (base)\n     • default_plugins (base)\n\nFinal Configuration (after merging all layers):\n  version: "1.30.0" (from Infrastructure)\n  control_plane_servers: ["web-01"] (from Infrastructure)\n  worker_servers: ["web-02"] (from Infrastructure)\n  security_policies: {...} (from Workspace)\n  network_policies: {...} (from Workspace)\n  resource_quotas: {...} (from Workspace)\n  default_features: {...} (from Core)\n  default_plugins: {...} (from Core)\n\nResolution: ✅ Success\n```\n\n## Using Templates\n\n### List Available Templates\n\n```\n# List all templates\nprovisioning tpl list\n```\n\n**Expected Output:**\n\n```\n📋 Available Templates:\n\nTASKSERVS:\n  • production-kubernetes    - Production-ready Kubernetes setup\n  • production-postgres      - Production PostgreSQL with replication\n  • production-redis         - Redis cluster with sentinel\n  • development-kubernetes   - Development Kubernetes (minimal)\n  • ci-cd-pipeline          - Complete CI/CD pipeline\n\nPROVIDERS:\n  • upcloud-production      - UpCloud production settings\n  • upcloud-development     - UpCloud development settings\n  • aws-production          - AWS production VPC setup\n  • aws-development         - AWS development environment\n  • local-docker            - Local Docker-based setup\n\nCLUSTERS:\n  • buildkit-cluster        - BuildKit for container builds\n  • monitoring-stack        - Prometheus + Grafana + Loki\n  • security-stack          - Security monitoring tools\n\nTotal: 13 templates\n```\n\n```\n# List templates by type\nprovisioning tpl list --type taskservs\nprovisioning tpl list --type providers\nprovisioning tpl list --type clusters\n```\n\n### View Template Details\n\n```\n# Show template details\nprovisioning tpl show production-kubernetes\n```\n\n**Expected Output:**\n\n```\n📄 Template: production-kubernetes\n\nDescription: Production-ready Kubernetes configuration with\n             security hardening, network policies, and monitoring\n\nCategory: taskservs\nVersion: 1.0.0\n\nConfiguration Provided:\n  • Kubernetes version: 1.30.0\n  • Security policies: Pod Security Standards (restricted)\n  • Network policies: Default deny + allow rules\n  • Resource quotas: Per-namespace limits\n  • Monitoring: Prometheus integration\n  • Logging: Loki integration\n  • Backup: Velero configuration\n\nRequirements:\n  • Minimum 2 servers\n  • 4 GB RAM per server\n  • Network plugin (Cilium recommended)\n\nLocation: provisioning/workspace/templates/production-kubernetes.ncl\n\nExample Usage:\n  provisioning tpl apply production-kubernetes my-production\n```\n\n### Apply Template\n\n```\n# Apply template to your infrastructure\nprovisioning tpl apply production-kubernetes my-production\n```\n\n**Expected Output:**\n\n```\n🚀 Applying template: production-kubernetes → my-production\n\nChecking compatibility... ⏳\n✅ Infrastructure compatible with template\n\nMerging configuration... ⏳\n✅ Configuration merged\n\nFiles created/updated:\n  • workspace/infra/my-production/taskservs/kubernetes.ncl (updated)\n  • workspace/infra/my-production/policies/security.ncl (created)\n  • workspace/infra/my-production/policies/network.ncl (created)\n  • workspace/infra/my-production/monitoring/prometheus.ncl (created)\n\n🎉 Template applied successfully!\n\nNext steps:\n  1. Review generated configuration\n  2. Adjust as needed\n  3. Deploy: provisioning t create kubernetes --infra my-production\n```\n\n### Validate Template Usage\n\n```\n# Validate template was applied correctly\nprovisioning tpl validate my-production\n```\n\n**Expected Output:**\n\n```\n✅ Template Validation: my-production\n\nTemplates Applied:\n  ✅ production-kubernetes (v1.0.0)\n  ✅ production-postgres (v1.0.0)\n\nConfiguration Status:\n  ✅ All required fields present\n  ✅ No conflicting settings\n  ✅ Dependencies satisfied\n\nCompliance:\n  ✅ Security policies configured\n  ✅ Network policies configured\n  ✅ Resource quotas set\n  ✅ Monitoring enabled\n\nStatus: ✅ Valid\n```\n\n## Creating Custom Templates\n\n### Step 1: Create Template Structure\n\n```\n# Create custom template directory\nmkdir -p provisioning/workspace/templates/my-custom-template\n```\n\n### Step 2: Write Template Configuration\n\n**File: `provisioning/workspace/templates/my-custom-template/main.ncl`**\n\n```\n# Custom Kubernetes template with specific settings\nlet kubernetes_config = {\n  # Version\n  version = "1.30.0",\n\n  # Custom feature gates\n  feature_gates = {\n    "GracefulNodeShutdown" = true,\n    "SeccompDefault" = true,\n    "StatefulSetAutoDeletePVC" = true,\n  },\n\n  # Custom kubelet configuration\n  kubelet_config = {\n    max_pods = 110,\n    pod_pids_limit = 4096,\n    container_log_max_size = "10Mi",\n    container_log_max_files = 5,\n  },\n\n  # Custom API server flags\n  apiserver_extra_args = {\n    "enable-admission-plugins" = "NodeRestriction,PodSecurity,LimitRanger",\n    "audit-log-maxage" = "30",\n    "audit-log-maxbackup" = "10",\n  },\n\n  # Custom scheduler configuration\n  scheduler_config = {\n    profiles = [\n      {\n        name = "high-availability",\n        plugins = {\n          score = {\n            enabled = [\n              {name = "NodeResourcesBalancedAllocation", weight = 2},\n              {name = "NodeResourcesLeastAllocated", weight = 1},\n            ],\n          },\n        },\n      },\n    ],\n  },\n\n  # Network configuration\n  network = {\n    service_cidr = "10.96.0.0/12",\n    pod_cidr = "10.244.0.0/16",\n    dns_domain = "cluster.local",\n  },\n\n  # Security configuration\n  security = {\n    pod_security_standard = "restricted",\n    encrypt_etcd = true,\n    rotate_certificates = true,\n  },\n} in\nkubernetes_config\n```\n\n### Step 3: Create Template Metadata\n\n**File: `provisioning/workspace/templates/my-custom-template/metadata.toml`**\n\n```\n[template]\nname = "my-custom-template"\nversion = "1.0.0"\ndescription = "Custom Kubernetes template with enhanced security"\ncategory = "taskservs"\nauthor = "Your Name"\n\n[requirements]\nmin_servers = 2\nmin_memory_gb = 4\nrequired_taskservs = ["containerd", "cilium"]\n\n[tags]\nenvironment = ["production", "staging"]\nfeatures = ["security", "monitoring", "high-availability"]\n```\n\n### Step 4: Test Custom Template\n\n```\n# List templates (should include your custom template)\nprovisioning tpl list\n\n# Show your template\nprovisioning tpl show my-custom-template\n\n# Apply to test infrastructure\nprovisioning tpl apply my-custom-template my-test\n```\n\n## Configuration Inheritance Examples\n\n### Example 1: Override Single Value\n\n**Core Layer** (`provisioning/extensions/taskservs/postgres/main.ncl`):\n\n```\nlet postgres_config = {\n  version = "15.5",\n  port = 5432,\n  max_connections = 100,\n} in\npostgres_config\n```\n\n**Infrastructure Layer** (`workspace/infra/my-production/taskservs/postgres.ncl`):\n\n```\nlet postgres_config = {\n  max_connections = 500,  # Override only max_connections\n} in\npostgres_config\n```\n\n**Result** (after layer resolution):\n\n```\nlet postgres_config = {\n  version = "15.5",          # From Core\n  port = 5432,               # From Core\n  max_connections = 500,     # From Infrastructure (overridden)\n} in\npostgres_config\n```\n\n### Example 2: Add Custom Configuration\n\n**Workspace Layer** (`provisioning/workspace/templates/production-postgres.ncl`):\n\n```\nlet postgres_config = {\n  replication = {\n    enabled = true,\n    replicas = 2,\n    sync_mode = "async",\n  },\n} in\npostgres_config\n```\n\n**Infrastructure Layer** (`workspace/infra/my-production/taskservs/postgres.ncl`):\n\n```\nlet postgres_config = {\n  replication = {\n    sync_mode = "sync",  # Override sync mode\n  },\n  custom_extensions = ["pgvector", "timescaledb"],  # Add custom config\n} in\npostgres_config\n```\n\n**Result**:\n\n```\nlet postgres_config = {\n  version = "15.5",          # From Core\n  port = 5432,               # From Core\n  max_connections = 100,     # From Core\n  replication = {\n    enabled = true,          # From Workspace\n    replicas = 2,            # From Workspace\n    sync_mode = "sync",      # From Infrastructure (overridden)\n  },\n  custom_extensions = ["pgvector", "timescaledb"],  # From Infrastructure (added)\n} in\npostgres_config\n```\n\n### Example 3: Environment-Specific Configuration\n\n**Workspace Layer** (`provisioning/workspace/templates/base-kubernetes.ncl`):\n\n```\nlet kubernetes_config = {\n  version = "1.30.0",\n  control_plane_count = 3,\n  worker_count = 5,\n  resources = {\n    control_plane = {cpu = "4", memory = "8Gi"},\n    worker = {cpu = "8", memory = "16Gi"},\n  },\n} in\nkubernetes_config\n```\n\n**Development Infrastructure** (`workspace/infra/my-dev/taskservs/kubernetes.ncl`):\n\n```\nlet kubernetes_config = {\n  control_plane_count = 1,  # Smaller for dev\n  worker_count = 2,\n  resources = {\n    control_plane = {cpu = "2", memory = "4Gi"},\n    worker = {cpu = "2", memory = "4Gi"},\n  },\n} in\nkubernetes_config\n```\n\n**Production Infrastructure** (`workspace/infra/my-prod/taskservs/kubernetes.ncl`):\n\n```\nlet kubernetes_config = {\n  control_plane_count = 5,  # Larger for prod\n  worker_count = 10,\n  resources = {\n    control_plane = {cpu = "8", memory = "16Gi"},\n    worker = {cpu = "16", memory = "32Gi"},\n  },\n} in\nkubernetes_config\n```\n\n## Advanced Customization Patterns\n\n### Pattern 1: Multi-Environment Setup\n\nCreate different configurations for each environment:\n\n```\n# Create environments\nprovisioning ws init my-app-dev\nprovisioning ws init my-app-staging\nprovisioning ws init my-app-prod\n\n# Apply environment-specific templates\nprovisioning tpl apply development-kubernetes my-app-dev\nprovisioning tpl apply staging-kubernetes my-app-staging\nprovisioning tpl apply production-kubernetes my-app-prod\n\n# Customize each environment\n# Edit: workspace/infra/my-app-dev/...\n# Edit: workspace/infra/my-app-staging/...\n# Edit: workspace/infra/my-app-prod/...\n```\n\n### Pattern 2: Shared Configuration Library\n\nCreate reusable configuration fragments:\n\n**File: `provisioning/workspace/templates/shared/security-policies.ncl`**\n\n```\nlet security_policies = {\n  pod_security = {\n    enforce = "restricted",\n    audit = "restricted",\n    warn = "restricted",\n  },\n  network_policies = [\n    {\n      name = "deny-all",\n      pod_selector = {},\n      policy_types = ["Ingress", "Egress"],\n    },\n    {\n      name = "allow-dns",\n      pod_selector = {},\n      egress = [\n        {\n          to = [{namespace_selector = {name = "kube-system"}}],\n          ports = [{protocol = "UDP", port = 53}],\n        },\n      ],\n    },\n  ],\n} in\nsecurity_policies\n```\n\nImport in your infrastructure:\n\n```\nlet security_policies = (import "../../../provisioning/workspace/templates/shared/security-policies.ncl") in\n\nlet kubernetes_config = {\n  version = "1.30.0",\n  image_repo = "k8s.gcr.io",\n  security = security_policies,  # Import shared policies\n} in\nkubernetes_config\n```\n\n### Pattern 3: Dynamic Configuration\n\nUse Nickel features for dynamic configuration:\n\n```\n# Calculate resources based on server count\nlet server_count = 5 in\nlet replicas_per_server = 2 in\nlet total_replicas = server_count * replicas_per_server in\n\nlet postgres_config = {\n  version = "16.1",\n  max_connections = total_replicas * 50,  # Dynamic calculation\n  shared_buffers = "1024 MB",\n} in\npostgres_config\n```\n\n### Pattern 4: Conditional Configuration\n\n```\nlet environment = "production" in  # or "development"\n\nlet kubernetes_config = {\n  version = "1.30.0",\n  control_plane_count = if environment == "production" then 3 else 1,\n  worker_count = if environment == "production" then 5 else 2,\n  monitoring = {\n    enabled = environment == "production",\n    retention = if environment == "production" then "30d" else "7d",\n  },\n} in\nkubernetes_config\n```\n\n## Layer Statistics\n\n```\n# Show layer system statistics\nprovisioning lyr stats\n```\n\n**Expected Output:**\n\n```\n📊 Layer System Statistics:\n\nInfrastructure Layer:\n  • Projects: 3\n  • Total files: 15\n  • Average overrides per project: 5\n\nWorkspace Layer:\n  • Templates: 13\n  • Most used: production-kubernetes (5 projects)\n  • Custom templates: 2\n\nCore Layer:\n  • Taskservs: 15\n  • Providers: 3\n  • Clusters: 3\n\nResolution Performance:\n  • Average resolution time: 45 ms\n  • Cache hit rate: 87%\n  • Total resolutions: 1,250\n```\n\n## Customization Workflow\n\n### Complete Customization Example\n\n```\n# 1. Create new infrastructure\nprovisioning ws init my-custom-app\n\n# 2. Understand layer system\nprovisioning lyr explain\n\n# 3. Discover templates\nprovisioning tpl list --type taskservs\n\n# 4. Apply base template\nprovisioning tpl apply production-kubernetes my-custom-app\n\n# 5. View applied configuration\nprovisioning lyr show my-custom-app\n\n# 6. Customize (edit files)\nprovisioning sops workspace/infra/my-custom-app/taskservs/kubernetes.ncl\n\n# 7. Test layer resolution\nprovisioning lyr test kubernetes my-custom-app\n\n# 8. Validate configuration\nprovisioning tpl validate my-custom-app\nprovisioning val config --infra my-custom-app\n\n# 9. Deploy customized infrastructure\nprovisioning s create --infra my-custom-app --check\nprovisioning s create --infra my-custom-app\nprovisioning t create kubernetes --infra my-custom-app\n```\n\n## Best Practices\n\n### 1. Use Layers Correctly\n\n- **Core Layer**: Only modify for system-wide changes\n- **Workspace Layer**: Use for organization-wide templates\n- **Infrastructure Layer**: Use for project-specific customizations\n\n### 2. Template Organization\n\n```\nprovisioning/workspace/templates/\n├── shared/           # Shared configuration fragments\n│   ├── security-policies.ncl\n│   ├── network-policies.ncl\n│   └── monitoring.ncl\n├── production/       # Production templates\n│   ├── kubernetes.ncl\n│   ├── postgres.ncl\n│   └── redis.ncl\n└── development/      # Development templates\n    ├── kubernetes.ncl\n    └── postgres.ncl\n```\n\n### 3. Documentation\n\nDocument your customizations:\n\n**File: `workspace/infra/my-production/README.md`**\n\n```\n# My Production Infrastructure\n\n## Customizations\n\n- Kubernetes: Using production template with 5 control plane nodes\n- PostgreSQL: Configured with streaming replication\n- Cilium: Native routing mode enabled\n\n## Layer Overrides\n\n- `taskservs/kubernetes.ncl`: Control plane count (3 → 5)\n- `taskservs/postgres.ncl`: Replication mode (async → sync)\n- `network/cilium.ncl`: Routing mode (tunnel → native)\n```\n\n### 4. Version Control\n\nKeep templates and configurations in version control:\n\n```\ncd provisioning/workspace/templates/\ngit add .\ngit commit -m "Add production Kubernetes template with enhanced security"\n\ncd workspace/infra/my-production/\ngit add .\ngit commit -m "Configure production environment for my-production"\n```\n\n## Troubleshooting Customizations\n\n### Issue: Configuration not applied\n\n```\n# Check layer resolution\nprovisioning lyr show my-production\n\n# Verify file exists\nls -la workspace/infra/my-production/taskservs/\n\n# Test specific resolution\nprovisioning lyr test kubernetes my-production\n```\n\n### Issue: Conflicting configurations\n\n```\n# Validate configuration\nprovisioning val config --infra my-production\n\n# Show configuration merge result\nprovisioning show config kubernetes --infra my-production\n```\n\n### Issue: Template not found\n\n```\n# List available templates\nprovisioning tpl list\n\n# Check template path\nls -la provisioning/workspace/templates/\n\n# Refresh template cache\nprovisioning tpl refresh\n```\n\n## Next Steps\n\n- **[From Scratch Guide](from-scratch.md)** - Deploy new infrastructure\n- **[Update Guide](update-infrastructure.md)** - Update existing infrastructure\n- **[Workflow Guide](../development/workflow.md)** - Automate with workflows\n- **[Nickel Guide](../development/nickel-module-guide.md)** - Learn Nickel configuration language\n\n## Quick Reference\n\n```\n# Layer system\nprovisioning lyr explain              # Explain layers\nprovisioning lyr show <project>       # Show layer resolution\nprovisioning lyr test <module> <project>  # Test resolution\nprovisioning lyr stats                # Layer statistics\n\n# Templates\nprovisioning tpl list                 # List all templates\nprovisioning tpl list --type <type>   # Filter by type\nprovisioning tpl show <template>      # Show template details\nprovisioning tpl apply <template> <project>  # Apply template\nprovisioning tpl validate <project>   # Validate template usage\n```\n\n---\n\n*This guide is part of the provisioning project documentation. Last updated: 2025-09-30*
