Customize Infrastructure
Goal: Customize infrastructure using layers, templates, and configuration patterns Time: 20-40 minutes Difficulty: Intermediate to Advanced
Overview
This guide covers:
- Understanding the layer system
- Using templates
- Creating custom modules
- Configuration inheritance
- Advanced customization patterns
The Layer System
Understanding Layers
The provisioning system uses a 3-layer architecture for configuration inheritance:
┌─────────────────────────────────────┐
│ Infrastructure Layer (Priority 300)│ ← Highest priority
│ workspace/infra/{name}/ │
│ • Project-specific configs │
│ • Environment customizations │
│ • Local overrides │
└─────────────────────────────────────┘
↓ overrides
┌─────────────────────────────────────┐
│ Workspace Layer (Priority 200) │
│ provisioning/workspace/templates/ │
│ • Reusable patterns │
│ • Organization standards │
│ • Team conventions │
└─────────────────────────────────────┘
↓ overrides
┌─────────────────────────────────────┐
│ Core Layer (Priority 100) │ ← Lowest priority
│ provisioning/extensions/ │
│ • System defaults │
│ • Provider implementations │
│ • Default taskserv configs │
└─────────────────────────────────────┘
Resolution Order: Infrastructure (300) → Workspace (200) → Core (100)
Higher numbers override lower numbers.
View Layer Resolution
# Explain layer concept
provisioning lyr explain
Expected Output:
📚 LAYER SYSTEM EXPLAINED
The layer system provides configuration inheritance across 3 levels:
🔵 CORE LAYER (100) - System Defaults
Location: provisioning/extensions/
• Base taskserv configurations
• Default provider settings
• Standard cluster templates
• Built-in extensions
🟢 WORKSPACE LAYER (200) - Shared Templates
Location: provisioning/workspace/templates/
• Organization-wide patterns
• Reusable configurations
• Team standards
• Custom extensions
🔴 INFRASTRUCTURE LAYER (300) - Project Specific
Location: workspace/infra/{project}/
• Project-specific overrides
• Environment customizations
• Local modifications
• Runtime settings
Resolution: Infrastructure → Workspace → Core
Higher priority layers override lower ones.
# Show layer resolution for your project
provisioning lyr show my-production
Expected Output:
📊 Layer Resolution for my-production:
LAYER PRIORITY SOURCE FILES
Infrastructure 300 workspace/infra/my-production/ 4 files
• servers.ncl (overrides)
• taskservs.ncl (overrides)
• clusters.ncl (custom)
• providers.ncl (overrides)
Workspace 200 provisioning/workspace/templates/ 2 files
• production.ncl (used)
• kubernetes.ncl (used)
Core 100 provisioning/extensions/ 15 files
• taskservs/* (base configs)
• providers/* (default settings)
• clusters/* (templates)
Resolution Order: Infrastructure → Workspace → Core
Status: ✅ All layers resolved successfully
Test Layer Resolution
# Test how a specific module resolves
provisioning lyr test kubernetes my-production
Expected Output:
🔍 Layer Resolution Test: kubernetes → my-production
Resolving kubernetes configuration...
🔴 Infrastructure Layer (300):
✅ Found: workspace/infra/my-production/taskservs/kubernetes.ncl
Provides:
• version = "1.30.0" (overrides)
• control_plane_servers = ["web-01"] (overrides)
• worker_servers = ["web-02"] (overrides)
🟢 Workspace Layer (200):
✅ Found: provisioning/workspace/templates/production-kubernetes.ncl
Provides:
• security_policies (inherited)
• network_policies (inherited)
• resource_quotas (inherited)
🔵 Core Layer (100):
✅ Found: provisioning/extensions/taskservs/kubernetes/main.ncl
Provides:
• default_version = "1.29.0" (base)
• default_features (base)
• default_plugins (base)
Final Configuration (after merging all layers):
version: "1.30.0" (from Infrastructure)
control_plane_servers: ["web-01"] (from Infrastructure)
worker_servers: ["web-02"] (from Infrastructure)
security_policies: {...} (from Workspace)
network_policies: {...} (from Workspace)
resource_quotas: {...} (from Workspace)
default_features: {...} (from Core)
default_plugins: {...} (from Core)
Resolution: ✅ Success
Using Templates
List Available Templates
# List all templates
provisioning tpl list
Expected Output:
📋 Available Templates:
TASKSERVS:
• production-kubernetes - Production-ready Kubernetes setup
• production-postgres - Production PostgreSQL with replication
• production-redis - Redis cluster with sentinel
• development-kubernetes - Development Kubernetes (minimal)
• ci-cd-pipeline - Complete CI/CD pipeline
PROVIDERS:
• upcloud-production - UpCloud production settings
• upcloud-development - UpCloud development settings
• aws-production - AWS production VPC setup
• aws-development - AWS development environment
• local-docker - Local Docker-based setup
CLUSTERS:
• buildkit-cluster - BuildKit for container builds
• monitoring-stack - Prometheus + Grafana + Loki
• security-stack - Security monitoring tools
Total: 13 templates
# List templates by type
provisioning tpl list --type taskservs
provisioning tpl list --type providers
provisioning tpl list --type clusters
View Template Details
# Show template details
provisioning tpl show production-kubernetes
Expected Output:
📄 Template: production-kubernetes
Description: Production-ready Kubernetes configuration with
security hardening, network policies, and monitoring
Category: taskservs
Version: 1.0.0
Configuration Provided:
• Kubernetes version: 1.30.0
• Security policies: Pod Security Standards (restricted)
• Network policies: Default deny + allow rules
• Resource quotas: Per-namespace limits
• Monitoring: Prometheus integration
• Logging: Loki integration
• Backup: Velero configuration
Requirements:
• Minimum 2 servers
• 4 GB RAM per server
• Network plugin (Cilium recommended)
Location: provisioning/workspace/templates/production-kubernetes.ncl
Example Usage:
provisioning tpl apply production-kubernetes my-production
Apply Template
# Apply template to your infrastructure
provisioning tpl apply production-kubernetes my-production
Expected Output:
🚀 Applying template: production-kubernetes → my-production
Checking compatibility... ⏳
✅ Infrastructure compatible with template
Merging configuration... ⏳
✅ Configuration merged
Files created/updated:
• workspace/infra/my-production/taskservs/kubernetes.ncl (updated)
• workspace/infra/my-production/policies/security.ncl (created)
• workspace/infra/my-production/policies/network.ncl (created)
• workspace/infra/my-production/monitoring/prometheus.ncl (created)
🎉 Template applied successfully!
Next steps:
1. Review generated configuration
2. Adjust as needed
3. Deploy: provisioning t create kubernetes --infra my-production
Validate Template Usage
# Validate template was applied correctly
provisioning tpl validate my-production
Expected Output:
✅ Template Validation: my-production
Templates Applied:
✅ production-kubernetes (v1.0.0)
✅ production-postgres (v1.0.0)
Configuration Status:
✅ All required fields present
✅ No conflicting settings
✅ Dependencies satisfied
Compliance:
✅ Security policies configured
✅ Network policies configured
✅ Resource quotas set
✅ Monitoring enabled
Status: ✅ Valid
Creating Custom Templates
Step 1: Create Template Structure
# Create custom template directory
mkdir -p provisioning/workspace/templates/my-custom-template
Step 2: Write Template Configuration
File: provisioning/workspace/templates/my-custom-template/main.ncl
# Custom Kubernetes template with specific settings
let kubernetes_config = {
# Version
version = "1.30.0",
# Custom feature gates
feature_gates = {
"GracefulNodeShutdown" = true,
"SeccompDefault" = true,
"StatefulSetAutoDeletePVC" = true,
},
# Custom kubelet configuration
kubelet_config = {
max_pods = 110,
pod_pids_limit = 4096,
container_log_max_size = "10Mi",
container_log_max_files = 5,
},
# Custom API server flags
apiserver_extra_args = {
"enable-admission-plugins" = "NodeRestriction,PodSecurity,LimitRanger",
"audit-log-maxage" = "30",
"audit-log-maxbackup" = "10",
},
# Custom scheduler configuration
scheduler_config = {
profiles = [
{
name = "high-availability",
plugins = {
score = {
enabled = [
{name = "NodeResourcesBalancedAllocation", weight = 2},
{name = "NodeResourcesLeastAllocated", weight = 1},
],
},
},
},
],
},
# Network configuration
network = {
service_cidr = "10.96.0.0/12",
pod_cidr = "10.244.0.0/16",
dns_domain = "cluster.local",
},
# Security configuration
security = {
pod_security_standard = "restricted",
encrypt_etcd = true,
rotate_certificates = true,
},
} in
kubernetes_config
Step 3: Create Template Metadata
File: provisioning/workspace/templates/my-custom-template/metadata.toml
[template]
name = "my-custom-template"
version = "1.0.0"
description = "Custom Kubernetes template with enhanced security"
category = "taskservs"
author = "Your Name"
[requirements]
min_servers = 2
min_memory_gb = 4
required_taskservs = ["containerd", "cilium"]
[tags]
environment = ["production", "staging"]
features = ["security", "monitoring", "high-availability"]
Step 4: Test Custom Template
# List templates (should include your custom template)
provisioning tpl list
# Show your template
provisioning tpl show my-custom-template
# Apply to test infrastructure
provisioning tpl apply my-custom-template my-test
Configuration Inheritance Examples
Example 1: Override Single Value
Core Layer (provisioning/extensions/taskservs/postgres/main.ncl):
let postgres_config = {
version = "15.5",
port = 5432,
max_connections = 100,
} in
postgres_config
Infrastructure Layer (workspace/infra/my-production/taskservs/postgres.ncl):
let postgres_config = {
max_connections = 500, # Override only max_connections
} in
postgres_config
Result (after layer resolution):
let postgres_config = {
version = "15.5", # From Core
port = 5432, # From Core
max_connections = 500, # From Infrastructure (overridden)
} in
postgres_config
Example 2: Add Custom Configuration
Workspace Layer (provisioning/workspace/templates/production-postgres.ncl):
let postgres_config = {
replication = {
enabled = true,
replicas = 2,
sync_mode = "async",
},
} in
postgres_config
Infrastructure Layer (workspace/infra/my-production/taskservs/postgres.ncl):
let postgres_config = {
replication = {
sync_mode = "sync", # Override sync mode
},
custom_extensions = ["pgvector", "timescaledb"], # Add custom config
} in
postgres_config
Result:
let postgres_config = {
version = "15.5", # From Core
port = 5432, # From Core
max_connections = 100, # From Core
replication = {
enabled = true, # From Workspace
replicas = 2, # From Workspace
sync_mode = "sync", # From Infrastructure (overridden)
},
custom_extensions = ["pgvector", "timescaledb"], # From Infrastructure (added)
} in
postgres_config
Example 3: Environment-Specific Configuration
Workspace Layer (provisioning/workspace/templates/base-kubernetes.ncl):
let kubernetes_config = {
version = "1.30.0",
control_plane_count = 3,
worker_count = 5,
resources = {
control_plane = {cpu = "4", memory = "8Gi"},
worker = {cpu = "8", memory = "16Gi"},
},
} in
kubernetes_config
Development Infrastructure (workspace/infra/my-dev/taskservs/kubernetes.ncl):
let kubernetes_config = {
control_plane_count = 1, # Smaller for dev
worker_count = 2,
resources = {
control_plane = {cpu = "2", memory = "4Gi"},
worker = {cpu = "2", memory = "4Gi"},
},
} in
kubernetes_config
Production Infrastructure (workspace/infra/my-prod/taskservs/kubernetes.ncl):
let kubernetes_config = {
control_plane_count = 5, # Larger for prod
worker_count = 10,
resources = {
control_plane = {cpu = "8", memory = "16Gi"},
worker = {cpu = "16", memory = "32Gi"},
},
} in
kubernetes_config
Advanced Customization Patterns
Pattern 1: Multi-Environment Setup
Create different configurations for each environment:
# Create environments
provisioning ws init my-app-dev
provisioning ws init my-app-staging
provisioning ws init my-app-prod
# Apply environment-specific templates
provisioning tpl apply development-kubernetes my-app-dev
provisioning tpl apply staging-kubernetes my-app-staging
provisioning tpl apply production-kubernetes my-app-prod
# Customize each environment
# Edit: workspace/infra/my-app-dev/...
# Edit: workspace/infra/my-app-staging/...
# Edit: workspace/infra/my-app-prod/...
Pattern 2: Shared Configuration Library
Create reusable configuration fragments:
File: provisioning/workspace/templates/shared/security-policies.ncl
let security_policies = {
pod_security = {
enforce = "restricted",
audit = "restricted",
warn = "restricted",
},
network_policies = [
{
name = "deny-all",
pod_selector = {},
policy_types = ["Ingress", "Egress"],
},
{
name = "allow-dns",
pod_selector = {},
egress = [
{
to = [{namespace_selector = {name = "kube-system"}}],
ports = [{protocol = "UDP", port = 53}],
},
],
},
],
} in
security_policies
Import in your infrastructure:
let security_policies = (import "../../../provisioning/workspace/templates/shared/security-policies.ncl") in
let kubernetes_config = {
version = "1.30.0",
image_repo = "k8s.gcr.io",
security = security_policies, # Import shared policies
} in
kubernetes_config
Pattern 3: Dynamic Configuration
Use Nickel features for dynamic configuration:
# Calculate resources based on server count
let server_count = 5 in
let replicas_per_server = 2 in
let total_replicas = server_count * replicas_per_server in
let postgres_config = {
version = "16.1",
max_connections = total_replicas * 50, # Dynamic calculation
shared_buffers = "1024 MB",
} in
postgres_config
Pattern 4: Conditional Configuration
let environment = "production" in # or "development"
let kubernetes_config = {
version = "1.30.0",
control_plane_count = if environment == "production" then 3 else 1,
worker_count = if environment == "production" then 5 else 2,
monitoring = {
enabled = environment == "production",
retention = if environment == "production" then "30d" else "7d",
},
} in
kubernetes_config
Layer Statistics
# Show layer system statistics
provisioning lyr stats
Expected Output:
📊 Layer System Statistics:
Infrastructure Layer:
• Projects: 3
• Total files: 15
• Average overrides per project: 5
Workspace Layer:
• Templates: 13
• Most used: production-kubernetes (5 projects)
• Custom templates: 2
Core Layer:
• Taskservs: 15
• Providers: 3
• Clusters: 3
Resolution Performance:
• Average resolution time: 45 ms
• Cache hit rate: 87%
• Total resolutions: 1,250
Customization Workflow
Complete Customization Example
# 1. Create new infrastructure
provisioning ws init my-custom-app
# 2. Understand layer system
provisioning lyr explain
# 3. Discover templates
provisioning tpl list --type taskservs
# 4. Apply base template
provisioning tpl apply production-kubernetes my-custom-app
# 5. View applied configuration
provisioning lyr show my-custom-app
# 6. Customize (edit files)
provisioning sops workspace/infra/my-custom-app/taskservs/kubernetes.ncl
# 7. Test layer resolution
provisioning lyr test kubernetes my-custom-app
# 8. Validate configuration
provisioning tpl validate my-custom-app
provisioning val config --infra my-custom-app
# 9. Deploy customized infrastructure
provisioning s create --infra my-custom-app --check
provisioning s create --infra my-custom-app
provisioning t create kubernetes --infra my-custom-app
Best Practices
1. Use Layers Correctly
- Core Layer: Only modify for system-wide changes
- Workspace Layer: Use for organization-wide templates
- Infrastructure Layer: Use for project-specific customizations
2. Template Organization
provisioning/workspace/templates/
├── shared/ # Shared configuration fragments
│ ├── security-policies.ncl
│ ├── network-policies.ncl
│ └── monitoring.ncl
├── production/ # Production templates
│ ├── kubernetes.ncl
│ ├── postgres.ncl
│ └── redis.ncl
└── development/ # Development templates
├── kubernetes.ncl
└── postgres.ncl
3. Documentation
Document your customizations:
File: workspace/infra/my-production/README.md
# My Production Infrastructure
## Customizations
- Kubernetes: Using production template with 5 control plane nodes
- PostgreSQL: Configured with streaming replication
- Cilium: Native routing mode enabled
## Layer Overrides
- `taskservs/kubernetes.ncl`: Control plane count (3 → 5)
- `taskservs/postgres.ncl`: Replication mode (async → sync)
- `network/cilium.ncl`: Routing mode (tunnel → native)
4. Version Control
Keep templates and configurations in version control:
cd provisioning/workspace/templates/
git add .
git commit -m "Add production Kubernetes template with enhanced security"
cd workspace/infra/my-production/
git add .
git commit -m "Configure production environment for my-production"
Troubleshooting Customizations
Issue: Configuration not applied
# Check layer resolution
provisioning lyr show my-production
# Verify file exists
ls -la workspace/infra/my-production/taskservs/
# Test specific resolution
provisioning lyr test kubernetes my-production
Issue: Conflicting configurations
# Validate configuration
provisioning val config --infra my-production
# Show configuration merge result
provisioning show config kubernetes --infra my-production
Issue: Template not found
# List available templates
provisioning tpl list
# Check template path
ls -la provisioning/workspace/templates/
# Refresh template cache
provisioning tpl refresh
Next Steps
- From Scratch Guide - Deploy new infrastructure
- Update Guide - Update existing infrastructure
- Workflow Guide - Automate with workflows
- Nickel Guide - Learn Nickel configuration language
Quick Reference
# Layer system
provisioning lyr explain # Explain layers
provisioning lyr show <project> # Show layer resolution
provisioning lyr test <module> <project> # Test resolution
provisioning lyr stats # Layer statistics
# Templates
provisioning tpl list # List all templates
provisioning tpl list --type <type> # Filter by type
provisioning tpl show <template> # Show template details
provisioning tpl apply <template> <project> # Apply template
provisioning tpl validate <project> # Validate template usage
This guide is part of the provisioning project documentation. Last updated: 2025-09-30