Customize Infrastructure
Goal: Customize infrastructure using layers, templates, and configuration patterns Time: 20-40 minutes Difficulty: Intermediate to Advanced
Overview
This guide covers:
- Understanding the layer system
- Using templates
- Creating custom modules
- Configuration inheritance
- Advanced customization patterns
The Layer System
Understanding Layers
The provisioning system uses a 3-layer architecture for configuration inheritance:
┌─────────────────────────────────────┐
│ Infrastructure Layer (Priority 300)│ ← Highest priority
│ workspace/infra/{name}/ │
│ • Project-specific configs │
│ • Environment customizations │
│ • Local overrides │
└─────────────────────────────────────┘
↓ overrides
┌─────────────────────────────────────┐
│ Workspace Layer (Priority 200) │
│ provisioning/workspace/templates/ │
│ • Reusable patterns │
│ • Organization standards │
│ • Team conventions │
└─────────────────────────────────────┘
↓ overrides
┌─────────────────────────────────────┐
│ Core Layer (Priority 100) │ ← Lowest priority
│ provisioning/extensions/ │
│ • System defaults │
│ • Provider implementations │
│ • Default taskserv configs │
└─────────────────────────────────────┘
```plaintext
**Resolution Order**: Infrastructure (300) → Workspace (200) → Core (100)
Higher numbers override lower numbers.
### View Layer Resolution
```bash
# Explain layer concept
provisioning lyr explain
```plaintext
**Expected Output:**
```plaintext
📚 LAYER SYSTEM EXPLAINED
The layer system provides configuration inheritance across 3 levels:
🔵 CORE LAYER (100) - System Defaults
Location: provisioning/extensions/
• Base taskserv configurations
• Default provider settings
• Standard cluster templates
• Built-in extensions
🟢 WORKSPACE LAYER (200) - Shared Templates
Location: provisioning/workspace/templates/
• Organization-wide patterns
• Reusable configurations
• Team standards
• Custom extensions
🔴 INFRASTRUCTURE LAYER (300) - Project Specific
Location: workspace/infra/{project}/
• Project-specific overrides
• Environment customizations
• Local modifications
• Runtime settings
Resolution: Infrastructure → Workspace → Core
Higher priority layers override lower ones.
```plaintext
```bash
# Show layer resolution for your project
provisioning lyr show my-production
```plaintext
**Expected Output:**
```plaintext
📊 Layer Resolution for my-production:
LAYER PRIORITY SOURCE FILES
Infrastructure 300 workspace/infra/my-production/ 4 files
• servers.k (overrides)
• taskservs.k (overrides)
• clusters.k (custom)
• providers.k (overrides)
Workspace 200 provisioning/workspace/templates/ 2 files
• production.k (used)
• kubernetes.k (used)
Core 100 provisioning/extensions/ 15 files
• taskservs/* (base configs)
• providers/* (default settings)
• clusters/* (templates)
Resolution Order: Infrastructure → Workspace → Core
Status: ✅ All layers resolved successfully
```plaintext
### Test Layer Resolution
```bash
# Test how a specific module resolves
provisioning lyr test kubernetes my-production
```plaintext
**Expected Output:**
```plaintext
🔍 Layer Resolution Test: kubernetes → my-production
Resolving kubernetes configuration...
🔴 Infrastructure Layer (300):
✅ Found: workspace/infra/my-production/taskservs/kubernetes.k
Provides:
• version = "1.30.0" (overrides)
• control_plane_servers = ["web-01"] (overrides)
• worker_servers = ["web-02"] (overrides)
🟢 Workspace Layer (200):
✅ Found: provisioning/workspace/templates/production-kubernetes.k
Provides:
• security_policies (inherited)
• network_policies (inherited)
• resource_quotas (inherited)
🔵 Core Layer (100):
✅ Found: provisioning/extensions/taskservs/kubernetes/config.k
Provides:
• default_version = "1.29.0" (base)
• default_features (base)
• default_plugins (base)
Final Configuration (after merging all layers):
version: "1.30.0" (from Infrastructure)
control_plane_servers: ["web-01"] (from Infrastructure)
worker_servers: ["web-02"] (from Infrastructure)
security_policies: {...} (from Workspace)
network_policies: {...} (from Workspace)
resource_quotas: {...} (from Workspace)
default_features: {...} (from Core)
default_plugins: {...} (from Core)
Resolution: ✅ Success
```plaintext
## Using Templates
### List Available Templates
```bash
# List all templates
provisioning tpl list
```plaintext
**Expected Output:**
```plaintext
📋 Available Templates:
TASKSERVS:
• production-kubernetes - Production-ready Kubernetes setup
• production-postgres - Production PostgreSQL with replication
• production-redis - Redis cluster with sentinel
• development-kubernetes - Development Kubernetes (minimal)
• ci-cd-pipeline - Complete CI/CD pipeline
PROVIDERS:
• upcloud-production - UpCloud production settings
• upcloud-development - UpCloud development settings
• aws-production - AWS production VPC setup
• aws-development - AWS development environment
• local-docker - Local Docker-based setup
CLUSTERS:
• buildkit-cluster - BuildKit for container builds
• monitoring-stack - Prometheus + Grafana + Loki
• security-stack - Security monitoring tools
Total: 13 templates
```plaintext
```bash
# List templates by type
provisioning tpl list --type taskservs
provisioning tpl list --type providers
provisioning tpl list --type clusters
```plaintext
### View Template Details
```bash
# Show template details
provisioning tpl show production-kubernetes
```plaintext
**Expected Output:**
```plaintext
📄 Template: production-kubernetes
Description: Production-ready Kubernetes configuration with
security hardening, network policies, and monitoring
Category: taskservs
Version: 1.0.0
Configuration Provided:
• Kubernetes version: 1.30.0
• Security policies: Pod Security Standards (restricted)
• Network policies: Default deny + allow rules
• Resource quotas: Per-namespace limits
• Monitoring: Prometheus integration
• Logging: Loki integration
• Backup: Velero configuration
Requirements:
• Minimum 2 servers
• 4GB RAM per server
• Network plugin (Cilium recommended)
Location: provisioning/workspace/templates/production-kubernetes.k
Example Usage:
provisioning tpl apply production-kubernetes my-production
```plaintext
### Apply Template
```bash
# Apply template to your infrastructure
provisioning tpl apply production-kubernetes my-production
```plaintext
**Expected Output:**
```plaintext
🚀 Applying template: production-kubernetes → my-production
Checking compatibility... ⏳
✅ Infrastructure compatible with template
Merging configuration... ⏳
✅ Configuration merged
Files created/updated:
• workspace/infra/my-production/taskservs/kubernetes.k (updated)
• workspace/infra/my-production/policies/security.k (created)
• workspace/infra/my-production/policies/network.k (created)
• workspace/infra/my-production/monitoring/prometheus.k (created)
🎉 Template applied successfully!
Next steps:
1. Review generated configuration
2. Adjust as needed
3. Deploy: provisioning t create kubernetes --infra my-production
```plaintext
### Validate Template Usage
```bash
# Validate template was applied correctly
provisioning tpl validate my-production
```plaintext
**Expected Output:**
```plaintext
✅ Template Validation: my-production
Templates Applied:
✅ production-kubernetes (v1.0.0)
✅ production-postgres (v1.0.0)
Configuration Status:
✅ All required fields present
✅ No conflicting settings
✅ Dependencies satisfied
Compliance:
✅ Security policies configured
✅ Network policies configured
✅ Resource quotas set
✅ Monitoring enabled
Status: ✅ Valid
```plaintext
## Creating Custom Templates
### Step 1: Create Template Structure
```bash
# Create custom template directory
mkdir -p provisioning/workspace/templates/my-custom-template
```plaintext
### Step 2: Write Template Configuration
**File: `provisioning/workspace/templates/my-custom-template/config.k`**
```kcl
# Custom Kubernetes template with specific settings
kubernetes_config = {
# Version
version = "1.30.0"
# Custom feature gates
feature_gates = {
"GracefulNodeShutdown" = True
"SeccompDefault" = True
"StatefulSetAutoDeletePVC" = True
}
# Custom kubelet configuration
kubelet_config = {
max_pods = 110
pod_pids_limit = 4096
container_log_max_size = "10Mi"
container_log_max_files = 5
}
# Custom API server flags
apiserver_extra_args = {
"enable-admission-plugins" = "NodeRestriction,PodSecurity,LimitRanger"
"audit-log-maxage" = "30"
"audit-log-maxbackup" = "10"
}
# Custom scheduler configuration
scheduler_config = {
profiles = [
{
name = "high-availability"
plugins = {
score = {
enabled = [
{name = "NodeResourcesBalancedAllocation", weight = 2}
{name = "NodeResourcesLeastAllocated", weight = 1}
]
}
}
}
]
}
# Network configuration
network = {
service_cidr = "10.96.0.0/12"
pod_cidr = "10.244.0.0/16"
dns_domain = "cluster.local"
}
# Security configuration
security = {
pod_security_standard = "restricted"
encrypt_etcd = True
rotate_certificates = True
}
}
```plaintext
### Step 3: Create Template Metadata
**File: `provisioning/workspace/templates/my-custom-template/metadata.toml`**
```toml
[template]
name = "my-custom-template"
version = "1.0.0"
description = "Custom Kubernetes template with enhanced security"
category = "taskservs"
author = "Your Name"
[requirements]
min_servers = 2
min_memory_gb = 4
required_taskservs = ["containerd", "cilium"]
[tags]
environment = ["production", "staging"]
features = ["security", "monitoring", "high-availability"]
```plaintext
### Step 4: Test Custom Template
```bash
# List templates (should include your custom template)
provisioning tpl list
# Show your template
provisioning tpl show my-custom-template
# Apply to test infrastructure
provisioning tpl apply my-custom-template my-test
```plaintext
## Configuration Inheritance Examples
### Example 1: Override Single Value
**Core Layer** (`provisioning/extensions/taskservs/postgres/config.k`):
```kcl
postgres_config = {
version = "15.5"
port = 5432
max_connections = 100
}
```plaintext
**Infrastructure Layer** (`workspace/infra/my-production/taskservs/postgres.k`):
```kcl
postgres_config = {
max_connections = 500 # Override only max_connections
}
```plaintext
**Result** (after layer resolution):
```kcl
postgres_config = {
version = "15.5" # From Core
port = 5432 # From Core
max_connections = 500 # From Infrastructure (overridden)
}
```plaintext
### Example 2: Add Custom Configuration
**Workspace Layer** (`provisioning/workspace/templates/production-postgres.k`):
```kcl
postgres_config = {
replication = {
enabled = True
replicas = 2
sync_mode = "async"
}
}
```plaintext
**Infrastructure Layer** (`workspace/infra/my-production/taskservs/postgres.k`):
```kcl
postgres_config = {
replication = {
sync_mode = "sync" # Override sync mode
}
custom_extensions = ["pgvector", "timescaledb"] # Add custom config
}
```plaintext
**Result**:
```kcl
postgres_config = {
version = "15.5" # From Core
port = 5432 # From Core
max_connections = 100 # From Core
replication = {
enabled = True # From Workspace
replicas = 2 # From Workspace
sync_mode = "sync" # From Infrastructure (overridden)
}
custom_extensions = ["pgvector", "timescaledb"] # From Infrastructure (added)
}
```plaintext
### Example 3: Environment-Specific Configuration
**Workspace Layer** (`provisioning/workspace/templates/base-kubernetes.k`):
```kcl
kubernetes_config = {
version = "1.30.0"
control_plane_count = 3
worker_count = 5
resources = {
control_plane = {cpu = "4", memory = "8Gi"}
worker = {cpu = "8", memory = "16Gi"}
}
}
```plaintext
**Development Infrastructure** (`workspace/infra/my-dev/taskservs/kubernetes.k`):
```kcl
kubernetes_config = {
control_plane_count = 1 # Smaller for dev
worker_count = 2
resources = {
control_plane = {cpu = "2", memory = "4Gi"}
worker = {cpu = "2", memory = "4Gi"}
}
}
```plaintext
**Production Infrastructure** (`workspace/infra/my-prod/taskservs/kubernetes.k`):
```kcl
kubernetes_config = {
control_plane_count = 5 # Larger for prod
worker_count = 10
resources = {
control_plane = {cpu = "8", memory = "16Gi"}
worker = {cpu = "16", memory = "32Gi"}
}
}
```plaintext
## Advanced Customization Patterns
### Pattern 1: Multi-Environment Setup
Create different configurations for each environment:
```bash
# Create environments
provisioning ws init my-app-dev
provisioning ws init my-app-staging
provisioning ws init my-app-prod
# Apply environment-specific templates
provisioning tpl apply development-kubernetes my-app-dev
provisioning tpl apply staging-kubernetes my-app-staging
provisioning tpl apply production-kubernetes my-app-prod
# Customize each environment
# Edit: workspace/infra/my-app-dev/...
# Edit: workspace/infra/my-app-staging/...
# Edit: workspace/infra/my-app-prod/...
```plaintext
### Pattern 2: Shared Configuration Library
Create reusable configuration fragments:
**File: `provisioning/workspace/templates/shared/security-policies.k`**
```kcl
security_policies = {
pod_security = {
enforce = "restricted"
audit = "restricted"
warn = "restricted"
}
network_policies = [
{
name = "deny-all"
pod_selector = {}
policy_types = ["Ingress", "Egress"]
},
{
name = "allow-dns"
pod_selector = {}
egress = [
{
to = [{namespace_selector = {name = "kube-system"}}]
ports = [{protocol = "UDP", port = 53}]
}
]
}
]
}
```plaintext
Import in your infrastructure:
```kcl
import "../../../provisioning/workspace/templates/shared/security-policies.k"
kubernetes_config = {
version = "1.30.0"
# ... other config
security = security_policies # Import shared policies
}
```plaintext
### Pattern 3: Dynamic Configuration
Use KCL features for dynamic configuration:
```kcl
# Calculate resources based on server count
server_count = 5
replicas_per_server = 2
total_replicas = server_count * replicas_per_server
postgres_config = {
version = "16.1"
max_connections = total_replicas * 50 # Dynamic calculation
shared_buffers = "${total_replicas * 128}MB"
}
```plaintext
### Pattern 4: Conditional Configuration
```kcl
environment = "production" # or "development"
kubernetes_config = {
version = "1.30.0"
control_plane_count = if environment == "production" { 3 } else { 1 }
worker_count = if environment == "production" { 5 } else { 2 }
monitoring = {
enabled = environment == "production"
retention = if environment == "production" { "30d" } else { "7d" }
}
}
```plaintext
## Layer Statistics
```bash
# Show layer system statistics
provisioning lyr stats
```plaintext
**Expected Output:**
```plaintext
📊 Layer System Statistics:
Infrastructure Layer:
• Projects: 3
• Total files: 15
• Average overrides per project: 5
Workspace Layer:
• Templates: 13
• Most used: production-kubernetes (5 projects)
• Custom templates: 2
Core Layer:
• Taskservs: 15
• Providers: 3
• Clusters: 3
Resolution Performance:
• Average resolution time: 45ms
• Cache hit rate: 87%
• Total resolutions: 1,250
```plaintext
## Customization Workflow
### Complete Customization Example
```bash
# 1. Create new infrastructure
provisioning ws init my-custom-app
# 2. Understand layer system
provisioning lyr explain
# 3. Discover templates
provisioning tpl list --type taskservs
# 4. Apply base template
provisioning tpl apply production-kubernetes my-custom-app
# 5. View applied configuration
provisioning lyr show my-custom-app
# 6. Customize (edit files)
provisioning sops workspace/infra/my-custom-app/taskservs/kubernetes.k
# 7. Test layer resolution
provisioning lyr test kubernetes my-custom-app
# 8. Validate configuration
provisioning tpl validate my-custom-app
provisioning val config --infra my-custom-app
# 9. Deploy customized infrastructure
provisioning s create --infra my-custom-app --check
provisioning s create --infra my-custom-app
provisioning t create kubernetes --infra my-custom-app
```plaintext
## Best Practices
### 1. Use Layers Correctly
- **Core Layer**: Only modify for system-wide changes
- **Workspace Layer**: Use for organization-wide templates
- **Infrastructure Layer**: Use for project-specific customizations
### 2. Template Organization
```plaintext
provisioning/workspace/templates/
├── shared/ # Shared configuration fragments
│ ├── security-policies.k
│ ├── network-policies.k
│ └── monitoring.k
├── production/ # Production templates
│ ├── kubernetes.k
│ ├── postgres.k
│ └── redis.k
└── development/ # Development templates
├── kubernetes.k
└── postgres.k
```plaintext
### 3. Documentation
Document your customizations:
**File: `workspace/infra/my-production/README.md`**
```markdown
# My Production Infrastructure
## Customizations
- Kubernetes: Using production template with 5 control plane nodes
- PostgreSQL: Configured with streaming replication
- Cilium: Native routing mode enabled
## Layer Overrides
- `taskservs/kubernetes.k`: Control plane count (3 → 5)
- `taskservs/postgres.k`: Replication mode (async → sync)
- `network/cilium.k`: Routing mode (tunnel → native)
```plaintext
### 4. Version Control
Keep templates and configurations in version control:
```bash
cd provisioning/workspace/templates/
git add .
git commit -m "Add production Kubernetes template with enhanced security"
cd workspace/infra/my-production/
git add .
git commit -m "Configure production environment for my-production"
```plaintext
## Troubleshooting Customizations
### Issue: Configuration not applied
```bash
# Check layer resolution
provisioning lyr show my-production
# Verify file exists
ls -la workspace/infra/my-production/taskservs/
# Test specific resolution
provisioning lyr test kubernetes my-production
```plaintext
### Issue: Conflicting configurations
```bash
# Validate configuration
provisioning val config --infra my-production
# Show configuration merge result
provisioning show config kubernetes --infra my-production
```plaintext
### Issue: Template not found
```bash
# List available templates
provisioning tpl list
# Check template path
ls -la provisioning/workspace/templates/
# Refresh template cache
provisioning tpl refresh
```plaintext
## Next Steps
- **[From Scratch Guide](from-scratch.md)** - Deploy new infrastructure
- **[Update Guide](update-infrastructure.md)** - Update existing infrastructure
- **[Workflow Guide](../development/workflow.md)** - Automate with workflows
- **[KCL Guide](../development/KCL_MODULE_GUIDE.md)** - Learn KCL configuration language
## Quick Reference
```bash
# Layer system
provisioning lyr explain # Explain layers
provisioning lyr show <project> # Show layer resolution
provisioning lyr test <module> <project> # Test resolution
provisioning lyr stats # Layer statistics
# Templates
provisioning tpl list # List all templates
provisioning tpl list --type <type> # Filter by type
provisioning tpl show <template> # Show template details
provisioning tpl apply <template> <project> # Apply template
provisioning tpl validate <project> # Validate template usage
```plaintext
---
*This guide is part of the provisioning project documentation. Last updated: 2025-09-30*