268 lines
8.9 KiB
Markdown
268 lines
8.9 KiB
Markdown
# Layered Template Architecture System
|
|
|
|
This workspace provides a combined **Layered Extension Architecture with Override System** and **Template-Based Infrastructure Pattern Library** that maintains PAP principles while enabling maximum reusability of infrastructure configurations.
|
|
|
|
## 🏗️ Architecture Overview
|
|
|
|
### Layer Hierarchy
|
|
The system resolves configurations through a three-tier layer system:
|
|
|
|
1. **Core Layer (Priority 100)** - `provisioning/extensions/`
|
|
- Base provisioning system extensions
|
|
- Core taskservs, providers, and clusters
|
|
|
|
2. **Workspace Layer (Priority 200)** - `provisioning/workspace/templates/`
|
|
- Shared templates extracted from proven infrastructure patterns
|
|
- Reusable configurations across multiple infrastructures
|
|
|
|
3. **Infrastructure Layer (Priority 300)** - `workspace/infra/{name}/`
|
|
- Infrastructure-specific configurations and overrides
|
|
- Custom implementations per infrastructure
|
|
|
|
### Directory Structure
|
|
|
|
```
|
|
provisioning/workspace/
|
|
├── templates/ # Template library
|
|
│ ├── taskservs/ # Taskserv configuration templates
|
|
│ │ ├── kubernetes/ # Kubernetes templates
|
|
│ │ │ ├── base.k # Base configuration
|
|
│ │ │ └── variants/ # HA, single-node variants
|
|
│ │ ├── storage/ # Storage system templates
|
|
│ │ ├── networking/ # Network configuration templates
|
|
│ │ └── container-runtime/ # Container runtime templates
|
|
│ ├── providers/ # Provider templates
|
|
│ │ ├── upcloud/ # UpCloud provider templates
|
|
│ │ └── aws/ # AWS provider templates
|
|
│ ├── servers/ # Server configuration patterns
|
|
│ └── clusters/ # Complete cluster templates
|
|
├── layers/ # Layer definitions
|
|
│ ├── core.layer.k # Core layer definition
|
|
│ ├── workspace.layer.k # Workspace layer definition
|
|
│ └── infra.layer.k # Infrastructure layer definition
|
|
├── registry/ # Extension registry
|
|
│ ├── manifest.yaml # Template catalog and metadata
|
|
│ └── imports.k # Central import aliases
|
|
├── templates/lib/ # Composition utilities
|
|
│ ├── compose.k # KCL composition functions
|
|
│ └── override.k # Override and layer utilities
|
|
└── tools/ # Migration and management tools
|
|
└── migrate-infra.nu # Infrastructure migration tool
|
|
```
|
|
|
|
## 🚀 Getting Started
|
|
|
|
### 1. Extract Existing Patterns
|
|
|
|
Extract patterns from existing infrastructure (e.g., wuji) to create reusable templates:
|
|
|
|
```bash
|
|
# Extract all patterns from wuji infrastructure
|
|
cd provisioning/workspace/tools
|
|
./migrate-infra.nu extract wuji
|
|
|
|
# Extract specific types only
|
|
./migrate-infra.nu extract wuji --type taskservs
|
|
```
|
|
|
|
### 2. Use Enhanced Module Loader
|
|
|
|
The enhanced module loader provides template and layer management:
|
|
|
|
```bash
|
|
# List available templates
|
|
provisioning/core/cli/module-loader-enhanced template list
|
|
|
|
# Show layer resolution order
|
|
provisioning/core/cli/module-loader-enhanced layer show
|
|
|
|
# Test layer resolution for a specific module
|
|
provisioning/core/cli/module-loader-enhanced layer test kubernetes --infra wuji
|
|
```
|
|
|
|
### 3. Apply Templates to New Infrastructure
|
|
|
|
```bash
|
|
# Apply kubernetes template to new infrastructure
|
|
provisioning/core/cli/module-loader-enhanced template apply kubernetes-base new-infra --provider upcloud
|
|
|
|
# Load taskservs using templates
|
|
provisioning/core/cli/module-loader-enhanced load enhanced taskservs workspace/infra/new-infra [kubernetes, cilium] --layer workspace
|
|
```
|
|
|
|
## 📋 Usage Examples
|
|
|
|
### Creating a New Infrastructure from Templates
|
|
|
|
```bash
|
|
# 1. Create directory structure
|
|
mkdir -p workspace/infra/my-new-infra/{taskservs,defs,overrides}
|
|
|
|
# 2. Apply base templates
|
|
cd provisioning
|
|
./core/cli/module-loader-enhanced template apply kubernetes-base my-new-infra
|
|
|
|
# 3. Customize for your needs
|
|
# Edit workspace/infra/my-new-infra/taskservs/kubernetes.k
|
|
|
|
# 4. Test layer resolution
|
|
./core/cli/module-loader-enhanced layer test kubernetes --infra my-new-infra
|
|
```
|
|
|
|
### Converting Existing Infrastructure
|
|
|
|
```bash
|
|
# 1. Extract patterns to templates
|
|
cd provisioning/workspace/tools
|
|
./migrate-infra.nu extract existing-infra
|
|
|
|
# 2. Convert infrastructure to use templates
|
|
./migrate-infra.nu convert existing-infra
|
|
|
|
# 3. Validate conversion
|
|
./migrate-infra.nu validate existing-infra
|
|
```
|
|
|
|
### Template Development
|
|
|
|
```kcl
|
|
# Create a new taskserv template
|
|
# provisioning/workspace/templates/taskservs/my-service/base.k
|
|
|
|
import taskservs.my_service.kcl.my_service as service_core
|
|
import ../../../workspace/templates/lib/compose as comp
|
|
|
|
schema MyServiceBase {
|
|
version: str = "1.0.0"
|
|
cluster_name: str
|
|
# ... configuration options
|
|
}
|
|
|
|
def create_my_service [cluster_name: str, overrides: any = {}] -> any {
|
|
let base_config = MyServiceBase { cluster_name = $cluster_name }
|
|
let final_config = comp.deep_merge $base_config $overrides
|
|
|
|
service_core.MyService $final_config
|
|
}
|
|
```
|
|
|
|
## 🔧 Configuration Composition
|
|
|
|
### Using Templates in Infrastructure
|
|
|
|
```kcl
|
|
# workspace/infra/my-infra/taskservs/kubernetes.k
|
|
|
|
import provisioning.workspace.registry.imports as reg
|
|
import provisioning.workspace.templates.lib.override as ovr
|
|
|
|
# Use base template with infrastructure-specific overrides
|
|
_taskserv = ovr.infrastructure_overrides.taskserv_override(
|
|
reg.tpl_kubernetes_base.kubernetes_base,
|
|
"my-infra",
|
|
"kubernetes",
|
|
{
|
|
cluster_name: "my-infra"
|
|
version: "1.31.0" # Override version
|
|
cri: "crio" # Override container runtime
|
|
# Custom network configuration
|
|
network_config: {
|
|
pod_cidr: "10.244.0.0/16"
|
|
service_cidr: "10.96.0.0/12"
|
|
}
|
|
}
|
|
)
|
|
```
|
|
|
|
### Layer Composition
|
|
|
|
```kcl
|
|
# Compose configuration through all layers
|
|
import provisioning.workspace.templates.lib.compose as comp
|
|
|
|
# Manual layer composition
|
|
final_config = comp.compose_templates(
|
|
$core_config, # From provisioning/extensions
|
|
$workspace_config, # From provisioning/workspace/templates
|
|
$infra_config # From workspace/infra/{name}
|
|
)
|
|
```
|
|
|
|
## 🛠️ Advanced Features
|
|
|
|
### Provider-Aware Composition
|
|
|
|
```kcl
|
|
import provisioning.workspace.templates.lib.override as ovr
|
|
|
|
# Apply provider-specific configurations
|
|
config = ovr.override_patterns.env_override(
|
|
$base_config,
|
|
"upcloud",
|
|
{
|
|
upcloud: { zone: "es-mad1", plan: "2xCPU-4GB" },
|
|
aws: { region: "us-west-2", instance_type: "t3.medium" },
|
|
local: { memory: "4GB", cpus: 2 }
|
|
}
|
|
)
|
|
```
|
|
|
|
### Conditional Overrides
|
|
|
|
```kcl
|
|
# Infrastructure-specific conditional overrides
|
|
config = ovr.layer_resolution.infra_conditional(
|
|
$base_config,
|
|
$infra_name,
|
|
{
|
|
"production": { ha: true, replicas: 3 },
|
|
"development": { ha: false, replicas: 1 },
|
|
"default": { ha: false, replicas: 1 }
|
|
}
|
|
)
|
|
```
|
|
|
|
## 📚 Benefits
|
|
|
|
### ✅ Maintains PAP Principles
|
|
- **Configuration-driven**: No hardcoded values
|
|
- **Modular**: Clear separation of concerns
|
|
- **Declarative**: Infrastructure as code
|
|
- **Reusable**: DRY principle throughout
|
|
|
|
### ✅ Flexible Override System
|
|
- **Layer-based resolution**: Clean precedence order
|
|
- **Selective overrides**: Override only what's needed
|
|
- **Provider-agnostic**: Works across all providers
|
|
- **Environment-aware**: Dev/test/prod configurations
|
|
|
|
### ✅ Template Reusability
|
|
- **Pattern extraction**: Learn from existing infrastructures
|
|
- **Template versioning**: Track evolution over time
|
|
- **Composition utilities**: Rich KCL composition functions
|
|
- **Migration tools**: Easy conversion process
|
|
|
|
### ✅ No Core Changes
|
|
- **Non-invasive**: Core provisioning system unchanged
|
|
- **Backward compatible**: Existing infrastructures continue working
|
|
- **Progressive adoption**: Migrate at your own pace
|
|
- **Extensible**: Add new templates and layers easily
|
|
|
|
## 🔄 Migration Path
|
|
|
|
1. **Extract patterns** from existing infrastructures using `migrate-infra.nu extract`
|
|
2. **Create templates** in `provisioning/workspace/templates/`
|
|
3. **Convert infrastructures** to use templates with `migrate-infra.nu convert`
|
|
4. **Validate** the conversion with `migrate-infra.nu validate`
|
|
5. **Test** layer resolution with enhanced module loader
|
|
6. **Iterate** and improve templates based on usage
|
|
|
|
## 📖 Further Reading
|
|
|
|
- **Layer Definitions**: See `layers/*.layer.k` for layer configuration
|
|
- **Template Examples**: Browse `templates/` for real-world patterns
|
|
- **Composition Utilities**: Check `templates/lib/` for KCL utilities
|
|
- **Migration Tools**: Use `tools/migrate-infra.nu` for infrastructure conversion
|
|
- **Registry System**: Explore `registry/` for template metadata and imports
|
|
|
|
This system provides the perfect balance of flexibility, reusability, and maintainability while preserving the core provisioning system's integrity. |