Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Configuration Management

This document provides comprehensive guidance on provisioning’s configuration architecture, environment-specific configurations, validation, error handling, and migration strategies.

Table of Contents

  1. Overview
  2. Configuration Architecture
  3. Configuration Files
  4. Environment-Specific Configuration
  5. User Overrides and Customization
  6. Validation and Error Handling
  7. Interpolation and Dynamic Values
  8. Migration Strategies
  9. Troubleshooting

Overview

Provisioning implements a sophisticated configuration management system that has migrated from environment variable-based configuration to a hierarchical TOML configuration system with comprehensive validation and interpolation support.

Key Features:

  • Hierarchical Configuration: Multi-layer configuration with clear precedence
  • Environment-Specific: Dedicated configurations for dev, test, and production
  • Dynamic Interpolation: Template-based value resolution
  • Type Safety: Comprehensive validation and error handling
  • Migration Support: Backward compatibility with existing ENV variables
  • Workspace Integration: Seamless integration with development workspaces

Migration Status: ✅ Complete (2025-09-23)

  • 65+ files migrated across entire codebase
  • 200+ ENV variables replaced with 476 config accessors
  • 16 token-efficient agents used for systematic migration
  • 92% token efficiency achieved vs monolithic approach

Configuration Architecture

Hierarchical Loading Order

The configuration system implements a clear precedence hierarchy (lowest to highest precedence):

Configuration Hierarchy (Low → High Precedence)
┌─────────────────────────────────────────────────┐
│ 1. config.defaults.toml                         │ ← System defaults
│    (System-wide default values)                 │
├─────────────────────────────────────────────────┤
│ 2. ~/.config/provisioning/config.toml          │ ← User configuration
│    (User-specific preferences)                  │
├─────────────────────────────────────────────────┤
│ 3. ./provisioning.toml                         │ ← Project configuration
│    (Project-specific settings)                  │
├─────────────────────────────────────────────────┤
│ 4. ./.provisioning.toml                        │ ← Infrastructure config
│    (Infrastructure-specific settings)           │
├─────────────────────────────────────────────────┤
│ 5. Environment-specific configs                 │ ← Environment overrides
│    (config.{dev,test,prod}.toml)               │
├─────────────────────────────────────────────────┤
│ 6. Runtime environment variables                │ ← Runtime overrides
│    (PROVISIONING_* variables)                   │
└─────────────────────────────────────────────────┘

Configuration Access Patterns

Configuration Accessor Functions:

# Core configuration access
use core/nulib/lib_provisioning/config/accessor.nu

# Get configuration value with fallback
let api_url = (get-config-value "providers.upcloud.api_url" "https://api.upcloud.com")

# Get required configuration (errors if missing)
let api_key = (get-config-required "providers.upcloud.api_key")

# Get nested configuration
let server_defaults = (get-config-section "defaults.servers")

# Environment-aware configuration
let log_level = (get-config-env "logging.level" "info")

# Interpolated configuration
let data_path = (get-config-interpolated "paths.data")  # Resolves {{paths.base}}/data

Migration from ENV Variables

Before (ENV-based):

export PROVISIONING_UPCLOUD_API_KEY="your-key"
export PROVISIONING_UPCLOUD_API_URL="https://api.upcloud.com"
export PROVISIONING_LOG_LEVEL="debug"
export PROVISIONING_BASE_PATH="/usr/local/provisioning"

After (Config-based):

# config.user.toml
[providers.upcloud]
api_key = "your-key"
api_url = "https://api.upcloud.com"

[logging]
level = "debug"

[paths]
base = "/usr/local/provisioning"

Configuration Files

System Defaults (config.defaults.toml)

Purpose: Provides sensible defaults for all system components Location: Root of the repository Modification: Should only be modified by system maintainers

# System-wide defaults - DO NOT MODIFY in production
# Copy values to config.user.toml for customization

[core]
version = "1.0.0"
name = "provisioning-system"

[paths]
# Base path - all other paths derived from this
base = "/usr/local/provisioning"
config = "{{paths.base}}/config"
data = "{{paths.base}}/data"
logs = "{{paths.base}}/logs"
cache = "{{paths.base}}/cache"
runtime = "{{paths.base}}/runtime"

[logging]
level = "info"
file = "{{paths.logs}}/provisioning.log"
rotation = true
max_size = "100MB"
max_files = 5

[http]
timeout = 30
retries = 3
user_agent = "provisioning-system/{{core.version}}"
use_curl = false

[providers]
default = "local"

[providers.upcloud]
api_url = "https://api.upcloud.com/1.3"
timeout = 30
max_retries = 3

[providers.aws]
region = "us-east-1"
timeout = 30

[providers.local]
enabled = true
base_path = "{{paths.data}}/local"

[defaults]
[defaults.servers]
plan = "1xCPU-2GB"
zone = "auto"
template = "ubuntu-22.04"

[cache]
enabled = true
ttl = 3600
path = "{{paths.cache}}"

[orchestrator]
enabled = false
port = 8080
bind = "127.0.0.1"
data_path = "{{paths.data}}/orchestrator"

[workflow]
storage_backend = "filesystem"
parallel_limit = 5
rollback_enabled = true

[telemetry]
enabled = false
endpoint = ""
sample_rate = 0.1

User Configuration (~/.config/provisioning/config.toml)

Purpose: User-specific customizations and preferences Location: User’s configuration directory Modification: Users should customize this file for their needs

# User configuration - customizations and personal preferences
# This file overrides system defaults

[core]
name = "provisioning-{{env.USER}}"

[paths]
# Personal installation path
base = "{{env.HOME}}/.local/share/provisioning"

[logging]
level = "debug"
file = "{{paths.logs}}/provisioning-{{env.USER}}.log"

[providers]
default = "upcloud"

[providers.upcloud]
api_key = "your-personal-api-key"
api_secret = "your-personal-api-secret"

[defaults.servers]
plan = "2xCPU-4GB"
zone = "us-nyc1"

[development]
auto_reload = true
hot_reload_templates = true
verbose_errors = true

[notifications]
slack_webhook = "https://hooks.slack.com/your-webhook"
email = "your-email@domain.com"

[git]
auto_commit = true
commit_prefix = "[{{env.USER}}]"

Project Configuration (./provisioning.toml)

Purpose: Project-specific settings shared across team Location: Project root directory Version Control: Should be committed to version control

# Project-specific configuration
# Shared settings for this project/repository

[core]
name = "my-project-provisioning"
version = "1.2.0"

[infra]
default = "staging"
environments = ["dev", "staging", "production"]

[providers]
default = "upcloud"
allowed = ["upcloud", "aws", "local"]

[providers.upcloud]
# Project-specific UpCloud settings
default_zone = "us-nyc1"
template = "ubuntu-22.04-lts"

[defaults.servers]
plan = "2xCPU-4GB"
storage = 50
firewall_enabled = true

[security]
enforce_https = true
require_mfa = true
allowed_cidr = ["10.0.0.0/8", "172.16.0.0/12"]

[compliance]
data_region = "us-east"
encryption_at_rest = true
audit_logging = true

[team]
admins = ["alice@company.com", "bob@company.com"]
developers = ["dev-team@company.com"]

Infrastructure Configuration (./.provisioning.toml)

Purpose: Infrastructure-specific overrides Location: Infrastructure directory Usage: Overrides for specific infrastructure deployments

# Infrastructure-specific configuration
# Overrides for this specific infrastructure deployment

[core]
name = "production-east-provisioning"

[infra]
name = "production-east"
environment = "production"
region = "us-east-1"

[providers.upcloud]
zone = "us-nyc1"
private_network = true

[providers.aws]
region = "us-east-1"
availability_zones = ["us-east-1a", "us-east-1b", "us-east-1c"]

[defaults.servers]
plan = "4xCPU-8GB"
storage = 100
backup_enabled = true
monitoring_enabled = true

[security]
firewall_strict_mode = true
encryption_required = true
audit_all_actions = true

[monitoring]
prometheus_enabled = true
grafana_enabled = true
alertmanager_enabled = true

[backup]
enabled = true
schedule = "0 2 * * *"  # Daily at 2 AM
retention_days = 30

Environment-Specific Configuration

Development Environment (config.dev.toml)

Purpose: Development-optimized settings Features: Enhanced debugging, local providers, relaxed validation

# Development environment configuration
# Optimized for local development and testing

[core]
name = "provisioning-dev"
version = "dev-{{git.branch}}"

[paths]
base = "{{env.PWD}}/dev-environment"

[logging]
level = "debug"
console_output = true
structured_logging = true
debug_http = true

[providers]
default = "local"

[providers.local]
enabled = true
fast_mode = true
mock_delays = false

[http]
timeout = 10
retries = 1
debug_requests = true

[cache]
enabled = true
ttl = 60  # Short TTL for development
debug_cache = true

[development]
auto_reload = true
hot_reload_templates = true
validate_strict = false
experimental_features = true
debug_mode = true

[orchestrator]
enabled = true
port = 8080
debug = true
file_watcher = true

[testing]
parallel_tests = true
cleanup_after_tests = true
mock_external_apis = true

Testing Environment (config.test.toml)

Purpose: Testing-specific configuration Features: Mock services, isolated environments, comprehensive logging

# Testing environment configuration
# Optimized for automated testing and CI/CD

[core]
name = "provisioning-test"
version = "test-{{build.timestamp}}"

[logging]
level = "info"
test_output = true
capture_stderr = true

[providers]
default = "local"

[providers.local]
enabled = true
mock_mode = true
deterministic = true

[http]
timeout = 5
retries = 0
mock_responses = true

[cache]
enabled = false

[testing]
isolated_environments = true
cleanup_after_each_test = true
parallel_execution = true
mock_all_external_calls = true
deterministic_ids = true

[orchestrator]
enabled = false

[validation]
strict_mode = true
fail_fast = true

Production Environment (config.prod.toml)

Purpose: Production-optimized settings Features: Performance optimization, security hardening, comprehensive monitoring

# Production environment configuration
# Optimized for performance, reliability, and security

[core]
name = "provisioning-production"
version = "{{release.version}}"

[logging]
level = "warn"
structured_logging = true
sensitive_data_filtering = true
audit_logging = true

[providers]
default = "upcloud"

[http]
timeout = 60
retries = 5
connection_pool = 20
keep_alive = true

[cache]
enabled = true
ttl = 3600
size_limit = "500MB"
persistence = true

[security]
strict_mode = true
encrypt_at_rest = true
encrypt_in_transit = true
audit_all_actions = true

[monitoring]
metrics_enabled = true
tracing_enabled = true
health_checks = true
alerting = true

[orchestrator]
enabled = true
port = 8080
bind = "0.0.0.0"
workers = 4
max_connections = 100

[performance]
parallel_operations = true
batch_operations = true
connection_pooling = true

User Overrides and Customization

Personal Development Setup

Creating User Configuration:

# Create user config directory
mkdir -p ~/.config/provisioning

# Copy template
cp src/provisioning/config-examples/config.user.toml ~/.config/provisioning/config.toml

# Customize for your environment
$EDITOR ~/.config/provisioning/config.toml

Common User Customizations:

# Personal configuration customizations

[paths]
base = "{{env.HOME}}/dev/provisioning"

[development]
editor = "code"
auto_backup = true
backup_interval = "1h"

[git]
auto_commit = false
commit_template = "[{{env.USER}}] {{change.type}}: {{change.description}}"

[providers.upcloud]
api_key = "{{env.UPCLOUD_API_KEY}}"
api_secret = "{{env.UPCLOUD_API_SECRET}}"
default_zone = "de-fra1"

[shortcuts]
# Custom command aliases
quick_server = "server create {{name}} 2xCPU-4GB --zone us-nyc1"
dev_cluster = "cluster create development --infra {{env.USER}}-dev"

[notifications]
desktop_notifications = true
sound_notifications = false
slack_webhook = "{{env.SLACK_WEBHOOK_URL}}"

Workspace-Specific Configuration

Workspace Integration:

# Workspace-aware configuration
# workspace/config/developer.toml

[workspace]
user = "developer"
type = "development"

[paths]
base = "{{workspace.root}}"
extensions = "{{workspace.root}}/extensions"
runtime = "{{workspace.root}}/runtime/{{workspace.user}}"

[development]
workspace_isolation = true
per_user_cache = true
shared_extensions = false

[infra]
current = "{{workspace.user}}-development"
auto_create = true

Validation and Error Handling

Configuration Validation

Built-in Validation:

# Validate current configuration
provisioning validate config

# Validate specific configuration file
provisioning validate config --file config.dev.toml

# Show configuration with validation
provisioning config show --validate

# Debug configuration loading
provisioning config debug

Validation Rules:

# Configuration validation in Nushell
def validate_configuration [config: record] -> record {
    let errors = []

    # Validate required fields
    if not ("paths" in $config and "base" in $config.paths) {
        $errors = ($errors | append "paths.base is required")
    }

    # Validate provider configuration
    if "providers" in $config {
        for provider in ($config.providers | columns) {
            if $provider == "upcloud" {
                if not ("api_key" in $config.providers.upcloud) {
                    $errors = ($errors | append "providers.upcloud.api_key is required")
                }
            }
        }
    }

    # Validate numeric values
    if "http" in $config and "timeout" in $config.http {
        if $config.http.timeout <= 0 {
            $errors = ($errors | append "http.timeout must be positive")
        }
    }

    {
        valid: ($errors | length) == 0,
        errors: $errors
    }
}

Error Handling

Configuration-Driven Error Handling:

# Never patch with hardcoded fallbacks - use configuration
def get_api_endpoint [provider: string] -> string {
    # Good: Configuration-driven with clear error
    let config_key = $"providers.($provider).api_url"
    let endpoint = try {
        get-config-required $config_key
    } catch {
        error make {
            msg: $"API endpoint not configured for provider ($provider)",
            help: $"Add '($config_key)' to your configuration file"
        }
    }

    $endpoint
}

# Bad: Hardcoded fallback defeats IaC purpose
def get_api_endpoint_bad [provider: string] -> string {
    try {
        get-config-required $"providers.($provider).api_url"
    } catch {
        # DON'T DO THIS - defeats configuration-driven architecture
        "https://default-api.com"
    }
}

Comprehensive Error Context:

def load_provider_config [provider: string] -> record {
    let config_section = $"providers.($provider)"

    try {
        get-config-section $config_section
    } catch { |e|
        error make {
            msg: $"Failed to load configuration for provider ($provider): ($e.msg)",
            label: {
                text: "configuration missing",
                span: (metadata $provider).span
            },
            help: [
                $"Add [$config_section] section to your configuration",
                "Example configuration files available in config-examples/",
                "Run 'provisioning config show' to see current configuration"
            ]
        }
    }
}

Interpolation and Dynamic Values

Interpolation Syntax

Supported Interpolation Variables:

# Environment variables
base_path = "{{env.HOME}}/provisioning"
user_name = "{{env.USER}}"

# Configuration references
data_path = "{{paths.base}}/data"
log_file = "{{paths.logs}}/{{core.name}}.log"

# Date/time values
backup_name = "backup-{{now.date}}-{{now.time}}"
version = "{{core.version}}-{{now.timestamp}}"

# Git information
branch_name = "{{git.branch}}"
commit_hash = "{{git.commit}}"
version_with_git = "{{core.version}}-{{git.commit}}"

# System information
hostname = "{{system.hostname}}"
platform = "{{system.platform}}"
architecture = "{{system.arch}}"

Complex Interpolation Examples

Dynamic Path Resolution:

[paths]
base = "{{env.HOME}}/.local/share/provisioning"
config = "{{paths.base}}/config"
data = "{{paths.base}}/data/{{system.hostname}}"
logs = "{{paths.base}}/logs/{{env.USER}}/{{now.date}}"
runtime = "{{paths.base}}/runtime/{{git.branch}}"

[providers.upcloud]
cache_path = "{{paths.cache}}/providers/upcloud/{{env.USER}}"
log_file = "{{paths.logs}}/upcloud-{{now.date}}.log"

Environment-Aware Configuration:

[core]
name = "provisioning-{{system.hostname}}-{{env.USER}}"
version = "{{release.version}}+{{git.commit}}.{{now.timestamp}}"

[database]
name = "provisioning_{{env.USER}}_{{git.branch}}"
backup_prefix = "{{core.name}}-backup-{{now.date}}"

[monitoring]
instance_id = "{{system.hostname}}-{{core.version}}"
tags = {
    environment = "{{infra.environment}}",
    user = "{{env.USER}}",
    version = "{{core.version}}",
    deployment_time = "{{now.iso8601}}"
}

Interpolation Functions

Custom Interpolation Logic:

# Interpolation resolver
def resolve_interpolation [template: string, context: record] -> string {
    let interpolations = ($template | parse --regex '\{\{([^}]+)\}\}')

    mut result = $template

    for interpolation in $interpolations {
        let key_path = ($interpolation.capture0 | str trim)
        let value = resolve_interpolation_key $key_path $context

        $result = ($result | str replace $"{{($interpolation.capture0)}}" $value)
    }

    $result
}

def resolve_interpolation_key [key_path: string, context: record] -> string {
    match ($key_path | split row ".") {
        ["env", $var] => ($env | get $var | default ""),
        ["paths", $path] => (resolve_path_key $path $context),
        ["now", $format] => (resolve_time_format $format),
        ["git", $info] => (resolve_git_info $info),
        ["system", $info] => (resolve_system_info $info),
        $path => (get_nested_config_value $path $context)
    }
}

Migration Strategies

ENV to Config Migration

Migration Status: The system has successfully migrated from ENV-based to config-driven architecture:

Migration Statistics:

  • Files Migrated: 65+ files across entire codebase
  • Variables Replaced: 200+ ENV variables → 476 config accessors
  • Agent-Based Development: 16 token-efficient agents used
  • Efficiency Gained: 92% token efficiency vs monolithic approach

Legacy Support

Backward Compatibility:

# Configuration accessor with ENV fallback
def get-config-with-env-fallback [
    config_key: string,
    env_var: string,
    default: string = ""
] -> string {
    # Try configuration first
    let config_value = try {
        get-config-value $config_key
    } catch { null }

    if $config_value != null {
        return $config_value
    }

    # Fall back to environment variable
    let env_value = ($env | get $env_var | default null)
    if $env_value != null {
        return $env_value
    }

    # Use default if provided
    if $default != "" {
        return $default
    }

    # Error if no value found
    error make {
        msg: $"Configuration value not found: ($config_key)",
        help: $"Set ($config_key) in configuration or ($env_var) environment variable"
    }
}

Migration Tools

Available Migration Scripts:

# Migrate existing ENV-based setup to configuration
nu src/tools/migration/env-to-config.nu --scan-environment --create-config

# Validate migration completeness
nu src/tools/migration/validate-migration.nu --check-env-usage

# Generate configuration from current environment
nu src/tools/migration/generate-config.nu --output-file config.migrated.toml

Troubleshooting

Common Configuration Issues

Configuration Not Found

Error: Configuration file not found

# Solution: Check configuration file paths
provisioning config paths

# Create default configuration
provisioning config init --template user

# Verify configuration loading order
provisioning config debug

Invalid Configuration Syntax

Error: Invalid TOML syntax in configuration file

# Solution: Validate TOML syntax
nu -c "open config.user.toml | from toml"

# Use configuration validation
provisioning validate config --file config.user.toml

# Show parsing errors
provisioning config check --verbose

Interpolation Errors

Error: Failed to resolve interpolation: {{env.MISSING_VAR}}

# Solution: Check available interpolation variables
provisioning config interpolation --list-variables

# Debug specific interpolation
provisioning config interpolation --test "{{env.USER}}"

# Show interpolation context
provisioning config debug --show-interpolation

Provider Configuration Issues

Error: Provider 'upcloud' configuration invalid

# Solution: Validate provider configuration
provisioning validate config --section providers.upcloud

# Show required provider fields
provisioning providers upcloud config --show-schema

# Test provider configuration
provisioning providers upcloud test --dry-run

Debug Commands

Configuration Debugging:

# Show complete resolved configuration
provisioning config show --resolved

# Show configuration loading order
provisioning config debug --show-hierarchy

# Show configuration sources
provisioning config sources

# Test specific configuration keys
provisioning config get paths.base --trace

# Show interpolation resolution
provisioning config interpolation --debug "{{paths.data}}/{{env.USER}}"

Performance Optimization

Configuration Caching:

# Enable configuration caching
export PROVISIONING_CONFIG_CACHE=true

# Clear configuration cache
provisioning config cache --clear

# Show cache statistics
provisioning config cache --stats

Startup Optimization:

# Optimize configuration loading
[performance]
lazy_loading = true
cache_compiled_config = true
skip_unused_sections = true

[cache]
config_cache_ttl = 3600
interpolation_cache = true

This configuration management system provides a robust, flexible foundation that supports development workflows while maintaining production reliability and security requirements.