provisioning/docs/src/development/dev-configuration.md

# Configuration Management

This document provides comprehensive guidance on provisioning's configuration architecture, environment-specific configurations, validation, error
handling, and migration strategies.

## Table of Contents

1. [Overview](#overview)
2. [Configuration Architecture](#configuration-architecture)
3. [Configuration Files](#configuration-files)
4. [Environment-Specific Configuration](#environment-specific-configuration)
5. [User Overrides and Customization](#user-overrides-and-customization)
6. [Validation and Error Handling](#validation-and-error-handling)
7. [Interpolation and Dynamic Values](#interpolation-and-dynamic-values)
8. [Migration Strategies](#migration-strategies)
9. [Troubleshooting](#troubleshooting)

## Overview

Provisioning implements a sophisticated configuration management system that has migrated from environment variable-based configuration to a
hierarchical TOML configuration system with comprehensive validation and interpolation support.

**Key Features**:

- **Hierarchical Configuration**: Multi-layer configuration with clear precedence
- **Environment-Specific**: Dedicated configurations for dev, test, and production
- **Dynamic Interpolation**: Template-based value resolution
- **Type Safety**: Comprehensive validation and error handling
- **Migration Support**: Backward compatibility with existing ENV variables
- **Workspace Integration**: Seamless integration with development workspaces

**Migration Status**: ✅ **Complete** (2025-09-23)

- **65+ files migrated** across entire codebase
- **200+ ENV variables replaced** with 476 config accessors
- **16 token-efficient agents** used for systematic migration
- **92% token efficiency** achieved vs monolithic approach

## Configuration Architecture

### Hierarchical Loading Order

The configuration system implements a clear precedence hierarchy (lowest to highest precedence):

```text
Configuration Hierarchy (Low → High Precedence)
┌─────────────────────────────────────────────────┐
│ 1. config.defaults.toml                         │ ← System defaults
│    (System-wide default values)                 │
├─────────────────────────────────────────────────┤
│ 2. ~/.config/provisioning/config.toml          │ ← User configuration
│    (User-specific preferences)                  │
├─────────────────────────────────────────────────┤
│ 3. ./provisioning.toml                         │ ← Project configuration
│    (Project-specific settings)                  │
├─────────────────────────────────────────────────┤
│ 4. ./.provisioning.toml                        │ ← Infrastructure config
│    (Infrastructure-specific settings)           │
├─────────────────────────────────────────────────┤
│ 5. Environment-specific configs                 │ ← Environment overrides
│    (config.{dev,test,prod}.toml)               │
├─────────────────────────────────────────────────┤
│ 6. Runtime environment variables                │ ← Runtime overrides
│    (PROVISIONING_* variables)                   │
└─────────────────────────────────────────────────┘
```

### Configuration Access Patterns

**Configuration Accessor Functions**:

```text
# Core configuration access
use core/nulib/lib_provisioning/config/accessor.nu

# Get configuration value with fallback
let api_url = (get-config-value "providers.upcloud.api_url" "https://api.upcloud.com")

# Get required configuration (errors if missing)
let api_key = (get-config-required "providers.upcloud.api_key")

# Get nested configuration
let server_defaults = (get-config-section "defaults.servers")

# Environment-aware configuration
let log_level = (get-config-env "logging.level" "info")

# Interpolated configuration
let data_path = (get-config-interpolated "paths.data")  # Resolves {{paths.base}}/data
```

### Migration from ENV Variables

**Before (ENV-based)**:

```text
export PROVISIONING_UPCLOUD_API_KEY="your-key"
export PROVISIONING_UPCLOUD_API_URL="https://api.upcloud.com"
export PROVISIONING_LOG_LEVEL="debug"
export PROVISIONING_BASE_PATH="/usr/local/provisioning"
```

**After (Config-based)**:

```text
# config.user.toml
[providers.upcloud]
api_key = "your-key"
api_url = "https://api.upcloud.com"

[logging]
level = "debug"

[paths]
base = "/usr/local/provisioning"
```

## Configuration Files

### System Defaults (`config.defaults.toml`)

**Purpose**: Provides sensible defaults for all system components
**Location**: Root of the repository
**Modification**: Should only be modified by system maintainers

```text
# System-wide defaults - DO NOT MODIFY in production
# Copy values to config.user.toml for customization

[core]
version = "1.0.0"
name = "provisioning-system"

[paths]
# Base path - all other paths derived from this
base = "/usr/local/provisioning"
config = "{{paths.base}}/config"
data = "{{paths.base}}/data"
logs = "{{paths.base}}/logs"
cache = "{{paths.base}}/cache"
runtime = "{{paths.base}}/runtime"

[logging]
level = "info"
file = "{{paths.logs}}/provisioning.log"
rotation = true
max_size = "100 MB"
max_files = 5

[http]
timeout = 30
retries = 3
user_agent = "provisioning-system/{{core.version}}"
use_curl = false

[providers]
default = "local"

[providers.upcloud]
api_url = "https://api.upcloud.com/1.3"
timeout = 30
max_retries = 3

[providers.aws]
region = "us-east-1"
timeout = 30

[providers.local]
enabled = true
base_path = "{{paths.data}}/local"

[defaults]
[defaults.servers]
plan = "1xCPU-2 GB"
zone = "auto"
template = "ubuntu-22.04"

[cache]
enabled = true
ttl = 3600
path = "{{paths.cache}}"

[orchestrator]
enabled = false
port = 8080
bind = "127.0.0.1"
data_path = "{{paths.data}}/orchestrator"

[workflow]
storage_backend = "filesystem"
parallel_limit = 5
rollback_enabled = true

[telemetry]
enabled = false
endpoint = ""
sample_rate = 0.1
```

### User Configuration (`~/.config/provisioning/config.toml`)

**Purpose**: User-specific customizations and preferences
**Location**: User's configuration directory
**Modification**: Users should customize this file for their needs

```text
# User configuration - customizations and personal preferences
# This file overrides system defaults

[core]
name = "provisioning-{{env.USER}}"

[paths]
# Personal installation path
base = "{{env.HOME}}/.local/share/provisioning"

[logging]
level = "debug"
file = "{{paths.logs}}/provisioning-{{env.USER}}.log"

[providers]
default = "upcloud"

[providers.upcloud]
api_key = "your-personal-api-key"
api_secret = "your-personal-api-secret"

[defaults.servers]
plan = "2xCPU-4 GB"
zone = "us-nyc1"

[development]
auto_reload = true
hot_reload_templates = true
verbose_errors = true

[notifications]
slack_webhook = "https://hooks.slack.com/your-webhook"
email = "your-email@domain.com"

[git]
auto_commit = true
commit_prefix = "[{{env.USER}}]"
```

### Project Configuration (`./provisioning.toml`)

**Purpose**: Project-specific settings shared across team
**Location**: Project root directory
**Version Control**: Should be committed to version control

```text
# Project-specific configuration
# Shared settings for this project/repository

[core]
name = "my-project-provisioning"
version = "1.2.0"

[infra]
default = "staging"
environments = ["dev", "staging", "production"]

[providers]
default = "upcloud"
allowed = ["upcloud", "aws", "local"]

[providers.upcloud]
# Project-specific UpCloud settings
default_zone = "us-nyc1"
template = "ubuntu-22.04-lts"

[defaults.servers]
plan = "2xCPU-4 GB"
storage = 50
firewall_enabled = true

[security]
enforce_https = true
require_mfa = true
allowed_cidr = ["10.0.0.0/8", "172.16.0.0/12"]

[compliance]
data_region = "us-east"
encryption_at_rest = true
audit_logging = true

[team]
admins = ["alice@company.com", "bob@company.com"]
developers = ["dev-team@company.com"]
```

### Infrastructure Configuration (`./.provisioning.toml`)

**Purpose**: Infrastructure-specific overrides
**Location**: Infrastructure directory
**Usage**: Overrides for specific infrastructure deployments

```text
# Infrastructure-specific configuration
# Overrides for this specific infrastructure deployment

[core]
name = "production-east-provisioning"

[infra]
name = "production-east"
environment = "production"
region = "us-east-1"

[providers.upcloud]
zone = "us-nyc1"
private_network = true

[providers.aws]
region = "us-east-1"
availability_zones = ["us-east-1a", "us-east-1b", "us-east-1c"]

[defaults.servers]
plan = "4xCPU-8 GB"
storage = 100
backup_enabled = true
monitoring_enabled = true

[security]
firewall_strict_mode = true
encryption_required = true
audit_all_actions = true

[monitoring]
prometheus_enabled = true
grafana_enabled = true
alertmanager_enabled = true

[backup]
enabled = true
schedule = "0 2 * * *"  # Daily at 2 AM
retention_days = 30
```

## Environment-Specific Configuration

### Development Environment (`config.dev.toml`)

**Purpose**: Development-optimized settings
**Features**: Enhanced debugging, local providers, relaxed validation

```text
# Development environment configuration
# Optimized for local development and testing

[core]
name = "provisioning-dev"
version = "dev-{{git.branch}}"

[paths]
base = "{{env.PWD}}/dev-environment"

[logging]
level = "debug"
console_output = true
structured_logging = true
debug_http = true

[providers]
default = "local"

[providers.local]
enabled = true
fast_mode = true
mock_delays = false

[http]
timeout = 10
retries = 1
debug_requests = true

[cache]
enabled = true
ttl = 60  # Short TTL for development
debug_cache = true

[development]
auto_reload = true
hot_reload_templates = true
validate_strict = false
experimental_features = true
debug_mode = true

[orchestrator]
enabled = true
port = 8080
debug = true
file_watcher = true

[testing]
parallel_tests = true
cleanup_after_tests = true
mock_external_apis = true
```

### Testing Environment (`config.test.toml`)

**Purpose**: Testing-specific configuration
**Features**: Mock services, isolated environments, comprehensive logging

```text
# Testing environment configuration
# Optimized for automated testing and CI/CD

[core]
name = "provisioning-test"
version = "test-{{build.timestamp}}"

[logging]
level = "info"
test_output = true
capture_stderr = true

[providers]
default = "local"

[providers.local]
enabled = true
mock_mode = true
deterministic = true

[http]
timeout = 5
retries = 0
mock_responses = true

[cache]
enabled = false

[testing]
isolated_environments = true
cleanup_after_each_test = true
parallel_execution = true
mock_all_external_calls = true
deterministic_ids = true

[orchestrator]
enabled = false

[validation]
strict_mode = true
fail_fast = true
```

### Production Environment (`config.prod.toml`)

**Purpose**: Production-optimized settings
**Features**: Performance optimization, security hardening, comprehensive monitoring

```text
# Production environment configuration
# Optimized for performance, reliability, and security

[core]
name = "provisioning-production"
version = "{{release.version}}"

[logging]
level = "warn"
structured_logging = true
sensitive_data_filtering = true
audit_logging = true

[providers]
default = "upcloud"

[http]
timeout = 60
retries = 5
connection_pool = 20
keep_alive = true

[cache]
enabled = true
ttl = 3600
size_limit = "500 MB"
persistence = true

[security]
strict_mode = true
encrypt_at_rest = true
encrypt_in_transit = true
audit_all_actions = true

[monitoring]
metrics_enabled = true
tracing_enabled = true
health_checks = true
alerting = true

[orchestrator]
enabled = true
port = 8080
bind = "0.0.0.0"
workers = 4
max_connections = 100

[performance]
parallel_operations = true
batch_operations = true
connection_pooling = true
```

## User Overrides and Customization

### Personal Development Setup

**Creating User Configuration**:

```text
# Create user config directory
mkdir -p ~/.config/provisioning

# Copy template
cp src/provisioning/config-examples/config.user.toml ~/.config/provisioning/config.toml

# Customize for your environment
$EDITOR ~/.config/provisioning/config.toml
```

**Common User Customizations**:

```text
# Personal configuration customizations

[paths]
base = "{{env.HOME}}/dev/provisioning"

[development]
editor = "code"
auto_backup = true
backup_interval = "1h"

[git]
auto_commit = false
commit_template = "[{{env.USER}}] {{change.type}}: {{change.description}}"

[providers.upcloud]
api_key = "{{env.UPCLOUD_API_KEY}}"
api_secret = "{{env.UPCLOUD_API_SECRET}}"
default_zone = "de-fra1"

[shortcuts]
# Custom command aliases
quick_server = "server create {{name}} 2xCPU-4 GB --zone us-nyc1"
dev_cluster = "cluster create development --infra {{env.USER}}-dev"

[notifications]
desktop_notifications = true
sound_notifications = false
slack_webhook = "{{env.SLACK_WEBHOOK_URL}}"
```

### Workspace-Specific Configuration

**Workspace Integration**:

```text
# Workspace-aware configuration
# workspace/config/developer.toml

[workspace]
user = "developer"
type = "development"

[paths]
base = "{{workspace.root}}"
extensions = "{{workspace.root}}/extensions"
runtime = "{{workspace.root}}/runtime/{{workspace.user}}"

[development]
workspace_isolation = true
per_user_cache = true
shared_extensions = false

[infra]
current = "{{workspace.user}}-development"
auto_create = true
```

## Validation and Error Handling

### Configuration Validation

**Built-in Validation**:

```text
# Validate current configuration
provisioning validate config

# Validate specific configuration file
provisioning validate config --file config.dev.toml

# Show configuration with validation
provisioning config show --validate

# Debug configuration loading
provisioning config debug
```

**Validation Rules**:

```text
# Configuration validation in Nushell
def validate_configuration [config: record] -> record {
    let errors = []

    # Validate required fields
    if not ("paths" in $config and "base" in $config.paths) {
        $errors = ($errors | append "paths.base is required")
    }

    # Validate provider configuration
    if "providers" in $config {
        for provider in ($config.providers | columns) {
            if $provider == "upcloud" {
                if not ("api_key" in $config.providers.upcloud) {
                    $errors = ($errors | append "providers.upcloud.api_key is required")
                }
            }
        }
    }

    # Validate numeric values
    if "http" in $config and "timeout" in $config.http {
        if $config.http.timeout <= 0 {
            $errors = ($errors | append "http.timeout must be positive")
        }
    }

    {
        valid: ($errors | length) == 0,
        errors: $errors
    }
}
```

### Error Handling

**Configuration-Driven Error Handling**:

```text
# Never patch with hardcoded fallbacks - use configuration
def get_api_endpoint [provider: string] -> string {
    # Good: Configuration-driven with clear error
    let config_key = $"providers.($provider).api_url"
    let endpoint = try {
        get-config-required $config_key
    } catch {
        error make {
            msg: $"API endpoint not configured for provider ($provider)",
            help: $"Add '($config_key)' to your configuration file"
        }
    }

    $endpoint
}

# Bad: Hardcoded fallback defeats IaC purpose
def get_api_endpoint_bad [provider: string] -> string {
    try {
        get-config-required $"providers.($provider).api_url"
    } catch {
        # DON'T DO THIS - defeats configuration-driven architecture
        "https://default-api.com"
    }
}
```

**Comprehensive Error Context**:

```text
def load_provider_config [provider: string] -> record {
    let config_section = $"providers.($provider)"

    try {
        get-config-section $config_section
    } catch { |e|
        error make {
            msg: $"Failed to load configuration for provider ($provider): ($e.msg)",
            label: {
                text: "configuration missing",
                span: (metadata $provider).span
            },
            help: [
                $"Add [$config_section] section to your configuration",
                "Example configuration files available in config-examples/",
                "Run 'provisioning config show' to see current configuration"
            ]
        }
    }
}
```

## Interpolation and Dynamic Values

### Interpolation Syntax

**Supported Interpolation Variables**:

```text
# Environment variables
base_path = "{{env.HOME}}/provisioning"
user_name = "{{env.USER}}"

# Configuration references
data_path = "{{paths.base}}/data"
log_file = "{{paths.logs}}/{{core.name}}.log"

# Date/time values
backup_name = "backup-{{now.date}}-{{now.time}}"
version = "{{core.version}}-{{now.timestamp}}"

# Git information
branch_name = "{{git.branch}}"
commit_hash = "{{git.commit}}"
version_with_git = "{{core.version}}-{{git.commit}}"

# System information
hostname = "{{system.hostname}}"
platform = "{{system.platform}}"
architecture = "{{system.arch}}"
```

### Complex Interpolation Examples

**Dynamic Path Resolution**:

```text
[paths]
base = "{{env.HOME}}/.local/share/provisioning"
config = "{{paths.base}}/config"
data = "{{paths.base}}/data/{{system.hostname}}"
logs = "{{paths.base}}/logs/{{env.USER}}/{{now.date}}"
runtime = "{{paths.base}}/runtime/{{git.branch}}"

[providers.upcloud]
cache_path = "{{paths.cache}}/providers/upcloud/{{env.USER}}"
log_file = "{{paths.logs}}/upcloud-{{now.date}}.log"
```

**Environment-Aware Configuration**:

```text
[core]
name = "provisioning-{{system.hostname}}-{{env.USER}}"
version = "{{release.version}}+{{git.commit}}.{{now.timestamp}}"

[database]
name = "provisioning_{{env.USER}}_{{git.branch}}"
backup_prefix = "{{core.name}}-backup-{{now.date}}"

[monitoring]
instance_id = "{{system.hostname}}-{{core.version}}"
tags = {
    environment = "{{infra.environment}}",
    user = "{{env.USER}}",
    version = "{{core.version}}",
    deployment_time = "{{now.iso8601}}"
}
```

### Interpolation Functions

**Custom Interpolation Logic**:

```text
# Interpolation resolver
def resolve_interpolation [template: string, context: record] -> string {
    let interpolations = ($template | parse --regex '\{\{([^}]+)\}\}')

    mut result = $template

    for interpolation in $interpolations {
        let key_path = ($interpolation.capture0 | str trim)
        let value = resolve_interpolation_key $key_path $context

        $result = ($result | str replace $"{{($interpolation.capture0)}}" $value)
    }

    $result
}

def resolve_interpolation_key [key_path: string, context: record] -> string {
    match ($key_path | split row ".") {
        ["env", $var] => ($env | get $var | default ""),
        ["paths", $path] => (resolve_path_key $path $context),
        ["now", $format] => (resolve_time_format $format),
        ["git", $info] => (resolve_git_info $info),
        ["system", $info] => (resolve_system_info $info),
        $path => (get_nested_config_value $path $context)
    }
}
```

## Migration Strategies

### ENV to Config Migration

**Migration Status**: The system has successfully migrated from ENV-based to config-driven architecture:

**Migration Statistics**:

- **Files Migrated**: 65+ files across entire codebase
- **Variables Replaced**: 200+ ENV variables → 476 config accessors
- **Agent-Based Development**: 16 token-efficient agents used
- **Efficiency Gained**: 92% token efficiency vs monolithic approach

### Legacy Support

**Backward Compatibility**:

```text
# Configuration accessor with ENV fallback
def get-config-with-env-fallback [
    config_key: string,
    env_var: string,
    default: string = ""
] -> string {
    # Try configuration first
    let config_value = try {
        get-config-value $config_key
    } catch { null }

    if $config_value != null {
        return $config_value
    }

    # Fall back to environment variable
    let env_value = ($env | get $env_var | default null)
    if $env_value != null {
        return $env_value
    }

    # Use default if provided
    if $default != "" {
        return $default
    }

    # Error if no value found
    error make {
        msg: $"Configuration value not found: ($config_key)",
        help: $"Set ($config_key) in configuration or ($env_var) environment variable"
    }
}
```

### Migration Tools

**Available Migration Scripts**:

```text
# Migrate existing ENV-based setup to configuration
nu src/tools/migration/env-to-config.nu --scan-environment --create-config

# Validate migration completeness
nu src/tools/migration/validate-migration.nu --check-env-usage

# Generate configuration from current environment
nu src/tools/migration/generate-config.nu --output-file config.migrated.toml
```

## Troubleshooting

### Common Configuration Issues

#### Configuration Not Found

**Error**: `Configuration file not found`

```text
# Solution: Check configuration file paths
provisioning config paths

# Create default configuration
provisioning config init --template user

# Verify configuration loading order
provisioning config debug
```

#### Invalid Configuration Syntax

**Error**: `Invalid TOML syntax in configuration file`

```text
# Solution: Validate TOML syntax
nu -c "open config.user.toml | from toml"

# Use configuration validation
provisioning validate config --file config.user.toml

# Show parsing errors
provisioning config check --verbose
```

#### Interpolation Errors

**Error**: `Failed to resolve interpolation: {{env.MISSING_VAR}}`

```text
# Solution: Check available interpolation variables
provisioning config interpolation --list-variables

# Debug specific interpolation
provisioning config interpolation --test "{{env.USER}}"

# Show interpolation context
provisioning config debug --show-interpolation
```

#### Provider Configuration Issues

**Error**: `Provider 'upcloud' configuration invalid`

```text
# Solution: Validate provider configuration
provisioning validate config --section providers.upcloud

# Show required provider fields
provisioning providers upcloud config --show-schema

# Test provider configuration
provisioning providers upcloud test --dry-run
```

### Debug Commands

**Configuration Debugging**:

```text
# Show complete resolved configuration
provisioning config show --resolved

# Show configuration loading order
provisioning config debug --show-hierarchy

# Show configuration sources
provisioning config sources

# Test specific configuration keys
provisioning config get paths.base --trace

# Show interpolation resolution
provisioning config interpolation --debug "{{paths.data}}/{{env.USER}}"
```

### Performance Optimization

**Configuration Caching**:

```text
# Enable configuration caching
export PROVISIONING_CONFIG_CACHE=true

# Clear configuration cache
provisioning config cache --clear

# Show cache statistics
provisioning config cache --stats
```

**Startup Optimization**:

```text
# Optimize configuration loading
[performance]
lazy_loading = true
cache_compiled_config = true
skip_unused_sections = true

[cache]
config_cache_ttl = 3600
interpolation_cache = true
```

This configuration management system provides a robust, flexible foundation that supports development workflows while maintaining production
reliability and security requirements.