provisioning/docs/src/infrastructure/configuration.md

Configuration Guide

This comprehensive guide explains the configuration system of the Infrastructure Automation platform, helping you understand, customize, and manage all configuration aspects.

What You'll Learn

• Understanding the configuration hierarchy and precedence
• Working with different configuration file types
• Configuration interpolation and templating
• Environment-specific configurations
• User customization and overrides
• Validation and troubleshooting
• Advanced configuration patterns

Configuration Architecture

Configuration Hierarchy

The system uses a layered configuration approach with clear precedence rules:

Runtime CLI arguments (highest precedence)
    ↓ (overrides)
Environment Variables
    ↓ (overrides)
Infrastructure Config (./.provisioning.toml)
    ↓ (overrides)
Project Config (./provisioning.toml)
    ↓ (overrides)
User Config (~/.config/provisioning/config.toml)
    ↓ (overrides)
System Defaults (config.defaults.toml) (lowest precedence)

Configuration File Types

File Type	Purpose	Location	Format
System Defaults	Base system configuration	config.defaults.toml	TOML
User Config	Personal preferences	~/.config/provisioning/config.toml	TOML
Project Config	Project-wide settings	./provisioning.toml	TOML
Infrastructure Config	Infra-specific settings	./.provisioning.toml	TOML
Environment Config	Environment overrides	config.{env}.toml	TOML
Infrastructure Definitions	Infrastructure as Code	main.ncl, *.ncl	Nickel

Understanding Configuration Sections

Core System Configuration

[core]
version = "1.0.0"           # System version
name = "provisioning"       # System identifier

Path Configuration

The most critical configuration section that defines where everything is located:

[paths]
# Base directory - all other paths derive from this
base = "/usr/local/provisioning"

# Derived paths (usually don't need to change these)
kloud = "{{paths.base}}/infra"
providers = "{{paths.base}}/providers"
taskservs = "{{paths.base}}/taskservs"
clusters = "{{paths.base}}/cluster"
resources = "{{paths.base}}/resources"
templates = "{{paths.base}}/templates"
tools = "{{paths.base}}/tools"
core = "{{paths.base}}/core"

[paths.files]
# Important file locations
settings_file = "settings.ncl"
keys = "{{paths.base}}/keys.yaml"
requirements = "{{paths.base}}/requirements.yaml"

Debug and Logging

[debug]
enabled = false             # Enable debug mode
metadata = false           # Show internal metadata
check = false              # Default to check mode (dry run)
remote = false            # Enable remote debugging
log_level = "info"        # Logging verbosity
no_terminal = false       # Disable terminal features

Output Configuration

[output]
file_viewer = "less"       # File viewer command
format = "yaml"           # Default output format (json, yaml, toml, text)

Provider Configuration

[providers]
default = "local"         # Default provider

[providers.aws]
api_url = ""              # AWS API endpoint (blank = default)
auth = ""                 # Authentication method
interface = "CLI"         # Interface type (CLI or API)

[providers.upcloud]
api_url = "https://api.upcloud.com/1.3"
auth = ""
interface = "CLI"

[providers.local]
api_url = ""
auth = ""
interface = "CLI"

Encryption (SOPS) Configuration

[sops]
use_sops = true           # Enable SOPS encryption
config_path = "{{paths.base}}/.sops.yaml"

# Search paths for Age encryption keys
key_search_paths = [
    "{{paths.base}}/keys/age.txt",
    "~/.config/sops/age/keys.txt"
]

Configuration Interpolation

The system supports powerful interpolation patterns for dynamic configuration values.

Basic Interpolation Patterns

Path Interpolation

# Reference other path values
templates = "{{paths.base}}/my-templates"
custom_path = "{{paths.providers}}/custom"

Environment Variable Interpolation

# Access environment variables
user_home = "{{env.HOME}}"
current_user = "{{env.USER}}"
custom_path = "{{env.CUSTOM_PATH || /default/path}}"  # With fallback

Date/Time Interpolation

# Dynamic date/time values
log_file = "{{paths.base}}/logs/app-{{now.date}}.log"
backup_dir = "{{paths.base}}/backups/{{now.timestamp}}"

Git Information Interpolation

# Git repository information
deployment_branch = "{{git.branch}}"
version_tag = "{{git.tag}}"
commit_hash = "{{git.commit}}"

Cross-Section References

# Reference values from other sections
database_host = "{{providers.aws.database_endpoint}}"
api_key = "{{sops.decrypted_key}}"

Advanced Interpolation

Function Calls

# Built-in functions
config_path = "{{path.join(env.HOME, .config, provisioning)}}"
safe_name = "{{str.lower(str.replace(project.name, ' ', '-'))}}"

Conditional Expressions

# Conditional logic
debug_level = "{{debug.enabled && 'debug' || 'info'}}"
storage_path = "{{env.STORAGE_PATH || path.join(paths.base, 'storage')}}"

Interpolation Examples

[paths]
base = "/opt/provisioning"
workspace = "{{env.HOME}}/provisioning-workspace"
current_project = "{{paths.workspace}}/{{env.PROJECT_NAME || 'default'}}"

[deployment]
environment = "{{env.DEPLOY_ENV || 'development'}}"
timestamp = "{{now.iso8601}}"
version = "{{git.tag || git.commit}}"

[database]
connection_string = "postgresql://{{env.DB_USER}}:{{env.DB_PASS}}@{{env.DB_HOST || 'localhost'}}/{{env.DB_NAME}}"

[notifications]
slack_channel = "#{{env.TEAM_NAME || 'general'}}-notifications"
email_subject = "Deployment {{deployment.environment}} - {{deployment.timestamp}}"

Environment-Specific Configuration

Environment Detection

The system automatically detects the environment using:

1. PROVISIONING_ENV environment variable
2. Git branch patterns (dev, staging, main/master)
3. Directory patterns (development, staging, production)
4. Explicit configuration

Environment Configuration Files

Create environment-specific configurations:

Development Environment (config.dev.toml)

[core]
name = "provisioning-dev"

[debug]
enabled = true
log_level = "debug"
metadata = true

[providers]
default = "local"

[cache]
enabled = false  # Disable caching for development

[notifications]
enabled = false  # No notifications in dev

Testing Environment (config.test.toml)

[core]
name = "provisioning-test"

[debug]
enabled = true
check = true     # Default to check mode in testing
log_level = "info"

[providers]
default = "local"

[infrastructure]
auto_cleanup = true  # Clean up test resources
resource_prefix = "test-{{git.branch}}-"

Production Environment (config.prod.toml)

[core]
name = "provisioning-prod"

[debug]
enabled = false
log_level = "warn"

[providers]
default = "aws"

[security]
require_approval = true
audit_logging = true
encrypt_backups = true

[notifications]
enabled = true
critical_only = true

Environment Switching

# Set environment for session
export PROVISIONING_ENV=dev
provisioning env

# Use environment for single command
provisioning --environment prod server create

# Switch environment permanently
provisioning env set prod

User Configuration Customization

Creating Your User Configuration

# Initialize user configuration from template
provisioning init config

# Or copy and customize
cp config-examples/config.user.toml ~/.config/provisioning/config.toml

Common User Customizations

Developer Setup

[paths]
base = "/Users/alice/dev/provisioning"

[debug]
enabled = true
log_level = "debug"

[providers]
default = "local"

[output]
format = "json"
file_viewer = "code"

[sops]
key_search_paths = [
    "/Users/alice/.config/sops/age/keys.txt"
]

Operations Engineer Setup

[paths]
base = "/opt/provisioning"

[debug]
enabled = false
log_level = "info"

[providers]
default = "aws"

[output]
format = "yaml"

[notifications]
enabled = true
email = "ops-team@company.com"

Team Lead Setup

[paths]
base = "/home/teamlead/provisioning"

[debug]
enabled = true
metadata = true
log_level = "info"

[providers]
default = "upcloud"

[security]
require_confirmation = true
audit_logging = true

[sops]
key_search_paths = [
    "/secure/keys/team-lead.txt",
    "~/.config/sops/age/keys.txt"
]

Project-Specific Configuration

Project Configuration File (provisioning.toml)

[project]
name = "web-application"
description = "Main web application infrastructure"
version = "2.1.0"
team = "platform-team"

[paths]
# Project-specific path overrides
infra = "./infrastructure"
templates = "./custom-templates"

[defaults]
# Project defaults
provider = "aws"
region = "us-west-2"
environment = "development"

[cost_controls]
max_monthly_budget = 5000.00
alert_threshold = 0.8

[compliance]
required_tags = ["team", "environment", "cost-center"]
encryption_required = true
backup_required = true

[notifications]
slack_webhook = "https://hooks.slack.com/services/..."
team_email = "platform-team@company.com"

Infrastructure-Specific Configuration (.provisioning.toml)

[infrastructure]
name = "production-web-app"
environment = "production"
region = "us-west-2"

[overrides]
# Infrastructure-specific overrides
debug.enabled = false
debug.log_level = "error"
cache.enabled = true

[scaling]
auto_scaling_enabled = true
min_instances = 3
max_instances = 20

[security]
vpc_id = "vpc-12345678"
subnet_ids = ["subnet-12345678", "subnet-87654321"]
security_group_id = "sg-12345678"

[monitoring]
enabled = true
retention_days = 90
alerting_enabled = true

Configuration Validation

Built-in Validation

# Validate current configuration
provisioning validate config

# Detailed validation with warnings
provisioning validate config --detailed

# Strict validation mode
provisioning validate config strict

# Validate specific environment
provisioning validate config --environment prod

Custom Validation Rules

Create custom validation in your configuration:

[validation]
# Custom validation rules
required_sections = ["paths", "providers", "debug"]
required_env_vars = ["AWS_REGION", "PROJECT_NAME"]
forbidden_values = ["password123", "admin"]

[validation.paths]
# Path validation rules
base_must_exist = true
writable_required = ["paths.base", "paths.cache"]

[validation.security]
# Security validation
require_encryption = true
min_key_length = 32

Troubleshooting Configuration

Common Configuration Issues

Issue 1: Path Not Found Errors

# Problem: Base path doesn't exist
# Check current configuration
provisioning env | grep paths.base

# Verify path exists
ls -la /path/shown/above

# Fix: Update user config
nano ~/.config/provisioning/config.toml
# Set correct paths.base = "/correct/path"

Issue 2: Interpolation Failures

# Problem: {{env.VARIABLE}} not resolving
# Check environment variables
env | grep VARIABLE

# Check interpolation
provisioning validate interpolation test

# Debug interpolation
provisioning --debug validate interpolation validate

Issue 3: SOPS Encryption Errors

# Problem: Cannot decrypt SOPS files
# Check SOPS configuration
provisioning sops config

# Verify key files
ls -la ~/.config/sops/age/keys.txt

# Test decryption
sops -d encrypted-file.ncl

Issue 4: Provider Authentication

# Problem: Provider authentication failed
# Check provider configuration
provisioning show providers

# Test provider connection
provisioning provider test aws

# Verify credentials
aws configure list  # For AWS

Configuration Debugging

# Show current configuration hierarchy
provisioning config show --hierarchy

# Show configuration sources
provisioning config sources

# Show interpolated values
provisioning config interpolated

# Debug specific section
provisioning config debug paths
provisioning config debug providers

Configuration Reset

# Reset to defaults
provisioning config reset

# Reset specific section
provisioning config reset providers

# Backup current config before reset
provisioning config backup

Advanced Configuration Patterns

Dynamic Configuration Loading

[dynamic]
# Load configuration from external sources
config_urls = [
    "https://config.company.com/provisioning/base.toml",
    "file:///etc/provisioning/shared.toml"
]

# Conditional configuration loading
load_if_exists = [
    "./local-overrides.toml",
    "../shared/team-config.toml"
]

Configuration Templating

[templates]
# Template-based configuration
base_template = "aws-web-app"
template_vars = {
    region = "us-west-2"
    instance_type = "t3.medium"
    team_name = "platform"
}

# Template inheritance
extends = ["base-web", "monitoring", "security"]

Multi-Region Configuration

[regions]
primary = "us-west-2"
secondary = "us-east-1"

[regions.us-west-2]
providers.aws.region = "us-west-2"
availability_zones = ["us-west-2a", "us-west-2b", "us-west-2c"]

[regions.us-east-1]
providers.aws.region = "us-east-1"
availability_zones = ["us-east-1a", "us-east-1b", "us-east-1c"]

Configuration Profiles

[profiles]
active = "development"

[profiles.development]
debug.enabled = true
providers.default = "local"
cost_controls.enabled = false

[profiles.staging]
debug.enabled = true
providers.default = "aws"
cost_controls.max_budget = 1000.00

[profiles.production]
debug.enabled = false
providers.default = "aws"
security.strict_mode = true

Configuration Management Best Practices

1. Version Control

# Track configuration changes
git add provisioning.toml
git commit -m "feat(config): add production settings"

# Use branches for configuration experiments
git checkout -b config/new-provider

2. Documentation

# Document your configuration choices
[paths]
# Using custom base path for team shared installation
base = "/opt/team-provisioning"

[debug]
# Debug enabled for troubleshooting infrastructure issues
enabled = true
log_level = "debug"  # Temporary while debugging network problems

3. Validation

# Always validate before committing
provisioning validate config
git add . && git commit -m "update config"

4. Backup

# Regular configuration backups
provisioning config export --format yaml > config-backup-$(date +%Y%m%d).yaml

# Automated backup script
echo '0 2 * * * provisioning config export > ~/backups/config-$(date +\%Y\%m\%d).yaml' | crontab -

5. Security

• Never commit sensitive values in plain text
• Use SOPS for encrypting secrets
• Rotate encryption keys regularly
• Audit configuration access

# Encrypt sensitive configuration
sops -e settings.ncl > settings.encrypted.ncl

# Audit configuration changes
git log -p -- provisioning.toml

Configuration Migration

Migrating from Environment Variables

# Old: Environment variables
export PROVISIONING_DEBUG=true
export PROVISIONING_PROVIDER=aws

# New: Configuration file
[debug]
enabled = true

[providers]
default = "aws"

Upgrading Configuration Format

# Check for configuration updates needed
provisioning config check-version

# Migrate to new format
provisioning config migrate --from 1.0 --to 2.0

# Validate migrated configuration
provisioning validate config

Next Steps

Now that you understand the configuration system:

1. Create your user configuration: provisioning init config
2. Set up environment-specific configs for your workflow
3. Learn CLI commands: CLI Reference
4. Practice with examples: Examples and Tutorials
5. Troubleshoot issues: Troubleshooting Guide

You now have complete control over how provisioning behaves in your environment!