Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Target-Based Configuration System - Complete Implementation

Version: 4.0.0 Date: 2025-10-06 Status: ✅ PRODUCTION READY

Executive Summary

A comprehensive target-based configuration system has been successfully implemented, replacing the monolithic config.defaults.toml with a modular, workspace-centric architecture. Each provider, platform service, and KMS component now has independent configuration, and workspaces are fully self-contained with their own config/provisioning.yaml.

🎯 Objectives Achieved

✅ Independent Target Configs: Providers, platform services, and KMS have separate configs ✅ Workspace-Centric: Each workspace has complete, self-contained configuration ✅ User Context Priority: ws_{name}.yaml files provide high-priority overrides ✅ No Runtime config.defaults.toml: Template-only, never loaded at runtime ✅ Migration Automation: Safe migration scripts with dry-run and backup ✅ Schema Validation: Comprehensive validation for all config types ✅ CLI Integration: Complete command suite for config management ✅ Legacy Nomenclature: All cn_provisioning/kloud references updated

📐 Architecture Overview

Configuration Hierarchy (Priority: Low → High)

1. Workspace Config      workspace/{name}/config/provisioning.yaml
2. Provider Configs      workspace/{name}/config/providers/*.toml
3. Platform Configs      workspace/{name}/config/platform/*.toml
4. User Context          ~/Library/Application Support/provisioning/ws_{name}.yaml
5. Environment Variables PROVISIONING_*

Directory Structure

workspace/{name}/
├── config/
│   ├── provisioning.yaml          # Main workspace config (YAML)
│   ├── providers/
│   │   ├── aws.toml               # AWS provider config
│   │   ├── upcloud.toml           # UpCloud provider config
│   │   └── local.toml             # Local provider config
│   ├── platform/
│   │   ├── orchestrator.toml      # Orchestrator service config
│   │   ├── control-center.toml    # Control Center config
│   │   └── mcp-server.toml        # MCP Server config
│   └── kms.toml                   # KMS configuration
├── infra/                         # Infrastructure definitions
├── .cache/                        # Cache directory
├── .runtime/                      # Runtime data
├── .providers/                    # Provider-specific runtime
├── .orchestrator/                 # Orchestrator data
└── .kms/                          # KMS keys and cache

🚀 Implementation Details

Phase 1: Nomenclature Migration ✅

Files Updated: 9 core files (29+ changes)

Mappings:

  • cn_provisioning → provisioning
  • kloud → workspace
  • kloud_path → workspace_path
  • kloud_list → workspace_list
  • dflt_set → default_settings
  • PROVISIONING_KLOUD_PATH → PROVISIONING_WORKSPACE_PATH

Files Modified:

  1. lib_provisioning/defs/lists.nu
  2. lib_provisioning/sops/lib.nu
  3. lib_provisioning/kms/lib.nu
  4. lib_provisioning/cmd/lib.nu
  5. lib_provisioning/config/migration.nu
  6. lib_provisioning/config/loader.nu
  7. lib_provisioning/config/accessor.nu
  8. lib_provisioning/utils/settings.nu
  9. templates/default_context.yaml

Phase 2: Independent Target Configs ✅

2.1 Provider Configs

Files Created: 6 files (3 providers × 2 files each)

ProviderConfigSchemaFeatures
AWSextensions/providers/aws/config.defaults.tomlconfig.schema.tomlCLI/API, multi-auth, cost tracking
UpCloudextensions/providers/upcloud/config.defaults.tomlconfig.schema.tomlAPI-first, firewall, backups
Localextensions/providers/local/config.defaults.tomlconfig.schema.tomlMulti-backend (libvirt/docker/podman)

Interpolation Variables: {{workspace.path}}, {{provider.paths.base}}

2.2 Platform Service Configs

Files Created: 10 files

ServiceConfigSchemaIntegration
Orchestratorplatform/orchestrator/config.defaults.tomlconfig.schema.tomlRust config loader (src/config.rs)
Control Centerplatform/control-center/config.defaults.tomlconfig.schema.tomlEnhanced with workspace paths
MCP Serverplatform/mcp-server/config.defaults.tomlconfig.schema.tomlNew configuration

Orchestrator Rust Integration:

  • Added toml dependency to Cargo.toml
  • Created src/config.rs (291 lines)
  • CLI args override config values

2.3 KMS Config

Files Created: 6 files (2,510 lines total)

  • core/services/kms/config.defaults.toml (270 lines)
  • core/services/kms/config.schema.toml (330 lines)
  • core/services/kms/config.remote.example.toml (180 lines)
  • core/services/kms/config.local.example.toml (290 lines)
  • core/services/kms/README.md (500+ lines)
  • core/services/kms/MIGRATION.md (800+ lines)

Key Features:

  • Three modes: local, remote, hybrid
  • 59 new accessor functions in config/accessor.nu
  • Secure defaults (TLS 1.3, 0600 permissions)
  • Comprehensive security validation

Phase 3: Workspace Structure ✅

3.1 Workspace-Centric Architecture

Template Files Created: 7 files

  • config/templates/workspace-provisioning.yaml.template
  • config/templates/provider-aws.toml.template
  • config/templates/provider-local.toml.template
  • config/templates/provider-upcloud.toml.template
  • config/templates/kms.toml.template
  • config/templates/user-context.yaml.template
  • config/templates/README.md

Workspace Init Module: lib_provisioning/workspace/init.nu

Functions:

  • workspace-init - Initialize complete workspace structure
  • workspace-init-interactive - Interactive creation wizard
  • workspace-list - List all workspaces
  • workspace-activate - Activate a workspace
  • workspace-get-active - Get currently active workspace

3.2 User Context System

User Context Files: ~/Library/Application Support/provisioning/ws_{name}.yaml

Format:

workspace:
  name: "production"
  path: "/path/to/workspace"
  active: true

overrides:
  debug_enabled: false
  log_level: "info"
  kms_mode: "remote"
  # ... 9 override fields total

Functions Created:

  • create-workspace-context - Create ws_{name}.yaml
  • set-workspace-active - Mark workspace as active
  • list-workspace-contexts - List all contexts
  • get-active-workspace-context - Get active workspace
  • update-workspace-last-used - Update timestamp

Helper Functions: lib_provisioning/workspace/helpers.nu

  • apply-context-overrides - Apply overrides to config
  • validate-workspace-context - Validate context structure
  • has-workspace-context - Check context existence

3.3 Workspace Activation

CLI Flags Added:

  • --activate (-a) - Activate workspace on creation
  • --interactive (-I) - Interactive creation wizard

Commands:

# Create and activate
provisioning workspace init my-app ~/workspaces/my-app --activate

# Interactive mode
provisioning workspace init --interactive

# Activate existing
provisioning workspace activate my-app

Phase 4: Configuration Loading ✅

4.1 Config Loader Refactored

File: lib_provisioning/config/loader.nu

Critical Changes:

  • ❌ REMOVED: get-defaults-config-path() function
  • ✅ ADDED: get-active-workspace() function
  • ✅ ADDED: apply-user-context-overrides() function
  • ✅ ADDED: YAML format support

New Loading Sequence:

  1. Get active workspace from user context
  2. Load workspace/{name}/config/provisioning.yaml
  3. Load provider configs from workspace/{name}/config/providers/*.toml
  4. Load platform configs from workspace/{name}/config/platform/*.toml
  5. Load user context ws_{name}.yaml (stored separately)
  6. Apply user context overrides (highest config priority)
  7. Apply environment-specific overrides
  8. Apply environment variable overrides (highest priority)
  9. Interpolate paths
  10. Validate configuration

4.2 Path Interpolation

Variables Supported:

  • {{workspace.path}} - Active workspace base path
  • {{workspace.name}} - Active workspace name
  • {{provider.paths.base}} - Provider-specific paths
  • {{env.*}} - Environment variables (safe list)
  • {{now.date}}, {{now.timestamp}}, {{now.iso}} - Date/time
  • {{git.branch}}, {{git.commit}} - Git info
  • {{path.join(...)}} - Path joining function

Implementation: Already present in loader.nu (lines 698-1262)

Phase 5: CLI Commands ✅

Module Created: lib_provisioning/workspace/config_commands.nu (380 lines)

Commands Implemented:

# Show configuration
provisioning workspace config show [name] [--format yaml|json|toml]

# Validate configuration
provisioning workspace config validate [name]

# Generate provider config
provisioning workspace config generate provider <name>

# Edit configuration
provisioning workspace config edit <type> [name]
  # Types: main, provider, platform, kms

# Show hierarchy
provisioning workspace config hierarchy [name]

# List configs
provisioning workspace config list [name] [--type all|provider|platform|kms]

Help System Updated: main_provisioning/help_system.nu

Phase 6: Migration & Validation ✅

6.1 Migration Script

File: scripts/migrate-to-target-configs.nu (200+ lines)

Features:

  • Automatic detection of old config.defaults.toml
  • Workspace structure creation
  • Config transformation (TOML → YAML)
  • Provider config generation from templates
  • User context creation
  • Safety features: --dry-run, --backup, confirmation prompts

Usage:

# Dry run
./scripts/migrate-to-target-configs.nu --workspace-name "prod" --dry-run

# Execute with backup
./scripts/migrate-to-target-configs.nu --workspace-name "prod" --backup

6.2 Schema Validation

Module: lib_provisioning/config/schema_validator.nu (150+ lines)

Validation Features:

  • Required fields checking
  • Type validation (string, int, bool, record)
  • Enum value validation
  • Numeric range validation (min/max)
  • Pattern matching with regex
  • Deprecation warnings
  • Pretty-printed error messages

Functions:

# Generic validation
validate-config-with-schema $config $schema_file

# Domain-specific
validate-provider-config "aws" $config
validate-platform-config "orchestrator" $config
validate-kms-config $config
validate-workspace-config $config

Test Suite: tests/config_validation_tests.nu (200+ lines)

📊 Statistics

Files Created

CategoryCountTotal Lines
Provider Configs622,900 bytes
Platform Configs10~1,500 lines
KMS Configs62,510 lines
Workspace Templates7~800 lines
Migration Scripts1200+ lines
Validation System2350+ lines
CLI Commands1380 lines
Documentation15+8,000+ lines
TOTAL48+~13,740 lines

Files Modified

CategoryCountChanges
Core Libraries829+ occurrences
Config Loader1Major refactor
Context System2Enhanced
CLI Integration5Flags & commands
TOTAL16Significant

🎓 Key Features

1. Independent Configuration

✅ Each provider has own config ✅ Each platform service has own config ✅ KMS has independent config ✅ No shared monolithic config

2. Workspace Self-Containment

✅ Each workspace has complete config ✅ No dependency on global config ✅ Portable workspace directories ✅ Easy backup/restore

3. User Context Priority

✅ Per-workspace overrides ✅ Highest config file priority ✅ Active workspace tracking ✅ Last used timestamp

4. Migration Safety

✅ Dry-run mode ✅ Automatic backups ✅ Confirmation prompts ✅ Rollback procedures

5. Comprehensive Validation

✅ Schema-based validation ✅ Type checking ✅ Pattern matching ✅ Deprecation warnings

6. CLI Integration

✅ Workspace creation with activation ✅ Interactive mode ✅ Config management commands ✅ Validation commands

📖 Documentation

Created Documentation

  1. Architecture: docs/configuration/workspace-config-architecture.md
  2. Migration Guide: docs/MIGRATION_GUIDE.md
  3. Validation Guide: docs/CONFIG_VALIDATION.md
  4. Migration Example: docs/MIGRATION_EXAMPLE.md
  5. CLI Commands: docs/user/workspace-config-commands.md
  6. KMS README: core/services/kms/README.md
  7. KMS Migration: core/services/kms/MIGRATION.md
  8. Platform Summary: platform/PLATFORM_CONFIG_SUMMARY.md
  9. Workspace Implementation: docs/WORKSPACE_CONFIG_IMPLEMENTATION_SUMMARY.md
  10. Template Guide: config/templates/README.md

🧪 Testing

Test Suites Created

  1. Config Validation Tests: tests/config_validation_tests.nu

    • Required fields validation
    • Type validation
    • Enum validation
    • Range validation
    • Pattern validation
    • Deprecation warnings

  2. Workspace Verification: lib_provisioning/workspace/verify.nu

    • Template directory checks
    • Template file existence
    • Module loading verification
    • Config loader validation

Running Tests

# Run validation tests
nu tests/config_validation_tests.nu

# Run workspace verification
nu lib_provisioning/workspace/verify.nu

# Validate specific workspace
provisioning workspace config validate my-app

🔄 Migration Path

Step-by-Step Migration

  1. Backup

    cp -r provisioning/config provisioning/config.backup.$(date +%Y%m%d)

  2. Dry Run

    ./scripts/migrate-to-target-configs.nu --workspace-name "production" --dry-run

  3. Execute Migration

    ./scripts/migrate-to-target-configs.nu --workspace-name "production" --backup

  4. Validate

    provisioning workspace config validate

  5. Test

    provisioning --check server list

  6. Clean Up

    # Only after verifying everything works
rm provisioning/config/config.defaults.toml

⚠️ Breaking Changes

Version 4.0.0 Changes

  1. config.defaults.toml is template-only

    • Never loaded at runtime
    • Used only to generate workspace configs

  2. Workspace required

    • Must have active workspace
    • Or be in workspace directory

  3. Environment variables renamed

    • PROVISIONING_KLOUD_PATH → PROVISIONING_WORKSPACE_PATH
    • PROVISIONING_DFLT_SET → PROVISIONING_DEFAULT_SETTINGS

  4. User context location

    • ~/Library/Application Support/provisioning/ws_{name}.yaml
    • Not default_context.yaml

🎯 Success Criteria

All success criteria MET ✅:

  1. ✅ Zero occurrences of legacy nomenclature
  2. ✅ Each provider has independent config + schema
  3. ✅ Each platform service has independent config
  4. ✅ KMS has independent config (local/remote)
  5. ✅ Workspace creation generates complete config structure
  6. ✅ User context system ws_{name}.yaml functional
  7. ✅ provisioning workspace create --activate works
  8. ✅ Config hierarchy respected correctly
  9. ✅ paths.base adjusts dynamically per workspace
  10. ✅ Migration script tested and functional
  11. ✅ Documentation complete
  12. ✅ Tests passing

📞 Support

Common Issues

Issue: “No active workspace found” Solution: Initialize or activate a workspace

provisioning workspace init my-app ~/workspaces/my-app --activate

Issue: “Config file not found” Solution: Ensure workspace is properly initialized

provisioning workspace config validate

Issue: “Old config still being loaded” Solution: Verify config.defaults.toml is not in runtime path

# Check loader.nu - get-defaults-config-path should be REMOVED
grep "get-defaults-config-path" lib_provisioning/config/loader.nu
# Should return: (empty)

Getting Help

# General help
provisioning help

# Workspace help
provisioning help workspace

# Config commands help
provisioning workspace config help

🏁 Conclusion

The target-based configuration system is complete, tested, and production-ready. It provides:

  • Modularity: Independent configs per target
  • Flexibility: Workspace-centric with user overrides
  • Safety: Migration scripts with dry-run and backups
  • Validation: Comprehensive schema validation
  • Usability: Complete CLI integration
  • Documentation: Extensive guides and examples

All objectives achieved. System ready for deployment.

Maintained By: Infrastructure Team Version: 4.0.0 Status: ✅ Production Ready Last Updated: 2025-10-06