provisioning/schemas/platform/configuration-workflow.md

Configuration Workflow: TypeDialog → Nickel → TOML → Rust

Complete documentation of the configuration pipeline that transforms interactive user input into production Rust service configurations.

Overview

The provisioning platform uses a four-stage configuration workflow that leverages TypeDialog for interactive configuration, Nickel for type-safe composition, and TOML for service consumption:

┌─────────────────────────────────────────────────────────────────┐
│ Stage 1: User Interaction (TypeDialog)                          │
│  - Can use Nickel configuration as default values               │
│    if use provisioning/platform/config/ it will be updated      │
│  - Interactive form (web/tui/cli)                               │
│  - Real-time constraint validation                              │
│  - Generates Nickel configuration                               │
└────────────────┬────────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────────┐
│ Stage 2: Composition (Nickel)                                   │
│  - Base defaults imported                                       │
│  - Mode overlay applied                                         │
│  - Validators enforce business rules                            │
│  - Produces Nickel config file                                  │
└────────────────┬────────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────────┐
│ Stage 3: Export (Nickel → TOML)                                 │
│  - Nickel config evaluated                                      │
│  - Exported to TOML format                                      │
│  - Saved to provisioning/platform/config/                       │
└────────────────┬────────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────────┐
│ Stage 4: Runtime (Rust Services)                                │
│  - Services load TOML configuration                             │
│  - Environment variables override specific values               │
│  - Start services with final configuration                      │
└─────────────────────────────────────────────────────────────────┘

---

Stage 1: User Interaction (TypeDialog)

Purpose

Collect configuration from users through an interactive, constraint-aware interface.

Workflow

# Launch interactive configuration wizard
nu scripts/configure.nu orchestrator solo --backend web

What Happens

1. Form Loads

   • TypeDialog reads forms/orchestrator-form.toml
   • Form displays configuration sections
   • Constraints from constraints.toml enforce min/max values
   • Environment variables populate initial defaults

2. User Interaction

   • User fills in form fields (workspace name, server port, etc.)
   • Real-time validation on each field
   • Constraint interpolation shows valid ranges:
     • ${constraint.orchestrator.workers.min} → 1
     • ${constraint.orchestrator.workers.max} → 32

3. Configuration Submission

   • User submits form
   • TypeDialog validates all fields against schemas
   • Generates Nickel configuration output

4. Output Generation

   • Nickel config saved to values/{service}.{mode}.ncl
   • Example: values/orchestrator.solo.ncl
   • File becomes source of truth for user customizations

Form Structure Example

# forms/orchestrator-form.toml
name = "orchestrator_configuration"
description = "Configure orchestrator service"

[[items]]
name = "workspace_group"
type = "group"
includes = ["fragments/workspace-section.toml"]

[[items]]
name = "server_group"
type = "group"
includes = ["fragments/server-section.toml"]

[[items]]
name = "queue_group"
type = "group"
includes = ["fragments/orchestrator/queue-section.toml"]

Fragment with Constraint Interpolation

# forms/fragments/orchestrator/queue-section.toml
[[elements]]
name = "max_concurrent_tasks"
type = "number"
prompt = "Maximum Concurrent Tasks"
default = 5
min = "${constraint.orchestrator.queue.concurrent_tasks.min}"
max = "${constraint.orchestrator.queue.concurrent_tasks.max}"
required = true
help = "Range: ${constraint.orchestrator.queue.concurrent_tasks.min}-${constraint.orchestrator.queue.concurrent_tasks.max}"
nickel_path = ["orchestrator", "queue", "max_concurrent_tasks"]

Generated Nickel Output (from TypeDialog)

TypeDialog's nickel-roundtrip pattern generates:

# values/orchestrator.solo.ncl
# Auto-generated by TypeDialog
{
  orchestrator = {
    workspace = {
      name = "dev-workspace",
      path = "/home/developer/provisioning/data/orchestrator",
      enabled = true,
    },
    server = {
      host = "127.0.0.1",
      port = 9090,
      workers = 2,
    },
    queue = {
      max_concurrent_tasks = 3,
      retry_attempts = 2,
      retry_delay = 1000,
    },
  },
}

---

Stage 2: Composition (Nickel)

Purpose

Compose the user input with defaults, validators, and schemas to create a complete, validated configuration.

Workflow

# The nickel typecheck command validates the composition
nickel typecheck values/orchestrator.solo.ncl

Composition Layers

The final configuration is built by merging layers in priority order:

Layer 1: Schema Import

# Ensures type safety and required fields
let schemas = import "../schemas/orchestrator.ncl" in

Layer 2: Base Defaults

# Default values for all orchestrator configurations
let defaults = import "../defaults/orchestrator-defaults.ncl" in

Layer 3: Mode Overlay

# Solo-specific overrides and adjustments
let solo_defaults = import "../defaults/deployment/solo-defaults.ncl" in

Layer 4: Validators Import

# Business rule validation (ranges, uniqueness, dependencies)
let validators = import "../validators/orchestrator-validator.ncl" in

Layer 5: User Values

# User input from TypeDialog (values/orchestrator.solo.ncl)
# Loaded and merged with defaults

Composition Example

# configs/orchestrator.solo.ncl (generated composition)

let schemas = import "../schemas/orchestrator.ncl" in
let defaults = import "../defaults/orchestrator-defaults.ncl" in
let solo_defaults = import "../defaults/deployment/solo-defaults.ncl" in
let validators = import "../validators/orchestrator-validator.ncl" in

# Composition: Base defaults + mode overlay + user input
{
  orchestrator = defaults.orchestrator & {
    # User input from TypeDialog values/orchestrator.solo.ncl
    workspace = {
      name = "dev-workspace",
      path = "/home/developer/provisioning/data/orchestrator",
    },

    # Solo mode overrides
    server = {
      workers = validators.ValidWorkers 2,
      max_connections = 128,
    },

    queue = {
      max_concurrent_tasks = validators.ValidConcurrentTasks 3,
    },

    # Fallback to defaults for unspecified fields
  },
} | schemas.OrchestratorConfig  # Validate against schema

Validation During Composition

Each field is validated through multiple validation layers:

# validators/orchestrator-validator.ncl
let constraints = import "../constraints/constraints.toml" in

{
  # Validate workers within allowed range
  ValidWorkers = fun workers =>
    if workers < constraints.orchestrator.workers.min then
      error "Workers below minimum"
    else if workers > constraints.orchestrator.workers.max then
      error "Workers above maximum"
    else
      workers,

  # Validate concurrent tasks
  ValidConcurrentTasks = fun tasks =>
    if tasks < constraints.orchestrator.queue.concurrent_tasks.min then
      error "Tasks below minimum"
    else if tasks > constraints.orchestrator.queue.concurrent_tasks.max then
      error "Tasks above maximum"
    else
      tasks,
}

Constraints: Single Source of Truth

# constraints/constraints.toml
[orchestrator.workers]
min = 1
max = 32

[orchestrator.queue.concurrent_tasks]
min = 1
max = 100

[common.server.port]
min = 1024
max = 65535

These values are referenced in:

• Form constraints (constraint interpolation)
• Validators (ValidWorkers, ValidConcurrentTasks)
• Default values (appropriate for each mode)

---

Stage 3: Export (Nickel → TOML)

Purpose

Convert validated Nickel configuration to TOML format for consumption by Rust services.

Workflow

# Export Nickel to TOML
nu scripts/generate-configs.nu orchestrator solo

Command Chain

# What happens internally:

# 1. Typecheck the Nickel config (catch errors early)
nickel typecheck provisioning/.typedialog/provisioning/platform/configs/orchestrator.solo.ncl

# 2. Export to TOML format
nickel export --format toml provisioning/.typedialog/provisioning/platform/configs/orchestrator.solo.ncl

# 3. Save to output location
# → provisioning/platform/config/orchestrator.solo.toml

Input: Nickel Configuration

# From: configs/orchestrator.solo.ncl
{
  orchestrator = {
    workspace = {
      name = "dev-workspace",
      path = "/home/developer/provisioning/data/orchestrator",
      enabled = true,
      multi_workspace = false,
    },
    server = {
      host = "127.0.0.1",
      port = 9090,
      workers = 2,
      keep_alive = 75,
      max_connections = 128,
    },
    storage = {
      backend = "filesystem",
      path = "/home/developer/provisioning/data/orchestrator",
    },
    queue = {
      max_concurrent_tasks = 3,
      retry_attempts = 2,
      retry_delay = 1000,
      task_timeout = 1800000,
    },
    monitoring = {
      enabled = true,
      metrics = {
        enabled = false,
      },
      health_check = {
        enabled = true,
        interval = 60,
      },
    },
    logging = {
      level = "debug",
      format = "text",
      outputs = [
        {
          destination = "stdout",
          level = "debug",
        },
      ],
    },
  },
}

Output: TOML Configuration

# To: provisioning/platform/config/orchestrator.solo.toml
[orchestrator.workspace]
name = "dev-workspace"
path = "/home/developer/provisioning/data/orchestrator"
enabled = true
multi_workspace = false

[orchestrator.server]
host = "127.0.0.1"
port = 9090
workers = 2
keep_alive = 75
max_connections = 128

[orchestrator.storage]
backend = "filesystem"
path = "/home/developer/provisioning/data/orchestrator"

[orchestrator.queue]
max_concurrent_tasks = 3
retry_attempts = 2
retry_delay = 1000
task_timeout = 1800000

[orchestrator.monitoring]
enabled = true

[orchestrator.monitoring.metrics]
enabled = false

[orchestrator.monitoring.health_check]
enabled = true
interval = 60

[orchestrator.logging]
level = "debug"
format = "text"

[[orchestrator.logging.outputs]]
destination = "stdout"
level = "debug"

Output Location

provisioning/platform/config/
├── orchestrator.solo.toml          # Exported from configs/orchestrator.solo.ncl
├── orchestrator.multiuser.toml     # Exported from configs/orchestrator.multiuser.ncl
├── orchestrator.cicd.toml          # Exported from configs/orchestrator.cicd.ncl
├── orchestrator.enterprise.toml    # Exported from configs/orchestrator.enterprise.ncl
├── control-center.solo.toml        # Similar structure for each service
├── control-center.multiuser.toml
├── mcp-server.solo.toml
└── mcp-server.enterprise.toml

Validation During Export

The generate-configs.nu script:

1. Typechecks - Ensures Nickel is syntactically valid
2. Evaluates - Computes final values
3. Exports - Converts to TOML format
4. Saves - Writes to provisioning/platform/config/

---

Stage 4: Runtime (Rust Services)

Purpose

Load TOML configuration and start Rust services with validated settings.

Configuration Loading Hierarchy

Rust services load configuration in this priority order:

1. Runtime Arguments (Highest Priority)

ORCHESTRATOR_CONFIG=/path/to/config.toml cargo run --bin orchestrator

2. Environment Variables

# Environment variable overrides specific TOML values
export ORCHESTRATOR_SERVER_PORT=9999
export ORCHESTRATOR_LOG_LEVEL=debug

ORCHESTRATOR_CONFIG=orchestrator.solo.toml cargo run --bin orchestrator

Environment variable format: ORCHESTRATOR_{SECTION}_{KEY}=value

Example mappings:

• ORCHESTRATOR_SERVER_PORT=9999 → orchestrator.server.port = 9999
• ORCHESTRATOR_LOG_LEVEL=debug → orchestrator.logging.level = "debug"
• ORCHESTRATOR_QUEUE_MAX_CONCURRENT_TASKS=10 → orchestrator.queue.max_concurrent_tasks = 10

3. TOML Configuration File

# Load from TOML (medium priority)
ORCHESTRATOR_CONFIG=orchestrator.solo.toml cargo run --bin orchestrator

4. Compiled Defaults (Lowest Priority)

// In Rust code - fallback for unspecified values
let config = Config::from_file(config_path)
    .unwrap_or_else(|_| Config::default());

Example: Solo Mode Startup

# Step 1: User generates config through TypeDialog
nu scripts/configure.nu orchestrator solo --backend web

# Step 2: Export to TOML
nu scripts/generate-configs.nu orchestrator solo

# Step 3: Set environment variables for environment-specific overrides
export ORCHESTRATOR_SERVER_PORT=9090
export ORCHESTRATOR_LOG_LEVEL=debug

# Step 4: Start the Rust service
ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator

Rust Service Configuration Loading

// In orchestrator/src/config.rs

use config::{Config, ConfigError, Environment, File};
use serde::Deserialize;

#[derive(Debug, Deserialize)]
pub struct OrchestratorConfig {
    pub orchestrator: OrchestratorService,
}

#[derive(Debug, Deserialize)]
pub struct OrchestratorService {
    pub workspace: Workspace,
    pub server: Server,
    pub storage: Storage,
    pub queue: Queue,
}

impl OrchestratorConfig {
    pub fn load(config_path: Option<&str>) -> Result<Self, ConfigError> {
        let mut builder = Config::builder();

        // 1. Load TOML file if provided
        if let Some(path) = config_path {
            builder = builder.add_source(File::from(Path::new(path)));
        } else {
            // Fallback to defaults
            builder = builder.add_source(File::with_name("config/orchestrator.defaults.toml"));
        }

        // 2. Apply environment variable overrides
        builder = builder.add_source(
            Environment::with_prefix("ORCHESTRATOR")
                .separator("_")
        );

        let config = builder.build()?;
        config.try_deserialize()
    }
}

Configuration Validation in Rust

impl OrchestratorConfig {
    pub fn validate(&self) -> Result<(), ConfigError> {
        // Validate server configuration
        if self.orchestrator.server.port < 1024 || self.orchestrator.server.port > 65535 {
            return Err(ConfigError::Message(
                "Server port must be between 1024 and 65535".to_string()
            ));
        }

        // Validate queue configuration
        if self.orchestrator.queue.max_concurrent_tasks == 0 {
            return Err(ConfigError::Message(
                "max_concurrent_tasks must be > 0".to_string()
            ));
        }

        // Validate storage configuration
        match self.orchestrator.storage.backend.as_str() {
            "filesystem" | "surrealdb" | "rocksdb" => {
                // Valid backend
            },
            backend => {
                return Err(ConfigError::Message(
                    format!("Unknown storage backend: {}", backend)
                ));
            }
        }

        Ok(())
    }
}

Runtime Startup Sequence

#[tokio::main]
async fn main() -> Result<()> {
    // Load configuration
    let config = OrchestratorConfig::load(
        std::env::var("ORCHESTRATOR_CONFIG").ok().as_deref()
    )?;

    // Validate configuration
    config.validate()?;

    // Initialize logging
    init_logging(&config.orchestrator.logging)?;

    // Start HTTP server
    let server = Server::new(
        config.orchestrator.server.host.clone(),
        config.orchestrator.server.port,
    );

    // Initialize storage backend
    let storage = Storage::new(&config.orchestrator.storage)?;

    // Start the service
    server.start(storage).await?;

    Ok(())
}

---

Complete Example: Solo Mode End-to-End

Step 1: Interactive Configuration

$ nu scripts/configure.nu orchestrator solo --backend web

# TypeDialog launches web interface
# User fills in form:
# - Workspace name: "dev-workspace"
# - Server host: "127.0.0.1"
# - Server port: 9090
# - Storage backend: "filesystem"
# - Storage path: "/home/developer/provisioning/data/orchestrator"
# - Max concurrent tasks: 3
# - Log level: "debug"

# Saves to: values/orchestrator.solo.ncl

Step 2: Generated Nickel Configuration

# values/orchestrator.solo.ncl
{
  orchestrator = {
    workspace = {
      name = "dev-workspace",
      path = "/home/developer/provisioning/data/orchestrator",
      enabled = true,
      multi_workspace = false,
    },
    server = {
      host = "127.0.0.1",
      port = 9090,
      workers = 2,
      keep_alive = 75,
      max_connections = 128,
    },
    storage = {
      backend = "filesystem",
      path = "/home/developer/provisioning/data/orchestrator",
    },
    queue = {
      max_concurrent_tasks = 3,
      retry_attempts = 2,
      retry_delay = 1000,
      task_timeout = 1800000,
    },
    logging = {
      level = "debug",
      format = "text",
      outputs = [{
        destination = "stdout",
        level = "debug",
      }],
    },
  },
}

Step 3: Composition and Validation

$ nickel typecheck provisioning/.typedialog/provisioning/platform/configs/orchestrator.solo.ncl

# Validation passes:
# - Workspace name: valid string ✓
# - Port 9090: within range 1024-65535 ✓
# - Max concurrent tasks 3: within range 1-100 ✓
# - Log level: recognized level ✓

Step 4: Export to TOML

$ nu scripts/generate-configs.nu orchestrator solo

# Generates: provisioning/platform/config/orchestrator.solo.toml

Step 5: TOML File Created

# provisioning/platform/config/orchestrator.solo.toml
[orchestrator.workspace]
name = "dev-workspace"
path = "/home/developer/provisioning/data/orchestrator"
enabled = true
multi_workspace = false

[orchestrator.server]
host = "127.0.0.1"
port = 9090
workers = 2
keep_alive = 75
max_connections = 128

[orchestrator.storage]
backend = "filesystem"
path = "/home/developer/provisioning/data/orchestrator"

[orchestrator.queue]
max_concurrent_tasks = 3
retry_attempts = 2
retry_delay = 1000
task_timeout = 1800000

[orchestrator.logging]
level = "debug"
format = "text"

[[orchestrator.logging.outputs]]
destination = "stdout"
level = "debug"

Step 6: Runtime Startup

$ export ORCHESTRATOR_LOG_LEVEL=debug
$ ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator

# Service loads orchestrator.solo.toml
# Environment variable overrides ORCHESTRATOR_LOG_LEVEL to "debug"
# Service starts and begins accepting requests on 127.0.0.1:9090

---

Configuration Modification Workflow

Scenario: User Wants to Change Port

Option A: Modify TypeDialog Form and Regenerate

# 1. Re-run interactive configuration
nu scripts/configure.nu orchestrator solo --backend web

# 2. User changes port to 9999 in form
# 3. TypeDialog generates new values/orchestrator.solo.ncl

# 4. Export updated config
nu scripts/generate-configs.nu orchestrator solo

# 5. New TOML created with port: 9999
# 6. Restart service
ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator

Option B: Direct TOML Edit

# 1. Edit TOML directly
vi provisioning/platform/config/orchestrator.solo.toml
# Change: port = 9999

# 2. Restart service (no Nickel re-export needed)
ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator

Option C: Environment Variable Override

# 1. No file changes needed
# 2. Just override environment variable
export ORCHESTRATOR_SERVER_PORT=9999

# 3. Restart service
ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator

---

Architecture Relationships

Component Interactions

TypeDialog Forms              Nickel Schemas
(forms/*.toml)    ←shares→   (schemas/*.ncl)
       │                            │
       │ user input                 │ type definitions
       │                            │
       ▼                            ▼
values/*.ncl  ←─ constraint validation ─→  constraints.toml
       │                                      (single source of truth)
       │                                            │
       │                                            │
       ├──→ imported into composition ────────────┤
       │     (configs/*.ncl)                       │
       │                                            │
       │     base defaults ───→ defaults/*.ncl     │
       │     mode overlay ─────→ deployment/*.ncl  │
       │     validators ──────→ validators/*.ncl   │
       │                                            │
       └──→ typecheck + export ──────────────→─────┘
             nickel export --format toml
             │
             ▼
      provisioning/platform/config/
      *.toml files
             │
             │ loaded by Rust services
             │ at runtime
             ▼
      Running Service
      (orchestrator, control-center, mcp-server)

---

Best Practices

1. Always Validate Before Deploying

# Typecheck Nickel before export
nickel typecheck provisioning/.typedialog/provisioning/platform/configs/orchestrator.solo.ncl

# Validate TOML before loading in Rust
cargo run --bin orchestrator -- --validate-config orchestrator.solo.toml

2. Use Version Control for TOML Configs

# Commit generated TOML files
git add provisioning/platform/config/orchestrator.solo.toml
git commit -m "Update orchestrator solo configuration"

# But NOT the values/*.ncl files
echo "values/*.ncl" >> provisioning/.typedialog/provisioning/platform/.gitignore

3. Document Configuration Changes

# In TypeDialog form, add comments
[[items]]
name = "max_concurrent_tasks"
type = "number"
prompt = "Max concurrent tasks (3 for dev, 50+ for production)"
help = "Increased from 3 to 10 for higher throughput testing"

4. Environment Variables for Sensitive Data

Never hardcode secrets in TOML:

# Instead of:
# [orchestrator.security]
# jwt_secret = "hardcoded-secret"

# Use environment variable:
export ORCHESTRATOR_SECURITY_JWT_SECRET="actual-secret"

# TOML can reference it:
# [orchestrator.security]
# jwt_secret = "${JWT_SECRET}"

5. Test Configuration Changes in Staging First

# Generate staging config
nu scripts/configure.nu orchestrator multiuser --backend web

# Export to staging TOML
nu scripts/generate-configs.nu orchestrator multiuser

# Test in staging environment
ORCHESTRATOR_CONFIG=orchestrator.multiuser.toml cargo run --bin orchestrator
# Monitor logs and verify behavior

# Then deploy to production

---

Summary

The four-stage workflow provides:

1. User-Friendly Interface: TypeDialog forms with real-time validation
2. Type Safety: Nickel schemas and validators catch configuration errors early
3. Flexibility: TOML format can be edited manually or generated programmatically
4. Runtime Configurability: Environment variables allow deployment-time overrides
5. Single Source of Truth: Constraints, schemas, and validators all reference shared definitions

This layered approach ensures that:

• Invalid configurations are caught before deployment
• Users can modify configuration safely
• Different deployment modes have appropriate defaults
• Configuration changes can be version-controlled
• Services can be reconfigured without code changes