provisioning/schemas/platform/configuration-workflow.md

# Configuration Workflow: TypeDialog → Nickel → TOML → Rust

Complete documentation of the configuration pipeline that transforms interactive user input into production Rust service configurations.

## Overview

The provisioning platform uses a **four-stage configuration workflow** that leverages TypeDialog for interactive configuration,
Nickel for type-safe composition, and TOML for service consumption:

```nickel
┌─────────────────────────────────────────────────────────────────┐
│ Stage 1: User Interaction (TypeDialog)                          │
│  - Can use Nickel configuration as default values               │
│    if use provisioning/platform/config/ it will be updated      │
│  - Interactive form (web/tui/cli)                               │
│  - Real-time constraint validation                              │
│  - Generates Nickel configuration                               │
└────────────────┬────────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────────┐
│ Stage 2: Composition (Nickel)                                   │
│  - Base defaults imported                                       │
│  - Mode overlay applied                                         │
│  - Validators enforce business rules                            │
│  - Produces Nickel config file                                  │
└────────────────┬────────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────────┐
│ Stage 3: Export (Nickel → TOML)                                 │
│  - Nickel config evaluated                                      │
│  - Exported to TOML format                                      │
│  - Saved to provisioning/platform/config/                       │
└────────────────┬────────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────────┐
│ Stage 4: Runtime (Rust Services)                                │
│  - Services load TOML configuration                             │
│  - Environment variables override specific values               │
│  - Start services with final configuration                      │
└─────────────────────────────────────────────────────────────────┘
```

---

## Stage 1: User Interaction (TypeDialog)

### Purpose

Collect configuration from users through an interactive, constraint-aware interface.

### Workflow

```bash
# Launch interactive configuration wizard
nu scripts/configure.nu orchestrator solo --backend web
```

### What Happens

1. **Form Loads**
   - TypeDialog reads `forms/orchestrator-form.toml`
   - Form displays configuration sections
   - Constraints from `constraints.toml` enforce min/max values
   - Environment variables populate initial defaults

2. **User Interaction**
   - User fills in form fields (workspace name, server port, etc.)
   - Real-time validation on each field
   - Constraint interpolation shows valid ranges:
     - `${constraint.orchestrator.workers.min}` → `1`
     - `${constraint.orchestrator.workers.max}` → `32`

3. **Configuration Submission**
   - User submits form
   - TypeDialog validates all fields against schemas
   - Generates Nickel configuration output

4. **Output Generation**
   - Nickel config saved to `values/{service}.{mode}.ncl`
   - Example: `values/orchestrator.solo.ncl`
   - File becomes source of truth for user customizations

### Form Structure Example

```bash
# forms/orchestrator-form.toml
name = "orchestrator_configuration"
description = "Configure orchestrator service"

[[items]]
name = "workspace_group"
type = "group"
includes = ["fragments/workspace-section.toml"]

[[items]]
name = "server_group"
type = "group"
includes = ["fragments/server-section.toml"]

[[items]]
name = "queue_group"
type = "group"
includes = ["fragments/orchestrator/queue-section.toml"]
```

### Fragment with Constraint Interpolation

```bash
# forms/fragments/orchestrator/queue-section.toml
[[elements]]
name = "max_concurrent_tasks"
type = "number"
prompt = "Maximum Concurrent Tasks"
default = 5
min = "${constraint.orchestrator.queue.concurrent_tasks.min}"
max = "${constraint.orchestrator.queue.concurrent_tasks.max}"
required = true
help = "Range: ${constraint.orchestrator.queue.concurrent_tasks.min}-${constraint.orchestrator.queue.concurrent_tasks.max}"
nickel_path = ["orchestrator", "queue", "max_concurrent_tasks"]
```

### Generated Nickel Output (from TypeDialog)

TypeDialog's `nickel-roundtrip` pattern generates:

```nickel
# values/orchestrator.solo.ncl
# Auto-generated by TypeDialog
{
  orchestrator = {
    workspace = {
      name = "dev-workspace",
      path = "/home/developer/provisioning/data/orchestrator",
      enabled = true,
    },
    server = {
      host = "127.0.0.1",
      port = 9090,
      workers = 2,
    },
    queue = {
      max_concurrent_tasks = 3,
      retry_attempts = 2,
      retry_delay = 1000,
    },
  },
}
```

---

## Stage 2: Composition (Nickel)

### Purpose

Compose the user input with defaults, validators, and schemas to create a complete, validated configuration.

### Workflow

```bash
# The nickel typecheck command validates the composition
nickel typecheck values/orchestrator.solo.ncl
```

### Composition Layers

The final configuration is built by merging layers in priority order:

#### Layer 1: Schema Import

```bash
# Ensures type safety and required fields
let schemas = import "../schemas/orchestrator.ncl" in
```

#### Layer 2: Base Defaults

```bash
# Default values for all orchestrator configurations
let defaults = import "../defaults/orchestrator-defaults.ncl" in
```

#### Layer 3: Mode Overlay

```bash
# Solo-specific overrides and adjustments
let solo_defaults = import "../defaults/deployment/solo-defaults.ncl" in
```

#### Layer 4: Validators Import

```bash
# Business rule validation (ranges, uniqueness, dependencies)
let validators = import "../validators/orchestrator-validator.ncl" in
```

#### Layer 5: User Values

```bash
# User input from TypeDialog (values/orchestrator.solo.ncl)
# Loaded and merged with defaults
```

### Composition Example

```bash
# configs/orchestrator.solo.ncl (generated composition)

let schemas = import "../schemas/orchestrator.ncl" in
let defaults = import "../defaults/orchestrator-defaults.ncl" in
let solo_defaults = import "../defaults/deployment/solo-defaults.ncl" in
let validators = import "../validators/orchestrator-validator.ncl" in

# Composition: Base defaults + mode overlay + user input
{
  orchestrator = defaults.orchestrator & {
    # User input from TypeDialog values/orchestrator.solo.ncl
    workspace = {
      name = "dev-workspace",
      path = "/home/developer/provisioning/data/orchestrator",
    },

    # Solo mode overrides
    server = {
      workers = validators.ValidWorkers 2,
      max_connections = 128,
    },

    queue = {
      max_concurrent_tasks = validators.ValidConcurrentTasks 3,
    },

    # Fallback to defaults for unspecified fields
  },
} | schemas.OrchestratorConfig  # Validate against schema
```

### Validation During Composition

Each field is validated through multiple validation layers:

```bash
# validators/orchestrator-validator.ncl
let constraints = import "../constraints/constraints.toml" in

{
  # Validate workers within allowed range
  ValidWorkers = fun workers =>
    if workers < constraints.orchestrator.workers.min then
      error "Workers below minimum"
    else if workers > constraints.orchestrator.workers.max then
      error "Workers above maximum"
    else
      workers,

  # Validate concurrent tasks
  ValidConcurrentTasks = fun tasks =>
    if tasks < constraints.orchestrator.queue.concurrent_tasks.min then
      error "Tasks below minimum"
    else if tasks > constraints.orchestrator.queue.concurrent_tasks.max then
      error "Tasks above maximum"
    else
      tasks,
}
```

### Constraints: Single Source of Truth

```bash
# constraints/constraints.toml
[orchestrator.workers]
min = 1
max = 32

[orchestrator.queue.concurrent_tasks]
min = 1
max = 100

[common.server.port]
min = 1024
max = 65535
```

These values are referenced in:
- Form constraints (constraint interpolation)
- Validators (ValidWorkers, ValidConcurrentTasks)
- Default values (appropriate for each mode)

---

## Stage 3: Export (Nickel → TOML)

### Purpose

Convert validated Nickel configuration to TOML format for consumption by Rust services.

### Workflow

```bash
# Export Nickel to TOML
nu scripts/generate-configs.nu orchestrator solo
```

### Command Chain

```bash
# What happens internally:

# 1. Typecheck the Nickel config (catch errors early)
nickel typecheck provisioning/.typedialog/provisioning/platform/configs/orchestrator.solo.ncl

# 2. Export to TOML format
nickel export --format toml provisioning/.typedialog/provisioning/platform/configs/orchestrator.solo.ncl

# 3. Save to output location
# → provisioning/platform/config/orchestrator.solo.toml
```

### Input: Nickel Configuration

```nickel
# From: configs/orchestrator.solo.ncl
{
  orchestrator = {
    workspace = {
      name = "dev-workspace",
      path = "/home/developer/provisioning/data/orchestrator",
      enabled = true,
      multi_workspace = false,
    },
    server = {
      host = "127.0.0.1",
      port = 9090,
      workers = 2,
      keep_alive = 75,
      max_connections = 128,
    },
    storage = {
      backend = "filesystem",
      path = "/home/developer/provisioning/data/orchestrator",
    },
    queue = {
      max_concurrent_tasks = 3,
      retry_attempts = 2,
      retry_delay = 1000,
      task_timeout = 1800000,
    },
    monitoring = {
      enabled = true,
      metrics = {
        enabled = false,
      },
      health_check = {
        enabled = true,
        interval = 60,
      },
    },
    logging = {
      level = "debug",
      format = "text",
      outputs = [
        {
          destination = "stdout",
          level = "debug",
        },
      ],
    },
  },
}
```

### Output: TOML Configuration

```toml
# To: provisioning/platform/config/orchestrator.solo.toml
[orchestrator.workspace]
name = "dev-workspace"
path = "/home/developer/provisioning/data/orchestrator"
enabled = true
multi_workspace = false

[orchestrator.server]
host = "127.0.0.1"
port = 9090
workers = 2
keep_alive = 75
max_connections = 128

[orchestrator.storage]
backend = "filesystem"
path = "/home/developer/provisioning/data/orchestrator"

[orchestrator.queue]
max_concurrent_tasks = 3
retry_attempts = 2
retry_delay = 1000
task_timeout = 1800000

[orchestrator.monitoring]
enabled = true

[orchestrator.monitoring.metrics]
enabled = false

[orchestrator.monitoring.health_check]
enabled = true
interval = 60

[orchestrator.logging]
level = "debug"
format = "text"

[[orchestrator.logging.outputs]]
destination = "stdout"
level = "debug"
```

### Output Location

```bash
provisioning/platform/config/
├── orchestrator.solo.toml          # Exported from configs/orchestrator.solo.ncl
├── orchestrator.multiuser.toml     # Exported from configs/orchestrator.multiuser.ncl
├── orchestrator.cicd.toml          # Exported from configs/orchestrator.cicd.ncl
├── orchestrator.enterprise.toml    # Exported from configs/orchestrator.enterprise.ncl
├── control-center.solo.toml        # Similar structure for each service
├── control-center.multiuser.toml
├── mcp-server.solo.toml
└── mcp-server.enterprise.toml
```

### Validation During Export

The `generate-configs.nu` script:

1. **Typechecks** - Ensures Nickel is syntactically valid
2. **Evaluates** - Computes final values
3. **Exports** - Converts to TOML format
4. **Saves** - Writes to `provisioning/platform/config/`

---

## Stage 4: Runtime (Rust Services)

### Purpose

Load TOML configuration and start Rust services with validated settings.

### Configuration Loading Hierarchy

Rust services load configuration in this priority order:

#### 1. Runtime Arguments (Highest Priority)

```bash
ORCHESTRATOR_CONFIG=/path/to/config.toml cargo run --bin orchestrator
```

#### 2. Environment Variables

```bash
# Environment variable overrides specific TOML values
export ORCHESTRATOR_SERVER_PORT=9999
export ORCHESTRATOR_LOG_LEVEL=debug

ORCHESTRATOR_CONFIG=orchestrator.solo.toml cargo run --bin orchestrator
```

Environment variable format: `ORCHESTRATOR_{SECTION}_{KEY}=value`

Example mappings:
- `ORCHESTRATOR_SERVER_PORT=9999` → `orchestrator.server.port = 9999`
- `ORCHESTRATOR_LOG_LEVEL=debug` → `orchestrator.logging.level = "debug"`
- `ORCHESTRATOR_QUEUE_MAX_CONCURRENT_TASKS=10` → `orchestrator.queue.max_concurrent_tasks = 10`

#### 3. TOML Configuration File

```toml
# Load from TOML (medium priority)
ORCHESTRATOR_CONFIG=orchestrator.solo.toml cargo run --bin orchestrator
```

#### 4. Compiled Defaults (Lowest Priority)

```bash
// In Rust code - fallback for unspecified values
let config = Config::from_file(config_path)
    .unwrap_or_else(|_| Config::default());
```

### Example: Solo Mode Startup

```bash
# Step 1: User generates config through TypeDialog
nu scripts/configure.nu orchestrator solo --backend web

# Step 2: Export to TOML
nu scripts/generate-configs.nu orchestrator solo

# Step 3: Set environment variables for environment-specific overrides
export ORCHESTRATOR_SERVER_PORT=9090
export ORCHESTRATOR_LOG_LEVEL=debug

# Step 4: Start the Rust service
ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator
```

### Rust Service Configuration Loading

```rust
// In orchestrator/src/config.rs

use config::{Config, ConfigError, Environment, File};
use serde::Deserialize;

#[derive(Debug, Deserialize)]
pub struct OrchestratorConfig {
    pub orchestrator: OrchestratorService,
}

#[derive(Debug, Deserialize)]
pub struct OrchestratorService {
    pub workspace: Workspace,
    pub server: Server,
    pub storage: Storage,
    pub queue: Queue,
}

impl OrchestratorConfig {
    pub fn load(config_path: Option<&str>) -> Result<Self, ConfigError> {
        let mut builder = Config::builder();

        // 1. Load TOML file if provided
        if let Some(path) = config_path {
            builder = builder.add_source(File::from(Path::new(path)));
        } else {
            // Fallback to defaults
            builder = builder.add_source(File::with_name("config/orchestrator.defaults.toml"));
        }

        // 2. Apply environment variable overrides
        builder = builder.add_source(
            Environment::with_prefix("ORCHESTRATOR")
                .separator("_")
        );

        let config = builder.build()?;
        config.try_deserialize()
    }
}
```

### Configuration Validation in Rust

```rust
impl OrchestratorConfig {
    pub fn validate(&self) -> Result<(), ConfigError> {
        // Validate server configuration
        if self.orchestrator.server.port < 1024 || self.orchestrator.server.port > 65535 {
            return Err(ConfigError::Message(
                "Server port must be between 1024 and 65535".to_string()
            ));
        }

        // Validate queue configuration
        if self.orchestrator.queue.max_concurrent_tasks == 0 {
            return Err(ConfigError::Message(
                "max_concurrent_tasks must be > 0".to_string()
            ));
        }

        // Validate storage configuration
        match self.orchestrator.storage.backend.as_str() {
            "filesystem" | "surrealdb" | "rocksdb" => {
                // Valid backend
            },
            backend => {
                return Err(ConfigError::Message(
                    format!("Unknown storage backend: {}", backend)
                ));
            }
        }

        Ok(())
    }
}
```

### Runtime Startup Sequence

```bash
#[tokio::main]
async fn main() -> Result<()> {
    // Load configuration
    let config = OrchestratorConfig::load(
        std::env::var("ORCHESTRATOR_CONFIG").ok().as_deref()
    )?;

    // Validate configuration
    config.validate()?;

    // Initialize logging
    init_logging(&config.orchestrator.logging)?;

    // Start HTTP server
    let server = Server::new(
        config.orchestrator.server.host.clone(),
        config.orchestrator.server.port,
    );

    // Initialize storage backend
    let storage = Storage::new(&config.orchestrator.storage)?;

    // Start the service
    server.start(storage).await?;

    Ok(())
}
```

---

## Complete Example: Solo Mode End-to-End

### Step 1: Interactive Configuration

```toml
$ nu scripts/configure.nu orchestrator solo --backend web

# TypeDialog launches web interface
# User fills in form:
# - Workspace name: "dev-workspace"
# - Server host: "127.0.0.1"
# - Server port: 9090
# - Storage backend: "filesystem"
# - Storage path: "/home/developer/provisioning/data/orchestrator"
# - Max concurrent tasks: 3
# - Log level: "debug"

# Saves to: values/orchestrator.solo.ncl
```

### Step 2: Generated Nickel Configuration

```nickel
# values/orchestrator.solo.ncl
{
  orchestrator = {
    workspace = {
      name = "dev-workspace",
      path = "/home/developer/provisioning/data/orchestrator",
      enabled = true,
      multi_workspace = false,
    },
    server = {
      host = "127.0.0.1",
      port = 9090,
      workers = 2,
      keep_alive = 75,
      max_connections = 128,
    },
    storage = {
      backend = "filesystem",
      path = "/home/developer/provisioning/data/orchestrator",
    },
    queue = {
      max_concurrent_tasks = 3,
      retry_attempts = 2,
      retry_delay = 1000,
      task_timeout = 1800000,
    },
    logging = {
      level = "debug",
      format = "text",
      outputs = [{
        destination = "stdout",
        level = "debug",
      }],
    },
  },
}
```

### Step 3: Composition and Validation

```bash
$ nickel typecheck provisioning/.typedialog/provisioning/platform/configs/orchestrator.solo.ncl

# Validation passes:
# - Workspace name: valid string ✓
# - Port 9090: within range 1024-65535 ✓
# - Max concurrent tasks 3: within range 1-100 ✓
# - Log level: recognized level ✓
```

### Step 4: Export to TOML

```toml
$ nu scripts/generate-configs.nu orchestrator solo

# Generates: provisioning/platform/config/orchestrator.solo.toml
```

### Step 5: TOML File Created

```toml
# provisioning/platform/config/orchestrator.solo.toml
[orchestrator.workspace]
name = "dev-workspace"
path = "/home/developer/provisioning/data/orchestrator"
enabled = true
multi_workspace = false

[orchestrator.server]
host = "127.0.0.1"
port = 9090
workers = 2
keep_alive = 75
max_connections = 128

[orchestrator.storage]
backend = "filesystem"
path = "/home/developer/provisioning/data/orchestrator"

[orchestrator.queue]
max_concurrent_tasks = 3
retry_attempts = 2
retry_delay = 1000
task_timeout = 1800000

[orchestrator.logging]
level = "debug"
format = "text"

[[orchestrator.logging.outputs]]
destination = "stdout"
level = "debug"
```

### Step 6: Runtime Startup

```bash
$ export ORCHESTRATOR_LOG_LEVEL=debug
$ ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator

# Service loads orchestrator.solo.toml
# Environment variable overrides ORCHESTRATOR_LOG_LEVEL to "debug"
# Service starts and begins accepting requests on 127.0.0.1:9090
```

---

## Configuration Modification Workflow

### Scenario: User Wants to Change Port

#### Option A: Modify TypeDialog Form and Regenerate

```bash
# 1. Re-run interactive configuration
nu scripts/configure.nu orchestrator solo --backend web

# 2. User changes port to 9999 in form
# 3. TypeDialog generates new values/orchestrator.solo.ncl

# 4. Export updated config
nu scripts/generate-configs.nu orchestrator solo

# 5. New TOML created with port: 9999
# 6. Restart service
ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator
```

#### Option B: Direct TOML Edit

```toml
# 1. Edit TOML directly
vi provisioning/platform/config/orchestrator.solo.toml
# Change: port = 9999

# 2. Restart service (no Nickel re-export needed)
ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator
```

#### Option C: Environment Variable Override

```bash
# 1. No file changes needed
# 2. Just override environment variable
export ORCHESTRATOR_SERVER_PORT=9999

# 3. Restart service
ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator
```

---

## Architecture Relationships

### Component Interactions

```bash
TypeDialog Forms              Nickel Schemas
(forms/*.toml)    ←shares→   (schemas/*.ncl)
       │                            │
       │ user input                 │ type definitions
       │                            │
       ▼                            ▼
values/*.ncl  ←─ constraint validation ─→  constraints.toml
       │                                      (single source of truth)
       │                                            │
       │                                            │
       ├──→ imported into composition ────────────┤
       │     (configs/*.ncl)                       │
       │                                            │
       │     base defaults ───→ defaults/*.ncl     │
       │     mode overlay ─────→ deployment/*.ncl  │
       │     validators ──────→ validators/*.ncl   │
       │                                            │
       └──→ typecheck + export ──────────────→─────┘
             nickel export --format toml
             │
             ▼
      provisioning/platform/config/
      *.toml files
             │
             │ loaded by Rust services
             │ at runtime
             ▼
      Running Service
      (orchestrator, control-center, mcp-server)
```

---

## Best Practices

### 1. Always Validate Before Deploying

```bash
# Typecheck Nickel before export
nickel typecheck provisioning/.typedialog/provisioning/platform/configs/orchestrator.solo.ncl

# Validate TOML before loading in Rust
cargo run --bin orchestrator -- --validate-config orchestrator.solo.toml
```

### 2. Use Version Control for TOML Configs

```toml
# Commit generated TOML files
git add provisioning/platform/config/orchestrator.solo.toml
git commit -m "Update orchestrator solo configuration"

# But NOT the values/*.ncl files
echo "values/*.ncl" >> provisioning/.typedialog/provisioning/platform/.gitignore
```

### 3. Document Configuration Changes

```toml
# In TypeDialog form, add comments
[[items]]
name = "max_concurrent_tasks"
type = "number"
prompt = "Max concurrent tasks (3 for dev, 50+ for production)"
help = "Increased from 3 to 10 for higher throughput testing"
```

### 4. Environment Variables for Sensitive Data

Never hardcode secrets in TOML:

```toml
# Instead of:
# [orchestrator.security]
# jwt_secret = "hardcoded-secret"

# Use environment variable:
export ORCHESTRATOR_SECURITY_JWT_SECRET="actual-secret"

# TOML can reference it:
# [orchestrator.security]
# jwt_secret = "${JWT_SECRET}"
```

### 5. Test Configuration Changes in Staging First

```toml
# Generate staging config
nu scripts/configure.nu orchestrator multiuser --backend web

# Export to staging TOML
nu scripts/generate-configs.nu orchestrator multiuser

# Test in staging environment
ORCHESTRATOR_CONFIG=orchestrator.multiuser.toml cargo run --bin orchestrator
# Monitor logs and verify behavior

# Then deploy to production
```

---

## Summary

The four-stage workflow provides:

1. **User-Friendly Interface**: TypeDialog forms with real-time validation
2. **Type Safety**: Nickel schemas and validators catch configuration errors early
3. **Flexibility**: TOML format can be edited manually or generated programmatically
4. **Runtime Configurability**: Environment variables allow deployment-time overrides
5. **Single Source of Truth**: Constraints, schemas, and validators all reference shared definitions

This layered approach ensures that:
- Invalid configurations are caught before deployment
- Users can modify configuration safely
- Different deployment modes have appropriate defaults
- Configuration changes can be version-controlled
- Services can be reconfigured without code changes