Jesús Pérez a17595aaaf fix(docs): fix file endings and formatting in documentation

2026-01-14 04:56:20 +00:00

25 KiB

Raw Blame History

Configuration Workflow: TypeDialog → Nickel → TOML → Rust

Complete documentation of the configuration pipeline that transforms interactive user input into production Rust service configurations.

Overview

The provisioning platform uses a four-stage configuration workflow that leverages TypeDialog for interactive configuration, Nickel for type-safe composition, and TOML for service consumption:

┌─────────────────────────────────────────────────────────────────┐
│ Stage 1: User Interaction (TypeDialog)                          │
│  - Can use Nickel configuration as default values               │
│    if use provisioning/platform/config/ it will be updated      │
│  - Interactive form (web/tui/cli)                               │
│  - Real-time constraint validation                              │
│  - Generates Nickel configuration                               │
└────────────────┬────────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────────┐
│ Stage 2: Composition (Nickel)                                   │
│  - Base defaults imported                                       │
│  - Mode overlay applied                                         │
│  - Validators enforce business rules                            │
│  - Produces Nickel config file                                  │
└────────────────┬────────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────────┐
│ Stage 3: Export (Nickel → TOML)                                 │
│  - Nickel config evaluated                                      │
│  - Exported to TOML format                                      │
│  - Saved to provisioning/platform/config/                       │
└────────────────┬────────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────────┐
│ Stage 4: Runtime (Rust Services)                                │
│  - Services load TOML configuration                             │
│  - Environment variables override specific values               │
│  - Start services with final configuration                      │
└─────────────────────────────────────────────────────────────────┘

Stage 1: User Interaction (TypeDialog)

Purpose

Collect configuration from users through an interactive, constraint-aware interface.

Workflow

# Launch interactive configuration wizard
nu scripts/configure.nu orchestrator solo --backend web

What Happens

Form Loads
- TypeDialog reads forms/orchestrator-form.toml
- Form displays configuration sections
- Constraints from constraints.toml enforce min/max values
- Environment variables populate initial defaults
User Interaction
- User fills in form fields (workspace name, server port, etc.)
- Real-time validation on each field
- Constraint interpolation shows valid ranges:
  - ${constraint.orchestrator.workers.min} → 1
  - ${constraint.orchestrator.workers.max} → 32
Configuration Submission
- User submits form
- TypeDialog validates all fields against schemas
- Generates Nickel configuration output
Output Generation
- Nickel config saved to values/{service}.{mode}.ncl
- Example: values/orchestrator.solo.ncl
- File becomes source of truth for user customizations

Form Structure Example

# forms/orchestrator-form.toml
name = "orchestrator_configuration"
description = "Configure orchestrator service"

[[items]]
name = "workspace_group"
type = "group"
includes = ["fragments/workspace-section.toml"]

[[items]]
name = "server_group"
type = "group"
includes = ["fragments/server-section.toml"]

[[items]]
name = "queue_group"
type = "group"
includes = ["fragments/orchestrator/queue-section.toml"]

Fragment with Constraint Interpolation

# forms/fragments/orchestrator/queue-section.toml
[[elements]]
name = "max_concurrent_tasks"
type = "number"
prompt = "Maximum Concurrent Tasks"
default = 5
min = "${constraint.orchestrator.queue.concurrent_tasks.min}"
max = "${constraint.orchestrator.queue.concurrent_tasks.max}"
required = true
help = "Range: ${constraint.orchestrator.queue.concurrent_tasks.min}-${constraint.orchestrator.queue.concurrent_tasks.max}"
nickel_path = ["orchestrator", "queue", "max_concurrent_tasks"]

Generated Nickel Output (from TypeDialog)

TypeDialog's nickel-roundtrip pattern generates:

# values/orchestrator.solo.ncl
# Auto-generated by TypeDialog
{
  orchestrator = {
    workspace = {
      name = "dev-workspace",
      path = "/home/developer/provisioning/data/orchestrator",
      enabled = true,
    },
    server = {
      host = "127.0.0.1",
      port = 9090,
      workers = 2,
    },
    queue = {
      max_concurrent_tasks = 3,
      retry_attempts = 2,
      retry_delay = 1000,
    },
  },
}

Stage 2: Composition (Nickel)

Purpose

Compose the user input with defaults, validators, and schemas to create a complete, validated configuration.

Workflow

# The nickel typecheck command validates the composition
nickel typecheck values/orchestrator.solo.ncl

Composition Layers

The final configuration is built by merging layers in priority order:

Layer 1: Schema Import

# Ensures type safety and required fields
let schemas = import "../schemas/orchestrator.ncl" in

Layer 2: Base Defaults

# Default values for all orchestrator configurations
let defaults = import "../defaults/orchestrator-defaults.ncl" in

Layer 3: Mode Overlay

# Solo-specific overrides and adjustments
let solo_defaults = import "../defaults/deployment/solo-defaults.ncl" in

Layer 4: Validators Import

# Business rule validation (ranges, uniqueness, dependencies)
let validators = import "../validators/orchestrator-validator.ncl" in

Layer 5: User Values

# User input from TypeDialog (values/orchestrator.solo.ncl)
# Loaded and merged with defaults

Composition Example

# configs/orchestrator.solo.ncl (generated composition)

let schemas = import "../schemas/orchestrator.ncl" in
let defaults = import "../defaults/orchestrator-defaults.ncl" in
let solo_defaults = import "../defaults/deployment/solo-defaults.ncl" in
let validators = import "../validators/orchestrator-validator.ncl" in

# Composition: Base defaults + mode overlay + user input
{
  orchestrator = defaults.orchestrator & {
    # User input from TypeDialog values/orchestrator.solo.ncl
    workspace = {
      name = "dev-workspace",
      path = "/home/developer/provisioning/data/orchestrator",
    },

    # Solo mode overrides
    server = {
      workers = validators.ValidWorkers 2,
      max_connections = 128,
    },

    queue = {
      max_concurrent_tasks = validators.ValidConcurrentTasks 3,
    },

    # Fallback to defaults for unspecified fields
  },
} | schemas.OrchestratorConfig  # Validate against schema

Validation During Composition

Each field is validated through multiple validation layers:

# validators/orchestrator-validator.ncl
let constraints = import "../constraints/constraints.toml" in

{
  # Validate workers within allowed range
  ValidWorkers = fun workers =>
    if workers < constraints.orchestrator.workers.min then
      error "Workers below minimum"
    else if workers > constraints.orchestrator.workers.max then
      error "Workers above maximum"
    else
      workers,

  # Validate concurrent tasks
  ValidConcurrentTasks = fun tasks =>
    if tasks < constraints.orchestrator.queue.concurrent_tasks.min then
      error "Tasks below minimum"
    else if tasks > constraints.orchestrator.queue.concurrent_tasks.max then
      error "Tasks above maximum"
    else
      tasks,
}

Constraints: Single Source of Truth

# constraints/constraints.toml
[orchestrator.workers]
min = 1
max = 32

[orchestrator.queue.concurrent_tasks]
min = 1
max = 100

[common.server.port]
min = 1024
max = 65535

These values are referenced in:

Form constraints (constraint interpolation)
Validators (ValidWorkers, ValidConcurrentTasks)
Default values (appropriate for each mode)

Stage 3: Export (Nickel → TOML)

Purpose

Convert validated Nickel configuration to TOML format for consumption by Rust services.

Workflow

# Export Nickel to TOML
nu scripts/generate-configs.nu orchestrator solo

Command Chain

# What happens internally:

# 1. Typecheck the Nickel config (catch errors early)
nickel typecheck provisioning/.typedialog/provisioning/platform/configs/orchestrator.solo.ncl

# 2. Export to TOML format
nickel export --format toml provisioning/.typedialog/provisioning/platform/configs/orchestrator.solo.ncl

# 3. Save to output location
# → provisioning/platform/config/orchestrator.solo.toml

Input: Nickel Configuration

# From: configs/orchestrator.solo.ncl
{
  orchestrator = {
    workspace = {
      name = "dev-workspace",
      path = "/home/developer/provisioning/data/orchestrator",
      enabled = true,
      multi_workspace = false,
    },
    server = {
      host = "127.0.0.1",
      port = 9090,
      workers = 2,
      keep_alive = 75,
      max_connections = 128,
    },
    storage = {
      backend = "filesystem",
      path = "/home/developer/provisioning/data/orchestrator",
    },
    queue = {
      max_concurrent_tasks = 3,
      retry_attempts = 2,
      retry_delay = 1000,
      task_timeout = 1800000,
    },
    monitoring = {
      enabled = true,
      metrics = {
        enabled = false,
      },
      health_check = {
        enabled = true,
        interval = 60,
      },
    },
    logging = {
      level = "debug",
      format = "text",
      outputs = [
        {
          destination = "stdout",
          level = "debug",
        },
      ],
    },
  },
}

Output: TOML Configuration

# To: provisioning/platform/config/orchestrator.solo.toml
[orchestrator.workspace]
name = "dev-workspace"
path = "/home/developer/provisioning/data/orchestrator"
enabled = true
multi_workspace = false

[orchestrator.server]
host = "127.0.0.1"
port = 9090
workers = 2
keep_alive = 75
max_connections = 128

[orchestrator.storage]
backend = "filesystem"
path = "/home/developer/provisioning/data/orchestrator"

[orchestrator.queue]
max_concurrent_tasks = 3
retry_attempts = 2
retry_delay = 1000
task_timeout = 1800000

[orchestrator.monitoring]
enabled = true

[orchestrator.monitoring.metrics]
enabled = false

[orchestrator.monitoring.health_check]
enabled = true
interval = 60

[orchestrator.logging]
level = "debug"
format = "text"

[[orchestrator.logging.outputs]]
destination = "stdout"
level = "debug"

Output Location

provisioning/platform/config/
├── orchestrator.solo.toml          # Exported from configs/orchestrator.solo.ncl
├── orchestrator.multiuser.toml     # Exported from configs/orchestrator.multiuser.ncl
├── orchestrator.cicd.toml          # Exported from configs/orchestrator.cicd.ncl
├── orchestrator.enterprise.toml    # Exported from configs/orchestrator.enterprise.ncl
├── control-center.solo.toml        # Similar structure for each service
├── control-center.multiuser.toml
├── mcp-server.solo.toml
└── mcp-server.enterprise.toml

Validation During Export

The generate-configs.nu script:

Typechecks - Ensures Nickel is syntactically valid
Evaluates - Computes final values
Exports - Converts to TOML format
Saves - Writes to provisioning/platform/config/

Stage 4: Runtime (Rust Services)

Purpose

Load TOML configuration and start Rust services with validated settings.

Configuration Loading Hierarchy

Rust services load configuration in this priority order:

1. Runtime Arguments (Highest Priority)

ORCHESTRATOR_CONFIG=/path/to/config.toml cargo run --bin orchestrator

2. Environment Variables

# Environment variable overrides specific TOML values
export ORCHESTRATOR_SERVER_PORT=9999
export ORCHESTRATOR_LOG_LEVEL=debug

ORCHESTRATOR_CONFIG=orchestrator.solo.toml cargo run --bin orchestrator

Environment variable format: ORCHESTRATOR_{SECTION}_{KEY}=value

Example mappings:

ORCHESTRATOR_SERVER_PORT=9999 → orchestrator.server.port = 9999
ORCHESTRATOR_LOG_LEVEL=debug → orchestrator.logging.level = "debug"
ORCHESTRATOR_QUEUE_MAX_CONCURRENT_TASKS=10 → orchestrator.queue.max_concurrent_tasks = 10

3. TOML Configuration File

# Load from TOML (medium priority)
ORCHESTRATOR_CONFIG=orchestrator.solo.toml cargo run --bin orchestrator

4. Compiled Defaults (Lowest Priority)

// In Rust code - fallback for unspecified values
let config = Config::from_file(config_path)
    .unwrap_or_else(|_| Config::default());

Example: Solo Mode Startup

# Step 1: User generates config through TypeDialog
nu scripts/configure.nu orchestrator solo --backend web

# Step 2: Export to TOML
nu scripts/generate-configs.nu orchestrator solo

# Step 3: Set environment variables for environment-specific overrides
export ORCHESTRATOR_SERVER_PORT=9090
export ORCHESTRATOR_LOG_LEVEL=debug

# Step 4: Start the Rust service
ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator

Rust Service Configuration Loading

// In orchestrator/src/config.rs

use config::{Config, ConfigError, Environment, File};
use serde::Deserialize;

#[derive(Debug, Deserialize)]
pub struct OrchestratorConfig {
    pub orchestrator: OrchestratorService,
}

#[derive(Debug, Deserialize)]
pub struct OrchestratorService {
    pub workspace: Workspace,
    pub server: Server,
    pub storage: Storage,
    pub queue: Queue,
}

impl OrchestratorConfig {
    pub fn load(config_path: Option<&str>) -> Result<Self, ConfigError> {
        let mut builder = Config::builder();

        // 1. Load TOML file if provided
        if let Some(path) = config_path {
            builder = builder.add_source(File::from(Path::new(path)));
        } else {
            // Fallback to defaults
            builder = builder.add_source(File::with_name("config/orchestrator.defaults.toml"));
        }

        // 2. Apply environment variable overrides
        builder = builder.add_source(
            Environment::with_prefix("ORCHESTRATOR")
                .separator("_")
        );

        let config = builder.build()?;
        config.try_deserialize()
    }
}

Configuration Validation in Rust

impl OrchestratorConfig {
    pub fn validate(&self) -> Result<(), ConfigError> {
        // Validate server configuration
        if self.orchestrator.server.port < 1024 || self.orchestrator.server.port > 65535 {
            return Err(ConfigError::Message(
                "Server port must be between 1024 and 65535".to_string()
            ));
        }

        // Validate queue configuration
        if self.orchestrator.queue.max_concurrent_tasks == 0 {
            return Err(ConfigError::Message(
                "max_concurrent_tasks must be > 0".to_string()
            ));
        }

        // Validate storage configuration
        match self.orchestrator.storage.backend.as_str() {
            "filesystem" | "surrealdb" | "rocksdb" => {
                // Valid backend
            },
            backend => {
                return Err(ConfigError::Message(
                    format!("Unknown storage backend: {}", backend)
                ));
            }
        }

        Ok(())
    }
}

Runtime Startup Sequence

#[tokio::main]
async fn main() -> Result<()> {
    // Load configuration
    let config = OrchestratorConfig::load(
        std::env::var("ORCHESTRATOR_CONFIG").ok().as_deref()
    )?;

    // Validate configuration
    config.validate()?;

    // Initialize logging
    init_logging(&config.orchestrator.logging)?;

    // Start HTTP server
    let server = Server::new(
        config.orchestrator.server.host.clone(),
        config.orchestrator.server.port,
    );

    // Initialize storage backend
    let storage = Storage::new(&config.orchestrator.storage)?;

    // Start the service
    server.start(storage).await?;

    Ok(())
}

Complete Example: Solo Mode End-to-End

Step 1: Interactive Configuration

$ nu scripts/configure.nu orchestrator solo --backend web

# TypeDialog launches web interface
# User fills in form:
# - Workspace name: "dev-workspace"
# - Server host: "127.0.0.1"
# - Server port: 9090
# - Storage backend: "filesystem"
# - Storage path: "/home/developer/provisioning/data/orchestrator"
# - Max concurrent tasks: 3
# - Log level: "debug"

# Saves to: values/orchestrator.solo.ncl

Step 2: Generated Nickel Configuration

# values/orchestrator.solo.ncl
{
  orchestrator = {
    workspace = {
      name = "dev-workspace",
      path = "/home/developer/provisioning/data/orchestrator",
      enabled = true,
      multi_workspace = false,
    },
    server = {
      host = "127.0.0.1",
      port = 9090,
      workers = 2,
      keep_alive = 75,
      max_connections = 128,
    },
    storage = {
      backend = "filesystem",
      path = "/home/developer/provisioning/data/orchestrator",
    },
    queue = {
      max_concurrent_tasks = 3,
      retry_attempts = 2,
      retry_delay = 1000,
      task_timeout = 1800000,
    },
    logging = {
      level = "debug",
      format = "text",
      outputs = [{
        destination = "stdout",
        level = "debug",
      }],
    },
  },
}

Step 3: Composition and Validation

$ nickel typecheck provisioning/.typedialog/provisioning/platform/configs/orchestrator.solo.ncl

# Validation passes:
# - Workspace name: valid string ✓
# - Port 9090: within range 1024-65535 ✓
# - Max concurrent tasks 3: within range 1-100 ✓
# - Log level: recognized level ✓

Step 4: Export to TOML

$ nu scripts/generate-configs.nu orchestrator solo

# Generates: provisioning/platform/config/orchestrator.solo.toml

Step 5: TOML File Created

# provisioning/platform/config/orchestrator.solo.toml
[orchestrator.workspace]
name = "dev-workspace"
path = "/home/developer/provisioning/data/orchestrator"
enabled = true
multi_workspace = false

[orchestrator.server]
host = "127.0.0.1"
port = 9090
workers = 2
keep_alive = 75
max_connections = 128

[orchestrator.storage]
backend = "filesystem"
path = "/home/developer/provisioning/data/orchestrator"

[orchestrator.queue]
max_concurrent_tasks = 3
retry_attempts = 2
retry_delay = 1000
task_timeout = 1800000

[orchestrator.logging]
level = "debug"
format = "text"

[[orchestrator.logging.outputs]]
destination = "stdout"
level = "debug"

Step 6: Runtime Startup

$ export ORCHESTRATOR_LOG_LEVEL=debug
$ ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator

# Service loads orchestrator.solo.toml
# Environment variable overrides ORCHESTRATOR_LOG_LEVEL to "debug"
# Service starts and begins accepting requests on 127.0.0.1:9090

Configuration Modification Workflow

Scenario: User Wants to Change Port

Option A: Modify TypeDialog Form and Regenerate

# 1. Re-run interactive configuration
nu scripts/configure.nu orchestrator solo --backend web

# 2. User changes port to 9999 in form
# 3. TypeDialog generates new values/orchestrator.solo.ncl

# 4. Export updated config
nu scripts/generate-configs.nu orchestrator solo

# 5. New TOML created with port: 9999
# 6. Restart service
ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator

Option B: Direct TOML Edit

# 1. Edit TOML directly
vi provisioning/platform/config/orchestrator.solo.toml
# Change: port = 9999

# 2. Restart service (no Nickel re-export needed)
ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator

Option C: Environment Variable Override

# 1. No file changes needed
# 2. Just override environment variable
export ORCHESTRATOR_SERVER_PORT=9999

# 3. Restart service
ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator

Architecture Relationships

Component Interactions

TypeDialog Forms              Nickel Schemas
(forms/*.toml)    ←shares→   (schemas/*.ncl)
       │                            │
       │ user input                 │ type definitions
       │                            │
       ▼                            ▼
values/*.ncl  ←─ constraint validation ─→  constraints.toml
       │                                      (single source of truth)
       │                                            │
       │                                            │
       ├──→ imported into composition ────────────┤
       │     (configs/*.ncl)                       │
       │                                            │
       │     base defaults ───→ defaults/*.ncl     │
       │     mode overlay ─────→ deployment/*.ncl  │
       │     validators ──────→ validators/*.ncl   │
       │                                            │
       └──→ typecheck + export ──────────────→─────┘
             nickel export --format toml
             │
             ▼
      provisioning/platform/config/
      *.toml files
             │
             │ loaded by Rust services
             │ at runtime
             ▼
      Running Service
      (orchestrator, control-center, mcp-server)

Best Practices

1. Always Validate Before Deploying

# Typecheck Nickel before export
nickel typecheck provisioning/.typedialog/provisioning/platform/configs/orchestrator.solo.ncl

# Validate TOML before loading in Rust
cargo run --bin orchestrator -- --validate-config orchestrator.solo.toml

2. Use Version Control for TOML Configs

# Commit generated TOML files
git add provisioning/platform/config/orchestrator.solo.toml
git commit -m "Update orchestrator solo configuration"

# But NOT the values/*.ncl files
echo "values/*.ncl" >> provisioning/.typedialog/provisioning/platform/.gitignore

3. Document Configuration Changes

# In TypeDialog form, add comments
[[items]]
name = "max_concurrent_tasks"
type = "number"
prompt = "Max concurrent tasks (3 for dev, 50+ for production)"
help = "Increased from 3 to 10 for higher throughput testing"

4. Environment Variables for Sensitive Data

Never hardcode secrets in TOML:

# Instead of:
# [orchestrator.security]
# jwt_secret = "hardcoded-secret"

# Use environment variable:
export ORCHESTRATOR_SECURITY_JWT_SECRET="actual-secret"

# TOML can reference it:
# [orchestrator.security]
# jwt_secret = "${JWT_SECRET}"

5. Test Configuration Changes in Staging First

# Generate staging config
nu scripts/configure.nu orchestrator multiuser --backend web

# Export to staging TOML
nu scripts/generate-configs.nu orchestrator multiuser

# Test in staging environment
ORCHESTRATOR_CONFIG=orchestrator.multiuser.toml cargo run --bin orchestrator
# Monitor logs and verify behavior

# Then deploy to production

Summary

The four-stage workflow provides:

User-Friendly Interface: TypeDialog forms with real-time validation
Type Safety: Nickel schemas and validators catch configuration errors early
Flexibility: TOML format can be edited manually or generated programmatically
Runtime Configurability: Environment variables allow deployment-time overrides
Single Source of Truth: Constraints, schemas, and validators all reference shared definitions

This layered approach ensures that:

Invalid configurations are caught before deployment
Users can modify configuration safely
Different deployment modes have appropriate defaults
Configuration changes can be version-controlled
Services can be reconfigured without code changes

25 KiB Raw Blame History

Configuration Workflow: TypeDialog → Nickel → TOML → Rust

Overview

Stage 1: User Interaction (TypeDialog)

Purpose

Workflow

What Happens

Form Structure Example

Fragment with Constraint Interpolation

Generated Nickel Output (from TypeDialog)

Stage 2: Composition (Nickel)

Purpose

Workflow

Composition Layers

Layer 1: Schema Import

Layer 2: Base Defaults

Layer 3: Mode Overlay

Layer 4: Validators Import

Layer 5: User Values

Composition Example

Validation During Composition

Constraints: Single Source of Truth

Stage 3: Export (Nickel → TOML)

Purpose

Workflow

Command Chain

Input: Nickel Configuration

Output: TOML Configuration

Output Location

Validation During Export

Stage 4: Runtime (Rust Services)

Purpose

Configuration Loading Hierarchy

1. Runtime Arguments (Highest Priority)

2. Environment Variables

3. TOML Configuration File

4. Compiled Defaults (Lowest Priority)

Example: Solo Mode Startup

Rust Service Configuration Loading

Configuration Validation in Rust

Runtime Startup Sequence

Complete Example: Solo Mode End-to-End

Step 1: Interactive Configuration

Step 2: Generated Nickel Configuration

Step 3: Composition and Validation

Step 4: Export to TOML

Step 5: TOML File Created

Step 6: Runtime Startup

Configuration Modification Workflow

Scenario: User Wants to Change Port

Option A: Modify TypeDialog Form and Regenerate

Option B: Direct TOML Edit

Option C: Environment Variable Override

Architecture Relationships

Component Interactions

Best Practices

1. Always Validate Before Deploying

2. Use Version Control for TOML Configs

3. Document Configuration Changes

4. Environment Variables for Sensitive Data

5. Test Configuration Changes in Staging First

Summary

25 KiB

Raw Blame History