diff --git a/schemas/infrastructure/README.md b/schemas/infrastructure/README.md index a8cb02c..8d6b7c3 100644 --- a/schemas/infrastructure/README.md +++ b/schemas/infrastructure/README.md @@ -61,7 +61,7 @@ nickel export --format toml provisioning/schemas/infrastructure/systemd.ncl Define service configuration once, generate multiple infrastructure outputs: -``` +```plaintext orchestrator.ncl (Platform Service Schema) ↓ Infrastructure Schemas (Docker, Kubernetes, Nginx, etc.) @@ -123,7 +123,7 @@ nickel export --format yaml provisioning/schemas/infrastructure/kubernetes.ncl \ ## File Structure -``` +```plaintext infrastructure/ ├── README.md # This file ├── docker-compose.ncl # Docker Compose schema (232 lines) @@ -273,6 +273,7 @@ oci_registry_config # Zot backend with filesystem storage - Other services: 0.25-0.5 CPU, 256-512M RAM **Export to JSON**: + ```bash nickel export --format json provisioning/schemas/infrastructure/examples-solo-deployment.ncl # Output: 198 lines of configuration @@ -298,6 +299,7 @@ oci_registry_config # Harbor backend with S3 replication - Services scale appropriately for production load **Export to JSON**: + ```bash nickel export --format json provisioning/schemas/infrastructure/examples-enterprise-deployment.ncl # Output: 313 lines of configuration @@ -306,7 +308,7 @@ nickel export --format json provisioning/schemas/infrastructure/examples-enterpr ### Example Comparison | Aspect | Solo | Enterprise | -|--------|------|-----------| +| -------- | ------ | ----------- | | **Services** | 5 | 6 | | **Orchestrator CPU** | 1.0 | 4.0 | | **Orchestrator Memory** | 1024M | 4096M | @@ -395,7 +397,7 @@ Infrastructure schemas are independent from platform config schemas: ConfigLoader automatically loads platform configs. Infrastructure configs are generated separately and deployed via infrastructure tools: -``` +```plaintext Platform Schema (Nickel) ↓ nickel export → TOML ↓ ConfigLoader → Service reads config diff --git a/schemas/platform/README.md b/schemas/platform/README.md index 4f4961f..264c418 100644 --- a/schemas/platform/README.md +++ b/schemas/platform/README.md @@ -1,6 +1,7 @@ # TypeDialog + Nickel Configuration System for Platform Services -Complete configuration system for provisioning platform services (orchestrator, control-center, mcp-server, vault-service, extension-registry, rag, ai-service, provisioning-daemon) across multiple deployment modes (solo, multiuser, cicd, enterprise). +Complete configuration system for provisioning platform services (orchestrator, control-center, mcp-server, vault-service, +extension-registry, rag, ai-service, provisioning-daemon) across multiple deployment modes (solo, multiuser, cicd, enterprise). ## Architecture Overview @@ -15,7 +16,7 @@ This system implements a **TypeDialog + Nickel configuration workflow** that pro ## Directory Structure -``` +```plaintext provisioning/.typedialog/provisioning/platform/ ├── constraints/ # Single source of truth for validation limits ├── schemas/ # Nickel type contracts (services + common + deployment modes) @@ -48,7 +49,7 @@ nu scripts/configure.nu orchestrator solo --backend web ### 2. Configuration Composition -``` +```plaintext Base Defaults (defaults/*.ncl) ↓ + Mode Overlay (defaults/deployment/{mode}-defaults.ncl) @@ -249,7 +250,7 @@ nu provisioning/.typedialog/provisioning/platform/scripts/install-services.nu so ## Configuration Loading Hierarchy (Rust Services) -``` +```plaintext 1. Environment variables (ORCHESTRATOR_*) 2. User config (values/{service}.{mode}.ncl → TOML) 3. Mode-specific defaults (configs/{service}.{mode}.toml) @@ -259,6 +260,7 @@ nu provisioning/.typedialog/provisioning/platform/scripts/install-services.nu so ## Constraint Interpolation Example **constraints.toml**: + ```toml [orchestrator.queue.concurrent_tasks] min = 1 @@ -266,6 +268,7 @@ max = 100 ``` **Form element** (fragments/orchestrator-queue-section.toml): + ```toml [[elements]] name = "max_concurrent_tasks" @@ -276,6 +279,7 @@ nickel_path = ["orchestrator", "queue", "max_concurrent_tasks"] ``` **Jinja2 template** (orchestrator-config.ncl.j2): + ```nickel orchestrator = { queue = { @@ -289,16 +293,19 @@ orchestrator = { ## Getting Started 1. **Run configuration wizard**: + ```bash nu provisioning/.typedialog/provisioning/platform/scripts/configure.nu orchestrator solo ``` 2. **Generate TOML configs**: + ```bash nu provisioning/.typedialog/provisioning/platform/scripts/generate-configs.nu orchestrator solo ``` 3. **Deploy services**: + ```bash nu provisioning/.typedialog/provisioning/platform/scripts/install-services.nu solo ``` @@ -318,7 +325,7 @@ orchestrator = { ## Key Files | File | Purpose | -|------|---------| +| ------ | --------- | | `constraints/constraints.toml` | Single source of truth for validation limits | | `schemas/orchestrator.ncl` | Orchestrator type schema | | `defaults/orchestrator-defaults.ncl` | Orchestrator default values | diff --git a/schemas/platform/configs/README.md b/schemas/platform/configs/README.md index 53314fb..946c744 100644 --- a/schemas/platform/configs/README.md +++ b/schemas/platform/configs/README.md @@ -13,7 +13,7 @@ Configurations are **automatically generated** by composing: ## File Organization -``` +```plaintext configs/ ├── README.md # This file ├── orchestrator.solo.ncl # Orchestrator solo mode @@ -56,6 +56,7 @@ let validators = import "../validators/orchestrator-validator.ncl" in ## Example Configuration ### Base Defaults + ```nickel # defaults/orchestrator-defaults.ncl orchestrator = { @@ -76,6 +77,7 @@ orchestrator = { ``` ### Solo Mode Override + ```nickel # defaults/deployment/solo-defaults.ncl services.orchestrator = { @@ -86,6 +88,7 @@ services.orchestrator = { ``` ### Generated Config + ```nickel # configs/orchestrator.solo.ncl (auto-generated) { @@ -119,6 +122,7 @@ services.orchestrator = { ### Regenerating Configs #### Via TypeDialog (Recommended) + ```bash nu provisioning/.typedialog/provisioning/platform/scripts/configure.nu orchestrator solo ``` @@ -130,6 +134,7 @@ Automatically: 4. Generates updated config #### Manual Rebuild + ```bash # (Future) Script to rebuild all configs from sources nu provisioning/.typedialog/provisioning/platform/scripts/generate-configs.nu orchestrator solo @@ -166,7 +171,7 @@ nu provisioning/.typedialog/provisioning/platform/scripts/generate-configs.nu or ## Configuration Values Hierarchy -``` +```plaintext 1. Explicit user customization (values/{service}.{mode}.ncl) 2. Mode-specific defaults (defaults/deployment/{mode}-defaults.ncl) 3. Service base defaults (defaults/{service}-defaults.ncl) @@ -179,12 +184,14 @@ Configurations are validated at three levels: ### 1. Schema Validation Type checking when config is evaluated: -``` + +```plaintext | schemas.OrchestratorConfig ``` ### 2. Constraint Validation Range checking via validators: + ```nickel max_concurrent_tasks = validators.ValidConcurrentTasks 5 ``` @@ -204,6 +211,7 @@ nu provisioning/.typedialog/provisioning/platform/scripts/generate-configs.nu or ``` Rust services load the TOML: + ```rust let config_path = "provisioning/platform/config/orchestrator.solo.toml"; let config = Config::from_file(config_path)?; diff --git a/schemas/platform/configuration-workflow.md b/schemas/platform/configuration-workflow.md index 011f631..5221060 100644 --- a/schemas/platform/configuration-workflow.md +++ b/schemas/platform/configuration-workflow.md @@ -4,9 +4,10 @@ Complete documentation of the configuration pipeline that transforms interactive ## Overview -The provisioning platform uses a **four-stage configuration workflow** that leverages TypeDialog for interactive configuration, Nickel for type-safe composition, and TOML for service consumption: +The provisioning platform uses a **four-stage configuration workflow** that leverages TypeDialog for interactive configuration, +Nickel for type-safe composition, and TOML for service consumption: -``` +```plaintext ┌─────────────────────────────────────────────────────────────────┐ │ Stage 1: User Interaction (TypeDialog) │ │ - Can use Nickel configuration as default values │ @@ -419,7 +420,7 @@ level = "debug" ### Output Location -``` +```plaintext provisioning/platform/config/ ├── orchestrator.solo.toml # Exported from configs/orchestrator.solo.ncl ├── orchestrator.multiuser.toml # Exported from configs/orchestrator.multiuser.ncl @@ -801,7 +802,7 @@ ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo ru ### Component Interactions -``` +```plaintext TypeDialog Forms Nickel Schemas (forms/*.toml) ←shares→ (schemas/*.ncl) │ │ diff --git a/schemas/platform/constraints/README.md b/schemas/platform/constraints/README.md index d66e06e..dec92cc 100644 --- a/schemas/platform/constraints/README.md +++ b/schemas/platform/constraints/README.md @@ -16,7 +16,7 @@ These constraints are used by: ## File Structure -``` +```plaintext constraints/ └── constraints.toml # All validation constraints in TOML format ``` @@ -26,6 +26,7 @@ constraints/ ### 1. Define Constraint **constraints.toml**: + ```toml [orchestrator.queue.concurrent_tasks] min = 1 @@ -35,6 +36,7 @@ max = 100 ### 2. Reference in Validator **validators/orchestrator-validator.ncl**: + ```nickel let constraints = import "../constraints/constraints.toml" in @@ -52,6 +54,7 @@ let constraints = import "../constraints/constraints.toml" in ### 3. Reference in Form **forms/fragments/orchestrator-queue-section.toml**: + ```toml [[elements]] name = "max_concurrent_tasks" @@ -107,6 +110,7 @@ When changing constraint bounds: ### Example: Increase Max Queue Tasks **Before**: + ```toml [orchestrator.queue.concurrent_tasks] min = 1 @@ -114,6 +118,7 @@ max = 100 ``` **After**: + ```toml [orchestrator.queue.concurrent_tasks] min = 1 diff --git a/schemas/platform/defaults/README.md b/schemas/platform/defaults/README.md index a52542d..74094cb 100644 --- a/schemas/platform/defaults/README.md +++ b/schemas/platform/defaults/README.md @@ -12,7 +12,7 @@ Defaults provide: ## File Organization -``` +```plaintext defaults/ ├── README.md # This file ├── common/ # Shared defaults @@ -36,7 +36,7 @@ defaults/ Configuration is built from layers: -``` +```plaintext Base Defaults (service-defaults.ncl) ↓ + Mode Overlay (deployment/{mode}-defaults.ncl) @@ -49,6 +49,7 @@ Base Defaults (service-defaults.ncl) ``` Example: + ```nickel # configs/orchestrator.solo.ncl let defaults = import "../defaults/orchestrator-defaults.ncl" in @@ -68,6 +69,7 @@ let solo_defaults = import "../defaults/deployment/solo-defaults.ncl" in ### 1. Service Base Defaults **orchestrator-defaults.ncl**: + ```nickel { orchestrator = { @@ -97,6 +99,7 @@ let solo_defaults = import "../defaults/deployment/solo-defaults.ncl" in ### 2. Mode-Specific Overrides **deployment/solo-defaults.ncl**: + ```nickel { resources = { @@ -114,6 +117,7 @@ let solo_defaults = import "../defaults/deployment/solo-defaults.ncl" in ``` **deployment/enterprise-defaults.ncl**: + ```nickel { resources = { @@ -134,6 +138,7 @@ let solo_defaults = import "../defaults/deployment/solo-defaults.ncl" in ## Common Defaults ### server-defaults.ncl + ```nickel { server = { @@ -147,6 +152,7 @@ let solo_defaults = import "../defaults/deployment/solo-defaults.ncl" in ``` ### database-defaults.ncl + ```nickel { database = { @@ -159,6 +165,7 @@ let solo_defaults = import "../defaults/deployment/solo-defaults.ncl" in ``` ### security-defaults.ncl + ```nickel { security = { @@ -172,6 +179,7 @@ let solo_defaults = import "../defaults/deployment/solo-defaults.ncl" in ``` ### monitoring-defaults.ncl + ```nickel { monitoring = { @@ -218,6 +226,7 @@ let solo_defaults = import "../defaults/deployment/solo-defaults.ncl" in ### Changing a Base Default **orchestrator-defaults.ncl**: + ```nickel # Before queue = { @@ -238,6 +247,7 @@ queue = { ### Changing Mode Override **deployment/solo-defaults.ncl**: + ```nickel # Before orchestrator = { diff --git a/schemas/platform/examples/README.md b/schemas/platform/examples/README.md index 808ffb5..cd8b2d7 100644 --- a/schemas/platform/examples/README.md +++ b/schemas/platform/examples/README.md @@ -68,6 +68,7 @@ nickel export --format toml full-platform-enterprise.ncl > platform.toml - No audit logging **Usage**: + ```bash # Export to TOML and run nickel export --format toml orchestrator-solo.ncl > orchestrator.solo.toml @@ -78,6 +79,7 @@ nu ../../scripts/configure.nu orchestrator solo --backend cli ``` **Customization Examples**: + ```nickel # Increase concurrency for testing (still development-friendly) queue.max_concurrent_tasks = 5 @@ -141,12 +143,14 @@ workspace.path = "/path/to/my/workspace" - Network isolation and load balancing setup required **Environment Variables Required**: + ```bash export JWT_SECRET="" export SURREALDB_PASSWORD="" ``` **Usage**: + ```bash # Deploy standalone with SurrealDB nickel export --format toml orchestrator-enterprise.ncl > orchestrator.enterprise.toml @@ -158,6 +162,7 @@ kubectl apply -f provisioning/platform/infrastructure/kubernetes/*.yaml ``` **Customization Examples**: + ```nickel # Adjust concurrency for your specific infrastructure queue.max_concurrent_tasks = 50 # Scale down if resource-constrained @@ -222,12 +227,14 @@ security.rate_limit.requests_per_second = 20000 - RBAC requires careful role definition **Environment Variables Required**: + ```bash export DB_PASSWORD="" export JWT_SECRET="" ``` **Usage**: + ```bash # Generate and deploy nickel export --format toml control-center-multiuser.ncl > control-center.multiuser.toml @@ -242,14 +249,16 @@ docker-compose -f docker-compose.multiuser.yml up -d ``` **RBAC Quick Reference**: + | Role | Intended Users | Key Permissions | -|------|----------------|-----------------| +| ------ | ---------------- | ----------------- | | admin | Infrastructure leads | All operations: full access | | operator | Operations engineers | Execute workflows, manage tasks, view policies | | developer | Application developers | View workflows, view policies (read-only) | | viewer | Non-technical (PM, QA) | View workflows only (minimal read) | **Customization Examples**: + ```nickel # Require MFA globally for higher security mfa.required = true @@ -312,6 +321,7 @@ policies.password = { - 100 RPS rate limiting with 20-request burst **Global Configuration**: + ```nickel let deployment_mode = "enterprise" let namespace = "provisioning" @@ -356,6 +366,7 @@ let environment = "production" - Network setup complexity (TLS, CORS, rate limiting) **Environment Variables Required**: + ```bash # Database credentials export DB_PASSWORD="" @@ -375,7 +386,8 @@ export TLS_KEY="" ``` **Architecture Diagram**: -``` + +```plaintext ┌───────────────────────────────────────────────┐ │ Nginx Load Balancer (TLS, CORS, RateLimit) │ │ https://orchestrator.example.com │ @@ -404,6 +416,7 @@ export TLS_KEY="" ``` **Usage**: + ```bash # Export complete configuration nickel export --format toml full-platform-enterprise.ncl > platform.toml @@ -424,6 +437,7 @@ docker-compose -f docker-compose.enterprise.yml up -d ``` **Customization Examples**: + ```nickel # Adjust deployment domain let domain = "my-company.com" @@ -450,7 +464,7 @@ mcp_server.performance.cache_ttl = 7200 ## Deployment Mode Comparison Matrix | Feature | Solo | MultiUser | Enterprise | -|---------|------|-----------|-----------| +| --------- | ------ | ----------- | ----------- | | **Ideal For** | Dev | Team/Staging | Production | | **Storage** | Filesystem | PostgreSQL | SurrealDB Cluster | | **Replicas** | 1 | 1 | 3+ (HA) | @@ -615,6 +629,7 @@ ORCHESTRATOR_CONFIG=orchestrator.solo.toml timeout 5 cargo run --bin orchestrato ### Port Already in Use **Fix**: Change the port in your configuration: + ```nickel server.port = 9999 # Instead of 9090 ``` @@ -630,6 +645,7 @@ server.port = 9999 # Instead of 9090 ### Import Not Found **Fix**: Ensure all relative paths in imports are correct: + ```nickel # Correct (relative to examples/) let defaults = import "../defaults/orchestrator-defaults.ncl" in @@ -696,11 +712,13 @@ nickel export --format toml orchestrator-custom.ncl > orchestrator.custom.toml ## Using Examples ### View Example + ```bash cat provisioning/.typedialog/provisioning/platform/examples/orchestrator-solo.ncl ``` ### Copy and Customize + ```bash # Start with solo example cp examples/orchestrator-solo.ncl values/orchestrator.solo.ncl @@ -713,6 +731,7 @@ nu scripts/validate-config.nu values/orchestrator.solo.ncl ``` ### Generate from Example + ```bash # Use example as base, regenerate with TypeDialog nu scripts/configure.nu orchestrator solo --backend web @@ -769,11 +788,13 @@ Each example is a complete Nickel configuration: ## Creating New Examples ### 1. Start with Existing Example + ```bash cp examples/orchestrator-solo.ncl examples/orchestrator-custom.ncl ``` ### 2. Modify for Your Use Case + ```toml # Update configuration values orchestrator.server.workers = 8 # More workers @@ -781,6 +802,7 @@ orchestrator.queue.max_concurrent_tasks = 20 # Higher concurrency ``` ### 3. Validate Configuration + ```bash nickel typecheck examples/orchestrator-custom.ncl nickel eval examples/orchestrator-custom.ncl @@ -794,6 +816,7 @@ Add comments explaining: - Customization needed ### 5. Save as Reference + ```bash mv examples/orchestrator-custom.ncl examples/orchestrator-{scenario}.ncl ``` diff --git a/schemas/platform/schemas/README.md b/schemas/platform/schemas/README.md index a81c72e..b0c5d1b 100644 --- a/schemas/platform/schemas/README.md +++ b/schemas/platform/schemas/README.md @@ -12,7 +12,7 @@ Schemas define: ## File Organization -``` +```plaintext schemas/ ├── README.md # This file ├── common/ # Shared schemas (server, database, security, etc.) @@ -138,6 +138,7 @@ Log configuration: ### orchestrator.ncl Workflow orchestration: + ```nickel OrchestratorConfig = { workspace | WorkspaceConfig, @@ -153,6 +154,7 @@ OrchestratorConfig = { ### control-center.ncl Policy and RBAC: + ```nickel ControlCenterConfig = { workspace | WorkspaceConfig, @@ -166,6 +168,7 @@ ControlCenterConfig = { ### mcp-server.ncl MCP protocol server: + ```nickel MCPServerConfig = { workspace | WorkspaceConfig, @@ -186,6 +189,7 @@ Deployment schemas define resource constraints for each mode: - **enterprise.ncl** - 16+ CPU, 32+ GB RAM, HA Example: + ```nickel # schemas/deployment/solo.ncl { diff --git a/schemas/platform/templates/README.md b/schemas/platform/templates/README.md index e5a3ece..05884c0 100644 --- a/schemas/platform/templates/README.md +++ b/schemas/platform/templates/README.md @@ -12,7 +12,7 @@ Templates provide: ## File Organization -``` +```plaintext templates/ ├── README.md # This file ├── orchestrator-config.ncl.j2 # Nickel output template (Jinja2) @@ -133,6 +133,7 @@ typedialog-web nickel-roundtrip "$CONFIG" "forms/{service}-form.toml" --output " ### Key Jinja2 Patterns **Conditional blocks** (only include if field is set): + ```nickel {%- if workspace_name %} name = "{{ workspace_name }}", @@ -140,6 +141,7 @@ name = "{{ workspace_name }}", ``` **String values** (with quotes): + ```nickel {%- if storage_backend %} backend = '{{ storage_backend }}, # Enum (atom syntax) @@ -147,6 +149,7 @@ backend = '{{ storage_backend }}, # Enum (atom syntax) ``` **Numeric values** (no quotes): + ```nickel {%- if server_port %} port = {{ server_port }}, # Number @@ -154,6 +157,7 @@ port = {{ server_port }}, # Number ``` **Boolean values** (lower case): + ```nickel {%- if workspace_enabled is defined %} enabled = {{ workspace_enabled | lower }}, # Boolean (true/false) @@ -161,6 +165,7 @@ enabled = {{ workspace_enabled | lower }}, # Boolean (true/false) ``` **Comments** (for generated files): + ```nickel # Auto-generated by provisioning TypeDialog # Edit via: nu scripts/configure.nu orchestrator {mode} @@ -295,16 +300,19 @@ let config = import "../../values/orchestrator.solo.ncl" in ## Rendering Templates ### Render to JSON + ```bash nickel export --format json templates/orchestrator-config.ncl.j2 > config.json ``` ### Render to YAML (via yq) + ```bash nickel export --format json templates/kubernetes/orchestrator-deployment.yaml.ncl | yq -P > deployment.yaml ``` ### Render to TOML + ```bash nickel export --format toml templates/configs/orchestrator-config.toml.ncl > config.toml ``` diff --git a/schemas/platform/templates/configs/README.md b/schemas/platform/templates/configs/README.md index 60c64de..0edaf15 100644 --- a/schemas/platform/templates/configs/README.md +++ b/schemas/platform/templates/configs/README.md @@ -158,6 +158,7 @@ max_connections = 512 ### Database Configuration (Control Center) **RocksDB** (solo, cicd modes): + ```toml [database] backend = "rocksdb" @@ -170,6 +171,7 @@ compression = "snappy" ``` **PostgreSQL** (multiuser, enterprise modes): + ```toml [database] backend = "postgres" @@ -186,6 +188,7 @@ ssl_mode = "require" ### Storage Configuration (Orchestrator) **Filesystem** (solo, cicd modes): + ```toml [storage] backend = "filesystem" @@ -193,6 +196,7 @@ path = "/var/lib/provisioning/orchestrator/data" ``` **SurrealDB Server** (multiuser mode): + ```toml [storage] backend = "surrealdb_server" @@ -202,6 +206,7 @@ surrealdb_database = "orchestrator" ``` **SurrealDB Cluster** (enterprise mode): + ```toml [storage] backend = "surrealdb_cluster" diff --git a/schemas/platform/templates/docker-compose/README.md b/schemas/platform/templates/docker-compose/README.md index 6e62713..d00ab2d 100644 --- a/schemas/platform/templates/docker-compose/README.md +++ b/schemas/platform/templates/docker-compose/README.md @@ -4,7 +4,8 @@ Nickel-based Docker Compose templates for deploying platform services across all ## Overview -This directory contains Nickel templates that generate Docker Compose files for different deployment scenarios. Each template imports configuration from `values/*.ncl` and expands to valid Docker Compose YAML. +This directory contains Nickel templates that generate Docker Compose files for different deployment scenarios. +Each template imports configuration from `values/*.ncl` and expands to valid Docker Compose YAML. **Key Pattern**: Templates use **Nickel composition** to build service definitions dynamically based on configuration, allowing parameterized infrastructure-as-code. @@ -36,6 +37,7 @@ This directory contains Nickel templates that generate Docker Compose files for - Restart Policy: `unless-stopped` (survives host reboot) **Usage**: + ```bash # Generate from Nickel template nickel export --format json platform-stack.solo.yml.ncl | yq -P > docker-compose.solo.yml @@ -51,6 +53,7 @@ docker-compose -f docker-compose.solo.yml down ``` **Environment Variables** (recommended in `.env` file): + ```bash ORCHESTRATOR_LOG_LEVEL=debug CONTROL_CENTER_LOG_LEVEL=info @@ -96,6 +99,7 @@ MCP_SERVER_LOG_LEVEL=info - Logging: JSON format with rotation **Usage**: + ```bash # Generate from Nickel template nickel export --format json platform-stack.multiuser.yml.ncl | yq -P > docker-compose.multiuser.yml @@ -118,6 +122,7 @@ docker-compose exec postgres psql -U postgres -c "CREATE DATABASE provisioning;" ``` **Database Initialization**: + ```bash # Connect to PostgreSQL for schema creation docker-compose exec postgres psql -U provisioning -d provisioning @@ -130,6 +135,7 @@ docker-compose exec surrealdb surreal sql --auth root:password ``` **Environment Variables** (in `.env.multiuser`): + ```bash # Database Credentials (CRITICAL - change before production) DB_PASSWORD=your-strong-password @@ -178,6 +184,7 @@ GITEA_ROOT_URL=http://localhost:3000/ - **Low Resource**: Minimal memory/CPU footprint **Usage**: + ```bash # Generate from Nickel template nickel export --format json platform-stack.cicd.yml.ncl | yq -P > docker-compose.cicd.yml @@ -197,6 +204,7 @@ docker-compose -f docker-compose.cicd.yml down -v ``` **CI/CD Integration Example**: + ```bash # GitHub Actions workflow - name: Start Provisioning Stack @@ -213,6 +221,7 @@ docker-compose -f docker-compose.cicd.yml down -v ``` **Environment Variables** (minimal): + ```bash # Logging (optional) ORCHESTRATOR_LOG_LEVEL=warn @@ -265,7 +274,8 @@ ORCHESTRATOR_LOG_LEVEL=warn - Logging: JSON format with 500MB files, kept 30 versions **Architecture**: -``` + +```plaintext ┌──────────────────────┐ │ External Client │ │ (HTTPS, Port 443) │ @@ -299,6 +309,7 @@ Observability Stack: ``` **Usage**: + ```bash # Generate from Nickel template nickel export --format json platform-stack.enterprise.yml.ncl | yq -P > docker-compose.enterprise.yml @@ -352,6 +363,7 @@ curl -H "Host: orchestrator.example.com" http://localhost/health - [ ] Configure external log aggregation (ELK stack, Splunk, etc.) **Environment Variables** (in `.env.enterprise`): + ```bash # Database Credentials (CRITICAL) DB_PASSWORD=your-strong-password-32-chars-min @@ -534,6 +546,7 @@ docker logs -f $(docker ps | grep orchestrator | awk '{print $1}') **Error**: `bind: address already in use` **Fix**: Change port in template or stop conflicting container: + ```bash # Find process using port lsof -i :9090 @@ -549,6 +562,7 @@ ports: ### Service Fails to Start **Check logs**: + ```bash docker-compose logs orchestrator ``` @@ -563,6 +577,7 @@ docker-compose logs orchestrator ### Persistent Volume Issues **Clean volumes** (WARNING: Deletes data): + ```bash docker-compose down -v docker volume prune -f diff --git a/schemas/platform/templates/kubernetes/README.md b/schemas/platform/templates/kubernetes/README.md index afd3504..2f7dbc7 100644 --- a/schemas/platform/templates/kubernetes/README.md +++ b/schemas/platform/templates/kubernetes/README.md @@ -221,6 +221,7 @@ kubectl describe service orchestrator -n provisioning ### Required ConfigMaps #### orchestrator-config + ```yaml apiVersion: v1 kind: ConfigMap @@ -235,6 +236,7 @@ data: ``` #### control-center-config + ```yaml apiVersion: v1 kind: ConfigMap @@ -251,6 +253,7 @@ data: ``` #### mcp-server-config + ```yaml apiVersion: v1 kind: ConfigMap @@ -272,6 +275,7 @@ data: ### Required Secrets #### control-center-secrets + ```yaml apiVersion: v1 kind: Secret diff --git a/schemas/platform/usage-guide.md b/schemas/platform/usage-guide.md index 1ad5ab3..0f3e635 100644 --- a/schemas/platform/usage-guide.md +++ b/schemas/platform/usage-guide.md @@ -101,7 +101,8 @@ ORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo ru ``` **Expected Output**: -``` + +```plaintext [INFO] Orchestrator starting... [INFO] Server listening on 127.0.0.1:9090 [INFO] Storage backend: filesystem @@ -127,6 +128,7 @@ curl -X POST http://localhost:9090/api/workflows \ To change configuration: **Option A: Re-run Interactive Form** + ```bash cd provisioning/.typedialog/provisioning/platform nu scripts/configure.nu orchestrator solo --backend cli @@ -136,6 +138,7 @@ nu scripts/generate-configs.nu orchestrator solo ``` **Option B: Edit TOML Directly** + ```bash # Edit the file directly vi provisioning/platform/config/orchestrator.solo.toml @@ -144,6 +147,7 @@ vi provisioning/platform/config/orchestrator.solo.toml ``` **Option C: Environment Variable Override** + ```bash # No file changes needed export ORCHESTRATOR_SERVER_PORT=9999 @@ -215,6 +219,7 @@ docker-compose -f provisioning/platform/infrastructure/docker/docker-compose.mul ``` **Verify Services**: + ```bash # Check all services are running docker-compose -f provisioning/platform/infrastructure/docker/docker-compose.multiuser.yml ps @@ -466,6 +471,7 @@ kubectl apply -f provisioning/platform/infrastructure/kubernetes/prometheus-aler ### Change Configuration Value **Without Service Restart** (Environment Variable): + ```bash # Override specific value via environment variable export ORCHESTRATOR_LOG_LEVEL=debug @@ -476,6 +482,7 @@ ORCHESTRATOR_CONFIG=config.toml cargo run --bin orchestrator ``` **With Service Restart** (TOML Edit): + ```bash # Edit TOML directly vi provisioning/platform/config/orchestrator.solo.toml @@ -486,6 +493,7 @@ ORCHESTRATOR_CONFIG=config.toml cargo run --bin orchestrator ``` **With Validation** (Regenerate from Form): + ```bash # Re-run interactive form to regenerate cd provisioning/.typedialog/provisioning/platform @@ -500,6 +508,7 @@ nu scripts/generate-configs.nu orchestrator solo ### Add Team Member **In Kubernetes PostgreSQL**: + ```bash kubectl exec -it -n provisioning deployment/postgres -- psql -U provisioning -d provisioning @@ -515,6 +524,7 @@ INSERT INTO role_assignments (user_id, role, granted_by, granted_at) VALUES ### Scale Service Replicas **In Kubernetes**: + ```bash # Scale orchestrator from 3 to 5 replicas kubectl scale deployment orchestrator -n provisioning --replicas=5 @@ -578,7 +588,7 @@ ORCHESTRATOR_CONFIG=config.toml cargo run --bin orchestrator 2>&1 | grep -i "con ## Configuration File Locations -``` +```plaintext provisioning/.typedialog/provisioning/platform/ ├── forms/ # User-facing interactive forms │ ├── orchestrator-form.toml @@ -628,7 +638,8 @@ provisioning/.typedialog/provisioning/platform/ ``` TOML output location: -``` + +```plaintext provisioning/platform/config/ ├── orchestrator.solo.toml # Consumed by orchestrator service ├── control-center.enterprise.toml # Consumed by control-center service diff --git a/schemas/platform/validators/README.md b/schemas/platform/validators/README.md index 9f0f530..34bb567 100644 --- a/schemas/platform/validators/README.md +++ b/schemas/platform/validators/README.md @@ -12,7 +12,7 @@ Validators provide: ## File Organization -``` +```plaintext validators/ ├── README.md # This file ├── common-validator.ncl # Ports, positive numbers, strings @@ -94,6 +94,7 @@ ValidRange = fun min max value => ## Common Validators ### common-validator.ncl + ```nickel let constraints = import "../constraints/constraints.toml" in @@ -123,6 +124,7 @@ let constraints = import "../constraints/constraints.toml" in ``` ### resource-validator.ncl + ```nickel let constraints = import "../constraints/constraints.toml" in let common = import "./common-validator.ncl" in @@ -143,6 +145,7 @@ let common = import "./common-validator.ncl" in ## Service-Specific Validators ### orchestrator-validator.ncl + ```nickel let constraints = import "../constraints/constraints.toml" in let common = import "./common-validator.ncl" in @@ -178,6 +181,7 @@ let common = import "./common-validator.ncl" in ``` ### control-center-validator.ncl + ```nickel { # JWT token expiration @@ -195,6 +199,7 @@ let common = import "./common-validator.ncl" in ``` ### mcp-server-validator.ncl + ```nickel { # Max concurrent tool executions @@ -281,6 +286,7 @@ nickel export --format toml provisioning/.typedialog/provisioning/platform/confi ## Adding a New Validator 1. **Create validator function** in appropriate file: + ```nickel ValidMyValue = fun value => if value < minimum then error "Too low" @@ -289,6 +295,7 @@ nickel export --format toml provisioning/.typedialog/provisioning/platform/confi ``` 2. **Add constraint** to constraints.toml if needed: + ```toml [service.feature.my_value] min = 1 @@ -296,11 +303,13 @@ nickel export --format toml provisioning/.typedialog/provisioning/platform/confi ``` 3. **Use in config**: + ```nickel my_value = validators.ValidMyValue 50, ``` 4. **Add form constraint** (if interactive): + ```toml [[elements]] name = "my_value" @@ -309,6 +318,7 @@ nickel export --format toml provisioning/.typedialog/provisioning/platform/confi ``` 5. **Test**: + ```bash nickel typecheck configs/service.mode.ncl ``` diff --git a/schemas/platform/values/README.md b/schemas/platform/values/README.md index 6663418..18d3660 100644 --- a/schemas/platform/values/README.md +++ b/schemas/platform/values/README.md @@ -12,7 +12,7 @@ The values directory stores: ## File Organization -``` +```plaintext values/ ├── .gitignore # Ignore *.ncl user configs ├── README.md # This file @@ -44,6 +44,7 @@ Each config file (`{service}.{mode}.ncl`) is: - **Runtime data** - Created/updated by scripts and forms Example: + ```nickel # values/orchestrator.solo.ncl (auto-generated, user-editable) { @@ -68,7 +69,7 @@ Example: ## .gitignore Pattern -``` +```plaintext # values/.gitignore *.ncl # Ignore all Nickel config files (user-specific) !*.example.ncl # EXCEPT example files (tracked for documentation) @@ -150,6 +151,7 @@ This ensures: ## Configuration Workflow ### 1. Generate Initial Config + ```bash nu scripts/configure.nu orchestrator solo ``` @@ -157,6 +159,7 @@ nu scripts/configure.nu orchestrator solo Creates `values/orchestrator.solo.ncl` from form input. ### 2. Edit Configuration + ```bash # Manually edit if needed vi values/orchestrator.solo.ncl @@ -166,11 +169,13 @@ nu scripts/configure.nu orchestrator solo --backend web ``` ### 3. Validate Configuration + ```bash nu scripts/validate-config.nu values/orchestrator.solo.ncl ``` ### 4. Generate TOML for Services + ```bash nu scripts/generate-configs.nu orchestrator solo ``` @@ -181,7 +186,7 @@ Exports to `provisioning/platform/config/orchestrator.solo.toml` (consumed by Ru User configs are composed with defaults during generation: -``` +```plaintext defaults/orchestrator-defaults.ncl (base values) ↓ & values/orchestrator.solo.ncl (user customizations) @@ -205,6 +210,7 @@ provisioning/platform/config/orchestrator.solo.toml (Rust service config) Since user configs are gitignored, sharing requires: ### Option 1: Share via File + ```bash # Export current config cat values/orchestrator.solo.ncl > /tmp/orchestrator-config.ncl @@ -215,6 +221,7 @@ cp /tmp/orchestrator-config.ncl values/orchestrator.solo.ncl ### Option 2: Use Example Template Share setup instructions instead of raw config: + ```bash # Document the setup steps cat > SETUP.md << EOF @@ -227,6 +234,7 @@ EOF ### Option 3: Store in Separate Repo For team configs, use a separate private repository: + ```bash # Clone team configs git clone private-repo/provisioning-configs values/ @@ -249,16 +257,19 @@ chmod 600 values/orchestrator.solo.ncl If you accidentally delete a user config: ### Option 1: Regenerate from TypeDialog + ```bash nu scripts/configure.nu orchestrator solo ``` ### Option 2: Copy from Backup + ```bash cp /backup/provisioning-values/orchestrator.solo.ncl values/ ``` ### Option 3: Use Example as Base + ```bash cp examples/orchestrator-solo.ncl values/orchestrator.solo.ncl # Customize as needed @@ -268,12 +279,14 @@ nu scripts/configure.nu orchestrator solo --backend web ## Troubleshooting ### Config File Missing + ```bash # Regenerate from defaults nu scripts/configure.nu orchestrator solo ``` ### Config Won't Validate + ```bash # Check for syntax errors nickel eval values/orchestrator.solo.ncl @@ -283,6 +296,7 @@ diff examples/orchestrator-solo.ncl values/orchestrator.solo.ncl ``` ### Changes Not Taking Effect + ```bash # Regenerate TOML from Nickel nu scripts/generate-configs.nu orchestrator solo