361 lines
13 KiB
Markdown
361 lines
13 KiB
Markdown
# TypeDialog + Nickel Configuration System for Platform Services
|
||
|
||
Complete configuration system for provisioning platform services (orchestrator, control-center, mcp-server, vault-service,
|
||
extension-registry, rag, ai-service, provisioning-daemon) across multiple deployment modes (solo, multiuser, cicd, enterprise).
|
||
|
||
## Architecture Overview
|
||
|
||
This system implements a **TypeDialog + Nickel configuration workflow** that provides:
|
||
|
||
- **Type-safe configuration** via Nickel schemas with validation
|
||
- **Interactive configuration** via TypeDialog forms with real-time constraint validation
|
||
- **Multi-mode deployment** (solo/multiuser/cicd/enterprise) with mode-specific defaults
|
||
- **Configuration composition** (base defaults + mode overlays + user customization + validation)
|
||
- **Automated TOML export** for Rust service consumption
|
||
- **Docker Compose + Kubernetes templates** for infrastructure deployment
|
||
|
||
## Directory Structure
|
||
|
||
```bash
|
||
provisioning/.typedialog/provisioning/platform/
|
||
├── constraints/ # Single source of truth for validation limits
|
||
├── schemas/ # Nickel type contracts (services + common + deployment modes)
|
||
├── defaults/ # Default configuration values (services + common + deployment modes)
|
||
├── validators/ # Validation logic (constraints, ranges, business rules)
|
||
├── configs/ # Generated mode-specific Nickel configurations (4 services × 4 modes = 16 configs)
|
||
├── forms/ # TypeDialog form definitions (4 main forms + flat fragments)
|
||
│ └── fragments/ # Reusable form fragments (workspace, server, database, etc.)
|
||
├── templates/ # Jinja2 + Nickel templates for config/deployment generation
|
||
│ ├── docker-compose/ # Docker Compose templates (solo/multiuser/cicd/enterprise)
|
||
│ ├── kubernetes/ # Kubernetes deployment templates
|
||
│ └── configs/ # Service configuration templates (TOML generation)
|
||
├── scripts/ # Nushell orchestration scripts (configure, generate, validate, deploy)
|
||
├── examples/ # Example configurations for different deployment scenarios
|
||
└── values/ # User configuration files (gitignored *.ncl)
|
||
```
|
||
|
||
## Configuration Workflow
|
||
|
||
### 1. User Interaction (TypeDialog)
|
||
|
||
```nushell
|
||
nu scripts/configure.nu orchestrator solo --backend web
|
||
```
|
||
|
||
- Launches interactive form (web/tui/cli)
|
||
- Loads existing config as default values (if exists)
|
||
- Validates user input against constraints
|
||
- Generates updated Nickel config
|
||
|
||
### 2. Configuration Composition
|
||
|
||
```toml
|
||
Base Defaults (defaults/*.ncl)
|
||
↓
|
||
+ Mode Overlay (defaults/deployment/{mode}-defaults.ncl)
|
||
↓
|
||
+ User Customization (values/{service}.{mode}.ncl)
|
||
↓
|
||
+ Schema Validation (schemas/*.ncl)
|
||
↓
|
||
+ Constraint Validation (validators/*.ncl)
|
||
↓
|
||
= Final Configuration (configs/{service}.{mode}.ncl)
|
||
```
|
||
|
||
### 3. TOML Export
|
||
|
||
```toml
|
||
nu scripts/generate-configs.nu orchestrator solo
|
||
```
|
||
|
||
Exports Nickel config to TOML:
|
||
- `provisioning/platform/config/orchestrator.solo.toml` (consumed by Rust services)
|
||
|
||
## Deployment Modes
|
||
|
||
### Solo (2 CPU, 4GB RAM)
|
||
- Single developer/testing
|
||
- Filesystem or embedded database
|
||
- Minimal security
|
||
- All services enabled
|
||
|
||
### MultiUser (4 CPU, 8GB RAM)
|
||
- Team collaboration, staging
|
||
- PostgreSQL or SurrealDB server
|
||
- RBAC enabled
|
||
- Gitea integration
|
||
|
||
### CI/CD (8 CPU, 16GB RAM)
|
||
- Automated pipelines, ephemeral
|
||
- API-driven configuration
|
||
- Fast cleanup, minimal storage
|
||
|
||
### Enterprise (16+ CPU, 32+ GB RAM)
|
||
- Production high availability
|
||
- SurrealDB cluster with replication
|
||
- MFA required, KMS integration
|
||
- Compliance (SOC2/HIPAA)
|
||
|
||
## Key Components
|
||
|
||
### Constraints (constraints/constraints.toml)
|
||
Single source of truth for validation limits across all services. Used for:
|
||
- Form field validation (min/max values)
|
||
- Constraint interpolation in TypeDialog forms
|
||
- Nickel validator bounds checking
|
||
|
||
### Schemas (schemas/*.ncl)
|
||
Type-safe configuration contracts defining:
|
||
- Required/optional fields
|
||
- Valid value types and enums
|
||
- Default values
|
||
- Input/output type signatures
|
||
|
||
**Organization**:
|
||
- `schemas/common/` - HTTP server, database, security, monitoring, logging
|
||
- `schemas/{orchestrator,control-center,mcp-server,vault-service,extension-registry,rag,ai-service,provisioning-daemon}.ncl` - Service-specific schemas
|
||
- `schemas/deployment/{solo,multiuser,cicd,enterprise}.ncl` - Mode-specific schemas
|
||
|
||
### Defaults (defaults/*.ncl)
|
||
Configuration base values composed with mode overlays:
|
||
- `defaults/{service}-defaults.ncl` - Service base defaults
|
||
- `defaults/common/` - Shared defaults (server, database, security)
|
||
- `defaults/deployment/{mode}-defaults.ncl` - Mode-specific value overrides
|
||
|
||
### Validators (validators/*.ncl)
|
||
Business logic validation using constraints:
|
||
- Port range validation (1024-65535)
|
||
- Resource allocation validation (CPU, memory)
|
||
- Workflow/policy validation (service-specific)
|
||
- Cross-field validation
|
||
|
||
### Configurations (configs/*.ncl)
|
||
Generated mode-specific Nickel configs (NOT manually edited):
|
||
- `orchestrator.{solo,multiuser,cicd,enterprise}.ncl`
|
||
- `control-center.{solo,multiuser,cicd,enterprise}.ncl`
|
||
- `mcp-server.{solo,multiuser,cicd,enterprise}.ncl`
|
||
- `vault-service.{solo,multiuser,cicd,enterprise}.ncl`
|
||
- `extension-registry.{solo,multiuser,cicd,enterprise}.ncl`
|
||
- `rag.{solo,multiuser,cicd,enterprise}.ncl`
|
||
- `ai-service.{solo,multiuser,cicd,enterprise}.ncl`
|
||
- `provisioning-daemon.{solo,multiuser,cicd,enterprise}.ncl`
|
||
|
||
### Forms (forms/*.toml)
|
||
TypeDialog form definitions with **flat fragments** referenced by paths:
|
||
- 4 main forms: `{service}-form.toml`
|
||
- Fragments: `fragments/{name}-section.toml` (workspace, server, database, security, monitoring, etc.)
|
||
- CRITICAL: Every form element has `nickel_path` for Nickel structure mapping
|
||
|
||
**Fragment Organization** (FLAT, referenced by paths):
|
||
- `workspace-section.toml`
|
||
- `server-section.toml`
|
||
- `database-rocksdb-section.toml`
|
||
- `database-surrealdb-section.toml`
|
||
- `database-postgres-section.toml`
|
||
- `security-section.toml`
|
||
- `monitoring-section.toml`
|
||
- `logging-section.toml`
|
||
- `orchestrator-queue-section.toml`
|
||
- `orchestrator-workflow-section.toml`
|
||
- ... (service-specific and mode-specific fragments)
|
||
|
||
### Templates (templates/)
|
||
Jinja2 + Nickel templates for automated generation:
|
||
- `{service}-config.ncl.j2` - Nickel output template (critical for TypeDialog nickel-roundtrip)
|
||
- `docker-compose/platform-stack.{mode}.yml.ncl` - Docker Compose templates
|
||
- `kubernetes/{service}-deployment.yaml.ncl` - Kubernetes templates
|
||
|
||
### Scripts (scripts/)
|
||
Nushell orchestration (NuShell 0.109+):
|
||
- `configure.nu` - Interactive TypeDialog wizard (nickel-roundtrip workflow)
|
||
- `generate-configs.nu` - Export Nickel → TOML
|
||
- `validate-config.nu` - Typecheck Nickel configs
|
||
- `render-docker-compose.nu` - Generate Docker Compose files
|
||
- `render-kubernetes.nu` - Generate Kubernetes manifests
|
||
- `install-services.nu` - Deploy platform services
|
||
- `detect-services.nu` - Auto-detect running services
|
||
|
||
### Examples (examples/)
|
||
Reference configurations for different scenarios:
|
||
- `orchestrator-solo.ncl` - Simple development setup
|
||
- `orchestrator-enterprise.ncl` - Complex production setup
|
||
- `full-platform-enterprise.ncl` - Complete enterprise stack
|
||
|
||
### Values (values/)
|
||
User configuration directory (gitignored):
|
||
- `{service}.{mode}.ncl` - User customizations (loaded in compose)
|
||
- `.gitignore` - Ignores `*.ncl` files
|
||
- `orchestrator.example.ncl` - Documented example template
|
||
|
||
## TypeDialog nickel-roundtrip Workflow
|
||
|
||
CRITICAL: Forms use Jinja2 templates for Nickel generation:
|
||
|
||
```nickel
|
||
# Command pattern
|
||
typedialog-web nickel-roundtrip "$CONFIG_FILE" "$FORM_FILE" --output "$CONFIG_FILE" --template "$NCL_TEMPLATE"
|
||
|
||
# Example
|
||
typedialog-web nickel-roundtrip
|
||
"provisioning/.typedialog/provisioning/platform/values/orchestrator.solo.ncl"
|
||
"provisioning/.typedialog/provisioning/platform/forms/orchestrator-form.toml"
|
||
--output "provisioning/.typedialog/provisioning/platform/values/orchestrator.solo.ncl"
|
||
--template "provisioning/.typedialog/provisioning/platform/templates/orchestrator-config.ncl.j2"
|
||
```
|
||
|
||
**Key Requirements**:
|
||
1. **Jinja2 template** (`config.ncl.j2`) - Defines Nickel output structure with conditional `{% if %}` blocks
|
||
2. **nickel_path** in form elements - Maps form fields to Nickel structure paths (e.g., `["orchestrator", "queue", "max_concurrent_tasks"]`)
|
||
3. **Constraint interpolation** - Form limits reference constraints (e.g., `${constraint.orchestrator.queue.concurrent_tasks.max}`)
|
||
4. **Base + overlay composition** - Nickel imports merge defaults + mode overlays + validators
|
||
|
||
## Usage Workflow
|
||
|
||
### 1. Configure Service (Interactive)
|
||
|
||
```toml
|
||
# Start TypeDialog wizard for orchestrator in solo mode
|
||
nu provisioning/.typedialog/provisioning/platform/scripts/configure.nu orchestrator solo --backend web
|
||
```
|
||
|
||
Wizard:
|
||
1. Loads existing config (if exists) as defaults
|
||
2. Shows form with validated constraints
|
||
3. User edits configuration
|
||
4. Generates updated Nickel config to `provisioning/.typedialog/provisioning/platform/values/orchestrator.solo.ncl`
|
||
|
||
### 2. Validate Configuration
|
||
|
||
```toml
|
||
# Typecheck Nickel config
|
||
nu provisioning/.typedialog/provisioning/platform/scripts/validate-config.nu provisioning/.typedialog/provisioning/platform/values/orchestrator.solo.ncl
|
||
```
|
||
|
||
### 3. Generate TOML for Rust Services
|
||
|
||
```toml
|
||
# Export Nickel → TOML
|
||
nu provisioning/.typedialog/provisioning/platform/scripts/generate-configs.nu orchestrator solo
|
||
```
|
||
|
||
Output: `provisioning/platform/config/orchestrator.solo.toml`
|
||
|
||
### 4. Deploy Services
|
||
|
||
```bash
|
||
# Install services (Docker Compose or Kubernetes)
|
||
nu provisioning/.typedialog/provisioning/platform/scripts/install-services.nu solo
|
||
```
|
||
|
||
## Configuration Loading Hierarchy (Rust Services)
|
||
|
||
```rust
|
||
1. Environment variables (ORCHESTRATOR_*)
|
||
2. User config (values/{service}.{mode}.ncl → TOML)
|
||
3. Mode-specific defaults (configs/{service}.{mode}.toml)
|
||
4. Service defaults (config/orchestrator.defaults.toml)
|
||
```
|
||
|
||
## Constraint Interpolation Example
|
||
|
||
**constraints.toml**:
|
||
|
||
```toml
|
||
[orchestrator.queue.concurrent_tasks]
|
||
min = 1
|
||
max = 100
|
||
```
|
||
|
||
**Form element** (fragments/orchestrator-queue-section.toml):
|
||
|
||
```toml
|
||
[[elements]]
|
||
name = "max_concurrent_tasks"
|
||
type = "number"
|
||
min = "${constraint.orchestrator.queue.concurrent_tasks.min}"
|
||
max = "${constraint.orchestrator.queue.concurrent_tasks.max}"
|
||
nickel_path = ["orchestrator", "queue", "max_concurrent_tasks"]
|
||
```
|
||
|
||
**Jinja2 template** (orchestrator-config.ncl.j2):
|
||
|
||
```nickel
|
||
orchestrator = {
|
||
queue = {
|
||
{%- if max_concurrent_tasks %}
|
||
max_concurrent_tasks = {{ max_concurrent_tasks }},
|
||
{%- endif %}
|
||
},
|
||
}
|
||
```
|
||
|
||
## Getting Started
|
||
|
||
1. **Run configuration wizard**:
|
||
|
||
```bash
|
||
nu provisioning/.typedialog/provisioning/platform/scripts/configure.nu orchestrator solo
|
||
```
|
||
|
||
2. **Generate TOML configs**:
|
||
|
||
```bash
|
||
nu provisioning/.typedialog/provisioning/platform/scripts/generate-configs.nu orchestrator solo
|
||
```
|
||
|
||
3. **Deploy services**:
|
||
|
||
```bash
|
||
nu provisioning/.typedialog/provisioning/platform/scripts/install-services.nu solo
|
||
```
|
||
|
||
## Documentation
|
||
|
||
- `constraints/README.md` - How to modify validation constraints
|
||
- `schemas/README.md` - Schema patterns and imports
|
||
- `defaults/README.md` - Defaults composition and merging strategy
|
||
- `validators/README.md` - Validator patterns and error handling
|
||
- `forms/README.md` - Form structure and fragment organization
|
||
- `forms/fragments/README.md` - Fragment usage and nickel_path mapping
|
||
- `scripts/README.md` - Script usage and dependencies
|
||
- `examples/README.md` - Example deployment scenarios
|
||
- `templates/README.md` - Template patterns and interpolation
|
||
|
||
## Key Files
|
||
|
||
| File | Purpose |
|
||
| ------ | --------- |
|
||
| `constraints/constraints.toml` | Single source of truth for validation limits |
|
||
| `schemas/orchestrator.ncl` | Orchestrator type schema |
|
||
| `defaults/orchestrator-defaults.ncl` | Orchestrator default values |
|
||
| `validators/orchestrator-validator.ncl` | Orchestrator validation logic |
|
||
| `configs/orchestrator.solo.ncl` | Generated solo mode config |
|
||
| `forms/orchestrator-form.toml` | Orchestrator form definition |
|
||
| `templates/orchestrator-config.ncl.j2` | Nickel output template |
|
||
| `scripts/configure.nu` | Interactive configuration wizard |
|
||
| `scripts/generate-configs.nu` | Nickel → TOML export |
|
||
| `values/orchestrator.solo.ncl` | User configuration (gitignored) |
|
||
|
||
## Tools Required
|
||
|
||
- **Nickel** (0.10+) - Configuration language
|
||
- **TypeDialog** - Interactive form backend
|
||
- **NuShell** (0.109+) - Script orchestration
|
||
- **Jinja2/tera** - Template rendering (via nu_plugin_tera)
|
||
- **TOML** - Config file format (for Rust services)
|
||
|
||
## Notes
|
||
|
||
- Configuration files in `values/` are **gitignored** (user-specific)
|
||
- Generated configs in `configs/` are composed automatically (not hand-edited)
|
||
- Each mode (solo/multiuser/cicd/enterprise) has different resource defaults
|
||
- Fragments are **flat** in `forms/fragments/` and referenced by paths in form definitions
|
||
- All form elements must have `nickel_path` for proper Nickel structure mapping
|
||
- Constraint interpolation enables dynamic form validation based on service requirements
|
||
|
||
---
|
||
|
||
**Version**: 1.0.0
|
||
**Created**: 2025-01-05
|
||
**Last Updated**: 2025-01-05 |