2 lines
21 KiB
Markdown
2 lines
21 KiB
Markdown
# Configuration System Usage Guide\n\nPractical guide for using the provisioning platform configuration system across common scenarios.\n\n## Quick Start (5 Minutes)\n\n### For Local Development\n\n```\n# 1. Enter configuration system directory\ncd provisioning/.typedialog/provisioning/platform\n\n# 2. Generate solo configuration (interactive)\nnu scripts/configure.nu orchestrator solo --backend cli\n\n# 3. Export to TOML\nnu scripts/generate-configs.nu orchestrator solo\n\n# 4. Start orchestrator\ncd ../../\nORCHESTRATOR_CONFIG=platform/config/orchestrator.solo.toml cargo run --bin orchestrator\n```\n\n### For Team Staging\n\n```\n# 1. Generate multiuser configuration\ncd provisioning/.typedialog/provisioning/platform\nnu scripts/configure.nu control-center multiuser --backend web\n\n# 2. Export configuration\nnu scripts/generate-configs.nu control-center multiuser\n\n# 3. Start with Docker Compose\ncd ../../\ndocker-compose -f platform/infrastructure/docker/docker-compose.multiuser.yml up -d\n```\n\n### For Production Enterprise\n\n```\n# 1. Generate enterprise configuration\ncd provisioning/.typedialog/provisioning/platform\nnu scripts/configure.nu orchestrator enterprise --backend web\n\n# 2. Export configuration\nnu scripts/generate-configs.nu orchestrator enterprise\n\n# 3. Deploy to Kubernetes\ncd ../../\nkubectl apply -f platform/infrastructure/kubernetes/namespace.yaml\nkubectl apply -f platform/infrastructure/kubernetes/*.yaml\n```\n\n---\n\n## Scenario 1: Single Developer Setup\n\n**Goal**: Set up local orchestrator for development testing\n**Time**: 5-10 minutes\n**Requirements**: Nushell, Nickel, Rust toolchain\n\n### Step 1: Interactive Configuration\n\n```\ncd provisioning/.typedialog/provisioning/platform\nnu scripts/configure.nu orchestrator solo --backend cli\n```\n\n**Form Fields**:\n- Workspace name: `dev-workspace` (default)\n- Workspace path: `/home/username/provisioning/data/orchestrator` (change to your path)\n- Server host: `127.0.0.1` (localhost only)\n- Server port: `9090` (default)\n- Storage backend: `filesystem` (selected by default)\n- Logging level: `debug` (recommended for dev)\n\n### Step 2: Validate Configuration\n\n```\n# Typecheck the generated Nickel\nnickel typecheck configs/orchestrator.solo.ncl\n\n# Should output: "✓ Type checking successful"\n```\n\n### Step 3: Export to TOML\n\n```\n# Generate TOML from Nickel\nnu scripts/generate-configs.nu orchestrator solo\n\n# Output: provisioning/platform/config/orchestrator.solo.toml\n```\n\n### Step 4: Start the Service\n\n```\ncd ../..\nORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator\n```\n\n**Expected Output**:\n\n```\n[INFO] Orchestrator starting...\n[INFO] Server listening on 127.0.0.1:9090\n[INFO] Storage backend: filesystem\n[INFO] Ready to accept requests\n```\n\n### Step 5: Test the Service\n\nIn another terminal:\n\n```\n# Check health\ncurl http://localhost:9090/health\n\n# Submit a workflow\ncurl -X POST http://localhost:9090/api/workflows \\n -H "Content-Type: application/json" \\n -d '{"name": "test-workflow", "steps": []}'\n```\n\n### Iteration: Modify Configuration\n\nTo change configuration:\n\n**Option A: Re-run Interactive Form**\n\n```\ncd provisioning/.typedialog/provisioning/platform\nnu scripts/configure.nu orchestrator solo --backend cli\n# Answer with new values\nnu scripts/generate-configs.nu orchestrator solo\n# Restart service\n```\n\n**Option B: Edit TOML Directly**\n\n```\n# Edit the file directly\nvi provisioning/platform/config/orchestrator.solo.toml\n# Change values as needed\n# Restart service\n```\n\n**Option C: Environment Variable Override**\n\n```\n# No file changes needed\nexport ORCHESTRATOR_SERVER_PORT=9999\nexport ORCHESTRATOR_LOG_LEVEL=info\n\nORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator\n```\n\n---\n\n## Scenario 2: Team Collaboration Setup\n\n**Goal**: Set up shared team environment with PostgreSQL and RBAC\n**Time**: 20-30 minutes\n**Requirements**: Docker, Docker Compose, PostgreSQL running\n\n### Step 1: Interactive Configuration\n\n```\ncd provisioning/.typedialog/provisioning/platform\n\n# Configure Control Center with RBAC\nnu scripts/configure.nu control-center multiuser --backend web\n```\n\n**Important Fields**:\n- Database backend: `postgres` (for persistent storage)\n- Database host: `postgres.provisioning.svc.cluster.local` or `localhost` for local\n- Database password: Generate strong password (store in `.env` file, don't hardcode)\n- JWT secret: Generate 256-bit random string\n- MFA required: `false` (optional for team, not required)\n- Default role: `viewer` (least privilege)\n\n### Step 2: Create Environment File\n\n```\n# Create .env for secrets\ncat > provisioning/platform/.env << 'EOF'\nDB_PASSWORD=generate-strong-password-here\nJWT_SECRET=generate-256-bit-random-base64-string\nSURREALDB_PASSWORD=another-strong-password\nEOF\n\n# Protect the file\nchmod 600 provisioning/platform/.env\n```\n\n### Step 3: Export Configurations\n\n```\n# Export all three services for team setup\nnu scripts/generate-configs.nu control-center multiuser\nnu scripts/generate-configs.nu orchestrator multiuser\nnu scripts/generate-configs.nu mcp-server multiuser\n```\n\n### Step 4: Start Services with Docker Compose\n\n```\ncd ../..\n\n# Generate Docker Compose from Nickel template\nnu provisioning/.typedialog/provisioning/platform/scripts/render-docker-compose.nu multiuser\n\n# Start all services\ndocker-compose -f provisioning/platform/infrastructure/docker/docker-compose.multiuser.yml \\n --env-file provisioning/platform/.env \\n up -d\n```\n\n**Verify Services**:\n\n```\n# Check all services are running\ndocker-compose -f provisioning/platform/infrastructure/docker/docker-compose.multiuser.yml ps\n\n# Check logs for errors\ndocker-compose -f provisioning/platform/infrastructure/docker/docker-compose.multiuser.yml logs -f control-center\n\n# Test Control Center UI\nopen http://localhost:8080\n# Login with default credentials (or configure initially)\n```\n\n### Step 5: Create Team Users and Roles\n\n```\n# Access PostgreSQL to set up users\ndocker-compose exec postgres psql -U provisioning -d provisioning\n\n-- Create users\nINSERT INTO users (username, email, role) VALUES\n ('alice@company.com', 'alice@company.com', 'admin'),\n ('bob@company.com', 'bob@company.com', 'operator'),\n ('charlie@company.com', 'charlie@company.com', 'developer');\n\n-- Create RBAC assignments\nINSERT INTO role_assignments (user_id, role) VALUES\n ((SELECT id FROM users WHERE username='alice@company.com'), 'admin'),\n ((SELECT id FROM users WHERE username='bob@company.com'), 'operator'),\n ((SELECT id FROM users WHERE username='charlie@company.com'), 'developer');\n```\n\n### Step 6: Team Access\n\n**Admin (Alice)**:\n- Full platform access\n- Can create/modify users\n- Can manage all workflows and policies\n\n**Operator (Bob)**:\n- Execute and manage workflows\n- View logs and metrics\n- Cannot modify policies or users\n\n**Developer (Charlie)**:\n- Read-only access to workflows\n- Cannot execute or modify\n- Can view logs\n\n---\n\n## Scenario 3: Production Enterprise Deployment\n\n**Goal**: Deploy complete platform to Kubernetes with HA and monitoring\n**Time**: 1-2 hours (includes infrastructure setup)\n**Requirements**: Kubernetes cluster, kubectl, Helm (optional)\n\n### Step 1: Pre-Deployment Checklist\n\n```\n# Verify Kubernetes access\nkubectl cluster-info\n\n# Create namespace\nkubectl create namespace provisioning\n\n# Verify persistent volumes available\nkubectl get pv\n\n# Check node resources\nkubectl top nodes\n# Minimum 16 CPU, 32GB RAM across cluster\n```\n\n### Step 2: Interactive Configuration (Enterprise Mode)\n\n```\ncd provisioning/.typedialog/provisioning/platform\n\nnu scripts/configure.nu orchestrator enterprise --backend web\nnu scripts/configure.nu control-center enterprise --backend web\nnu scripts/configure.nu mcp-server enterprise --backend web\n```\n\n**Critical Enterprise Settings**:\n- Deployment mode: `enterprise`\n- Replicas: Orchestrator (3), Control Center (2), MCP Server (1-2)\n- Storage:\n - Orchestrator: `surrealdb_cluster` with 3 nodes\n - Control Center: `postgres` with HA\n- Security:\n - Auth: `jwt` (required)\n - TLS: `true` (required)\n - MFA: `true` (required)\n- Monitoring: All enabled\n- Logging: JSON format with 365-day retention\n\n### Step 3: Generate Secrets\n\n```\n# Generate secure values\nJWT_SECRET=$(openssl rand -base64 32)\nDB_PASSWORD=$(openssl rand -base64 32)\nSURREALDB_PASSWORD=$(openssl rand -base64 32)\nADMIN_PASSWORD=$(openssl rand -base64 16)\n\n# Create Kubernetes secret\nkubectl create secret generic provisioning-secrets \\n -n provisioning \\n --from-literal=jwt-secret="$JWT_SECRET" \\n --from-literal=db-password="$DB_PASSWORD" \\n --from-literal=surrealdb-password="$SURREALDB_PASSWORD" \\n --from-literal=admin-password="$ADMIN_PASSWORD"\n\n# Verify secret created\nkubectl get secrets -n provisioning\n```\n\n### Step 4: TLS Certificate Setup\n\n```\n# Generate self-signed certificate (for testing)\nopenssl req -x509 -nodes -days 365 -newkey rsa:2048 \\n -keyout provisioning.key \\n -out provisioning.crt \\n -subj "/CN=provisioning.example.com"\n\n# Create TLS secret in Kubernetes\nkubectl create secret tls provisioning-tls \\n -n provisioning \\n --cert=provisioning.crt \\n --key=provisioning.key\n\n# For production: Use cert-manager or real certificates\n# kubectl create secret tls provisioning-tls \\n# -n provisioning \\n# --cert=/path/to/cert.pem \\n# --key=/path/to/key.pem\n```\n\n### Step 5: Export Configurations\n\n```\n# Export TOML configurations\nnu scripts/generate-configs.nu orchestrator enterprise\nnu scripts/generate-configs.nu control-center enterprise\nnu scripts/generate-configs.nu mcp-server enterprise\n```\n\n### Step 6: Create ConfigMaps for Configuration\n\n```\n# Create ConfigMaps with exported TOML\nkubectl create configmap orchestrator-config \\n -n provisioning \\n --from-file=provisioning/platform/config/orchestrator.enterprise.toml\n\nkubectl create configmap control-center-config \\n -n provisioning \\n --from-file=provisioning/platform/config/control-center.enterprise.toml\n\nkubectl create configmap mcp-server-config \\n -n provisioning \\n --from-file=provisioning/platform/config/mcp-server.enterprise.toml\n```\n\n### Step 7: Deploy Infrastructure\n\n```\ncd ../..\n\n# Deploy in order of dependencies\nkubectl apply -f provisioning/platform/infrastructure/kubernetes/namespace.yaml\nkubectl apply -f provisioning/platform/infrastructure/kubernetes/resource-quota.yaml\nkubectl apply -f provisioning/platform/infrastructure/kubernetes/rbac.yaml\nkubectl apply -f provisioning/platform/infrastructure/kubernetes/network-policy.yaml\n\n# Deploy storage (PostgreSQL, SurrealDB)\nkubectl apply -f provisioning/platform/infrastructure/kubernetes/postgres-*.yaml\nkubectl apply -f provisioning/platform/infrastructure/kubernetes/surrealdb-*.yaml\n\n# Wait for databases to be ready\nkubectl wait --for=condition=ready pod -l app=postgres -n provisioning --timeout=300s\nkubectl wait --for=condition=ready pod -l app=surrealdb -n provisioning --timeout=300s\n\n# Deploy platform services\nkubectl apply -f provisioning/platform/infrastructure/kubernetes/orchestrator-*.yaml\nkubectl apply -f provisioning/platform/infrastructure/kubernetes/control-center-*.yaml\nkubectl apply -f provisioning/platform/infrastructure/kubernetes/mcp-server-*.yaml\n\n# Deploy monitoring stack\nkubectl apply -f provisioning/platform/infrastructure/kubernetes/prometheus-*.yaml\nkubectl apply -f provisioning/platform/infrastructure/kubernetes/grafana-*.yaml\nkubectl apply -f provisioning/platform/infrastructure/kubernetes/loki-*.yaml\n\n# Deploy ingress\nkubectl apply -f provisioning/platform/infrastructure/kubernetes/platform-ingress.yaml\n```\n\n### Step 8: Verify Deployment\n\n```\n# Check all pods are running\nkubectl get pods -n provisioning\n\n# Check services\nkubectl get svc -n provisioning\n\n# Wait for all pods ready\nkubectl wait --for=condition=Ready pods --all -n provisioning --timeout=600s\n\n# Check ingress\nkubectl get ingress -n provisioning\n```\n\n### Step 9: Access the Platform\n\n```\n# Get Ingress IP\nkubectl get ingress -n provisioning\n\n# Configure DNS (or use /etc/hosts for testing)\necho "INGRESS_IP provisioning.example.com" | sudo tee -a /etc/hosts\n\n# Access services\n# Orchestrator: https://orchestrator.provisioning.example.com/api\n# Control Center: https://control-center.provisioning.example.com\n# MCP Server: https://mcp.provisioning.example.com\n# Grafana: https://grafana.provisioning.example.com (admin/password)\n# Prometheus: https://prometheus.provisioning.example.com (internal)\n```\n\n### Step 10: Post-Deployment Configuration\n\n```\n# Create database schema\nkubectl exec -it -n provisioning deployment/postgres -- psql -U provisioning -d provisioning -f /schema.sql\n\n# Initialize Grafana dashboards\nkubectl cp grafana-dashboards provisioning/grafana-0:/var/lib/grafana/dashboards/\n\n# Configure alerts\nkubectl apply -f provisioning/platform/infrastructure/kubernetes/prometheus-alerts.yaml\n```\n\n---\n\n## Common Tasks\n\n### Change Configuration Value\n\n**Without Service Restart** (Environment Variable):\n\n```\n# Override specific value via environment variable\nexport ORCHESTRATOR_LOG_LEVEL=debug\nexport ORCHESTRATOR_SERVER_PORT=9999\n\n# Service uses overridden values\nORCHESTRATOR_CONFIG=config.toml cargo run --bin orchestrator\n```\n\n**With Service Restart** (TOML Edit):\n\n```\n# Edit TOML directly\nvi provisioning/platform/config/orchestrator.solo.toml\n\n# Restart service\npkill -f "cargo run --bin orchestrator"\nORCHESTRATOR_CONFIG=config.toml cargo run --bin orchestrator\n```\n\n**With Validation** (Regenerate from Form):\n\n```\n# Re-run interactive form to regenerate\ncd provisioning/.typedialog/provisioning/platform\nnu scripts/configure.nu orchestrator solo --backend cli\n\n# Validation ensures consistency\nnu scripts/generate-configs.nu orchestrator solo\n\n# Restart service with validated config\n```\n\n### Add Team Member\n\n**In Kubernetes PostgreSQL**:\n\n```\nkubectl exec -it -n provisioning deployment/postgres -- psql -U provisioning -d provisioning\n\n-- Create user\nINSERT INTO users (username, email, password_hash, role, created_at) VALUES\n ('newuser@company.com', 'newuser@company.com', crypt('password', gen_salt('bf')), 'developer', now());\n\n-- Assign role\nINSERT INTO role_assignments (user_id, role, granted_by, granted_at) VALUES\n ((SELECT id FROM users WHERE username='newuser@company.com'), 'developer', 1, now());\n```\n\n### Scale Service Replicas\n\n**In Kubernetes**:\n\n```\n# Scale orchestrator from 3 to 5 replicas\nkubectl scale deployment orchestrator -n provisioning --replicas=5\n\n# Verify scaling\nkubectl get deployment orchestrator -n provisioning\nkubectl get pods -n provisioning | grep orchestrator\n```\n\n### Monitor Service Health\n\n```\n# Check pod status\nkubectl describe pod orchestrator-0 -n provisioning\n\n# Check service logs\nkubectl logs -f deployment/orchestrator -n provisioning --all-containers=true\n\n# Check resource usage\nkubectl top pods -n provisioning\n\n# Check service metrics (via Prometheus)\nkubectl port-forward -n provisioning svc/prometheus 9091:9091\nopen http://localhost:9091\n```\n\n### Backup Configuration\n\n```\n# Backup current TOML configs\ntar -czf configs-backup-$(date +%Y%m%d).tar.gz provisioning/platform/config/\n\n# Backup Kubernetes manifests\nkubectl get all -n provisioning -o yaml > k8s-backup-$(date +%Y%m%d).yaml\n\n# Backup database\nkubectl exec -n provisioning deployment/postgres -- pg_dump -U provisioning provisioning | gzip > db-backup-$(date +%Y%m%d).sql.gz\n```\n\n### Troubleshoot Configuration Issues\n\n```\n# Check Nickel syntax errors\nnickel typecheck provisioning/.typedialog/provisioning/platform/configs/orchestrator.solo.ncl\n\n# Validate TOML syntax\nnickel export --format toml provisioning/.typedialog/provisioning/platform/configs/orchestrator.solo.ncl\n\n# Check TOML is valid for Rust\nORCHESTRATOR_CONFIG=provisioning/platform/config/orchestrator.solo.toml cargo run --bin orchestrator -- --validate-config\n\n# Check environment variable overrides\necho $ORCHESTRATOR_SERVER_PORT\necho $ORCHESTRATOR_LOG_LEVEL\n\n# Examine actual config loaded (if service logs it)\nORCHESTRATOR_CONFIG=config.toml cargo run --bin orchestrator 2>&1 | grep -i "config\|configuration"\n```\n\n---\n\n## Configuration File Locations\n\n```\nprovisioning/.typedialog/provisioning/platform/\n├── forms/ # User-facing interactive forms\n│ ├── orchestrator-form.toml\n│ ├── control-center-form.toml\n│ └── fragments/ # Reusable form sections\n│\n├── values/ # User input files (gitignored)\n│ ├── orchestrator.solo.ncl\n│ ├── orchestrator.enterprise.ncl\n│ └── (auto-generated by TypeDialog)\n│\n├── configs/ # Composed Nickel configs\n│ ├── orchestrator.solo.ncl # Base + mode overlay + user input + validation\n│ ├── control-center.multiuser.ncl\n│ └── (4 services × 4 modes = 16 files)\n│\n├── schemas/ # Type definitions\n│ ├── orchestrator.ncl\n│ ├── control-center.ncl\n│ └── common/ # Shared schemas\n│\n├── defaults/ # Default values\n│ ├── orchestrator-defaults.ncl\n│ └── deployment/solo-defaults.ncl\n│\n├── validators/ # Business rules\n│ ├── orchestrator-validator.ncl\n│ └── (per-service validators)\n│\n├── constraints/\n│ └── constraints.toml # Min/max values (single source of truth)\n│\n├── templates/ # Deployment templates\n│ ├── docker-compose/\n│ │ ├── platform-stack.solo.yml.ncl\n│ │ └── (4 modes)\n│ └── kubernetes/\n│ ├── orchestrator-deployment.yaml.ncl\n│ └── (11 templates)\n│\n└── scripts/ # Automation\n ├── configure.nu # Interactive TypeDialog\n ├── generate-configs.nu # Nickel → TOML export\n ├── validate-config.nu # Typecheck Nickel\n ├── render-docker-compose.nu # Templates → Docker Compose\n └── render-kubernetes.nu # Templates → Kubernetes\n```\n\nTOML output location:\n\n```\nprovisioning/platform/config/\n├── orchestrator.solo.toml # Consumed by orchestrator service\n├── control-center.enterprise.toml # Consumed by control-center service\n└── (4 services × 4 modes = 16 files)\n```\n\n---\n\n## Tips & Best Practices\n\n### 1. Use Version Control\n\n```\n# Commit TOML configs to track changes\ngit add provisioning/platform/config/*.toml\ngit commit -m "Update orchestrator enterprise config: increase worker threads to 16"\n\n# Do NOT commit Nickel source files in values/\necho "provisioning/.typedialog/provisioning/platform/values/*.ncl" >> .gitignore\n```\n\n### 2. Test Before Production Deployment\n\n```\n# Test in solo mode first\nnu scripts/configure.nu orchestrator solo\ncargo run --bin orchestrator\n\n# Then test in staging (multiuser mode)\nnu scripts/configure.nu orchestrator multiuser\ndocker-compose -f docker-compose.multiuser.yml up\n\n# Finally deploy to production (enterprise)\nnu scripts/configure.nu orchestrator enterprise\n# Then Kubernetes deployment\n```\n\n### 3. Document Custom Configurations\n\n```\n# Add comments to configurations\n# In values/*.ncl or config/*.ncl:\n\n# Custom configuration for high-throughput testing\n# - Increased workers from 4 to 8\n# - Increased queue.max_concurrent_tasks from 5 to 20\n# - Lowered logging level from debug to info\n{\n orchestrator = {\n # Worker threads increased for testing parallel task processing\n server.workers = 8,\n queue.max_concurrent_tasks = 20,\n logging.level = "info",\n },\n}\n```\n\n### 4. Secrets Management\n\n**Never** hardcode secrets in configuration files:\n\n```\n# WRONG - Don't do this\n[orchestrator.security]\njwt_secret = "hardcoded-secret-exposed-in-git"\n\n# RIGHT - Use environment variables\nexport ORCHESTRATOR_SECURITY_JWT_SECRET="actual-secret-from-vault"\n\n# TOML references it:\n[orchestrator.security]\njwt_secret = "${JWT_SECRET}" # Loaded at runtime\n```\n\n### 5. Monitor Changes\n\n```\n# Track configuration changes over time\ngit log --oneline provisioning/platform/config/\n\n# See what changed\ngit diff <commit1> <commit2> provisioning/platform/config/orchestrator.solo.toml\n```\n\n---\n\n**Version**: 1.0\n**Last Updated**: 2025-01-05\n**Status**: Production Ready
|