# Provisioning Platform - Architecture Overview\n\n**Version**: 3.5.0\n**Date**: 2025-10-06\n**Status**: Production\n**Maintainers**: Architecture Team\n\n---\n\n## Table of Contents\n\n1. [Executive Summary](#executive-summary)\n2. [System Architecture](#system-architecture)\n3. [Component Architecture](#component-architecture)\n4. [Mode Architecture](#mode-architecture)\n5. [Network Architecture](#network-architecture)\n6. [Data Architecture](#data-architecture)\n7. [Security Architecture](#security-architecture)\n8. [Deployment Architecture](#deployment-architecture)\n9. [Integration Architecture](#integration-architecture)\n10. [Performance and Scalability](#performance-and-scalability)\n11. [Evolution and Roadmap](#evolution-and-roadmap)\n\n---\n\n## Executive Summary\n\n### What is the Provisioning Platform\n\nThe Provisioning Platform is a modern, cloud-native infrastructure automation system that combines:\n\n- the simplicity of declarative configuration (Nickel)\n- the power of shell scripting (Nushell)\n- high-performance coordination (Rust).\n\n### Key Characteristics\n\n- **Hybrid Architecture**: Rust for coordination, Nushell for business logic, Nickel for configuration\n- **Mode-Based**: Adapts from solo development to enterprise production\n- **OCI-Native**: Extends leveraging industry-standard OCI distribution\n- **Provider-Agnostic**: Supports multiple cloud providers (AWS, UpCloud) and local infrastructure\n- **Extension-Driven**: Core functionality enhanced through modular extensions\n\n### Architecture at a Glance\n\n```\n┌─────────────────────────────────────────────────────────────────────┐\n│                        Provisioning Platform                        │\n├─────────────────────────────────────────────────────────────────────┤\n│                                                                     │\n│   ┌──────────────┐   ┌─────────────┐    ┌──────────────┐            │\n│   │ User Layer   │   │  Extension  │    │   Service    │            │\n│   │  (CLI/UI)    │   │  Registry   │    │   Registry   │            │\n│   └──────┬───────┘   └──────┬──────┘    └──────┬───────┘            │\n│          │                  │                  │                    │\n│   ┌──────┴──────────────────┴──────────────────┴──--────┐           │\n│   │            Core Provisioning Engine                 │           │\n│   │  (Config | Dependency Resolution | Workflows)       │           │\n│   └──────┬──────────────────────────────────────┬───────┘           │\n│          │                                      │                   │\n│   ┌──────┴─────────┐                   ┌──────-─┴─────────┐         │\n│   │  Orchestrator  │                   │   Business Logic │         │\n│   │    (Rust)      │ ←─ Coordination → │    (Nushell)     │         │\n│   └──────┬─────────┘                   └───────┬──────────┘         │\n│          │                                     │                    │\n│   ┌──────┴─────────────────────────────────────┴---──────┐          │\n│   │                  Extension System                    │          │\n│   │      (Providers | Task Services | Clusters)          │          │\n│   └──────┬───────────────────────────────────────────────┘          │\n│          │                                                          │\n│   ┌──────┴──────────────────────────────────────────────────-─┐     │\n│   │        Infrastructure (Cloud | Local | Kubernetes)        │     │\n│   └───────────────────────────────────────────────────────────┘     │\n│                                                                     │\n└─────────────────────────────────────────────────────────────────────┘\n```\n\n### Key Metrics\n\n| Metric | Value | Description |\n| -------- | ------- | ------------- |\n| **Codebase Size** | ~50,000 LOC | Nushell (60%), Rust (30%), Nickel (10%) |\n| **Extensions** | 100+ | Providers, taskservs, clusters |\n| **Supported Providers** | 3 | AWS, UpCloud, Local |\n| **Task Services** | 50+ | Kubernetes, databases, monitoring, etc. |\n| **Deployment Modes** | 5 | Binary, Docker, Docker Compose, K8s, Remote |\n| **Operational Modes** | 4 | Solo, Multi-user, CI/CD, Enterprise |\n| **API Endpoints** | 80+ | REST, WebSocket, GraphQL (planned) |\n\n---\n\n## System Architecture\n\n### High-Level Architecture\n\n```\n┌────────────────────────────────────────────────────────────────────────────┐\n│                         PRESENTATION LAYER                                 │\n├────────────────────────────────────────────────────────────────────────────┤\n│                                                                            │\n│    ┌─────────────┐  ┌──────────────┐  ┌──────────────┐  ┌────────────┐     │\n│    │  CLI (Nu)   │  │ Control      │  │  REST API    │  │  MCP       │     │\n│    │             │  │ Center (Yew) │  │  Gateway     │  │  Server    │     │\n│    └─────────────┘  └──────────────┘  └──────────────┘  └────────────┘     │\n│                                                                            │\n└──────────────────────────────────┬─────────────────────────────────────────┘\n                                   │\n┌──────────────────────────────────┴─────────────────────────────────────────┐\n│                         CORE LAYER                                         │\n├────────────────────────────────────────────────────────────────────────────┤\n│                                                                            │\n│   ┌─────────────────────────────────────────────────────────────────┐      │\n│   │               Configuration Management                          │      │\n│   │   (Nickel Schemas | TOML Config | Hierarchical Loading)         │      │\n│   └─────────────────────────────────────────────────────────────────┘      │\n│                                                                            │\n│   ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐         │\n│   │   Dependency     │  │   Module/Layer   │  │   Workspace      │         │\n│   │   Resolution     │  │     System       │  │   Management     │         │\n│   └──────────────────┘  └──────────────────┘  └──────────────────┘         │\n│                                                                            │\n│  ┌──────────────────────────────────────────────────────────────────┐      │\n│  │                  Workflow Engine                                 │      │\n│  │   (Batch Operations | Checkpoints | Rollback)                    │      │\n│  └──────────────────────────────────────────────────────────────────┘      │\n│                                                                            │\n└──────────────────────────────────┬─────────────────────────────────────────┘\n                                   │\n┌──────────────────────────────────┴─────────────────────────────────────────┐\n│                      ORCHESTRATION LAYER                                   │\n├────────────────────────────────────────────────────────────────────────────┤\n│                                                                            │\n│  ┌──────────────────────────────────────────────────────────────────┐      │\n│  │                Orchestrator (Rust)                               │      │\n│  │   • Task Queue (File-based persistence)                          │      │\n│  │   • State Management (Checkpoints)                               │      │\n│  │   • Health Monitoring                                            │      │\n│  │   • REST API (HTTP/WS)                                           │      │\n│  └──────────────────────────────────────────────────────────────────┘      │\n│                                                                            │\n│  ┌──────────────────────────────────────────────────────────────────┐      │\n│  │           Business Logic (Nushell)                               │      │\n│  │   • Provider operations (AWS, UpCloud, Local)                    │      │\n│  │   • Server lifecycle (create, delete, configure)                 │      │\n│  │   • Taskserv installation (50+ services)                         │      │\n│  │   • Cluster deployment                                           │      │\n│  └──────────────────────────────────────────────────────────────────┘      │\n│                                                                            │\n└──────────────────────────────────┬─────────────────────────────────────────┘\n                                   │\n┌──────────────────────────────────┴─────────────────────────────────────────┐\n│                      EXTENSION LAYER                                       │\n├────────────────────────────────────────────────────────────────────────────┤\n│                                                                            │\n│   ┌────────────────┐  ┌──────────────────┐  ┌───────────────────┐          │\n│   │   Providers    │  │   Task Services  │  │    Clusters       │          │\n│   │   (3 types)    │  │   (50+ types)    │  │   (10+ types)     │          │\n│   │                │  │                  │  │                   │          │\n│   │  • AWS         │  │  • Kubernetes    │  │  • Buildkit       │          │\n│   │  • UpCloud     │  │  • Containerd    │  │  • Web cluster    │          │\n│   │  • Local       │  │  • Databases     │  │  • CI/CD          │          │\n│   │                │  │  • Monitoring    │  │                   │          │\n│   └────────────────┘  └──────────────────┘  └───────────────────┘          │\n│                                                                            │\n│  ┌──────────────────────────────────────────────────────────────────┐      │\n│  │            Extension Distribution (OCI Registry)                 │      │\n│  │   • Zot (local development)                                      │      │\n│  │   • Harbor (multi-user/enterprise)                               │      │\n│  └──────────────────────────────────────────────────────────────────┘      │\n│                                                                            │\n└──────────────────────────────────┬─────────────────────────────────────────┘\n                                   │\n┌──────────────────────────────────┴─────────────────────────────────────────┐\n│                      INFRASTRUCTURE LAYER                                  │\n├────────────────────────────────────────────────────────────────────────────┤\n│                                                                            │\n│   ┌────────────────┐  ┌──────────────────┐  ┌───────────────────┐          │\n│   │  Cloud (AWS)   │  │ Cloud (UpCloud)  │  │  Local (Docker)   │          │\n│   │                │  │                  │  │                   │          │\n│   │  • EC2         │  │  • Servers       │  │  • Containers     │          │\n│   │  • EKS         │  │  • LoadBalancer  │  │  • Local K8s      │          │\n│   │  • RDS         │  │  • Networking    │  │  • Processes      │          │\n│   └────────────────┘  └──────────────────┘  └───────────────────┘          │\n│                                                                            │\n└────────────────────────────────────────────────────────────────────────────┘\n```\n\n### Multi-Repository Architecture\n\nThe system is organized into three separate repositories:\n\n#### **provisioning-core**\n\n```\nCore system functionality\n├── CLI interface (Nushell entry point)\n├── Core libraries (lib_provisioning)\n├── Base Nickel schemas\n├── Configuration system\n├── Workflow engine\n└── Build/distribution tools\n```\n\n**Distribution**: `oci://registry/provisioning-core:v3.5.0`\n\n#### **provisioning-extensions**\n\n```\nAll provider, taskserv, cluster extensions\n├── providers/\n│   ├── aws/\n│   ├── upcloud/\n│   └── local/\n├── taskservs/\n│   ├── kubernetes/\n│   ├── containerd/\n│   ├── postgres/\n│   └── (50+ more)\n└── clusters/\n    ├── buildkit/\n    ├── web/\n    └── (10+ more)\n```\n\n**Distribution**: Each extension as separate OCI artifact\n\n- `oci://registry/provisioning-extensions/kubernetes:1.28.0`\n- `oci://registry/provisioning-extensions/aws:2.0.0`\n\n#### **provisioning-platform**\n\n```\nPlatform services\n├── orchestrator/      (Rust)\n├── control-center/    (Rust/Yew)\n├── mcp-server/        (Rust)\n└── api-gateway/       (Rust)\n```\n\n**Distribution**: Docker images in OCI registry\n\n- `oci://registry/provisioning-platform/orchestrator:v1.2.0`\n\n---\n\n## Component Architecture\n\n### Core Components\n\n#### 1. **CLI Interface** (Nushell)\n\n**Location**: `provisioning/core/cli/provisioning`\n\n**Purpose**: Primary user interface for all provisioning operations\n\n**Architecture**:\n\n```\nMain CLI (211 lines)\n    ↓\nCommand Dispatcher (264 lines)\n    ↓\nDomain Handlers (7 modules)\n    ├── infrastructure.nu (117 lines)\n    ├── orchestration.nu (64 lines)\n    ├── development.nu (72 lines)\n    ├── workspace.nu (56 lines)\n    ├── generation.nu (78 lines)\n    ├── utilities.nu (157 lines)\n    └── configuration.nu (316 lines)\n```\n\n**Key Features**:\n\n- 80+ command shortcuts\n- Bi-directional help system\n- Centralized flag handling\n- Domain-driven design\n\n#### 2. **Configuration System** (Nickel + TOML)\n\n**Hierarchical Loading**:\n\n```\n1. System defaults     (config.defaults.toml)\n2. User config         (~/.provisioning/config.user.toml)\n3. Workspace config    (workspace/config/provisioning.yaml)\n4. Environment config  (workspace/config/{env}-defaults.toml)\n5. Infrastructure config (workspace/infra/{name}/config.toml)\n6. Runtime overrides   (CLI flags, ENV variables)\n```\n\n**Variable Interpolation**:\n\n- `{{paths.base}}` - Path references\n- `{{env.HOME}}` - Environment variables\n- `{{now.date}}` - Dynamic values\n- `{{git.branch}}` - Git context\n\n#### 3. **Orchestrator** (Rust)\n\n**Location**: `provisioning/platform/orchestrator/`\n\n**Architecture**:\n\n```\nsrc/\n├── main.rs              // Entry point\n├── api/\n│   ├── routes.rs        // HTTP routes\n│   ├── workflows.rs     // Workflow endpoints\n│   └── batch.rs         // Batch endpoints\n├── workflow/\n│   ├── engine.rs        // Workflow execution\n│   ├── state.rs         // State management\n│   └── checkpoint.rs    // Checkpoint/recovery\n├── task_queue/\n│   ├── queue.rs         // File-based queue\n│   ├── priority.rs      // Priority scheduling\n│   └── retry.rs         // Retry logic\n├── health/\n│   └── monitor.rs       // Health checks\n├── nushell/\n│   └── bridge.rs        // Nu execution bridge\n└── test_environment/    // Test env management\n    ├── container_manager.rs\n    ├── test_orchestrator.rs\n    └── topologies.rs\n```\n\n**Key Features**:\n\n- File-based task queue (reliable, simple)\n- Checkpoint-based recovery\n- Priority scheduling\n- REST API (HTTP/WebSocket)\n- Nushell script execution bridge\n\n#### 4. **Workflow Engine** (Nushell)\n\n**Location**: `provisioning/core/nulib/workflows/`\n\n**Workflow Types**:\n\n```\nworkflows/\n├── server_create.nu     // Server provisioning\n├── taskserv.nu          // Task service management\n├── cluster.nu           // Cluster deployment\n├── batch.nu             // Batch operations\n└── management.nu        // Workflow monitoring\n```\n\n**Batch Workflow Features**:\n\n- Provider-agnostic (mix AWS, UpCloud, local)\n- Dependency resolution (hard/soft dependencies)\n- Parallel execution (configurable limits)\n- Rollback support\n- Real-time monitoring\n\n#### 5. **Extension System**\n\n**Extension Types**:\n\n| Type | Count | Purpose | Example |\n| ------ | ------- | --------- | --------- |\n| **Providers** | 3 | Cloud platform integration | AWS, UpCloud, Local |\n| **Task Services** | 50+ | Infrastructure components | Kubernetes, Postgres |\n| **Clusters** | 10+ | Complete configurations | Buildkit, Web cluster |\n\n**Extension Structure**:\n\n```\nextension-name/\n├── schemas/\n│   ├── main.ncl             // Main schema\n│   ├── contracts.ncl        // Contract definitions\n│   ├── defaults.ncl         // Default values\n│   └── version.ncl          // Version management\n├── scripts/\n│   ├── install.nu           // Installation logic\n│   ├── check.nu             // Health check\n│   └── uninstall.nu         // Cleanup\n├── templates/               // Config templates\n├── docs/                    // Documentation\n├── tests/                   // Extension tests\n└── manifest.yaml            // Extension metadata\n```\n\n**OCI Distribution**:\nEach extension packaged as OCI artifact:\n\n- Nickel schemas\n- Nushell scripts\n- Templates\n- Documentation\n- Manifest\n\n#### 6. **Module and Layer System**\n\n**Module System**:\n\n```\n# Discover available extensions\nprovisioning module discover taskservs\n\n# Load into workspace\nprovisioning module load taskserv my-workspace kubernetes containerd\n\n# List loaded modules\nprovisioning module list taskserv my-workspace\n```\n\n**Layer System** (Configuration Inheritance):\n\n```\nLayer 1: Core     (provisioning/extensions/{type}/{name})\n    ↓\nLayer 2: Workspace (workspace/extensions/{type}/{name})\n    ↓\nLayer 3: Infrastructure (workspace/infra/{infra}/extensions/{type}/{name})\n```\n\n**Resolution Priority**: Infrastructure → Workspace → Core\n\n#### 7. **Dependency Resolution**\n\n**Algorithm**: Topological sort with cycle detection\n\n**Features**:\n\n- Hard dependencies (must exist)\n- Soft dependencies (optional enhancement)\n- Conflict detection\n- Circular dependency prevention\n- Version compatibility checking\n\n**Example**:\n\n```\nlet { TaskservDependencies } = import "provisioning/dependencies.ncl" in\n{\n  kubernetes = TaskservDependencies {\n    name = "kubernetes",\n    version = "1.28.0",\n    requires = ["containerd", "etcd", "os"],\n    optional = ["cilium", "helm"],\n    conflicts = ["docker", "podman"],\n  }\n}\n```\n\n#### 8. **Service Management**\n\n**Supported Services**:\n\n| Service | Type | Category | Purpose |\n| --------- | ------ | ---------- | --------- |\n| orchestrator | Platform | Orchestration | Workflow coordination |\n| control-center | Platform | UI | Web management interface |\n| coredns | Infrastructure | DNS | Local DNS resolution |\n| gitea | Infrastructure | Git | Self-hosted Git service |\n| oci-registry | Infrastructure | Registry | OCI artifact storage |\n| mcp-server | Platform | API | Model Context Protocol |\n| api-gateway | Platform | API | Unified API access |\n\n**Lifecycle Management**:\n\n```\n# Start all auto-start services\nprovisioning platform start\n\n# Start specific service (with dependencies)\nprovisioning platform start orchestrator\n\n# Check health\nprovisioning platform health\n\n# View logs\nprovisioning platform logs orchestrator --follow\n```\n\n#### 9. **Test Environment Service**\n\n**Architecture**:\n\n```\nUser Command (CLI)\n    ↓\nTest Orchestrator (Rust)\n    ↓\nContainer Manager (bollard)\n    ↓\nDocker API\n    ↓\nIsolated Test Containers\n```\n\n**Test Types**:\n\n- Single taskserv testing\n- Server simulation (multiple taskservs)\n- Multi-node cluster topologies\n\n**Topology Templates**:\n\n- `kubernetes_3node` - 3-node HA cluster\n- `kubernetes_single` - All-in-one K8s\n- `etcd_cluster` - 3-node etcd\n- `postgres_redis` - Database stack\n\n---\n\n## Mode Architecture\n\n### Mode-Based System Overview\n\nThe platform supports four operational modes that adapt the system from individual development to enterprise production.\n\n### Mode Comparison\n\n```\n┌───────────────────────────────────────────────────────────────────────┐\n│                        MODE ARCHITECTURE                              │\n├───────────────┬───────────────┬───────────────┬───────────────────────┤\n│    SOLO       │  MULTI-USER   │    CI/CD      │    ENTERPRISE         │\n├───────────────┼───────────────┼───────────────┼───────────────────────┤\n│               │               │               │                       │\n│  Single Dev   │  Team (5-20)  │  Pipelines    │  Production           │\n│               │               │               │                       │\n│  ┌─────────┐  │ ┌──────────┐  │ ┌──────────┐  │ ┌──────────────────┐  │\n│  │ No Auth │  │ │Token(JWT)│  │ │Token(1h) │  │ │  mTLS (TLS 1.3)  │  │\n│  └─────────┘  │ └──────────┘  │ └──────────┘  │ └──────────────────┘  │\n│               │               │               │                       │\n│  ┌─────────┐  │ ┌──────────┐  │ ┌──────────┐  │ ┌──────────────────┐  │\n│  │ Local   │  │ │ Remote   │  │ │ Remote   │  │ │ Kubernetes (HA)  │  │\n│  │ Binary  │  │ │ Docker   │  │ │ K8s      │  │ │ Multi-AZ         │  │\n│  └─────────┘  │ └──────────┘  │ └──────────┘  │ └──────────────────┘  │\n│               │               │               │                       │\n│  ┌─────────┐  │ ┌──────────┐  │ ┌──────────┐  │ ┌──────────────────┐  │\n│  │ Local   │  │ │ OCI (Zot)│  │ │OCI(Harbor│  │ │ OCI (Harbor HA)  │  │\n│  │ Files   │  │ │ or Harbor│  │ │ required)│  │ │ + Replication    │  │\n│  └─────────┘  │ └──────────┘  │ └──────────┘  │ └──────────────────┘  │\n│               │               │               │                       │\n│  ┌─────────┐  │ ┌──────────┐  │ ┌──────────-┐ │ ┌──────────────────┐  │\n│  │ None    │  │ │ Gitea    │  │ │ Disabled  │ │ │ etcd (mandatory) │  │\n│  │         │  │ │(optional)│  │ │(stateless)| │ │                  │  │\n│  └─────────┘  │ └──────────┘  │ └─────────-─┘ │ └──────────────────┘  │\n│               │               │               │                       │\n│  Unlimited    │  10 srv, 32   │  5 srv, 16    │ 20 srv, 64 cores      │\n│               │ cores, 128 GB  │ cores, 64 GB   │ 256 GB per user        │\n│               │               │               │                       │\n└───────────────┴───────────────┴───────────────┴───────────────────────┘\n```\n\n### Mode Configuration\n\n**Mode Templates**: `workspace/config/modes/{mode}.yaml`\n\n**Active Mode**: `~/.provisioning/config/active-mode.yaml`\n\n**Switching Modes**:\n\n```\n# Check current mode\nprovisioning mode current\n\n# Switch to another mode\nprovisioning mode switch multi-user\n\n# Validate mode requirements\nprovisioning mode validate enterprise\n```\n\n### Mode-Specific Workflows\n\n#### Solo Mode\n\n```\n# 1. Default mode, no setup needed\nprovisioning workspace init\n\n# 2. Start local orchestrator\nprovisioning platform start orchestrator\n\n# 3. Create infrastructure\nprovisioning server create\n```\n\n#### Multi-User Mode\n\n```\n# 1. Switch mode and authenticate\nprovisioning mode switch multi-user\nprovisioning auth login\n\n# 2. Lock workspace\nprovisioning workspace lock my-infra\n\n# 3. Pull extensions from OCI\nprovisioning extension pull upcloud kubernetes\n\n# 4. Work...\n\n# 5. Unlock workspace\nprovisioning workspace unlock my-infra\n```\n\n#### CI/CD Mode\n\n```\n# GitLab CI\ndeploy:\n  stage: deploy\n  script:\n    - export PROVISIONING_MODE=cicd\n    - echo "$TOKEN" > /var/run/secrets/provisioning/token\n    - provisioning validate --all\n    - provisioning test quick kubernetes\n    - provisioning server create --check\n    - provisioning server create\n  after_script:\n    - provisioning workspace cleanup\n```\n\n#### Enterprise Mode\n\n```\n# 1. Switch to enterprise, verify K8s\nprovisioning mode switch enterprise\nkubectl get pods -n provisioning-system\n\n# 2. Request workspace (approval required)\nprovisioning workspace request prod-deployment\n\n# 3. After approval, lock with etcd\nprovisioning workspace lock prod-deployment --provider etcd\n\n# 4. Pull verified extensions\nprovisioning extension pull upcloud --verify-signature\n\n# 5. Deploy\nprovisioning infra create --check\nprovisioning infra create\n\n# 6. Release\nprovisioning workspace unlock prod-deployment\n```\n\n---\n\n## Network Architecture\n\n### Service Communication\n\n```\n┌──────────────────────────────────────────────────────────────────────┐\n│                         NETWORK LAYER                                 │\n├──────────────────────────────────────────────────────────────────────┤\n│                                                                        │\n│  ┌───────────────────────┐          ┌──────────────────────────┐     │\n│  │   Ingress/Load        │          │    API Gateway           │     │\n│  │   Balancer            │──────────│   (Optional)             │     │\n│  └───────────────────────┘          └──────────────────────────┘     │\n│              │                                    │                   │\n│              │                                    │                   │\n│  ┌───────────┴────────────────────────────────────┴──────────┐       │\n│  │                 Service Mesh (Optional)                    │       │\n│  │           (mTLS, Circuit Breaking, Retries)               │       │\n│  └────┬──────────┬───────────┬────────────┬──────────────┬───┘       │\n│       │          │           │            │              │            │\n│  ┌────┴─────┐ ┌─┴────────┐ ┌┴─────────┐ ┌┴──────────┐ ┌┴───────┐   │\n│  │ Orchestr │ │ Control  │ │ CoreDNS  │ │   Gitea   │ │  OCI   │   │\n│  │   ator   │ │ Center   │ │          │ │           │ │Registry│   │\n│  │          │ │          │ │          │ │           │ │        │   │\n│  │ :9090    │ │ :3000    │ │ :5353    │ │ :3001     │ │ :5000  │   │\n│  └──────────┘ └──────────┘ └──────────┘ └───────────┘ └────────┘   │\n│                                                                        │\n│  ┌────────────────────────────────────────────────────────────┐       │\n│  │              DNS Resolution (CoreDNS)                       │       │\n│  │  • *.prov.local  →  Internal services                      │       │\n│  │  • *.infra.local →  Infrastructure nodes                   │       │\n│  └────────────────────────────────────────────────────────────┘       │\n│                                                                        │\n└──────────────────────────────────────────────────────────────────────┘\n```\n\n### Port Allocation\n\n| Service | Port | Protocol | Purpose |\n| --------- | ------ | ---------- | --------- |\n| Orchestrator | 8080 | HTTP/WS | REST API, WebSocket |\n| Control Center | 3000 | HTTP | Web UI |\n| CoreDNS | 5353 | UDP/TCP | DNS resolution |\n| Gitea | 3001 | HTTP | Git operations |\n| OCI Registry (Zot) | 5000 | HTTP | OCI artifacts |\n| OCI Registry (Harbor) | 443 | HTTPS | OCI artifacts (prod) |\n| MCP Server | 8081 | HTTP | MCP protocol |\n| API Gateway | 8082 | HTTP | Unified API |\n\n### Network Security\n\n**Solo Mode**:\n\n- Localhost-only bindings\n- No authentication\n- No encryption\n\n**Multi-User Mode**:\n\n- Token-based authentication (JWT)\n- TLS for external access\n- Firewall rules\n\n**CI/CD Mode**:\n\n- Token authentication (short-lived)\n- Full TLS encryption\n- Network isolation\n\n**Enterprise Mode**:\n\n- mTLS for all connections\n- Network policies (Kubernetes)\n- Zero-trust networking\n- Audit logging\n\n---\n\n## Data Architecture\n\n### Data Storage\n\n```\n┌────────────────────────────────────────────────────────────────┐\n│                     DATA LAYER                                  │\n├────────────────────────────────────────────────────────────────┤\n│                                                                  │\n│  ┌─────────────────────────────────────────────────────────┐   │\n│  │            Configuration Data (Hierarchical)             │   │\n│  │                                                           │   │\n│  │  ~/.provisioning/                                        │   │\n│  │  ├── config.user.toml       (User preferences)          │   │\n│  │  └── config/                                             │   │\n│  │      ├── active-mode.yaml   (Active mode)               │   │\n│  │      └── user_config.yaml   (Workspaces, preferences)   │   │\n│  │                                                           │   │\n│  │  workspace/                                              │   │\n│  │  ├── config/                                             │   │\n│  │  │   ├── provisioning.yaml  (Workspace config)          │   │\n│  │  │   └── modes/*.yaml       (Mode templates)            │   │\n│  │  └── infra/{name}/                                       │   │\n│  │      ├── main.ncl           (Infrastructure Nickel)     │   │\n│  │      └── config.toml        (Infra-specific)            │   │\n│  └─────────────────────────────────────────────────────────┘   │\n│                                                                  │\n│  ┌─────────────────────────────────────────────────────────┐   │\n│  │            State Data (Runtime)                          │   │\n│  │                                                           │   │\n│  │  ~/.provisioning/orchestrator/data/                      │   │\n│  │  ├── tasks/                  (Task queue)                │   │\n│  │  ├── workflows/              (Workflow state)            │   │\n│  │  └── checkpoints/            (Recovery points)           │   │\n│  │                                                           │   │\n│  │  ~/.provisioning/services/                               │   │\n│  │  ├── pids/                   (Process IDs)               │   │\n│  │  ├── logs/                   (Service logs)              │   │\n│  │  └── state/                  (Service state)             │   │\n│  └─────────────────────────────────────────────────────────┘   │\n│                                                                  │\n│  ┌─────────────────────────────────────────────────────────┐   │\n│  │            Cache Data (Performance)                      │   │\n│  │                                                           │   │\n│  │  ~/.provisioning/cache/                                  │   │\n│  │  ├── oci/                    (OCI artifacts)             │   │\n│  │  ├── schemas/                (Nickel compiled)           │   │\n│  │  └── modules/                (Module cache)              │   │\n│  └─────────────────────────────────────────────────────────┘   │\n│                                                                  │\n│  ┌─────────────────────────────────────────────────────────┐   │\n│  │            Extension Data (OCI Artifacts)                │   │\n│  │                                                           │   │\n│  │  OCI Registry (localhost:5000 or harbor.company.com)    │   │\n│  │  ├── provisioning-core:v3.5.0                           │   │\n│  │  ├── provisioning-extensions/                           │   │\n│  │  │   ├── kubernetes:1.28.0                              │   │\n│  │  │   ├── aws:2.0.0                                      │   │\n│  │  │   └── (100+ artifacts)                               │   │\n│  │  └── provisioning-platform/                             │   │\n│  │      ├── orchestrator:v1.2.0                            │   │\n│  │      └── (4 service images)                             │   │\n│  └─────────────────────────────────────────────────────────┘   │\n│                                                                  │\n│  ┌─────────────────────────────────────────────────────────┐   │\n│  │            Secrets (Encrypted)                           │   │\n│  │                                                           │   │\n│  │  workspace/secrets/                                      │   │\n│  │  ├── keys.yaml.enc           (SOPS-encrypted)           │   │\n│  │  ├── ssh-keys/               (SSH keys)                 │   │\n│  │  └── tokens/                 (API tokens)               │   │\n│  │                                                           │   │\n│  │  KMS Integration (Enterprise):                          │   │\n│  │  • AWS KMS                                               │   │\n│  │  • HashiCorp Vault                                       │   │\n│  │  • Age encryption (local)                                │   │\n│  └─────────────────────────────────────────────────────────┘   │\n│                                                                  │\n└────────────────────────────────────────────────────────────────┘\n```\n\n### Data Flow\n\n**Configuration Loading**:\n\n```\n1. Load system defaults (config.defaults.toml)\n2. Merge user config (~/.provisioning/config.user.toml)\n3. Load workspace config (workspace/config/provisioning.yaml)\n4. Load environment config (workspace/config/{env}-defaults.toml)\n5. Load infrastructure config (workspace/infra/{name}/config.toml)\n6. Apply runtime overrides (ENV variables, CLI flags)\n```\n\n**State Persistence**:\n\n```\nWorkflow execution\n    ↓\nCreate checkpoint (JSON)\n    ↓\nSave to ~/.provisioning/orchestrator/data/checkpoints/\n    ↓\nOn failure, load checkpoint and resume\n```\n\n**OCI Artifact Flow**:\n\n```\n1. Package extension (oci-package.nu)\n2. Push to OCI registry (provisioning oci push)\n3. Extension stored as OCI artifact\n4. Pull when needed (provisioning oci pull)\n5. Cache locally (~/.provisioning/cache/oci/)\n```\n\n---\n\n## Security Architecture\n\n### Security Layers\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│                     SECURITY ARCHITECTURE                        │\n├─────────────────────────────────────────────────────────────────┤\n│                                                                   │\n│  ┌────────────────────────────────────────────────────────┐     │\n│  │  Layer 1: Authentication & Authorization               │     │\n│  │                                                          │     │\n│  │  Solo:       None (local development)                  │     │\n│  │  Multi-user: JWT tokens (24h expiry)                   │     │\n│  │  CI/CD:      CI-injected tokens (1h expiry)            │     │\n│  │  Enterprise: mTLS (TLS 1.3, mutual auth)               │     │\n│  └────────────────────────────────────────────────────────┘     │\n│                                                                   │\n│  ┌────────────────────────────────────────────────────────┐     │\n│  │  Layer 2: Encryption                                    │     │\n│  │                                                          │     │\n│  │  In Transit:                                            │     │\n│  │  • TLS 1.3 (multi-user, CI/CD, enterprise)             │     │\n│  │  • mTLS (enterprise)                                    │     │\n│  │                                                          │     │\n│  │  At Rest:                                               │     │\n│  │  • SOPS + Age (secrets encryption)                      │     │\n│  │  • KMS integration (CI/CD, enterprise)                  │     │\n│  │  • Encrypted filesystems (enterprise)                   │     │\n│  └────────────────────────────────────────────────────────┘     │\n│                                                                   │\n│  ┌────────────────────────────────────────────────────────┐     │\n│  │  Layer 3: Secret Management                             │     │\n│  │                                                          │     │\n│  │  • SOPS for file encryption                             │     │\n│  │  • Age for key management                               │     │\n│  │  • KMS integration (AWS KMS, Vault)                     │     │\n│  │  • SSH key storage (KMS-backed)                         │     │\n│  │  • API token management                                 │     │\n│  └────────────────────────────────────────────────────────┘     │\n│                                                                   │\n│  ┌────────────────────────────────────────────────────────┐     │\n│  │  Layer 4: Access Control                                │     │\n│  │                                                          │     │\n│  │  • RBAC (Role-Based Access Control)                     │     │\n│  │  • Workspace isolation                                   │     │\n│  │  • Workspace locking (Gitea, etcd)                      │     │\n│  │  • Resource quotas (per-user limits)                    │     │\n│  └────────────────────────────────────────────────────────┘     │\n│                                                                   │\n│  ┌────────────────────────────────────────────────────────┐     │\n│  │  Layer 5: Network Security                              │     │\n│  │                                                          │     │\n│  │  • Network policies (Kubernetes)                        │     │\n│  │  • Firewall rules                                       │     │\n│  │  • Zero-trust networking (enterprise)                   │     │\n│  │  • Service mesh (optional, mTLS)                        │     │\n│  └────────────────────────────────────────────────────────┘     │\n│                                                                   │\n│  ┌────────────────────────────────────────────────────────┐     │\n│  │  Layer 6: Audit & Compliance                            │     │\n│  │                                                          │     │\n│  │  • Audit logs (all operations)                          │     │\n│  │  • Compliance policies (SOC2, ISO27001)                 │     │\n│  │  • Image signing (cosign, notation)                     │     │\n│  │  • Vulnerability scanning (Harbor)                      │     │\n│  └────────────────────────────────────────────────────────┘     │\n│                                                                   │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n### Secret Management\n\n**SOPS Integration**:\n\n```\n# Edit encrypted file\nprovisioning sops workspace/secrets/keys.yaml.enc\n\n# Encryption happens automatically on save\n# Decryption happens automatically on load\n```\n\n**KMS Integration** (Enterprise):\n\n```\n# workspace/config/provisioning.yaml\nsecrets:\n  provider: "kms"\n  kms:\n    type: "aws"  # or "vault"\n    region: "us-east-1"\n    key_id: "arn:aws:kms:..."\n```\n\n### Image Signing and Verification\n\n**CI/CD Mode** (Required):\n\n```\n# Sign OCI artifact\ncosign sign oci://registry/kubernetes:1.28.0\n\n# Verify signature\ncosign verify oci://registry/kubernetes:1.28.0\n```\n\n**Enterprise Mode** (Mandatory):\n\n```\n# Pull with verification\nprovisioning extension pull kubernetes --verify-signature\n\n# System blocks unsigned artifacts\n```\n\n---\n\n## Deployment Architecture\n\n### Deployment Modes\n\n#### 1. **Binary Deployment** (Solo, Multi-user)\n\n```\nUser Machine\n├── ~/.provisioning/bin/\n│   ├── provisioning-orchestrator\n│   ├── provisioning-control-center\n│   └── ...\n├── ~/.provisioning/orchestrator/data/\n├── ~/.provisioning/services/\n└── Process Management (PID files, logs)\n```\n\n**Pros**: Simple, fast startup, no Docker dependency\n**Cons**: Platform-specific binaries, manual updates\n\n#### 2. **Docker Deployment** (Multi-user, CI/CD)\n\n```\nDocker Daemon\n├── Container: provisioning-orchestrator\n├── Container: provisioning-control-center\n├── Container: provisioning-coredns\n├── Container: provisioning-gitea\n├── Container: provisioning-oci-registry\n└── Volumes: ~/.provisioning/data/\n```\n\n**Pros**: Consistent environment, easy updates\n**Cons**: Requires Docker, resource overhead\n\n#### 3. **Docker Compose Deployment** (Multi-user)\n\n```\n# provisioning/platform/docker-compose.yaml\nservices:\n  orchestrator:\n    image: provisioning-platform/orchestrator:v1.2.0\n    ports:\n      - "8080:9090"\n    volumes:\n      - orchestrator-data:/data\n\n  control-center:\n    image: provisioning-platform/control-center:v1.2.0\n    ports:\n      - "3000:3000"\n    depends_on:\n      - orchestrator\n\n  coredns:\n    image: coredns/coredns:1.11.1\n    ports:\n      - "5353:53/udp"\n\n  gitea:\n    image: gitea/gitea:1.20\n    ports:\n      - "3001:3000"\n\n  oci-registry:\n    image: ghcr.io/project-zot/zot:latest\n    ports:\n      - "5000:5000"\n```\n\n**Pros**: Easy multi-service orchestration, declarative\n**Cons**: Local only, no HA\n\n#### 4. **Kubernetes Deployment** (CI/CD, Enterprise)\n\n```\n# Namespace: provisioning-system\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: orchestrator\nspec:\n  replicas: 3  # HA\n  selector:\n    matchLabels:\n      app: orchestrator\n  template:\n    metadata:\n      labels:\n        app: orchestrator\n    spec:\n      containers:\n      - name: orchestrator\n        image: harbor.company.com/provisioning-platform/orchestrator:v1.2.0\n        ports:\n        - containerPort: 8080\n        env:\n        - name: RUST_LOG\n          value: "info"\n        volumeMounts:\n        - name: data\n          mountPath: /data\n        livenessProbe:\n          httpGet:\n            path: /health\n            port: 8080\n        readinessProbe:\n          httpGet:\n            path: /health\n            port: 8080\n      volumes:\n      - name: data\n        persistentVolumeClaim:\n          claimName: orchestrator-data\n```\n\n**Pros**: HA, scalability, production-ready\n**Cons**: Complex setup, Kubernetes required\n\n#### 5. **Remote Deployment** (All modes)\n\n```\n# Connect to remotely-running services\nservices:\n  orchestrator:\n    deployment:\n      mode: "remote"\n      remote:\n        endpoint: "https://orchestrator.company.com"\n        tls_enabled: true\n        auth_token_path: "~/.provisioning/tokens/orchestrator.token"\n```\n\n**Pros**: No local resources, centralized\n**Cons**: Network dependency, latency\n\n---\n\n## Integration Architecture\n\n### Integration Patterns\n\n#### 1. **Hybrid Language Integration** (Rust ↔ Nushell)\n\n```\nRust Orchestrator\n    ↓ (HTTP API)\nNushell CLI\n    ↓ (exec via bridge)\nNushell Business Logic\n    ↓ (returns JSON)\nRust Orchestrator\n    ↓ (updates state)\nFile-based Task Queue\n```\n\n**Communication**: HTTP API + stdin/stdout JSON\n\n#### 2. **Provider Abstraction**\n\n```\nUnified Provider Interface\n├── create_server(config) -> Server\n├── delete_server(id) -> bool\n├── list_servers() -> [Server]\n└── get_server_status(id) -> Status\n\nProvider Implementations:\n├── AWS Provider (aws-sdk-rust, aws cli)\n├── UpCloud Provider (upcloud API)\n└── Local Provider (Docker, libvirt)\n```\n\n#### 3. **OCI Registry Integration**\n\n```\nExtension Development\n    ↓\nPackage (oci-package.nu)\n    ↓\nPush (provisioning oci push)\n    ↓\nOCI Registry (Zot/Harbor)\n    ↓\nPull (provisioning oci pull)\n    ↓\nCache (~/.provisioning/cache/oci/)\n    ↓\nLoad into Workspace\n```\n\n#### 4. **Gitea Integration** (Multi-user, Enterprise)\n\n```\nWorkspace Operations\n    ↓\nCheck Lock Status (Gitea API)\n    ↓\nAcquire Lock (Create lock file in Git)\n    ↓\nPerform Changes\n    ↓\nCommit + Push\n    ↓\nRelease Lock (Delete lock file)\n```\n\n**Benefits**:\n\n- Distributed locking\n- Change tracking via Git history\n- Collaboration features\n\n#### 5. **CoreDNS Integration**\n\n```\nService Registration\n    ↓\nUpdate CoreDNS Corefile\n    ↓\nReload CoreDNS\n    ↓\nDNS Resolution Available\n\nZones:\n├── *.prov.local     (Internal services)\n├── *.infra.local    (Infrastructure nodes)\n└── *.test.local     (Test environments)\n```\n\n---\n\n## Performance and Scalability\n\n### Performance Characteristics\n\n| Metric | Value | Notes |\n| -------- | ------- | ------- |\n| **CLI Startup Time** | < 100 ms | Nushell cold start |\n| **CLI Response Time** | < 50 ms | Most commands |\n| **Workflow Submission** | < 200 ms | To orchestrator |\n| **Task Processing** | 10-50/sec | Orchestrator throughput |\n| **Batch Operations** | Up to 100 servers | Parallel execution |\n| **OCI Pull Time** | 1-5s | Cached: <100 ms |\n| **Configuration Load** | < 500 ms | Full hierarchy |\n| **Health Check Interval** | 10s | Configurable |\n\n### Scalability Limits\n\n**Solo Mode**:\n\n- Unlimited local resources\n- Limited by machine capacity\n\n**Multi-User Mode**:\n\n- 10 servers per user\n- 32 cores, 128 GB RAM per user\n- 5-20 concurrent users\n\n**CI/CD Mode**:\n\n- 5 servers per pipeline\n- 16 cores, 64 GB RAM per pipeline\n- 100+ concurrent pipelines\n\n**Enterprise Mode**:\n\n- 20 servers per user\n- 64 cores, 256 GB RAM per user\n- 1000+ concurrent users\n- Horizontal scaling via Kubernetes\n\n### Optimization Strategies\n\n**Caching**:\n\n- OCI artifacts cached locally\n- Nickel compilation cached\n- Module resolution cached\n\n**Parallel Execution**:\n\n- Batch operations with configurable limits\n- Dependency-aware parallel starts\n- Workflow DAG execution\n\n**Incremental Operations**:\n\n- Only update changed resources\n- Checkpoint-based recovery\n- Delta synchronization\n\n---\n\n## Evolution and Roadmap\n\n### Version History\n\n| Version | Date | Major Features |\n| --------- | ------ | ---------------- |\n| **v3.5.0** | 2025-10-06 | Mode system, OCI distribution, comprehensive docs |\n| **v3.4.0** | 2025-10-06 | Test environment service |\n| **v3.3.0** | 2025-09-30 | Interactive guides |\n| **v3.2.0** | 2025-09-30 | Modular CLI refactoring |\n| **v3.1.0** | 2025-09-25 | Batch workflow system |\n| **v3.0.0** | 2025-09-25 | Hybrid orchestrator |\n| **v2.0.5** | 2025-10-02 | Workspace switching |\n| **v2.0.0** | 2025-09-23 | Configuration migration |\n\n### Roadmap (Future Versions)\n\n**v3.6.0** (Q1 2026):\n\n- GraphQL API\n- Advanced RBAC\n- Multi-tenancy\n- Observability enhancements (OpenTelemetry)\n\n**v4.0.0** (Q2 2026):\n\n- Multi-repository split complete\n- Extension marketplace\n- Advanced workflow features (conditional execution, loops)\n- Cost optimization engine\n\n**v4.1.0** (Q3 2026):\n\n- AI-assisted infrastructure generation\n- Policy-as-code (OPA integration)\n- Advanced compliance features\n\n**Long-term Vision**:\n\n- Serverless workflow execution\n- Edge computing support\n- Multi-cloud failover\n- Self-healing infrastructure\n\n---\n\n## Related Documentation\n\n### Architecture\n\n- **[Multi-Repo Architecture](MULTI_REPO_ARCHITECTURE.md)** - Repository organization\n- **[Design Principles](design-principles.md)** - Architectural philosophy\n- **[Integration Patterns](integration-patterns.md)** - Integration details\n- **[Orchestrator Model](orchestrator-integration-model.md)** - Hybrid orchestration\n\n### ADRs\n\n- **[ADR-001](adr-001-project-structure.md)** - Project structure\n- **[ADR-002](adr-002-distribution-strategy.md)** - Distribution strategy\n- **[ADR-003](adr-003-workspace-isolation.md)** - Workspace isolation\n- **[ADR-004](adr-004-hybrid-architecture.md)** - Hybrid architecture\n- **[ADR-005](adr-005-extension-framework.md)** - Extension framework\n- **[ADR-006](adr-006-provisioning-cli-refactoring.md)** - CLI refactoring\n\n### User Guides\n\n- **[Getting Started](../user/getting-started.md)** - First steps\n- **[Mode System](../user/MODE_SYSTEM_QUICK_REFERENCE.md)** - Modes overview\n- **[Service Management](../user/SERVICE_MANAGEMENT_GUIDE.md)** - Services\n- **[OCI Registry](../user/OCI_REGISTRY_GUIDE.md)** - OCI operations\n\n---\n\n**Maintained By**: Architecture Team\n**Review Cycle**: Quarterly\n**Next Review**: 2026-01-06
