# Multi-Repository Architecture with OCI Registry Support\n\n**Version**: 1.0.0\n**Date**: 2025-10-06\n**Status**: Implementation Complete\n\n## Overview\n\nThis document describes the multi-repository architecture for the provisioning system, enabling modular development, independent versioning, and\ndistributed extension management through OCI registry integration.\n\n## Architecture Goals\n\n1. **Separation of Concerns**: Core, Extensions, and Platform in separate repositories\n2. **Independent Versioning**: Each component can be versioned and released independently\n3. **Distributed Development**: Multiple teams can work on different repositories\n4. **OCI-Native Distribution**: Extensions distributed as OCI artifacts\n5. **Dependency Management**: Automated dependency resolution across repositories\n6. **Backward Compatibility**: Support legacy monorepo structure during transition\n\n## Repository Structure\n\n### Repository 1: `provisioning-core`\n\n**Purpose**: Core system functionality - CLI, libraries, base schemas\n\n```\nprovisioning-core/\n├── core/\n│   ├── cli/                    # Command-line interface\n│   │   ├── provisioning        # Main CLI entry point\n│   │   └── module-loader       # Dynamic module loader\n│   ├── nulib/                  # Core Nushell libraries\n│   │   ├── lib_provisioning/   # Core library modules\n│   │   │   ├── config/         # Configuration management\n│   │   │   ├── oci/            # OCI client integration\n│   │   │   ├── dependencies/   # Dependency resolution\n│   │   │   ├── module/         # Module system\n│   │   │   ├── layer/          # Layer system\n│   │   │   └── workspace/      # Workspace management\n│   │   └── workflows/          # Core workflow system\n│   ├── plugins/                # System plugins\n│   └── scripts/                # Utility scripts\n├── schemas/                    # Base Nickel schemas\n│   ├── main.ncl                # Main schema entry\n│   ├── lib.ncl                 # Core library types\n│   ├── settings.ncl            # Settings schema\n│   ├── dependencies.ncl        # Dependency schemas (with OCI support)\n│   ├── server.ncl              # Server schemas\n│   ├── cluster.ncl             # Cluster schemas\n│   └── workflows.ncl           # Workflow schemas\n├── config/                     # Core configuration templates\n├── templates/                  # Core templates\n├── tools/                      # Build and distribution tools\n│   ├── oci-package.nu          # OCI packaging tool\n│   ├── build-core.nu           # Core build script\n│   └── release-core.nu         # Core release script\n├── tests/                      # Core system tests\n└── docs/                       # Core documentation\n    ├── api/                    # API documentation\n    ├── architecture/           # Architecture docs\n    └── development/            # Development guides\n\n```\n\n**Distribution**:\n\n- Published as OCI artifact: `oci://registry/provisioning-core:v3.5.0`\n- Contains all core functionality needed to run the provisioning system\n- Version format: `v{major}.{minor}.{patch}` (for example, v3.5.0)\n\n**CI/CD**:\n\n- Build on commit to main\n- Publish OCI artifact on git tag (v*)\n- Run integration tests before publishing\n- Update changelog automatically\n\n---\n\n### Repository 2: `provisioning-extensions`\n\n**Purpose**: All provider, taskserv, and cluster extensions\n\n```\nprovisioning-extensions/\n├── providers/\n│   ├── aws/\n│   │   ├── schemas/            # Nickel schemas\n│   │   │   ├── manifest.toml   # Nickel dependencies\n│   │   │   ├── aws.ncl         # Main provider schema\n│   │   │   ├── defaults_aws.ncl # AWS defaults\n│   │   │   └── server_aws.ncl  # AWS server schema\n│   │   ├── scripts/            # Nushell scripts\n│   │   │   └── install.nu      # Installation script\n│   │   ├── templates/          # Provider templates\n│   │   ├── docs/               # Provider documentation\n│   │   └── manifest.yaml       # Extension manifest\n│   ├── upcloud/\n│   │   └── (same structure)\n│   └── local/\n│       └── (same structure)\n├── taskservs/\n│   ├── kubernetes/\n│   │   ├── schemas/\n│   │   │   ├── manifest.toml\n│   │   │   ├── kubernetes.ncl  # Main taskserv schema\n│   │   │   ├── version.ncl     # Version management\n│   │   │   └── dependencies.ncl # Taskserv dependencies\n│   │   ├── scripts/\n│   │   │   ├── install.nu      # Installation script\n│   │   │   ├── check.nu        # Health check script\n│   │   │   └── uninstall.nu    # Uninstall script\n│   │   ├── templates/          # Config templates\n│   │   ├── docs/               # Taskserv docs\n│   │   ├── tests/              # Taskserv tests\n│   │   └── manifest.yaml       # Extension manifest\n│   ├── containerd/\n│   ├── cilium/\n│   ├── postgres/\n│   └── (50+ more taskservs...)\n├── clusters/\n│   ├── buildkit/\n│   │   └── (same structure)\n│   ├── web/\n│   └── (other clusters...)\n├── tools/\n│   ├── extension-builder.nu   # Build individual extensions\n│   ├── mass-publish.nu         # Publish all extensions\n│   └── validate-extensions.nu # Validate all extensions\n└── docs/\n    ├── extension-guide.md      # Extension development guide\n    └── publishing.md           # Publishing guide\n\n```\n\n**Distribution**:\nEach extension published separately as OCI artifact:\n\n- `oci://registry/provisioning-extensions/kubernetes:1.28.0`\n- `oci://registry/provisioning-extensions/aws:2.0.0`\n- `oci://registry/provisioning-extensions/buildkit:0.12.0`\n\n**Extension Manifest** (`manifest.yaml`):\n\n```\nname: kubernetes\ntype: taskserv\nversion: 1.28.0\ndescription: Kubernetes container orchestration platform\nauthor: Provisioning Team\nlicense: MIT\nhomepage: https://kubernetes.io\nrepository: https://gitea.example.com/provisioning-extensions/kubernetes\n\ndependencies:\n  containerd: ">=1.7.0"\n  etcd: ">=3.5.0"\n\ntags:\n  - kubernetes\n  - container-orchestration\n  - cncf\n\nplatforms:\n  - linux/amd64\n  - linux/arm64\n\nmin_provisioning_version: "3.0.0"\n```\n\n**CI/CD**:\n\n- Build and publish each extension independently\n- Git tag format: `{extension-type}/{extension-name}/v{version}`\n  - Example: `taskservs/kubernetes/v1.28.0`\n- Automated publishing to OCI registry on tag\n- Run extension-specific tests before publishing\n\n---\n\n### Repository 3: `provisioning-platform`\n\n**Purpose**: Platform services (orchestrator, control-center, MCP server, API gateway)\n\n```\nprovisioning-platform/\n├── orchestrator/               # Rust orchestrator service\n│   ├── src/\n│   ├── Cargo.toml\n│   ├── Dockerfile\n│   └── README.md\n├── control-center/             # Web control center\n│   ├── src/\n│   ├── package.json\n│   ├── Dockerfile\n│   └── README.md\n├── mcp-server/                 # Model Context Protocol server\n│   ├── src/\n│   ├── Cargo.toml\n│   ├── Dockerfile\n│   └── README.md\n├── api-gateway/                # REST API gateway\n│   ├── src/\n│   ├── Cargo.toml\n│   ├── Dockerfile\n│   └── README.md\n├── docker-compose.yml          # Local development stack\n├── kubernetes/                 # K8s deployment manifests\n│   ├── orchestrator.yaml\n│   ├── control-center.yaml\n│   ├── mcp-server.yaml\n│   └── api-gateway.yaml\n└── docs/\n    ├── deployment.md\n    └── api-reference.md\n\n```\n\n**Distribution**:\nStandard Docker images in OCI registry:\n\n- `oci://registry/provisioning-platform/orchestrator:v1.2.0`\n- `oci://registry/provisioning-platform/control-center:v1.2.0`\n- `oci://registry/provisioning-platform/mcp-server:v1.0.0`\n- `oci://registry/provisioning-platform/api-gateway:v1.0.0`\n\n**CI/CD**:\n\n- Build Docker images on commit to main\n- Publish images on git tag (v*)\n- Multi-architecture builds (amd64, arm64)\n- Security scanning before publishing\n\n---\n\n## OCI Registry Integration\n\n### Registry Structure\n\n```\nOCI Registry (localhost:5000 or harbor.company.com)\n├── provisioning-core/\n│   ├── v3.5.0                  # Core system artifact\n│   ├── v3.4.0\n│   └── latest -> v3.5.0\n├── provisioning-extensions/\n│   ├── kubernetes:1.28.0       # Individual extension artifacts\n│   ├── kubernetes:1.27.0\n│   ├── containerd:1.7.0\n│   ├── aws:2.0.0\n│   ├── upcloud:1.5.0\n│   └── (100+ more extensions)\n└── provisioning-platform/\n    ├── orchestrator:v1.2.0     # Platform service images\n    ├── control-center:v1.2.0\n    ├── mcp-server:v1.0.0\n    └── api-gateway:v1.0.0\n\n```\n\n### OCI Artifact Structure\n\nEach extension packaged as OCI artifact:\n\n```\nkubernetes-1.28.0.tar.gz\n├── schemas/                    # Nickel schemas\n│   ├── kubernetes.ncl\n│   ├── version.ncl\n│   └── dependencies.ncl\n├── scripts/                    # Nushell scripts\n│   ├── install.nu\n│   ├── check.nu\n│   └── uninstall.nu\n├── templates/                  # Template files\n│   ├── kubeconfig.j2\n│   └── kubelet-config.yaml.j2\n├── docs/                       # Documentation\n│   └── README.md\n├── manifest.yaml               # Extension manifest\n└── oci-manifest.json           # OCI manifest metadata\n\n```\n\n---\n\n## Dependency Management\n\n### Workspace Configuration\n\n**File**: `workspace/config/provisioning.yaml`\n\n```\n# Core system dependency\ndependencies:\n  core:\n    source: "oci://harbor.company.com/provisioning-core:v3.5.0"\n    # Alternative: source: "gitea://provisioning-core"\n\n  # Extensions repository configuration\n  extensions:\n    source_type: "oci"          # oci, gitea, local\n\n    # OCI registry configuration\n    oci:\n      registry: "localhost:5000"\n      namespace: "provisioning-extensions"\n      tls_enabled: false\n      auth_token_path: "~/.provisioning/tokens/oci"\n\n    # Loaded extension modules\n    modules:\n      providers:\n        - "oci://localhost:5000/provisioning-extensions/aws:2.0.0"\n        - "oci://localhost:5000/provisioning-extensions/upcloud:1.5.0"\n\n      taskservs:\n        - "oci://localhost:5000/provisioning-extensions/kubernetes:1.28.0"\n        - "oci://localhost:5000/provisioning-extensions/containerd:1.7.0"\n        - "oci://localhost:5000/provisioning-extensions/cilium:1.14.0"\n\n      clusters:\n        - "oci://localhost:5000/provisioning-extensions/buildkit:0.12.0"\n\n  # Platform services\n  platform:\n    source_type: "oci"\n\n    oci:\n      registry: "harbor.company.com"\n      namespace: "provisioning-platform"\n\n      images:\n        orchestrator: "harbor.company.com/provisioning-platform/orchestrator:v1.2.0"\n        control_center: "harbor.company.com/provisioning-platform/control-center:v1.2.0"\n\n  # OCI registry configuration\n  registry:\n    type: "oci"                 # oci, gitea, http\n\n    oci:\n      endpoint: "localhost:5000"\n      namespaces:\n        extensions: "provisioning-extensions"\n        nickel: "provisioning-nickel"\n        platform: "provisioning-platform"\n        test: "provisioning-test"\n```\n\n### Dependency Resolution\n\nThe system resolves dependencies in this order:\n\n1. **Parse Configuration**: Read `provisioning.yaml` and extract dependencies\n2. **Resolve Core**: Ensure core system version is compatible\n3. **Resolve Extensions**: For each extension:\n   - Check if already installed and version matches\n   - Pull from OCI registry if needed\n   - Recursively resolve extension dependencies\n4. **Validate Graph**: Check for dependency cycles and conflicts\n5. **Install**: Install extensions in topological order\n\n### Dependency Resolution Commands\n\n```\n# Resolve and install all dependencies\nprovisioning dep resolve\n\n# Check for dependency updates\nprovisioning dep check-updates\n\n# Update specific extension\nprovisioning dep update kubernetes\n\n# Validate dependency graph\nprovisioning dep validate\n\n# Show dependency tree\nprovisioning dep tree kubernetes\n```\n\n---\n\n## OCI Client Operations\n\n### CLI Commands\n\n```\n# Pull extension from OCI registry\nprovisioning oci pull kubernetes:1.28.0\n\n# Push extension to OCI registry\nprovisioning oci push ./extensions/kubernetes kubernetes 1.28.0\n\n# List available extensions\nprovisioning oci list --namespace provisioning-extensions\n\n# Search for extensions\nprovisioning oci search kubernetes\n\n# Show extension versions\nprovisioning oci tags kubernetes\n\n# Inspect extension manifest\nprovisioning oci inspect kubernetes:1.28.0\n\n# Login to OCI registry\nprovisioning oci login localhost:5000 --username _token --password-stdin\n\n# Delete extension\nprovisioning oci delete kubernetes:1.28.0\n\n# Copy extension between registries\nprovisioning oci copy \\n  localhost:5000/provisioning-extensions/kubernetes:1.28.0 \\n  harbor.company.com/provisioning-extensions/kubernetes:1.28.0\n```\n\n### OCI Configuration\n\n```\n# Show OCI configuration\nprovisioning oci config\n\n# Output:\n{\n  tool: "oras"  # or "crane" or "skopeo"\n  registry: "localhost:5000"\n  namespace: {\n    extensions: "provisioning-extensions"\n    platform: "provisioning-platform"\n  }\n  cache_dir: "~/.provisioning/oci-cache"\n  tls_enabled: false\n}\n```\n\n---\n\n## Extension Development Workflow\n\n### 1. Develop Extension\n\n```\n# Create new extension from template\nprovisioning generate extension taskserv redis\n\n# Directory structure created:\n# extensions/taskservs/redis/\n# ├── schemas/\n# │   ├── manifest.toml\n# │   ├── redis.ncl\n# │   ├── version.ncl\n# │   └── dependencies.ncl\n# ├── scripts/\n# │   ├── install.nu\n# │   ├── check.nu\n# │   └── uninstall.nu\n# ├── templates/\n# ├── docs/\n# │   └── README.md\n# ├── tests/\n# └── manifest.yaml\n```\n\n### 2. Test Extension Locally\n\n```\n# Load extension from local path\nprovisioning module load taskserv workspace_dev redis --source local\n\n# Test installation\nprovisioning taskserv create redis --infra test-env --check\n\n# Run extension tests\nprovisioning test extension redis\n```\n\n### 3. Package Extension\n\n```\n# Validate extension structure\nprovisioning oci package validate ./extensions/taskservs/redis\n\n# Package as OCI artifact\nprovisioning oci package ./extensions/taskservs/redis\n\n# Output: redis-1.0.0.tar.gz\n```\n\n### 4. Publish Extension\n\n```\n# Login to registry (one-time)\nprovisioning oci login localhost:5000\n\n# Publish extension\nprovisioning oci push ./extensions/taskservs/redis redis 1.0.0\n\n# Verify publication\nprovisioning oci tags redis\n\n# Output:\n# ┬───────────┬─────────┬───────────────────────────────────────────────────┐\n# │ artifact  │ version │ reference                                         │\n# ├───────────┼─────────┼───────────────────────────────────────────────────┤\n# │ redis     │ 1.0.0   │ localhost:5000/provisioning-extensions/redis:1.0.0│\n# └───────────┴─────────┴───────────────────────────────────────────────────┘\n```\n\n### 5. Use Published Extension\n\n```\n# Add to workspace configuration\n# workspace/config/provisioning.yaml:\n# dependencies:\n#   extensions:\n#     modules:\n#       taskservs:\n#         - "oci://localhost:5000/provisioning-extensions/redis:1.0.0"\n\n# Pull and install\nprovisioning dep resolve\n\n# Extension automatically downloaded and installed\n```\n\n---\n\n## Registry Deployment Options\n\n### Local Registry (Solo Development)\n\n**Using Zot (lightweight OCI registry)**:\n\n```\n# Start local OCI registry\nprovisioning oci-registry start\n\n# Configuration:\n# - Endpoint: localhost:5000\n# - Storage: ~/.provisioning/oci-registry/\n# - No authentication by default\n# - TLS disabled (local only)\n\n# Stop registry\nprovisioning oci-registry stop\n\n# Check status\nprovisioning oci-registry status\n```\n\n### Remote Registry (Multi-User/Enterprise)\n\n**Using Harbor**:\n\n```\n# workspace/config/provisioning.yaml\ndependencies:\n  registry:\n    type: "oci"\n    oci:\n      endpoint: "https://harbor.company.com"\n      namespaces:\n        extensions: "provisioning/extensions"\n        platform: "provisioning/platform"\n      tls_enabled: true\n      auth_token_path: "~/.provisioning/tokens/harbor"\n```\n\n**Features**:\n\n- Multi-user authentication\n- Role-based access control (RBAC)\n- Vulnerability scanning\n- Replication across registries\n- Webhook notifications\n- Image signing (cosign/notation)\n\n---\n\n## Migration from Monorepo\n\n### Phase 1: Parallel Structure (Current)\n\n- Monorepo still exists and works\n- OCI distribution layer added on top\n- Extensions can be loaded from local or OCI\n- No breaking changes\n\n### Phase 2: Gradual Migration\n\n```\n# Migrate extensions one by one\nfor ext in (ls provisioning/extensions/taskservs); do\n  provisioning oci publish $ext.name\ndone\n\n# Update workspace configurations to use OCI\nprovisioning workspace migrate-to-oci workspace_prod\n```\n\n### Phase 3: Repository Split\n\n1. Create `provisioning-core` repository\n   - Extract core/ and schemas/ directories\n   - Set up CI/CD for core publishing\n   - Publish initial OCI artifact\n\n2. Create `provisioning-extensions` repository\n   - Extract extensions/ directory\n   - Set up CI/CD for extension publishing\n   - Publish all extensions to OCI registry\n\n3. Create `provisioning-platform` repository\n   - Extract platform/ directory\n   - Set up Docker image builds\n   - Publish platform services\n\n4. Update workspaces\n   - Reconfigure to use OCI dependencies\n   - Test multi-repo setup\n   - Verify all functionality works\n\n### Phase 4: Deprecate Monorepo\n\n- Archive monorepo\n- Redirect to new repositories\n- Update documentation\n- Announce migration complete\n\n---\n\n## Benefits Summary\n\n### Modularity\n\n✅ Independent repositories for core, extensions, and platform\n✅ Extensions can be developed and versioned separately\n✅ Clear ownership and responsibility boundaries\n\n### Distribution\n\n✅ OCI-native distribution (industry standard)\n✅ Built-in versioning with OCI tags\n✅ Efficient caching with OCI layers\n✅ Works with standard tools (skopeo, crane, oras)\n\n### Security\n\n✅ TLS support for registries\n✅ Authentication and authorization\n✅ Vulnerability scanning (Harbor)\n✅ Image signing (cosign, notation)\n✅ RBAC for access control\n\n### Developer Experience\n\n✅ Simple CLI commands for extension management\n✅ Automatic dependency resolution\n✅ Local testing before publishing\n✅ Easy extension discovery and installation\n\n### Operations\n\n✅ Air-gapped deployments (mirror OCI registry)\n✅ Bandwidth efficient (only download what's needed)\n✅ Version pinning for reproducibility\n✅ Rollback support (use previous versions)\n\n### Ecosystem\n\n✅ Compatible with existing OCI tooling\n✅ Can use public registries (DockerHub, GitHub, etc.)\n✅ Mirror to multiple registries\n✅ Replication for high availability\n\n---\n\n## Implementation Status\n\n| Component | Status | Notes |\n| ----------- | -------- | ------- |\n| **Nickel Schemas** | ✅ Complete | OCI schemas in `dependencies.ncl` |\n| **OCI Client** | ✅ Complete | `oci/client.nu` with skopeo/crane/oras |\n| **OCI Commands** | ✅ Complete | `oci/commands.nu` CLI interface |\n| **Dependency Resolver** | ✅ Complete | `dependencies/resolver.nu` |\n| **OCI Packaging** | ✅ Complete | `tools/oci-package.nu` |\n| **Repository Design** | ✅ Complete | This document |\n| **Migration Plan** | ✅ Complete | Phased approach defined |\n| **Documentation** | ✅ Complete | User guides and API docs |\n| **CI/CD Setup** | ⏳ Pending | Automated publishing pipelines |\n| **Registry Deployment** | ⏳ Pending | Zot/Harbor setup |\n\n---\n\n## Related Documentation\n\n- OCI Packaging Tool - Extension packaging\n- OCI Client Library - OCI operations\n- Dependency Resolver - Dependency management\n- Nickel Schemas - Type definitions\n- [Extension Development Guide](../user/extension-development.md) - How to create extensions\n\n---\n\n**Maintained By**: Architecture Team\n**Review Cycle**: Quarterly\n**Next Review**: 2026-01-06
