provisioning/docs/src/architecture/multi-repo-architecture.md

Multi-Repository Architecture with OCI Registry Support

Version: 1.0.0 Date: 2025-10-06 Status: Implementation Complete

Overview

This document describes the multi-repository architecture for the provisioning system, enabling modular development, independent versioning, and distributed extension management through OCI registry integration.

Architecture Goals

1. Separation of Concerns: Core, Extensions, and Platform in separate repositories
2. Independent Versioning: Each component can be versioned and released independently
3. Distributed Development: Multiple teams can work on different repositories
4. OCI-Native Distribution: Extensions distributed as OCI artifacts
5. Dependency Management: Automated dependency resolution across repositories
6. Backward Compatibility: Support legacy monorepo structure during transition

Repository Structure

Repository 1: provisioning-core

Purpose: Core system functionality - CLI, libraries, base schemas

provisioning-core/
├── core/
│   ├── cli/                    # Command-line interface
│   │   ├── provisioning        # Main CLI entry point
│   │   └── module-loader       # Dynamic module loader
│   ├── nulib/                  # Core Nushell libraries
│   │   ├── lib_provisioning/   # Core library modules
│   │   │   ├── config/         # Configuration management
│   │   │   ├── oci/            # OCI client integration
│   │   │   ├── dependencies/   # Dependency resolution
│   │   │   ├── module/         # Module system
│   │   │   ├── layer/          # Layer system
│   │   │   └── workspace/      # Workspace management
│   │   └── workflows/          # Core workflow system
│   ├── plugins/                # System plugins
│   └── scripts/                # Utility scripts
├── schemas/                    # Base Nickel schemas
│   ├── main.ncl                # Main schema entry
│   ├── lib.ncl                 # Core library types
│   ├── settings.ncl            # Settings schema
│   ├── dependencies.ncl        # Dependency schemas (with OCI support)
│   ├── server.ncl              # Server schemas
│   ├── cluster.ncl             # Cluster schemas
│   └── workflows.ncl           # Workflow schemas
├── config/                     # Core configuration templates
├── templates/                  # Core templates
├── tools/                      # Build and distribution tools
│   ├── oci-package.nu          # OCI packaging tool
│   ├── build-core.nu           # Core build script
│   └── release-core.nu         # Core release script
├── tests/                      # Core system tests
└── docs/                       # Core documentation
    ├── api/                    # API documentation
    ├── architecture/           # Architecture docs
    └── development/            # Development guides

Distribution:

• Published as OCI artifact: oci://registry/provisioning-core:v3.5.0
• Contains all core functionality needed to run the provisioning system
• Version format: v{major}.{minor}.{patch} (for example, v3.5.0)

CI/CD:

• Build on commit to main
• Publish OCI artifact on git tag (v*)
• Run integration tests before publishing
• Update changelog automatically

---

Repository 2: provisioning-extensions

Purpose: All provider, taskserv, and cluster extensions

provisioning-extensions/
├── providers/
│   ├── aws/
│   │   ├── schemas/            # Nickel schemas
│   │   │   ├── manifest.toml   # Nickel dependencies
│   │   │   ├── aws.ncl         # Main provider schema
│   │   │   ├── defaults_aws.ncl # AWS defaults
│   │   │   └── server_aws.ncl  # AWS server schema
│   │   ├── scripts/            # Nushell scripts
│   │   │   └── install.nu      # Installation script
│   │   ├── templates/          # Provider templates
│   │   ├── docs/               # Provider documentation
│   │   └── manifest.yaml       # Extension manifest
│   ├── upcloud/
│   │   └── (same structure)
│   └── local/
│       └── (same structure)
├── taskservs/
│   ├── kubernetes/
│   │   ├── schemas/
│   │   │   ├── manifest.toml
│   │   │   ├── kubernetes.ncl  # Main taskserv schema
│   │   │   ├── version.ncl     # Version management
│   │   │   └── dependencies.ncl # Taskserv dependencies
│   │   ├── scripts/
│   │   │   ├── install.nu      # Installation script
│   │   │   ├── check.nu        # Health check script
│   │   │   └── uninstall.nu    # Uninstall script
│   │   ├── templates/          # Config templates
│   │   ├── docs/               # Taskserv docs
│   │   ├── tests/              # Taskserv tests
│   │   └── manifest.yaml       # Extension manifest
│   ├── containerd/
│   ├── cilium/
│   ├── postgres/
│   └── (50+ more taskservs...)
├── clusters/
│   ├── buildkit/
│   │   └── (same structure)
│   ├── web/
│   └── (other clusters...)
├── tools/
│   ├── extension-builder.nu   # Build individual extensions
│   ├── mass-publish.nu         # Publish all extensions
│   └── validate-extensions.nu # Validate all extensions
└── docs/
    ├── extension-guide.md      # Extension development guide
    └── publishing.md           # Publishing guide

Distribution: Each extension published separately as OCI artifact:

• oci://registry/provisioning-extensions/kubernetes:1.28.0
• oci://registry/provisioning-extensions/aws:2.0.0
• oci://registry/provisioning-extensions/buildkit:0.12.0

Extension Manifest (manifest.yaml):

name: kubernetes
type: taskserv
version: 1.28.0
description: Kubernetes container orchestration platform
author: Provisioning Team
license: MIT
homepage: https://kubernetes.io
repository: https://gitea.example.com/provisioning-extensions/kubernetes

dependencies:
  containerd: ">=1.7.0"
  etcd: ">=3.5.0"

tags:
  - kubernetes
  - container-orchestration
  - cncf

platforms:
  - linux/amd64
  - linux/arm64

min_provisioning_version: "3.0.0"

CI/CD:

• Build and publish each extension independently
• Git tag format: {extension-type}/{extension-name}/v{version}
  • Example: taskservs/kubernetes/v1.28.0
• Automated publishing to OCI registry on tag
• Run extension-specific tests before publishing

---

Repository 3: provisioning-platform

Purpose: Platform services (orchestrator, control-center, MCP server, API gateway)

provisioning-platform/
├── orchestrator/               # Rust orchestrator service
│   ├── src/
│   ├── Cargo.toml
│   ├── Dockerfile
│   └── README.md
├── control-center/             # Web control center
│   ├── src/
│   ├── package.json
│   ├── Dockerfile
│   └── README.md
├── mcp-server/                 # Model Context Protocol server
│   ├── src/
│   ├── Cargo.toml
│   ├── Dockerfile
│   └── README.md
├── api-gateway/                # REST API gateway
│   ├── src/
│   ├── Cargo.toml
│   ├── Dockerfile
│   └── README.md
├── docker-compose.yml          # Local development stack
├── kubernetes/                 # K8s deployment manifests
│   ├── orchestrator.yaml
│   ├── control-center.yaml
│   ├── mcp-server.yaml
│   └── api-gateway.yaml
└── docs/
    ├── deployment.md
    └── api-reference.md

Distribution: Standard Docker images in OCI registry:

• oci://registry/provisioning-platform/orchestrator:v1.2.0
• oci://registry/provisioning-platform/control-center:v1.2.0
• oci://registry/provisioning-platform/mcp-server:v1.0.0
• oci://registry/provisioning-platform/api-gateway:v1.0.0

CI/CD:

• Build Docker images on commit to main
• Publish images on git tag (v*)
• Multi-architecture builds (amd64, arm64)
• Security scanning before publishing

---

OCI Registry Integration

Registry Structure

OCI Registry (localhost:5000 or harbor.company.com)
├── provisioning-core/
│   ├── v3.5.0                  # Core system artifact
│   ├── v3.4.0
│   └── latest -> v3.5.0
├── provisioning-extensions/
│   ├── kubernetes:1.28.0       # Individual extension artifacts
│   ├── kubernetes:1.27.0
│   ├── containerd:1.7.0
│   ├── aws:2.0.0
│   ├── upcloud:1.5.0
│   └── (100+ more extensions)
└── provisioning-platform/
    ├── orchestrator:v1.2.0     # Platform service images
    ├── control-center:v1.2.0
    ├── mcp-server:v1.0.0
    └── api-gateway:v1.0.0

OCI Artifact Structure

Each extension packaged as OCI artifact:

kubernetes-1.28.0.tar.gz
├── schemas/                    # Nickel schemas
│   ├── kubernetes.ncl
│   ├── version.ncl
│   └── dependencies.ncl
├── scripts/                    # Nushell scripts
│   ├── install.nu
│   ├── check.nu
│   └── uninstall.nu
├── templates/                  # Template files
│   ├── kubeconfig.j2
│   └── kubelet-config.yaml.j2
├── docs/                       # Documentation
│   └── README.md
├── manifest.yaml               # Extension manifest
└── oci-manifest.json           # OCI manifest metadata

---

Dependency Management

Workspace Configuration

File: workspace/config/provisioning.yaml

# Core system dependency
dependencies:
  core:
    source: "oci://harbor.company.com/provisioning-core:v3.5.0"
    # Alternative: source: "gitea://provisioning-core"

  # Extensions repository configuration
  extensions:
    source_type: "oci"          # oci, gitea, local

    # OCI registry configuration
    oci:
      registry: "localhost:5000"
      namespace: "provisioning-extensions"
      tls_enabled: false
      auth_token_path: "~/.provisioning/tokens/oci"

    # Loaded extension modules
    modules:
      providers:
        - "oci://localhost:5000/provisioning-extensions/aws:2.0.0"
        - "oci://localhost:5000/provisioning-extensions/upcloud:1.5.0"

      taskservs:
        - "oci://localhost:5000/provisioning-extensions/kubernetes:1.28.0"
        - "oci://localhost:5000/provisioning-extensions/containerd:1.7.0"
        - "oci://localhost:5000/provisioning-extensions/cilium:1.14.0"

      clusters:
        - "oci://localhost:5000/provisioning-extensions/buildkit:0.12.0"

  # Platform services
  platform:
    source_type: "oci"

    oci:
      registry: "harbor.company.com"
      namespace: "provisioning-platform"

      images:
        orchestrator: "harbor.company.com/provisioning-platform/orchestrator:v1.2.0"
        control_center: "harbor.company.com/provisioning-platform/control-center:v1.2.0"

  # OCI registry configuration
  registry:
    type: "oci"                 # oci, gitea, http

    oci:
      endpoint: "localhost:5000"
      namespaces:
        extensions: "provisioning-extensions"
        nickel: "provisioning-nickel"
        platform: "provisioning-platform"
        test: "provisioning-test"

Dependency Resolution

The system resolves dependencies in this order:

1. Parse Configuration: Read provisioning.yaml and extract dependencies
2. Resolve Core: Ensure core system version is compatible
3. Resolve Extensions: For each extension:
   • Check if already installed and version matches
   • Pull from OCI registry if needed
   • Recursively resolve extension dependencies
4. Validate Graph: Check for dependency cycles and conflicts
5. Install: Install extensions in topological order

Dependency Resolution Commands

# Resolve and install all dependencies
provisioning dep resolve

# Check for dependency updates
provisioning dep check-updates

# Update specific extension
provisioning dep update kubernetes

# Validate dependency graph
provisioning dep validate

# Show dependency tree
provisioning dep tree kubernetes

---

OCI Client Operations

CLI Commands

# Pull extension from OCI registry
provisioning oci pull kubernetes:1.28.0

# Push extension to OCI registry
provisioning oci push ./extensions/kubernetes kubernetes 1.28.0

# List available extensions
provisioning oci list --namespace provisioning-extensions

# Search for extensions
provisioning oci search kubernetes

# Show extension versions
provisioning oci tags kubernetes

# Inspect extension manifest
provisioning oci inspect kubernetes:1.28.0

# Login to OCI registry
provisioning oci login localhost:5000 --username _token --password-stdin

# Delete extension
provisioning oci delete kubernetes:1.28.0

# Copy extension between registries
provisioning oci copy \
  localhost:5000/provisioning-extensions/kubernetes:1.28.0 \
  harbor.company.com/provisioning-extensions/kubernetes:1.28.0

OCI Configuration

# Show OCI configuration
provisioning oci config

# Output:
{
  tool: "oras"  # or "crane" or "skopeo"
  registry: "localhost:5000"
  namespace: {
    extensions: "provisioning-extensions"
    platform: "provisioning-platform"
  }
  cache_dir: "~/.provisioning/oci-cache"
  tls_enabled: false
}

---

Extension Development Workflow

1. Develop Extension

# Create new extension from template
provisioning generate extension taskserv redis

# Directory structure created:
# extensions/taskservs/redis/
# ├── schemas/
# │   ├── manifest.toml
# │   ├── redis.ncl
# │   ├── version.ncl
# │   └── dependencies.ncl
# ├── scripts/
# │   ├── install.nu
# │   ├── check.nu
# │   └── uninstall.nu
# ├── templates/
# ├── docs/
# │   └── README.md
# ├── tests/
# └── manifest.yaml

2. Test Extension Locally

# Load extension from local path
provisioning module load taskserv workspace_dev redis --source local

# Test installation
provisioning taskserv create redis --infra test-env --check

# Run extension tests
provisioning test extension redis

3. Package Extension

# Validate extension structure
provisioning oci package validate ./extensions/taskservs/redis

# Package as OCI artifact
provisioning oci package ./extensions/taskservs/redis

# Output: redis-1.0.0.tar.gz

4. Publish Extension

# Login to registry (one-time)
provisioning oci login localhost:5000

# Publish extension
provisioning oci push ./extensions/taskservs/redis redis 1.0.0

# Verify publication
provisioning oci tags redis

# Output:
# ┬───────────┬─────────┬───────────────────────────────────────────────────┐
# │ artifact  │ version │ reference                                         │
# ├───────────┼─────────┼───────────────────────────────────────────────────┤
# │ redis     │ 1.0.0   │ localhost:5000/provisioning-extensions/redis:1.0.0│
# └───────────┴─────────┴───────────────────────────────────────────────────┘

5. Use Published Extension

# Add to workspace configuration
# workspace/config/provisioning.yaml:
# dependencies:
#   extensions:
#     modules:
#       taskservs:
#         - "oci://localhost:5000/provisioning-extensions/redis:1.0.0"

# Pull and install
provisioning dep resolve

# Extension automatically downloaded and installed

---

Registry Deployment Options

Local Registry (Solo Development)

Using Zot (lightweight OCI registry):

# Start local OCI registry
provisioning oci-registry start

# Configuration:
# - Endpoint: localhost:5000
# - Storage: ~/.provisioning/oci-registry/
# - No authentication by default
# - TLS disabled (local only)

# Stop registry
provisioning oci-registry stop

# Check status
provisioning oci-registry status

Remote Registry (Multi-User/Enterprise)

Using Harbor:

# workspace/config/provisioning.yaml
dependencies:
  registry:
    type: "oci"
    oci:
      endpoint: "https://harbor.company.com"
      namespaces:
        extensions: "provisioning/extensions"
        platform: "provisioning/platform"
      tls_enabled: true
      auth_token_path: "~/.provisioning/tokens/harbor"

Features:

• Multi-user authentication
• Role-based access control (RBAC)
• Vulnerability scanning
• Replication across registries
• Webhook notifications
• Image signing (cosign/notation)

---

Migration from Monorepo

Phase 1: Parallel Structure (Current)

• Monorepo still exists and works
• OCI distribution layer added on top
• Extensions can be loaded from local or OCI
• No breaking changes

Phase 2: Gradual Migration

# Migrate extensions one by one
for ext in (ls provisioning/extensions/taskservs); do
  provisioning oci publish $ext.name
done

# Update workspace configurations to use OCI
provisioning workspace migrate-to-oci workspace_prod

Phase 3: Repository Split

1. Create provisioning-core repository

   • Extract core/ and schemas/ directories
   • Set up CI/CD for core publishing
   • Publish initial OCI artifact

2. Create provisioning-extensions repository

   • Extract extensions/ directory
   • Set up CI/CD for extension publishing
   • Publish all extensions to OCI registry

3. Create provisioning-platform repository

   • Extract platform/ directory
   • Set up Docker image builds
   • Publish platform services

4. Update workspaces

   • Reconfigure to use OCI dependencies
   • Test multi-repo setup
   • Verify all functionality works

Phase 4: Deprecate Monorepo

• Archive monorepo
• Redirect to new repositories
• Update documentation
• Announce migration complete

---

Benefits Summary

Modularity

✅ Independent repositories for core, extensions, and platform ✅ Extensions can be developed and versioned separately ✅ Clear ownership and responsibility boundaries

Distribution

✅ OCI-native distribution (industry standard) ✅ Built-in versioning with OCI tags ✅ Efficient caching with OCI layers ✅ Works with standard tools (skopeo, crane, oras)

Security

✅ TLS support for registries ✅ Authentication and authorization ✅ Vulnerability scanning (Harbor) ✅ Image signing (cosign, notation) ✅ RBAC for access control

Developer Experience

✅ Simple CLI commands for extension management ✅ Automatic dependency resolution ✅ Local testing before publishing ✅ Easy extension discovery and installation

Operations

✅ Air-gapped deployments (mirror OCI registry) ✅ Bandwidth efficient (only download what's needed) ✅ Version pinning for reproducibility ✅ Rollback support (use previous versions)

Ecosystem

✅ Compatible with existing OCI tooling ✅ Can use public registries (DockerHub, GitHub, etc.) ✅ Mirror to multiple registries ✅ Replication for high availability

---

Implementation Status

Component	Status	Notes
Nickel Schemas	✅ Complete	OCI schemas in dependencies.ncl
OCI Client	✅ Complete	oci/client.nu with skopeo/crane/oras
OCI Commands	✅ Complete	oci/commands.nu CLI interface
Dependency Resolver	✅ Complete	dependencies/resolver.nu
OCI Packaging	✅ Complete	tools/oci-package.nu
Repository Design	✅ Complete	This document
Migration Plan	✅ Complete	Phased approach defined
Documentation	✅ Complete	User guides and API docs
CI/CD Setup	⏳ Pending	Automated publishing pipelines
Registry Deployment	⏳ Pending	Zot/Harbor setup

---

Related Documentation

• OCI Packaging Tool - Extension packaging
• OCI Client Library - OCI operations
• Dependency Resolver - Dependency management
• Nickel Schemas - Type definitions
• Extension Development Guide - How to create extensions

---

Maintained By: Architecture Team Review Cycle: Quarterly Next Review: 2026-01-06