provisioning/docs/src/architecture/MULTI_REPO_ARCHITECTURE.md

# Multi-Repository Architecture with OCI Registry Support

**Version**: 1.0.0
**Date**: 2025-10-06
**Status**: Implementation Complete

## Overview

This document describes the multi-repository architecture for the provisioning system, enabling modular development, independent versioning, and distributed extension management through OCI registry integration.

## Architecture Goals

1. **Separation of Concerns**: Core, Extensions, and Platform in separate repositories
2. **Independent Versioning**: Each component can be versioned and released independently
3. **Distributed Development**: Multiple teams can work on different repositories
4. **OCI-Native Distribution**: Extensions distributed as OCI artifacts
5. **Dependency Management**: Automated dependency resolution across repositories
6. **Backward Compatibility**: Support legacy monorepo structure during transition

## Repository Structure

### Repository 1: `provisioning-core`

**Purpose**: Core system functionality - CLI, libraries, base schemas

```
provisioning-core/
├── core/
│   ├── cli/                    # Command-line interface
│   │   ├── provisioning        # Main CLI entry point
│   │   └── module-loader       # Dynamic module loader
│   ├── nulib/                  # Core Nushell libraries
│   │   ├── lib_provisioning/   # Core library modules
│   │   │   ├── config/         # Configuration management
│   │   │   ├── oci/            # OCI client integration
│   │   │   ├── dependencies/   # Dependency resolution
│   │   │   ├── module/         # Module system
│   │   │   ├── layer/          # Layer system
│   │   │   └── workspace/      # Workspace management
│   │   └── workflows/          # Core workflow system
│   ├── plugins/                # System plugins
│   └── scripts/                # Utility scripts
├── kcl/                        # Base KCL schemas
│   ├── main.k                  # Main schema entry
│   ├── lib.k                   # Core library types
│   ├── settings.k              # Settings schema
│   ├── dependencies.k          # Dependency schemas (with OCI support)
│   ├── server.k                # Server schemas
│   ├── cluster.k               # Cluster schemas
│   └── workflows.k             # Workflow schemas
├── config/                     # Core configuration templates
├── templates/                  # Core templates
├── tools/                      # Build and distribution tools
│   ├── oci-package.nu          # OCI packaging tool
│   ├── build-core.nu           # Core build script
│   └── release-core.nu         # Core release script
├── tests/                      # Core system tests
└── docs/                       # Core documentation
    ├── api/                    # API documentation
    ├── architecture/           # Architecture docs
    └── development/            # Development guides

```

**Distribution**:
- Published as OCI artifact: `oci://registry/provisioning-core:v3.5.0`
- Contains all core functionality needed to run the provisioning system
- Version format: `v{major}.{minor}.{patch}` (e.g., v3.5.0)

**CI/CD**:
- Build on commit to main
- Publish OCI artifact on git tag (v*)
- Run integration tests before publishing
- Update changelog automatically

---

### Repository 2: `provisioning-extensions`

**Purpose**: All provider, taskserv, and cluster extensions

```
provisioning-extensions/
├── providers/
│   ├── aws/
│   │   ├── kcl/                # KCL schemas
│   │   │   ├── kcl.mod         # KCL dependencies
│   │   │   ├── aws.k           # Main provider schema
│   │   │   ├── defaults_aws.k  # AWS defaults
│   │   │   └── server_aws.k    # AWS server schema
│   │   ├── scripts/            # Nushell scripts
│   │   │   └── install.nu      # Installation script
│   │   ├── templates/          # Provider templates
│   │   ├── docs/               # Provider documentation
│   │   └── manifest.yaml       # Extension manifest
│   ├── upcloud/
│   │   └── (same structure)
│   └── local/
│       └── (same structure)
├── taskservs/
│   ├── kubernetes/
│   │   ├── kcl/
│   │   │   ├── kcl.mod
│   │   │   ├── kubernetes.k    # Main taskserv schema
│   │   │   ├── version.k       # Version management
│   │   │   └── dependencies.k  # Taskserv dependencies
│   │   ├── scripts/
│   │   │   ├── install.nu      # Installation script
│   │   │   ├── check.nu        # Health check script
│   │   │   └── uninstall.nu    # Uninstall script
│   │   ├── templates/          # Config templates
│   │   ├── docs/               # Taskserv docs
│   │   ├── tests/              # Taskserv tests
│   │   └── manifest.yaml       # Extension manifest
│   ├── containerd/
│   ├── cilium/
│   ├── postgres/
│   └── (50+ more taskservs...)
├── clusters/
│   ├── buildkit/
│   │   └── (same structure)
│   ├── web/
│   └── (other clusters...)
├── tools/
│   ├── extension-builder.nu   # Build individual extensions
│   ├── mass-publish.nu         # Publish all extensions
│   └── validate-extensions.nu # Validate all extensions
└── docs/
    ├── extension-guide.md      # Extension development guide
    └── publishing.md           # Publishing guide

```

**Distribution**:
Each extension published separately as OCI artifact:
- `oci://registry/provisioning-extensions/kubernetes:1.28.0`
- `oci://registry/provisioning-extensions/aws:2.0.0`
- `oci://registry/provisioning-extensions/buildkit:0.12.0`

**Extension Manifest** (`manifest.yaml`):
```yaml
name: kubernetes
type: taskserv
version: 1.28.0
description: Kubernetes container orchestration platform
author: Provisioning Team
license: MIT
homepage: https://kubernetes.io
repository: https://gitea.example.com/provisioning-extensions/kubernetes

dependencies:
  containerd: ">=1.7.0"
  etcd: ">=3.5.0"

tags:
  - kubernetes
  - container-orchestration
  - cncf

platforms:
  - linux/amd64
  - linux/arm64

min_provisioning_version: "3.0.0"
```

**CI/CD**:
- Build and publish each extension independently
- Git tag format: `{extension-type}/{extension-name}/v{version}`
  - Example: `taskservs/kubernetes/v1.28.0`
- Automated publishing to OCI registry on tag
- Run extension-specific tests before publishing

---

### Repository 3: `provisioning-platform`

**Purpose**: Platform services (orchestrator, control-center, MCP server, API gateway)

```
provisioning-platform/
├── orchestrator/               # Rust orchestrator service
│   ├── src/
│   ├── Cargo.toml
│   ├── Dockerfile
│   └── README.md
├── control-center/             # Web control center
│   ├── src/
│   ├── package.json
│   ├── Dockerfile
│   └── README.md
├── mcp-server/                 # Model Context Protocol server
│   ├── src/
│   ├── Cargo.toml
│   ├── Dockerfile
│   └── README.md
├── api-gateway/                # REST API gateway
│   ├── src/
│   ├── Cargo.toml
│   ├── Dockerfile
│   └── README.md
├── docker-compose.yml          # Local development stack
├── kubernetes/                 # K8s deployment manifests
│   ├── orchestrator.yaml
│   ├── control-center.yaml
│   ├── mcp-server.yaml
│   └── api-gateway.yaml
└── docs/
    ├── deployment.md
    └── api-reference.md

```

**Distribution**:
Standard Docker images in OCI registry:
- `oci://registry/provisioning-platform/orchestrator:v1.2.0`
- `oci://registry/provisioning-platform/control-center:v1.2.0`
- `oci://registry/provisioning-platform/mcp-server:v1.0.0`
- `oci://registry/provisioning-platform/api-gateway:v1.0.0`

**CI/CD**:
- Build Docker images on commit to main
- Publish images on git tag (v*)
- Multi-architecture builds (amd64, arm64)
- Security scanning before publishing

---

## OCI Registry Integration

### Registry Structure

```
OCI Registry (localhost:5000 or harbor.company.com)
├── provisioning-core/
│   ├── v3.5.0                  # Core system artifact
│   ├── v3.4.0
│   └── latest -> v3.5.0
├── provisioning-extensions/
│   ├── kubernetes:1.28.0       # Individual extension artifacts
│   ├── kubernetes:1.27.0
│   ├── containerd:1.7.0
│   ├── aws:2.0.0
│   ├── upcloud:1.5.0
│   └── (100+ more extensions)
└── provisioning-platform/
    ├── orchestrator:v1.2.0     # Platform service images
    ├── control-center:v1.2.0
    ├── mcp-server:v1.0.0
    └── api-gateway:v1.0.0

```

### OCI Artifact Structure

Each extension packaged as OCI artifact:

```
kubernetes-1.28.0.tar.gz
├── kcl/                        # KCL schemas
│   ├── kubernetes.k
│   ├── version.k
│   └── dependencies.k
├── scripts/                    # Nushell scripts
│   ├── install.nu
│   ├── check.nu
│   └── uninstall.nu
├── templates/                  # Template files
│   ├── kubeconfig.j2
│   └── kubelet-config.yaml.j2
├── docs/                       # Documentation
│   └── README.md
├── manifest.yaml               # Extension manifest
└── oci-manifest.json           # OCI manifest metadata

```

---

## Dependency Management

### Workspace Configuration

**File**: `workspace/config/provisioning.yaml`

```yaml
# Core system dependency
dependencies:
  core:
    source: "oci://harbor.company.com/provisioning-core:v3.5.0"
    # Alternative: source: "gitea://provisioning-core"

  # Extensions repository configuration
  extensions:
    source_type: "oci"          # oci, gitea, local

    # OCI registry configuration
    oci:
      registry: "localhost:5000"
      namespace: "provisioning-extensions"
      tls_enabled: false
      auth_token_path: "~/.provisioning/tokens/oci"

    # Loaded extension modules
    modules:
      providers:
        - "oci://localhost:5000/provisioning-extensions/aws:2.0.0"
        - "oci://localhost:5000/provisioning-extensions/upcloud:1.5.0"

      taskservs:
        - "oci://localhost:5000/provisioning-extensions/kubernetes:1.28.0"
        - "oci://localhost:5000/provisioning-extensions/containerd:1.7.0"
        - "oci://localhost:5000/provisioning-extensions/cilium:1.14.0"

      clusters:
        - "oci://localhost:5000/provisioning-extensions/buildkit:0.12.0"

  # Platform services
  platform:
    source_type: "oci"

    oci:
      registry: "harbor.company.com"
      namespace: "provisioning-platform"

      images:
        orchestrator: "harbor.company.com/provisioning-platform/orchestrator:v1.2.0"
        control_center: "harbor.company.com/provisioning-platform/control-center:v1.2.0"

  # OCI registry configuration
  registry:
    type: "oci"                 # oci, gitea, http

    oci:
      endpoint: "localhost:5000"
      namespaces:
        extensions: "provisioning-extensions"
        kcl: "provisioning-kcl"
        platform: "provisioning-platform"
        test: "provisioning-test"
```

### Dependency Resolution

The system resolves dependencies in this order:

1. **Parse Configuration**: Read `provisioning.yaml` and extract dependencies
2. **Resolve Core**: Ensure core system version is compatible
3. **Resolve Extensions**: For each extension:
   - Check if already installed and version matches
   - Pull from OCI registry if needed
   - Recursively resolve extension dependencies
4. **Validate Graph**: Check for dependency cycles and conflicts
5. **Install**: Install extensions in topological order

### Dependency Resolution Commands

```bash
# Resolve and install all dependencies
provisioning dep resolve

# Check for dependency updates
provisioning dep check-updates

# Update specific extension
provisioning dep update kubernetes

# Validate dependency graph
provisioning dep validate

# Show dependency tree
provisioning dep tree kubernetes
```

---

## OCI Client Operations

### CLI Commands

```bash
# Pull extension from OCI registry
provisioning oci pull kubernetes:1.28.0

# Push extension to OCI registry
provisioning oci push ./extensions/kubernetes kubernetes 1.28.0

# List available extensions
provisioning oci list --namespace provisioning-extensions

# Search for extensions
provisioning oci search kubernetes

# Show extension versions
provisioning oci tags kubernetes

# Inspect extension manifest
provisioning oci inspect kubernetes:1.28.0

# Login to OCI registry
provisioning oci login localhost:5000 --username _token --password-stdin

# Delete extension
provisioning oci delete kubernetes:1.28.0

# Copy extension between registries
provisioning oci copy \
  localhost:5000/provisioning-extensions/kubernetes:1.28.0 \
  harbor.company.com/provisioning-extensions/kubernetes:1.28.0
```

### OCI Configuration

```bash
# Show OCI configuration
provisioning oci config

# Output:
{
  tool: "oras"  # or "crane" or "skopeo"
  registry: "localhost:5000"
  namespace: {
    extensions: "provisioning-extensions"
    platform: "provisioning-platform"
  }
  cache_dir: "~/.provisioning/oci-cache"
  tls_enabled: false
}
```

---

## Extension Development Workflow

### 1. Develop Extension

```bash
# Create new extension from template
provisioning generate extension taskserv redis

# Directory structure created:
# extensions/taskservs/redis/
# ├── kcl/
# │   ├── kcl.mod
# │   ├── redis.k
# │   ├── version.k
# │   └── dependencies.k
# ├── scripts/
# │   ├── install.nu
# │   ├── check.nu
# │   └── uninstall.nu
# ├── templates/
# ├── docs/
# │   └── README.md
# ├── tests/
# └── manifest.yaml
```

### 2. Test Extension Locally

```bash
# Load extension from local path
provisioning module load taskserv workspace_dev redis --source local

# Test installation
provisioning taskserv create redis --infra test-env --check

# Run extension tests
provisioning test extension redis
```

### 3. Package Extension

```bash
# Validate extension structure
provisioning oci package validate ./extensions/taskservs/redis

# Package as OCI artifact
provisioning oci package ./extensions/taskservs/redis

# Output: redis-1.0.0.tar.gz
```

### 4. Publish Extension

```bash
# Login to registry (one-time)
provisioning oci login localhost:5000

# Publish extension
provisioning oci push ./extensions/taskservs/redis redis 1.0.0

# Verify publication
provisioning oci tags redis

# Output:
# ┬───────────┬─────────┬───────────────────────────────────────────────────┐
# │ artifact  │ version │ reference                                         │
# ├───────────┼─────────┼───────────────────────────────────────────────────┤
# │ redis     │ 1.0.0   │ localhost:5000/provisioning-extensions/redis:1.0.0│
# └───────────┴─────────┴───────────────────────────────────────────────────┘
```

### 5. Use Published Extension

```bash
# Add to workspace configuration
# workspace/config/provisioning.yaml:
# dependencies:
#   extensions:
#     modules:
#       taskservs:
#         - "oci://localhost:5000/provisioning-extensions/redis:1.0.0"

# Pull and install
provisioning dep resolve

# Extension automatically downloaded and installed
```

---

## Registry Deployment Options

### Local Registry (Solo Development)

**Using Zot (lightweight OCI registry)**:

```bash
# Start local OCI registry
provisioning oci-registry start

# Configuration:
# - Endpoint: localhost:5000
# - Storage: ~/.provisioning/oci-registry/
# - No authentication by default
# - TLS disabled (local only)

# Stop registry
provisioning oci-registry stop

# Check status
provisioning oci-registry status
```

### Remote Registry (Multi-User/Enterprise)

**Using Harbor**:

```yaml
# workspace/config/provisioning.yaml
dependencies:
  registry:
    type: "oci"
    oci:
      endpoint: "https://harbor.company.com"
      namespaces:
        extensions: "provisioning/extensions"
        platform: "provisioning/platform"
      tls_enabled: true
      auth_token_path: "~/.provisioning/tokens/harbor"
```

**Features**:
- Multi-user authentication
- Role-based access control (RBAC)
- Vulnerability scanning
- Replication across registries
- Webhook notifications
- Image signing (cosign/notation)

---

## Migration from Monorepo

### Phase 1: Parallel Structure (Current)

- Monorepo still exists and works
- OCI distribution layer added on top
- Extensions can be loaded from local or OCI
- No breaking changes

### Phase 2: Gradual Migration

```bash
# Migrate extensions one by one
for ext in (ls provisioning/extensions/taskservs); do
  provisioning oci publish $ext.name
done

# Update workspace configurations to use OCI
provisioning workspace migrate-to-oci workspace_prod
```

### Phase 3: Repository Split

1. Create `provisioning-core` repository
   - Extract core/ and kcl/ directories
   - Set up CI/CD for core publishing
   - Publish initial OCI artifact

2. Create `provisioning-extensions` repository
   - Extract extensions/ directory
   - Set up CI/CD for extension publishing
   - Publish all extensions to OCI registry

3. Create `provisioning-platform` repository
   - Extract platform/ directory
   - Set up Docker image builds
   - Publish platform services

4. Update workspaces
   - Reconfigure to use OCI dependencies
   - Test multi-repo setup
   - Verify all functionality works

### Phase 4: Deprecate Monorepo

- Archive monorepo
- Redirect to new repositories
- Update documentation
- Announce migration complete

---

## Benefits Summary

### Modularity
✅ Independent repositories for core, extensions, and platform
✅ Extensions can be developed and versioned separately
✅ Clear ownership and responsibility boundaries

### Distribution
✅ OCI-native distribution (industry standard)
✅ Built-in versioning with OCI tags
✅ Efficient caching with OCI layers
✅ Works with standard tools (skopeo, crane, oras)

### Security
✅ TLS support for registries
✅ Authentication and authorization
✅ Vulnerability scanning (Harbor)
✅ Image signing (cosign, notation)
✅ RBAC for access control

### Developer Experience
✅ Simple CLI commands for extension management
✅ Automatic dependency resolution
✅ Local testing before publishing
✅ Easy extension discovery and installation

### Operations
✅ Air-gapped deployments (mirror OCI registry)
✅ Bandwidth efficient (only download what's needed)
✅ Version pinning for reproducibility
✅ Rollback support (use previous versions)

### Ecosystem
✅ Compatible with existing OCI tooling
✅ Can use public registries (DockerHub, GitHub, etc.)
✅ Mirror to multiple registries
✅ Replication for high availability

---

## Implementation Status

| Component | Status | Notes |
|-----------|--------|-------|
| **KCL Schemas** | ✅ Complete | OCI schemas in `dependencies.k` |
| **OCI Client** | ✅ Complete | `oci/client.nu` with skopeo/crane/oras |
| **OCI Commands** | ✅ Complete | `oci/commands.nu` CLI interface |
| **Dependency Resolver** | ✅ Complete | `dependencies/resolver.nu` |
| **OCI Packaging** | ✅ Complete | `tools/oci-package.nu` |
| **Repository Design** | ✅ Complete | This document |
| **Migration Plan** | ✅ Complete | Phased approach defined |
| **Documentation** | ✅ Complete | User guides and API docs |
| **CI/CD Setup** | ⏳ Pending | Automated publishing pipelines |
| **Registry Deployment** | ⏳ Pending | Zot/Harbor setup |

---

## Related Documentation

- [OCI Packaging Tool](../../tools/oci-package.nu) - Extension packaging
- [OCI Client Library](../../core/nulib/lib_provisioning/oci/client.nu) - OCI operations
- [Dependency Resolver](../../core/nulib/lib_provisioning/dependencies/resolver.nu) - Dependency management
- [KCL Schemas](../../kcl/dependencies.k) - Type definitions
- [Extension Development Guide](../user/extension-development.md) - How to create extensions

---

**Maintained By**: Architecture Team
**Review Cycle**: Quarterly
**Next Review**: 2026-01-06