provisioning/docs/src/development/extension-registry.md

Extension Registry Service

A high-performance Rust microservice that provides a unified REST API for extension discovery, versioning, and download from multiple Git-based sources and OCI registries.

Source: provisioning/platform/crates/extension-registry/

Features

• Multi-Backend Source Support: Fetch extensions from Gitea, Forgejo, and GitHub releases
• Multi-Registry Distribution Support: Distribute extensions to Zot, Harbor, Docker Hub, GHCR, Quay, and other OCI-compliant registries
• Unified REST API: Single API for all extension operations across all backends
• Smart Caching: LRU cache with TTL to reduce backend API calls
• Prometheus Metrics: Built-in metrics for monitoring
• Health Monitoring: Parallel health checks for all backends with aggregated status
• Aggregation & Fallback: Intelligent request routing with aggregation and fallback strategies
• Type-Safe: Strong typing for extension metadata
• Async/Await: High-performance async operations with Tokio
• Backward Compatible: Old single-instance configs auto-migrate to new multi-instance format

Architecture

Dual-Trait System

The extension registry uses a trait-based architecture separating source and distribution backends:

┌────────────────────────────────────────────────────────────────────┐
│                    Extension Registry API                           │
│                          (axum)                                     │
├────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌─ SourceClients ────────────┐  ┌─ DistributionClients ────────┐  │
│  │                             │  │                              │  │
│  │ • Gitea (Git releases)      │  │ • OCI Registries             │  │
│  │ • Forgejo (Git releases)    │  │   - Zot                      │  │
│  │ • GitHub (Releases API)     │  │   - Harbor                   │  │
│  │                             │  │   - Docker Hub               │  │
│  │ Strategy: Aggregation +     │  │   - GHCR / Quay              │  │
│  │ Fallback across all sources │  │   - Any OCI-compliant        │  │
│  │                             │  │                              │  │
│  └─────────────────────────────┘  └──────────────────────────────┘  │
│                                                                      │
│  ┌─ LRU Cache ───────────────────────────────────────────────────┐  │
│  │ • Metadata cache (with TTL)                                   │  │
│  │ • List cache (with TTL)                                       │  │
│  │ • Version cache (version strings only)                        │  │
│  └───────────────────────────────────────────────────────────────┘  │
│                                                                      │
└────────────────────────────────────────────────────────────────────┘

Request Strategies

Aggregation Strategy (list_extensions, list_versions, search)

1. Parallel Execution: Spawn concurrent tasks for all source and distribution clients
2. Merge Results: Combine results from all backends
3. Deduplication: Remove duplicates, preferring more recent versions
4. Pagination: Apply limit/offset to merged results
5. Caching: Store merged results with composite cache key

Fallback Strategy (get_extension, download_extension)

1. Sequential Retry: Try source clients first (in configured order)
2. Distribution Fallback: If all sources fail, try distribution clients
3. Return First Success: Return result from first successful client
4. Caching: Cache successful result with backend-specific key

Installation

cd provisioning/platform/extension-registry
cargo build --release

Configuration

Single-Instance Configuration (Legacy - Auto-Migrated)

Old format is automatically migrated to new multi-instance format:

[server]
host = "0.0.0.0"
port = 8082

# Single Gitea instance (auto-migrated to sources.gitea[0])
[gitea]
url = "https://gitea.example.com"
organization = "provisioning-extensions"
token_path = "/path/to/gitea-token.txt"

# Single OCI registry (auto-migrated to distributions.oci[0])
[oci]
registry = "registry.example.com"
namespace = "provisioning"
auth_token_path = "/path/to/oci-token.txt"

[cache]
capacity = 1000
ttl_seconds = 300

Multi-Instance Configuration (Recommended)

New format supporting multiple backends of each type:

[server]
host = "0.0.0.0"
port = 8082
workers = 4
enable_cors = false
enable_compression = true

# Multiple Gitea sources
[sources.gitea]

[[sources.gitea]]
id = "internal-gitea"
url = "https://gitea.internal.example.com"
organization = "provisioning"
token_path = "/etc/secrets/gitea-internal-token.txt"
timeout_seconds = 30
verify_ssl = true

[[sources.gitea]]
id = "public-gitea"
url = "https://gitea.public.example.com"
organization = "extensions"
token_path = "/etc/secrets/gitea-public-token.txt"
timeout_seconds = 30
verify_ssl = true

# Forgejo sources (API compatible with Gitea)
[sources.forgejo]

[[sources.forgejo]]
id = "community-forgejo"
url = "https://forgejo.community.example.com"
organization = "provisioning"
token_path = "/etc/secrets/forgejo-token.txt"
timeout_seconds = 30
verify_ssl = true

# GitHub sources
[sources.github]

[[sources.github]]
id = "org-github"
organization = "my-organization"
token_path = "/etc/secrets/github-token.txt"
timeout_seconds = 30
verify_ssl = true

# Multiple OCI distribution registries
[distributions.oci]

[[distributions.oci]]
id = "internal-zot"
registry = "zot.internal.example.com"
namespace = "extensions"
timeout_seconds = 30
verify_ssl = true

[[distributions.oci]]
id = "public-harbor"
registry = "harbor.public.example.com"
namespace = "extensions"
auth_token_path = "/etc/secrets/harbor-token.txt"
timeout_seconds = 30
verify_ssl = true

[[distributions.oci]]
id = "docker-hub"
registry = "docker.io"
namespace = "myorg"
auth_token_path = "/etc/secrets/docker-hub-token.txt"
timeout_seconds = 30
verify_ssl = true

# Cache configuration
[cache]
capacity = 1000
ttl_seconds = 300
enable_metadata_cache = true
enable_list_cache = true

Configuration Notes

• Backend Identifiers: Use id field to uniquely identify each backend instance (auto-generated if omitted)
• Gitea/Forgejo Compatible: Both use same config format; organization field is required for Git repos
• GitHub Configuration: Uses organization as owner; token_path points to GitHub Personal Access Token
• OCI Registries: Support any OCI-compliant registry (Zot, Harbor, Docker Hub, GHCR, Quay, etc.)
• Optional Fields: id, verify_ssl, timeout_seconds have sensible defaults
• Token Files: Should contain only the token with no extra whitespace; permissions should be 0600

Environment Variable Overrides

Legacy environment variable support (for backward compatibility):

REGISTRY_SERVER_HOST=127.0.0.1
REGISTRY_SERVER_PORT=8083
REGISTRY_SERVER_WORKERS=8
REGISTRY_GITEA_URL=https://gitea.example.com
REGISTRY_GITEA_ORG=extensions
REGISTRY_GITEA_TOKEN_PATH=/path/to/token
REGISTRY_OCI_REGISTRY=registry.example.com
REGISTRY_OCI_NAMESPACE=extensions
REGISTRY_CACHE_CAPACITY=2000
REGISTRY_CACHE_TTL=600

API Endpoints

Extension Operations

List Extensions

GET /api/v1/extensions?type=provider&limit=10

Get Extension

GET /api/v1/extensions/{type}/{name}

List Versions

GET /api/v1/extensions/{type}/{name}/versions

Download Extension

GET /api/v1/extensions/{type}/{name}/{version}

Search Extensions

GET /api/v1/extensions/search?q=kubernetes&type=taskserv

System Endpoints

Health Check

GET /api/v1/health

Response (with multi-backend aggregation):

{
  "status": "healthy|degraded|unhealthy",
  "version": "0.1.0",
  "uptime": 3600,
  "backends": {
    "gitea": {
      "enabled": true,
      "healthy": true,
      "error": null
    },
    "oci": {
      "enabled": true,
      "healthy": true,
      "error": null
    }
  }
}

Status Values:

• healthy: All configured backends are healthy
• degraded: At least one backend is healthy, but some are failing
• unhealthy: No backends are responding

Metrics

GET /api/v1/metrics

Cache Statistics

GET /api/v1/cache/stats

Response:

{
  "metadata_hits": 1024,
  "metadata_misses": 256,
  "list_hits": 512,
  "list_misses": 128,
  "version_hits": 2048,
  "version_misses": 512,
  "size": 4096
}

Extension Naming Conventions

Gitea Repositories

• Providers: {name}_prov (for example, aws_prov)
• Task Services: {name}_taskserv (for example, kubernetes_taskserv)
• Clusters: {name}_cluster (for example, buildkit_cluster)

OCI Artifacts

• Providers: {namespace}/{name}-provider
• Task Services: {namespace}/{name}-taskserv
• Clusters: {namespace}/{name}-cluster

Deployment

Docker

docker build -t extension-registry:latest .
docker run -d -p 8082:8082 -v $(pwd)/config.toml:/app/config.toml:ro extension-registry:latest

Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: extension-registry
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: extension-registry
        image: extension-registry:latest
        ports:
        - containerPort: 8082

Migration Guide: Single to Multi-Instance

Automatic Migration

Old single-instance configs are automatically detected and migrated to the new multi-instance format during startup:

1. Detection: Registry checks if old-style fields (gitea, oci) contain values
2. Migration: Single instances are moved to new Vec-based format (sources.gitea[0], distributions.oci[0])
3. Logging: Migration event is logged for audit purposes
4. Transparency: No user action required; old configs continue to work

Before Migration

[gitea]
url = "https://gitea.example.com"
organization = "extensions"
token_path = "/path/to/token"

[oci]
registry = "registry.example.com"
namespace = "extensions"

After Migration (Automatic)

[sources.gitea]
[[sources.gitea]]
url = "https://gitea.example.com"
organization = "extensions"
token_path = "/path/to/token"

[distributions.oci]
[[distributions.oci]]
registry = "registry.example.com"
namespace = "extensions"

Gradual Upgrade Path

To adopt the new format manually:

1. Backup current config - Keep old format as reference
2. Adopt new format - Replace old fields with new structure
3. Test - Verify all backends are reachable and extensions are discovered
4. Add new backends - Use new format to add Forgejo, GitHub, or additional OCI registries
5. Remove old fields - Delete deprecated gitea and oci top-level sections

Benefits of Upgrading

• Multiple Sources: Support Gitea, Forgejo, and GitHub simultaneously
• Multiple Registries: Distribute to multiple OCI registries
• Better Resilience: If one backend fails, others continue to work
• Flexible Configuration: Each backend can have different credentials and timeouts
• Future-Proof: New backends can be added without config restructuring

Related Documentation

• Extension Development: Module System
• Extension Development Quickstart: Getting Started Guide
• ADR-005: Extension Framework Architecture
• OCI Registry Integration: OCI Registry Guide