prvng_platform/infrastructure/oci-registry/README.md

OCI Registry Service

Comprehensive OCI (Open Container Initiative) registry deployment and management for the provisioning system. Supports multiple registry implementations: Zot (lightweight), Harbor (full-featured), and Distribution (OCI reference implementation).

Table of Contents

• Overview
• Registry Types
• Quick Start
• Installation
• Configuration
• Management
• Namespaces
• Access Control
• Monitoring
• Troubleshooting
• Advanced Usage

Overview

The OCI registry service provides artifact storage and distribution for:

• Extension Packages: Providers, taskservs, clusters
• KCL Schemas: Configuration schemas and modules
• Platform Images: Orchestrator, control-center, services
• Test Artifacts: Development and testing images

Features

• Multi-Registry Support: Zot, Harbor, Distribution
• Namespace Organization: Logical separation of artifacts
• Access Control: RBAC, policies, authentication
• Monitoring: Prometheus metrics, health checks
• Garbage Collection: Automatic cleanup of unused artifacts
• High Availability: Optional HA configurations
• TLS/SSL: Secure communication
• UI Interface: Web-based management (Zot, Harbor)

Registry Types

Zot (Recommended for Development)

Lightweight, fast, OCI-native registry with search and UI.

Pros:

• Fast startup and low resource usage
• Built-in UI and search
• Prometheus metrics
• Automatic garbage collection
• Good for development and small deployments

Cons:

• Less mature than Distribution
• Fewer enterprise features than Harbor

Use Cases:

• Development environments
• CI/CD pipelines
• Small to medium deployments
• Quick prototyping

Harbor (Recommended for Production)

Full-featured enterprise registry with replication, scanning, and RBAC.

Pros:

• Enterprise-grade features
• Vulnerability scanning (Trivy)
• Replication and mirroring
• Advanced RBAC
• Webhooks and notifications
• Mature and battle-tested

Cons:

• Higher resource requirements
• More complex setup
• Heavier than Zot/Distribution

Use Cases:

• Production deployments
• Multi-tenant environments
• Security-critical applications
• Large-scale deployments

Distribution (OCI Reference)

Official OCI registry reference implementation.

Pros:

• OCI standard compliance
• Lightweight and simple
• Well-documented
• Industry standard

Cons:

• No built-in UI
• No search functionality
• Manual garbage collection
• Basic feature set

Use Cases:

• OCI standard compliance required
• Minimal registry needs
• Custom integrations
• Educational purposes

Quick Start

Start Zot Registry (Default)

# Start Zot in background
cd provisioning/platform/oci-registry/zot
docker-compose up -d

# Initialize with namespaces and policies
nu ../scripts/init-registry.nu --registry-type zot

# Check health
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry health"

# Access UI
open http://localhost:5000
```plaintext

### Start Harbor Registry

```bash
# Start Harbor
cd provisioning/platform/oci-registry/harbor
docker-compose up -d

# Wait for services to be ready (takes ~2 minutes)
sleep 120

# Initialize
nu ../scripts/init-registry.nu --registry-type harbor --admin-password Harbor12345

# Access UI
open http://localhost
# Login: admin / Harbor12345
```plaintext

### Start Distribution Registry

```bash
# Start Distribution with UI
cd provisioning/platform/oci-registry/distribution
docker-compose up -d

# Initialize
nu ../scripts/init-registry.nu --registry-type distribution

# Access UI (if included)
open http://localhost:8080
```plaintext

## Installation

### Prerequisites

- **Docker** (20.10+)
- **Docker Compose** (2.0+)
- **Nushell** (0.107+)

### Setup

```bash
# Clone configurations (already included)
cd provisioning/platform/oci-registry

# Choose registry type
REGISTRY_TYPE="zot"  # or "harbor" or "distribution"

# Generate TLS certificates (optional, for HTTPS)
./scripts/generate-certs.nu

# Start registry
cd $REGISTRY_TYPE
docker-compose up -d

# Initialize
nu ../scripts/init-registry.nu --registry-type $REGISTRY_TYPE

# Verify
docker-compose ps
```plaintext

## Configuration

### Zot Configuration

**File**: `zot/config.json`

Key settings:

```json
{
  "storage": {
    "rootDirectory": "/var/lib/registry",
    "dedupe": true,
    "gc": true,
    "gcInterval": "24h"
  },
  "http": {
    "address": "0.0.0.0",
    "port": "5000"
  },
  "extensions": {
    "search": {"enable": true},
    "metrics": {"enable": true},
    "ui": {"enable": true}
  },
  "accessControl": {
    "repositories": {
      "provisioning-extensions/**": {
        "policies": [
          {
            "users": ["provisioning"],
            "actions": ["read", "create", "update", "delete"]
          }
        ]
      }
    }
  }
}
```plaintext

### Harbor Configuration

**File**: `harbor/harbor.yml`

Key settings:

```yaml
hostname: harbor.provisioning.local
harbor_admin_password: Harbor12345

database:
  password: root123

trivy:
  ignore_unfixed: false
  skip_update: false

log:
  level: info
```plaintext

### Distribution Configuration

**File**: `distribution/config.yml`

Key settings:

```yaml
storage:
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true

http:
  addr: :5000
  tls:
    certificate: /etc/docker/registry/certs/cert.pem
    key: /etc/docker/registry/certs/key.pem

auth:
  htpasswd:
    realm: Registry
    path: /etc/docker/registry/htpasswd
```plaintext

## Management

### Using Nushell Commands

```bash
# Start registry
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry start --type zot"

# Stop registry
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry stop --type zot"

# Check status
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry status --type zot"

# View logs
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry logs --type zot --follow"

# Health check
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry health --type zot"

# Initialize
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry init --type zot"

# List namespaces
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry namespaces"
```plaintext

### Using Docker Compose

```bash
# Start
cd provisioning/platform/oci-registry/zot
docker-compose up -d

# Stop
docker-compose down

# View logs
docker-compose logs -f

# Restart
docker-compose restart

# Remove (including volumes)
docker-compose down -v
```plaintext

## Namespaces

### Default Namespaces

| Namespace | Description | Public | Retention |
|-----------|-------------|--------|-----------|
| `provisioning-extensions` | Extension packages | No | 10 tags, 90 days |
| `provisioning-kcl` | KCL schemas | No | 20 tags, 180 days |
| `provisioning-platform` | Platform images | No | 5 tags, 30 days |
| `provisioning-test` | Test artifacts | Yes | 3 tags, 7 days |

### Manage Namespaces

```bash
# Setup all namespaces
nu scripts/setup-namespaces.nu --registry-type zot

# List namespaces
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry namespaces"

# Create namespace
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; \
  oci-registry namespace create my-namespace --type zot"

# Get namespace info
nu scripts/setup-namespaces.nu namespace info provisioning-extensions
```plaintext

## Access Control

### Policies

Default access policies:

**provisioning-extensions:**

- Authenticated: Read, Write, Delete
- Anonymous: None

**provisioning-kcl:**

- Authenticated: Read, Write
- Anonymous: None

**provisioning-platform:**

- Authenticated: Read only (except admin)
- Anonymous: None

**provisioning-test:**

- Authenticated: Read, Write, Delete
- Anonymous: Read only

### Configure Policies

```bash
# Apply all policies
nu scripts/configure-policies.nu --registry-type zot

# Show policy for namespace
nu scripts/configure-policies.nu policy show provisioning-extensions

# List all policies
nu scripts/configure-policies.nu policy list
```plaintext

### Authentication

**Zot/Distribution (htpasswd):**

```bash
# Create user
htpasswd -Bc htpasswd provisioning

# Login
docker login localhost:5000
```plaintext

**Harbor (Database):**

```bash
# Login via UI or CLI
docker login localhost
# Username: admin
# Password: Harbor12345

# Create users via Harbor UI
# Admin → Users → New User
```plaintext

## Monitoring

### Health Checks

```bash
# Full health check
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; \
  oci-registry health --type zot"

# API check
curl http://localhost:5000/v2/

# Catalog check
curl http://localhost:5000/v2/_catalog
```plaintext

### Metrics

**Zot:**

```bash
# Prometheus metrics
curl http://localhost:5000/metrics

# Visualize with Prometheus
# Add to prometheus.yml:
# - targets: ['localhost:5000']
```plaintext

**Distribution:**

```bash
# Metrics on debug port
curl http://localhost:5001/metrics
```plaintext

**Harbor:**

```bash
# Metrics endpoint
curl http://localhost:9090/metrics

# View in Harbor UI
# Admin → System Settings → Metrics
```plaintext

### Logs

```bash
# Zot logs
docker-compose logs -f zot

# Harbor logs
docker-compose logs -f core registry nginx

# Distribution logs
docker-compose logs -f registry

# Nushell command
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; \
  oci-registry logs --type zot --follow --tail 100"
```plaintext

## Troubleshooting

### Registry Not Starting

```bash
# Check Docker daemon
docker ps

# Check ports
lsof -i :5000

# View logs
docker-compose logs

# Rebuild
docker-compose down -v
docker-compose up -d --build
```plaintext

### Cannot Push Images

```bash
# Check authentication
docker login localhost:5000

# Check permissions
# Ensure user has write access to namespace

# Check storage
df -h  # Ensure disk space available

# Check registry health
curl http://localhost:5000/v2/
```plaintext

### Slow Performance

```bash
# Enable deduplication (Zot)
# In config.json: "dedupe": true

# Increase resources (Docker)
# Docker → Preferences → Resources

# Run garbage collection
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry/service; \
  run-oci-registry-gc --type zot"
```plaintext

### TLS/Certificate Issues

```bash
# Regenerate certificates
./scripts/generate-certs.nu

# Trust certificate
# macOS: Add to Keychain Access
# Linux: Copy to /usr/local/share/ca-certificates/

# Skip TLS verification (testing only)
docker login --insecure localhost:5000
```plaintext

## Advanced Usage

### High Availability (Harbor)

```yaml
# harbor/docker-compose.yml
# Add multiple registry instances
registry-1:
  image: goharbor/registry-photon:v2.9.0
  ...

registry-2:
  image: goharbor/registry-photon:v2.9.0
  ...

# Add load balancer
nginx:
  ...
  depends_on:
    - registry-1
    - registry-2
```plaintext

### S3 Backend (Distribution)

```yaml
# distribution/config.yml
storage:
  s3:
    accesskey: AKIAIOSFODNN7EXAMPLE
    secretkey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    region: us-west-1
    bucket: my-registry-bucket
    rootdirectory: /registry
```plaintext

### Replication (Harbor)

```bash
# Harbor UI → Replications → New Replication Rule
# Source: Local registry
# Destination: Remote registry
# Trigger: Manual/Scheduled/Event-based
```plaintext

### Webhooks

**Zot** (via config.json):

```json
{
  "http": {
    "notifications": {
      "endpoints": [
        {
          "name": "orchestrator",
          "url": "http://orchestrator:8080/registry/events",
          "headers": {
            "Authorization": ["Bearer token"]
          }
        }
      ]
    }
  }
}
```plaintext

**Harbor** (via scripts):

```bash
nu scripts/configure-policies.nu --registry-type harbor
# Webhooks configured automatically
```plaintext

### Garbage Collection

**Zot** (automatic):

```json
{
  "storage": {
    "gc": true,
    "gcInterval": "24h"
  }
}
```plaintext

**Distribution** (manual):

```bash
# Run GC
docker-compose exec registry \
  registry garbage-collect /etc/docker/registry/config.yml

# Or via Nushell
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry/service; \
  run-oci-registry-gc --type distribution"
```plaintext

**Harbor** (UI):

```plaintext
Admin → System Settings → Garbage Collection → Run GC
```plaintext

## API Reference

### OCI API (All Registries)

```bash
# List repositories
curl http://localhost:5000/v2/_catalog

# List tags
curl http://localhost:5000/v2/{repository}/tags/list

# Get manifest
curl http://localhost:5000/v2/{repository}/manifests/{tag}

# Delete image (requires delete enabled)
curl -X DELETE http://localhost:5000/v2/{repository}/manifests/{digest}
```plaintext

### Harbor API

```bash
# List projects
curl -u admin:Harbor12345 \
  http://localhost/api/v2.0/projects

# Create project
curl -X POST -u admin:Harbor12345 \
  -H "Content-Type: application/json" \
  -d '{"project_name":"test","metadata":{"public":"false"}}' \
  http://localhost/api/v2.0/projects

# Scan image
curl -X POST -u admin:Harbor12345 \
  http://localhost/api/v2.0/projects/{project}/repositories/{repo}/artifacts/{tag}/scan
```plaintext

## Performance Tuning

### Zot

```json
{
  "storage": {
    "dedupe": true,  // Enable deduplication
    "gc": true,      // Enable GC
    "gcInterval": "12h"  // More frequent GC
  },
  "http": {
    "http2": true  // Enable HTTP/2
  }
}
```plaintext

### Distribution

```yaml
storage:
  cache:
    blobdescriptor: redis  # Use Redis for caching

redis:
  addr: redis:6379
  pool:
    maxidle: 16
    maxactive: 64
```plaintext

### Harbor

```yaml
jobservice:
  max_job_workers: 20  # Increase concurrent jobs

database:
  max_idle_conns: 100
  max_open_conns: 900  # Increase DB connections
```plaintext

## Security Best Practices

1. **Use TLS/SSL** for all connections
2. **Strong passwords** for admin accounts
3. **Regular updates** of registry software
4. **Scan images** for vulnerabilities (Harbor/Trivy)
5. **Least privilege** access control
6. **Network isolation** (Docker networks)
7. **Regular backups** of registry data
8. **Audit logging** enabled
9. **Rate limiting** for API access
10. **Secrets management** (not in configs)

## Backup and Restore

### Backup

```bash
# Backup Zot
docker-compose stop zot
tar czf zot-backup-$(date +%Y%m%d).tar.gz \
  -C /var/lib/docker/volumes zot-data

# Backup Harbor
docker-compose stop
tar czf harbor-backup-$(date +%Y%m%d).tar.gz \
  -C /var/lib/docker/volumes \
  harbor-registry harbor-database

# Backup Distribution
docker-compose stop registry
tar czf dist-backup-$(date +%Y%m%d).tar.gz \
  -C /var/lib/docker/volumes registry-data
```plaintext

### Restore

```bash
# Restore (example for Zot)
docker-compose down -v
tar xzf zot-backup-20250106.tar.gz -C /var/lib/docker/volumes
docker-compose up -d
```plaintext

## Migration Between Registries

```bash
# Example: Zot → Harbor

# 1. Export from Zot
for repo in $(curl http://localhost:5000/v2/_catalog | jq -r '.repositories[]'); do
  for tag in $(curl http://localhost:5000/v2/$repo/tags/list | jq -r '.tags[]'); do
    docker pull localhost:5000/$repo:$tag
    docker tag localhost:5000/$repo:$tag harbor.local/$repo:$tag
    docker push harbor.local/$repo:$tag
  done
done

# 2. Or use skopeo
skopeo sync --src docker --dest docker \
  localhost:5000/provisioning-extensions \
  harbor.local/provisioning-extensions
```plaintext

## References

- **Zot**: <https://github.com/project-zot/zot>
- **Harbor**: <https://goharbor.io/docs/>
- **Distribution**: <https://github.com/distribution/distribution>
- **OCI Spec**: <https://github.com/opencontainers/distribution-spec>

## Support

For issues or questions:

1. Check logs: `docker-compose logs`
2. Review this documentation
3. Check GitHub issues for respective registry
4. Contact provisioning team

---

**Version**: 1.0.0
**Last Updated**: 2025-01-06
**Maintainer**: Provisioning Platform Team