prvng_platform/oci-registry/README.md

# OCI Registry Service

Comprehensive OCI (Open Container Initiative) registry deployment and management for the provisioning system. Supports multiple registry implementations: **Zot** (lightweight), **Harbor** (full-featured), and **Distribution** (OCI reference implementation).

## Table of Contents

- [Overview](#overview)
- [Registry Types](#registry-types)
- [Quick Start](#quick-start)
- [Installation](#installation)
- [Configuration](#configuration)
- [Management](#management)
- [Namespaces](#namespaces)
- [Access Control](#access-control)
- [Monitoring](#monitoring)
- [Troubleshooting](#troubleshooting)
- [Advanced Usage](#advanced-usage)

## Overview

The OCI registry service provides artifact storage and distribution for:

- **Extension Packages**: Providers, taskservs, clusters
- **KCL Schemas**: Configuration schemas and modules
- **Platform Images**: Orchestrator, control-center, services
- **Test Artifacts**: Development and testing images

### Features

- **Multi-Registry Support**: Zot, Harbor, Distribution
- **Namespace Organization**: Logical separation of artifacts
- **Access Control**: RBAC, policies, authentication
- **Monitoring**: Prometheus metrics, health checks
- **Garbage Collection**: Automatic cleanup of unused artifacts
- **High Availability**: Optional HA configurations
- **TLS/SSL**: Secure communication
- **UI Interface**: Web-based management (Zot, Harbor)

## Registry Types

### Zot (Recommended for Development)

**Lightweight, fast, OCI-native registry with search and UI.**

**Pros:**
- Fast startup and low resource usage
- Built-in UI and search
- Prometheus metrics
- Automatic garbage collection
- Good for development and small deployments

**Cons:**
- Less mature than Distribution
- Fewer enterprise features than Harbor

**Use Cases:**
- Development environments
- CI/CD pipelines
- Small to medium deployments
- Quick prototyping

### Harbor (Recommended for Production)

**Full-featured enterprise registry with replication, scanning, and RBAC.**

**Pros:**
- Enterprise-grade features
- Vulnerability scanning (Trivy)
- Replication and mirroring
- Advanced RBAC
- Webhooks and notifications
- Mature and battle-tested

**Cons:**
- Higher resource requirements
- More complex setup
- Heavier than Zot/Distribution

**Use Cases:**
- Production deployments
- Multi-tenant environments
- Security-critical applications
- Large-scale deployments

### Distribution (OCI Reference)

**Official OCI registry reference implementation.**

**Pros:**
- OCI standard compliance
- Lightweight and simple
- Well-documented
- Industry standard

**Cons:**
- No built-in UI
- No search functionality
- Manual garbage collection
- Basic feature set

**Use Cases:**
- OCI standard compliance required
- Minimal registry needs
- Custom integrations
- Educational purposes

## Quick Start

### Start Zot Registry (Default)

```bash
# Start Zot in background
cd provisioning/platform/oci-registry/zot
docker-compose up -d

# Initialize with namespaces and policies
nu ../scripts/init-registry.nu --registry-type zot

# Check health
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry health"

# Access UI
open http://localhost:5000
```

### Start Harbor Registry

```bash
# Start Harbor
cd provisioning/platform/oci-registry/harbor
docker-compose up -d

# Wait for services to be ready (takes ~2 minutes)
sleep 120

# Initialize
nu ../scripts/init-registry.nu --registry-type harbor --admin-password Harbor12345

# Access UI
open http://localhost
# Login: admin / Harbor12345
```

### Start Distribution Registry

```bash
# Start Distribution with UI
cd provisioning/platform/oci-registry/distribution
docker-compose up -d

# Initialize
nu ../scripts/init-registry.nu --registry-type distribution

# Access UI (if included)
open http://localhost:8080
```

## Installation

### Prerequisites

- **Docker** (20.10+)
- **Docker Compose** (2.0+)
- **Nushell** (0.107+)

### Setup

```bash
# Clone configurations (already included)
cd provisioning/platform/oci-registry

# Choose registry type
REGISTRY_TYPE="zot"  # or "harbor" or "distribution"

# Generate TLS certificates (optional, for HTTPS)
./scripts/generate-certs.nu

# Start registry
cd $REGISTRY_TYPE
docker-compose up -d

# Initialize
nu ../scripts/init-registry.nu --registry-type $REGISTRY_TYPE

# Verify
docker-compose ps
```

## Configuration

### Zot Configuration

**File**: `zot/config.json`

Key settings:

```json
{
  "storage": {
    "rootDirectory": "/var/lib/registry",
    "dedupe": true,
    "gc": true,
    "gcInterval": "24h"
  },
  "http": {
    "address": "0.0.0.0",
    "port": "5000"
  },
  "extensions": {
    "search": {"enable": true},
    "metrics": {"enable": true},
    "ui": {"enable": true}
  },
  "accessControl": {
    "repositories": {
      "provisioning-extensions/**": {
        "policies": [
          {
            "users": ["provisioning"],
            "actions": ["read", "create", "update", "delete"]
          }
        ]
      }
    }
  }
}
```

### Harbor Configuration

**File**: `harbor/harbor.yml`

Key settings:

```yaml
hostname: harbor.provisioning.local
harbor_admin_password: Harbor12345

database:
  password: root123

trivy:
  ignore_unfixed: false
  skip_update: false

log:
  level: info
```

### Distribution Configuration

**File**: `distribution/config.yml`

Key settings:

```yaml
storage:
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true

http:
  addr: :5000
  tls:
    certificate: /etc/docker/registry/certs/cert.pem
    key: /etc/docker/registry/certs/key.pem

auth:
  htpasswd:
    realm: Registry
    path: /etc/docker/registry/htpasswd
```

## Management

### Using Nushell Commands

```bash
# Start registry
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry start --type zot"

# Stop registry
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry stop --type zot"

# Check status
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry status --type zot"

# View logs
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry logs --type zot --follow"

# Health check
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry health --type zot"

# Initialize
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry init --type zot"

# List namespaces
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry namespaces"
```

### Using Docker Compose

```bash
# Start
cd provisioning/platform/oci-registry/zot
docker-compose up -d

# Stop
docker-compose down

# View logs
docker-compose logs -f

# Restart
docker-compose restart

# Remove (including volumes)
docker-compose down -v
```

## Namespaces

### Default Namespaces

| Namespace | Description | Public | Retention |
|-----------|-------------|--------|-----------|
| `provisioning-extensions` | Extension packages | No | 10 tags, 90 days |
| `provisioning-kcl` | KCL schemas | No | 20 tags, 180 days |
| `provisioning-platform` | Platform images | No | 5 tags, 30 days |
| `provisioning-test` | Test artifacts | Yes | 3 tags, 7 days |

### Manage Namespaces

```bash
# Setup all namespaces
nu scripts/setup-namespaces.nu --registry-type zot

# List namespaces
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; oci-registry namespaces"

# Create namespace
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; \
  oci-registry namespace create my-namespace --type zot"

# Get namespace info
nu scripts/setup-namespaces.nu namespace info provisioning-extensions
```

## Access Control

### Policies

Default access policies:

**provisioning-extensions:**
- Authenticated: Read, Write, Delete
- Anonymous: None

**provisioning-kcl:**
- Authenticated: Read, Write
- Anonymous: None

**provisioning-platform:**
- Authenticated: Read only (except admin)
- Anonymous: None

**provisioning-test:**
- Authenticated: Read, Write, Delete
- Anonymous: Read only

### Configure Policies

```bash
# Apply all policies
nu scripts/configure-policies.nu --registry-type zot

# Show policy for namespace
nu scripts/configure-policies.nu policy show provisioning-extensions

# List all policies
nu scripts/configure-policies.nu policy list
```

### Authentication

**Zot/Distribution (htpasswd):**

```bash
# Create user
htpasswd -Bc htpasswd provisioning

# Login
docker login localhost:5000
```

**Harbor (Database):**

```bash
# Login via UI or CLI
docker login localhost
# Username: admin
# Password: Harbor12345

# Create users via Harbor UI
# Admin → Users → New User
```

## Monitoring

### Health Checks

```bash
# Full health check
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; \
  oci-registry health --type zot"

# API check
curl http://localhost:5000/v2/

# Catalog check
curl http://localhost:5000/v2/_catalog
```

### Metrics

**Zot:**

```bash
# Prometheus metrics
curl http://localhost:5000/metrics

# Visualize with Prometheus
# Add to prometheus.yml:
# - targets: ['localhost:5000']
```

**Distribution:**

```bash
# Metrics on debug port
curl http://localhost:5001/metrics
```

**Harbor:**

```bash
# Metrics endpoint
curl http://localhost:9090/metrics

# View in Harbor UI
# Admin → System Settings → Metrics
```

### Logs

```bash
# Zot logs
docker-compose logs -f zot

# Harbor logs
docker-compose logs -f core registry nginx

# Distribution logs
docker-compose logs -f registry

# Nushell command
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry; \
  oci-registry logs --type zot --follow --tail 100"
```

## Troubleshooting

### Registry Not Starting

```bash
# Check Docker daemon
docker ps

# Check ports
lsof -i :5000

# View logs
docker-compose logs

# Rebuild
docker-compose down -v
docker-compose up -d --build
```

### Cannot Push Images

```bash
# Check authentication
docker login localhost:5000

# Check permissions
# Ensure user has write access to namespace

# Check storage
df -h  # Ensure disk space available

# Check registry health
curl http://localhost:5000/v2/
```

### Slow Performance

```bash
# Enable deduplication (Zot)
# In config.json: "dedupe": true

# Increase resources (Docker)
# Docker → Preferences → Resources

# Run garbage collection
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry/service; \
  run-oci-registry-gc --type zot"
```

### TLS/Certificate Issues

```bash
# Regenerate certificates
./scripts/generate-certs.nu

# Trust certificate
# macOS: Add to Keychain Access
# Linux: Copy to /usr/local/share/ca-certificates/

# Skip TLS verification (testing only)
docker login --insecure localhost:5000
```

## Advanced Usage

### High Availability (Harbor)

```yaml
# harbor/docker-compose.yml
# Add multiple registry instances
registry-1:
  image: goharbor/registry-photon:v2.9.0
  ...

registry-2:
  image: goharbor/registry-photon:v2.9.0
  ...

# Add load balancer
nginx:
  ...
  depends_on:
    - registry-1
    - registry-2
```

### S3 Backend (Distribution)

```yaml
# distribution/config.yml
storage:
  s3:
    accesskey: AKIAIOSFODNN7EXAMPLE
    secretkey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    region: us-west-1
    bucket: my-registry-bucket
    rootdirectory: /registry
```

### Replication (Harbor)

```bash
# Harbor UI → Replications → New Replication Rule
# Source: Local registry
# Destination: Remote registry
# Trigger: Manual/Scheduled/Event-based
```

### Webhooks

**Zot** (via config.json):

```json
{
  "http": {
    "notifications": {
      "endpoints": [
        {
          "name": "orchestrator",
          "url": "http://orchestrator:8080/registry/events",
          "headers": {
            "Authorization": ["Bearer token"]
          }
        }
      ]
    }
  }
}
```

**Harbor** (via scripts):

```bash
nu scripts/configure-policies.nu --registry-type harbor
# Webhooks configured automatically
```

### Garbage Collection

**Zot** (automatic):

```json
{
  "storage": {
    "gc": true,
    "gcInterval": "24h"
  }
}
```

**Distribution** (manual):

```bash
# Run GC
docker-compose exec registry \
  registry garbage-collect /etc/docker/registry/config.yml

# Or via Nushell
nu -c "use provisioning/core/nulib/lib_provisioning/oci_registry/service; \
  run-oci-registry-gc --type distribution"
```

**Harbor** (UI):

```
Admin → System Settings → Garbage Collection → Run GC
```

## API Reference

### OCI API (All Registries)

```bash
# List repositories
curl http://localhost:5000/v2/_catalog

# List tags
curl http://localhost:5000/v2/{repository}/tags/list

# Get manifest
curl http://localhost:5000/v2/{repository}/manifests/{tag}

# Delete image (requires delete enabled)
curl -X DELETE http://localhost:5000/v2/{repository}/manifests/{digest}
```

### Harbor API

```bash
# List projects
curl -u admin:Harbor12345 \
  http://localhost/api/v2.0/projects

# Create project
curl -X POST -u admin:Harbor12345 \
  -H "Content-Type: application/json" \
  -d '{"project_name":"test","metadata":{"public":"false"}}' \
  http://localhost/api/v2.0/projects

# Scan image
curl -X POST -u admin:Harbor12345 \
  http://localhost/api/v2.0/projects/{project}/repositories/{repo}/artifacts/{tag}/scan
```

## Performance Tuning

### Zot

```json
{
  "storage": {
    "dedupe": true,  // Enable deduplication
    "gc": true,      // Enable GC
    "gcInterval": "12h"  // More frequent GC
  },
  "http": {
    "http2": true  // Enable HTTP/2
  }
}
```

### Distribution

```yaml
storage:
  cache:
    blobdescriptor: redis  # Use Redis for caching

redis:
  addr: redis:6379
  pool:
    maxidle: 16
    maxactive: 64
```

### Harbor

```yaml
jobservice:
  max_job_workers: 20  # Increase concurrent jobs

database:
  max_idle_conns: 100
  max_open_conns: 900  # Increase DB connections
```

## Security Best Practices

1. **Use TLS/SSL** for all connections
2. **Strong passwords** for admin accounts
3. **Regular updates** of registry software
4. **Scan images** for vulnerabilities (Harbor/Trivy)
5. **Least privilege** access control
6. **Network isolation** (Docker networks)
7. **Regular backups** of registry data
8. **Audit logging** enabled
9. **Rate limiting** for API access
10. **Secrets management** (not in configs)

## Backup and Restore

### Backup

```bash
# Backup Zot
docker-compose stop zot
tar czf zot-backup-$(date +%Y%m%d).tar.gz \
  -C /var/lib/docker/volumes zot-data

# Backup Harbor
docker-compose stop
tar czf harbor-backup-$(date +%Y%m%d).tar.gz \
  -C /var/lib/docker/volumes \
  harbor-registry harbor-database

# Backup Distribution
docker-compose stop registry
tar czf dist-backup-$(date +%Y%m%d).tar.gz \
  -C /var/lib/docker/volumes registry-data
```

### Restore

```bash
# Restore (example for Zot)
docker-compose down -v
tar xzf zot-backup-20250106.tar.gz -C /var/lib/docker/volumes
docker-compose up -d
```

## Migration Between Registries

```bash
# Example: Zot → Harbor

# 1. Export from Zot
for repo in $(curl http://localhost:5000/v2/_catalog | jq -r '.repositories[]'); do
  for tag in $(curl http://localhost:5000/v2/$repo/tags/list | jq -r '.tags[]'); do
    docker pull localhost:5000/$repo:$tag
    docker tag localhost:5000/$repo:$tag harbor.local/$repo:$tag
    docker push harbor.local/$repo:$tag
  done
done

# 2. Or use skopeo
skopeo sync --src docker --dest docker \
  localhost:5000/provisioning-extensions \
  harbor.local/provisioning-extensions
```

## References

- **Zot**: https://github.com/project-zot/zot
- **Harbor**: https://goharbor.io/docs/
- **Distribution**: https://github.com/distribution/distribution
- **OCI Spec**: https://github.com/opencontainers/distribution-spec

## Support

For issues or questions:

1. Check logs: `docker-compose logs`
2. Review this documentation
3. Check GitHub issues for respective registry
4. Contact provisioning team

---

**Version**: 1.0.0
**Last Updated**: 2025-01-06
**Maintainer**: Provisioning Platform Team