provisioning/docs/src/integration/oci-registry-guide.md

# OCI Registry User Guide

**Version**: 1.0.0
**Date**: 2025-10-06
**Audience**: Users and Developers

## Table of Contents

1. [Overview](#overview)
2. [Quick Start](#quick-start)
3. [OCI Commands Reference](#oci-commands-reference)
4. [Dependency Management](#dependency-management)
5. [Extension Development](#extension-development)
6. [Registry Setup](#registry-setup)
7. [Troubleshooting](#troubleshooting)

---

## Overview

The OCI registry integration enables distribution and management of provisioning extensions as OCI artifacts. This provides:

- **Standard Distribution**: Use industry-standard OCI registries
- **Version Management**: Proper semantic versioning for all extensions
- **Dependency Resolution**: Automatic dependency management
- **Caching**: Efficient caching to reduce downloads
- **Security**: TLS, authentication, and vulnerability scanning support

### What are OCI Artifacts

OCI (Open Container Initiative) artifacts are packaged files distributed through container registries. Unlike Docker images which contain applications, OCI artifacts can contain any type of content - in our case, provisioning extensions (KCL schemas, Nushell scripts, templates, etc.).

---

## Quick Start

### Prerequisites

Install one of the following OCI tools:

```bash
# ORAS (recommended)
brew install oras

# Crane (Google's tool)
go install github.com/google/go-containerregistry/cmd/crane@latest

# Skopeo (RedHat's tool)
brew install skopeo
```plaintext

### 1. Start Local OCI Registry (Development)

```bash
# Start lightweight OCI registry (Zot)
provisioning oci-registry start

# Verify registry is running
curl http://localhost:5000/v2/_catalog
```plaintext

### 2. Pull an Extension

```bash
# Pull Kubernetes extension from registry
provisioning oci pull kubernetes:1.28.0

# Pull with specific registry
provisioning oci pull kubernetes:1.28.0 \
  --registry harbor.company.com \
  --namespace provisioning-extensions
```plaintext

### 3. List Available Extensions

```bash
# List all extensions
provisioning oci list

# Search for specific extension
provisioning oci search kubernetes

# Show available versions
provisioning oci tags kubernetes
```plaintext

### 4. Configure Workspace to Use OCI

Edit `workspace/config/provisioning.yaml`:

```yaml
dependencies:
  extensions:
    source_type: "oci"

    oci:
      registry: "localhost:5000"
      namespace: "provisioning-extensions"
      tls_enabled: false

    modules:
      taskservs:
        - "oci://localhost:5000/provisioning-extensions/kubernetes:1.28.0"
        - "oci://localhost:5000/provisioning-extensions/containerd:1.7.0"
```plaintext

### 5. Resolve Dependencies

```bash
# Resolve and install all dependencies
provisioning dep resolve

# Check what will be installed
provisioning dep resolve --dry-run

# Show dependency tree
provisioning dep tree kubernetes
```plaintext

---

## OCI Commands Reference

### Pull Extension

**Download extension from OCI registry**

```bash
provisioning oci pull <artifact>:<version> [OPTIONS]

# Examples:
provisioning oci pull kubernetes:1.28.0
provisioning oci pull redis:7.0.0 --registry harbor.company.com
provisioning oci pull postgres:15.0 --insecure  # Skip TLS verification
```plaintext

**Options**:

- `--registry <endpoint>`: Override registry (default: from config)
- `--namespace <name>`: Override namespace (default: provisioning-extensions)
- `--destination <path>`: Local installation path
- `--insecure`: Skip TLS certificate verification

---

### Push Extension

**Publish extension to OCI registry**

```bash
provisioning oci push <source-path> <name> <version> [OPTIONS]

# Examples:
provisioning oci push ./extensions/taskservs/redis redis 1.0.0
provisioning oci push ./my-provider aws 2.1.0 --registry localhost:5000
```plaintext

**Options**:

- `--registry <endpoint>`: Target registry
- `--namespace <name>`: Target namespace
- `--insecure`: Skip TLS verification

**Prerequisites**:

- Extension must have valid `manifest.yaml`
- Must be logged in to registry (see `oci login`)

---

### List Extensions

**Show available extensions in registry**

```bash
provisioning oci list [OPTIONS]

# Examples:
provisioning oci list
provisioning oci list --namespace provisioning-platform
provisioning oci list --registry harbor.company.com
```plaintext

**Output**:

```plaintext
┬───────────────┬──────────────────┬─────────────────────────┬─────────────────────────────────────────────┐
│ name          │ registry         │ namespace               │ reference                                   │
├───────────────┼──────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
│ kubernetes    │ localhost:5000   │ provisioning-extensions │ localhost:5000/provisioning-extensions/...  │
│ containerd    │ localhost:5000   │ provisioning-extensions │ localhost:5000/provisioning-extensions/...  │
│ cilium        │ localhost:5000   │ provisioning-extensions │ localhost:5000/provisioning-extensions/...  │
└───────────────┴──────────────────┴─────────────────────────┴─────────────────────────────────────────────┘
```plaintext

---

### Search Extensions

**Search for extensions matching query**

```bash
provisioning oci search <query> [OPTIONS]

# Examples:
provisioning oci search kube
provisioning oci search postgres
provisioning oci search "container-*"
```plaintext

---

### Show Tags (Versions)

**Display all available versions of an extension**

```bash
provisioning oci tags <artifact-name> [OPTIONS]

# Examples:
provisioning oci tags kubernetes
provisioning oci tags redis --registry harbor.company.com
```plaintext

**Output**:

```plaintext
┬────────────┬─────────┬──────────────────────────────────────────────────────┐
│ artifact   │ version │ reference                                            │
├────────────┼─────────┼──────────────────────────────────────────────────────┤
│ kubernetes │ 1.29.0  │ localhost:5000/provisioning-extensions/kubernetes... │
│ kubernetes │ 1.28.0  │ localhost:5000/provisioning-extensions/kubernetes... │
│ kubernetes │ 1.27.0  │ localhost:5000/provisioning-extensions/kubernetes... │
└────────────┴─────────┴──────────────────────────────────────────────────────┘
```plaintext

---

### Inspect Extension

**Show detailed manifest and metadata**

```bash
provisioning oci inspect <artifact>:<version> [OPTIONS]

# Examples:
provisioning oci inspect kubernetes:1.28.0
provisioning oci inspect redis:7.0.0 --format json
```plaintext

**Output**:

```yaml
name: kubernetes
type: taskserv
version: 1.28.0
description: Kubernetes container orchestration platform
author: Provisioning Team
license: MIT
dependencies:
  containerd: ">=1.7.0"
  etcd: ">=3.5.0"
platforms:
  - linux/amd64
  - linux/arm64
```plaintext

---

### Login to Registry

**Authenticate with OCI registry**

```bash
provisioning oci login <registry> [OPTIONS]

# Examples:
provisioning oci login localhost:5000
provisioning oci login harbor.company.com --username admin
provisioning oci login registry.io --password-stdin < token.txt
provisioning oci login registry.io --token-file ~/.provisioning/tokens/registry
```plaintext

**Options**:

- `--username <user>`: Username (default: `_token`)
- `--password-stdin`: Read password from stdin
- `--token-file <path>`: Read token from file

**Note**: Credentials are stored in Docker config (`~/.docker/config.json`)

---

### Logout from Registry

**Remove stored credentials**

```bash
provisioning oci logout <registry>

# Example:
provisioning oci logout harbor.company.com
```plaintext

---

### Delete Extension

**Remove extension from registry**

```bash
provisioning oci delete <artifact>:<version> [OPTIONS]

# Examples:
provisioning oci delete kubernetes:1.27.0
provisioning oci delete redis:6.0.0 --force  # Skip confirmation
```plaintext

**Options**:

- `--force`: Skip confirmation prompt
- `--registry <endpoint>`: Target registry
- `--namespace <name>`: Target namespace

**Warning**: This operation is irreversible. Use with caution.

---

### Copy Extension

**Copy extension between registries**

```bash
provisioning oci copy <source> <destination> [OPTIONS]

# Examples:
# Copy between namespaces in same registry
provisioning oci copy \
  localhost:5000/test/kubernetes:1.28.0 \
  localhost:5000/production/kubernetes:1.28.0

# Copy between different registries
provisioning oci copy \
  localhost:5000/provisioning-extensions/kubernetes:1.28.0 \
  harbor.company.com/provisioning/kubernetes:1.28.0
```plaintext

---

### Show OCI Configuration

**Display current OCI settings**

```bash
provisioning oci config

# Output:
{
  tool: "oras"
  registry: "localhost:5000"
  namespace: {
    extensions: "provisioning-extensions"
    platform: "provisioning-platform"
  }
  cache_dir: "~/.provisioning/oci-cache"
  tls_enabled: false
}
```plaintext

---

## Dependency Management

### Dependency Configuration

Dependencies are configured in `workspace/config/provisioning.yaml`:

```yaml
dependencies:
  # Core provisioning system
  core:
    source: "oci://harbor.company.com/provisioning-core:v3.5.0"

  # Extensions (providers, taskservs, clusters)
  extensions:
    source_type: "oci"

    oci:
      registry: "localhost:5000"
      namespace: "provisioning-extensions"
      tls_enabled: false
      auth_token_path: "~/.provisioning/tokens/oci"

    modules:
      providers:
        - "oci://localhost:5000/provisioning-extensions/aws:2.0.0"
        - "oci://localhost:5000/provisioning-extensions/upcloud:1.5.0"

      taskservs:
        - "oci://localhost:5000/provisioning-extensions/kubernetes:1.28.0"
        - "oci://localhost:5000/provisioning-extensions/containerd:1.7.0"
        - "oci://localhost:5000/provisioning-extensions/etcd:3.5.0"

      clusters:
        - "oci://localhost:5000/provisioning-extensions/buildkit:0.12.0"

  # Platform services
  platform:
    source_type: "oci"
    oci:
      registry: "harbor.company.com"
      namespace: "provisioning-platform"
```plaintext

### Resolve Dependencies

```bash
# Resolve and install all configured dependencies
provisioning dep resolve

# Dry-run (show what would be installed)
provisioning dep resolve --dry-run

# Resolve with specific version constraints
provisioning dep resolve --update  # Update to latest versions
```plaintext

### Check for Updates

```bash
# Check all dependencies for updates
provisioning dep check-updates

# Output:
┬─────────────┬─────────┬────────┬──────────────────┐
│ name        │ current │ latest │ update_available │
├─────────────┼─────────┼────────┼──────────────────┤
│ kubernetes  │ 1.28.0  │ 1.29.0 │ true             │
│ containerd  │ 1.7.0   │ 1.7.0  │ false            │
│ etcd        │ 3.5.0   │ 3.5.1  │ true             │
└─────────────┴─────────┴────────┴──────────────────┘
```plaintext

### Update Dependency

```bash
# Update specific extension to latest version
provisioning dep update kubernetes

# Update to specific version
provisioning dep update kubernetes --version 1.29.0
```plaintext

### Dependency Tree

```bash
# Show dependency tree for extension
provisioning dep tree kubernetes

# Output:
kubernetes:1.28.0
├── containerd:1.7.0
│   └── runc:1.1.0
├── etcd:3.5.0
└── kubectl:1.28.0
```plaintext

### Validate Dependencies

```bash
# Validate dependency graph (check for cycles, conflicts)
provisioning dep validate

# Validate specific extension
provisioning dep validate kubernetes
```plaintext

---

## Extension Development

### Create New Extension

```bash
# Generate extension from template
provisioning generate extension taskserv redis

# Directory structure created:
# extensions/taskservs/redis/
# ├── schemas/
# │   ├── manifest.toml
# │   ├── main.ncl
# │   ├── version.ncl
# │   └── dependencies.ncl
# ├── scripts/
# │   ├── install.nu
# │   ├── check.nu
# │   └── uninstall.nu
# ├── templates/
# ├── docs/
# │   └── README.md
# ├── tests/
# └── manifest.yaml
```plaintext

### Extension Manifest

Edit `manifest.yaml`:

```yaml
name: redis
type: taskserv
version: 1.0.0
description: Redis in-memory data structure store
author: Your Name
license: MIT
homepage: https://redis.io
repository: https://gitea.example.com/provisioning-extensions/redis

dependencies:
  os: ">=1.0.0"  # Required OS taskserv

tags:
  - database
  - cache
  - key-value

platforms:
  - linux/amd64
  - linux/arm64

min_provisioning_version: "3.0.0"
```plaintext

### Test Extension Locally

```bash
# Load extension from local path
provisioning module load taskserv workspace_dev redis --source local

# Test installation
provisioning taskserv create redis --infra test-env --check

# Run tests
provisioning test extension redis
```plaintext

### Validate Extension

```bash
# Validate extension structure
provisioning oci package validate ./extensions/taskservs/redis

# Output:
✓ Extension structure valid
Warnings:
  - Missing docs/README.md (recommended)
```plaintext

### Package Extension

```bash
# Package as OCI artifact
provisioning oci package ./extensions/taskservs/redis

# Output: redis-1.0.0.tar.gz

# Inspect package
provisioning oci inspect-artifact redis-1.0.0.tar.gz
```plaintext

### Publish Extension

```bash
# Login to registry (one-time)
provisioning oci login localhost:5000

# Publish extension
provisioning oci push ./extensions/taskservs/redis redis 1.0.0

# Verify publication
provisioning oci tags redis

# Share with team
echo "Published: oci://localhost:5000/provisioning-extensions/redis:1.0.0"
```plaintext

---

## Registry Setup

### Local Registry (Development)

**Using Zot (lightweight)**:

```bash
# Start Zot registry
provisioning oci-registry start

# Configuration:
# - Endpoint: localhost:5000
# - Storage: ~/.provisioning/oci-registry/
# - No authentication
# - TLS disabled

# Stop registry
provisioning oci-registry stop

# Check status
provisioning oci-registry status
```plaintext

**Manual Zot Setup**:

```bash
# Install Zot
brew install project-zot/tap/zot

# Create config
cat > zot-config.json <<EOF
{
  "storage": {
    "rootDirectory": "/tmp/zot"
  },
  "http": {
    "address": "0.0.0.0",
    "port": "5000"
  },
  "log": {
    "level": "info"
  }
}
EOF

# Run Zot
zot serve zot-config.json
```plaintext

---

### Remote Registry (Production)

**Using Harbor**:

1. **Deploy Harbor**:

   ```bash
   # Using Docker Compose
   wget https://github.com/goharbor/harbor/releases/download/v2.9.0/harbor-offline-installer-v2.9.0.tgz
   tar xvf harbor-offline-installer-v2.9.0.tgz
   cd harbor
   ./install.sh
   ```

1. **Configure Workspace**:

   ```yaml
   # workspace/config/provisioning.yaml
   dependencies:
     registry:
       type: "oci"
       oci:
         endpoint: "https://harbor.company.com"
         namespaces:
           extensions: "provisioning/extensions"
           platform: "provisioning/platform"
         tls_enabled: true
         auth_token_path: "~/.provisioning/tokens/harbor"
   ```

2. **Login**:

   ```bash
   provisioning oci login harbor.company.com --username admin
   ```

---

## Troubleshooting

### No OCI Tool Found

**Error**: "No OCI tool found. Install oras, crane, or skopeo"

**Solution**:

```bash
# Install ORAS (recommended)
brew install oras

# Or install Crane
go install github.com/google/go-containerregistry/cmd/crane@latest

# Or install Skopeo
brew install skopeo
```plaintext

---

### Connection Refused

**Error**: "Connection refused to localhost:5000"

**Solution**:

```bash
# Check if registry is running
curl http://localhost:5000/v2/_catalog

# Start local registry if not running
provisioning oci-registry start
```plaintext

---

### TLS Certificate Error

**Error**: "x509: certificate signed by unknown authority"

**Solution**:

```bash
# For development, use --insecure flag
provisioning oci pull kubernetes:1.28.0 --insecure

# For production, configure TLS properly in workspace config:
# dependencies:
#   extensions:
#     oci:
#       tls_enabled: true
#       # Add CA certificate to system trust store
```plaintext

---

### Authentication Failed

**Error**: "unauthorized: authentication required"

**Solution**:

```bash
# Login to registry
provisioning oci login localhost:5000

# Or provide auth token in config:
# dependencies:
#   extensions:
#     oci:
#       auth_token_path: "~/.provisioning/tokens/oci"
```plaintext

---

### Extension Not Found

**Error**: "Dependency not found: kubernetes"

**Solutions**:

1. **Check registry endpoint**:

   ```bash
   provisioning oci config
   ```

1. **List available extensions**:

   ```bash
   provisioning oci list
   ```

2. **Check namespace**:

   ```bash
   provisioning oci list --namespace provisioning-extensions
   ```

3. **Verify extension exists**:

   ```bash
   provisioning oci tags kubernetes
   ```

---

### Dependency Resolution Failed

**Error**: "Circular dependency detected"

**Solution**:

```bash
# Validate dependency graph
provisioning dep validate kubernetes

# Check dependency tree
provisioning dep tree kubernetes

# Fix circular dependencies in extension manifests
```plaintext

---

## Best Practices

### Version Pinning

✅ **DO**: Pin to specific versions in production

```yaml
modules:
  taskservs:
    - "oci://registry/kubernetes:1.28.0"  # Specific version
```plaintext

❌ **DON'T**: Use `latest` tag in production

```yaml
modules:
  taskservs:
    - "oci://registry/kubernetes:latest"  # Unpredictable
```plaintext

---

### Semantic Versioning

✅ **DO**: Follow semver (MAJOR.MINOR.PATCH)

- `1.0.0` → `1.0.1`: Backward-compatible bug fix
- `1.0.0` → `1.1.0`: Backward-compatible new feature
- `1.0.0` → `2.0.0`: Breaking change

❌ **DON'T**: Use arbitrary version numbers

- `v1`, `version-2`, `latest-stable`

---

### Dependency Management

✅ **DO**: Specify version constraints

```yaml
dependencies:
  containerd: ">=1.7.0"
  etcd: "^3.5.0"  # 3.5.x compatible
```plaintext

❌ **DON'T**: Leave dependencies unversioned

```yaml
dependencies:
  containerd: "*"  # Too permissive
```plaintext

---

### Security

✅ **DO**:

- Use TLS for remote registries
- Rotate authentication tokens regularly
- Scan images for vulnerabilities (Harbor)
- Sign artifacts (cosign)

❌ **DON'T**:

- Use `--insecure` in production
- Store passwords in config files
- Skip certificate verification

---

## Related Documentation

- [Multi-Repository Architecture](../architecture/MULTI_REPO_ARCHITECTURE.md) - Overall architecture
- [Extension Development Guide](extension-development.md) - Create extensions
- [Dependency Resolution](dependency-resolution.md) - How dependencies work
- OCI Client Library - Low-level API

---

**Maintained By**: Documentation Team
**Last Updated**: 2025-10-06
**Next Review**: 2026-01-06