provisioning/docs/src/integration/oci-registry-guide.md

OCI Registry User Guide

Version: 1.0.0 Date: 2025-10-06 Audience: Users and Developers

Table of Contents

1. Overview
2. Quick Start
3. OCI Commands Reference
4. Dependency Management
5. Extension Development
6. Registry Setup
7. Troubleshooting

---

Overview

The OCI registry integration enables distribution and management of provisioning extensions as OCI artifacts. This provides:

• Standard Distribution: Use industry-standard OCI registries
• Version Management: Proper semantic versioning for all extensions
• Dependency Resolution: Automatic dependency management
• Caching: Efficient caching to reduce downloads
• Security: TLS, authentication, and vulnerability scanning support

What are OCI Artifacts

OCI (Open Container Initiative) artifacts are packaged files distributed through container registries. Unlike Docker images which contain applications, OCI artifacts can contain any type of content - in our case, provisioning extensions (KCL schemas, Nushell scripts, templates, etc.).

---

Quick Start

Prerequisites

Install one of the following OCI tools:

# ORAS (recommended)
brew install oras

# Crane (Google's tool)
go install github.com/google/go-containerregistry/cmd/crane@latest

# Skopeo (RedHat's tool)
brew install skopeo
```plaintext

### 1. Start Local OCI Registry (Development)

```bash
# Start lightweight OCI registry (Zot)
provisioning oci-registry start

# Verify registry is running
curl http://localhost:5000/v2/_catalog
```plaintext

### 2. Pull an Extension

```bash
# Pull Kubernetes extension from registry
provisioning oci pull kubernetes:1.28.0

# Pull with specific registry
provisioning oci pull kubernetes:1.28.0 \
  --registry harbor.company.com \
  --namespace provisioning-extensions
```plaintext

### 3. List Available Extensions

```bash
# List all extensions
provisioning oci list

# Search for specific extension
provisioning oci search kubernetes

# Show available versions
provisioning oci tags kubernetes
```plaintext

### 4. Configure Workspace to Use OCI

Edit `workspace/config/provisioning.yaml`:

```yaml
dependencies:
  extensions:
    source_type: "oci"

    oci:
      registry: "localhost:5000"
      namespace: "provisioning-extensions"
      tls_enabled: false

    modules:
      taskservs:
        - "oci://localhost:5000/provisioning-extensions/kubernetes:1.28.0"
        - "oci://localhost:5000/provisioning-extensions/containerd:1.7.0"
```plaintext

### 5. Resolve Dependencies

```bash
# Resolve and install all dependencies
provisioning dep resolve

# Check what will be installed
provisioning dep resolve --dry-run

# Show dependency tree
provisioning dep tree kubernetes
```plaintext

---

## OCI Commands Reference

### Pull Extension

**Download extension from OCI registry**

```bash
provisioning oci pull <artifact>:<version> [OPTIONS]

# Examples:
provisioning oci pull kubernetes:1.28.0
provisioning oci pull redis:7.0.0 --registry harbor.company.com
provisioning oci pull postgres:15.0 --insecure  # Skip TLS verification
```plaintext

**Options**:

- `--registry <endpoint>`: Override registry (default: from config)
- `--namespace <name>`: Override namespace (default: provisioning-extensions)
- `--destination <path>`: Local installation path
- `--insecure`: Skip TLS certificate verification

---

### Push Extension

**Publish extension to OCI registry**

```bash
provisioning oci push <source-path> <name> <version> [OPTIONS]

# Examples:
provisioning oci push ./extensions/taskservs/redis redis 1.0.0
provisioning oci push ./my-provider aws 2.1.0 --registry localhost:5000
```plaintext

**Options**:

- `--registry <endpoint>`: Target registry
- `--namespace <name>`: Target namespace
- `--insecure`: Skip TLS verification

**Prerequisites**:

- Extension must have valid `manifest.yaml`
- Must be logged in to registry (see `oci login`)

---

### List Extensions

**Show available extensions in registry**

```bash
provisioning oci list [OPTIONS]

# Examples:
provisioning oci list
provisioning oci list --namespace provisioning-platform
provisioning oci list --registry harbor.company.com
```plaintext

**Output**:

```plaintext
┬───────────────┬──────────────────┬─────────────────────────┬─────────────────────────────────────────────┐
│ name          │ registry         │ namespace               │ reference                                   │
├───────────────┼──────────────────┼─────────────────────────┼─────────────────────────────────────────────┤
│ kubernetes    │ localhost:5000   │ provisioning-extensions │ localhost:5000/provisioning-extensions/...  │
│ containerd    │ localhost:5000   │ provisioning-extensions │ localhost:5000/provisioning-extensions/...  │
│ cilium        │ localhost:5000   │ provisioning-extensions │ localhost:5000/provisioning-extensions/...  │
└───────────────┴──────────────────┴─────────────────────────┴─────────────────────────────────────────────┘
```plaintext

---

### Search Extensions

**Search for extensions matching query**

```bash
provisioning oci search <query> [OPTIONS]

# Examples:
provisioning oci search kube
provisioning oci search postgres
provisioning oci search "container-*"
```plaintext

---

### Show Tags (Versions)

**Display all available versions of an extension**

```bash
provisioning oci tags <artifact-name> [OPTIONS]

# Examples:
provisioning oci tags kubernetes
provisioning oci tags redis --registry harbor.company.com
```plaintext

**Output**:

```plaintext
┬────────────┬─────────┬──────────────────────────────────────────────────────┐
│ artifact   │ version │ reference                                            │
├────────────┼─────────┼──────────────────────────────────────────────────────┤
│ kubernetes │ 1.29.0  │ localhost:5000/provisioning-extensions/kubernetes... │
│ kubernetes │ 1.28.0  │ localhost:5000/provisioning-extensions/kubernetes... │
│ kubernetes │ 1.27.0  │ localhost:5000/provisioning-extensions/kubernetes... │
└────────────┴─────────┴──────────────────────────────────────────────────────┘
```plaintext

---

### Inspect Extension

**Show detailed manifest and metadata**

```bash
provisioning oci inspect <artifact>:<version> [OPTIONS]

# Examples:
provisioning oci inspect kubernetes:1.28.0
provisioning oci inspect redis:7.0.0 --format json
```plaintext

**Output**:

```yaml
name: kubernetes
type: taskserv
version: 1.28.0
description: Kubernetes container orchestration platform
author: Provisioning Team
license: MIT
dependencies:
  containerd: ">=1.7.0"
  etcd: ">=3.5.0"
platforms:
  - linux/amd64
  - linux/arm64
```plaintext

---

### Login to Registry

**Authenticate with OCI registry**

```bash
provisioning oci login <registry> [OPTIONS]

# Examples:
provisioning oci login localhost:5000
provisioning oci login harbor.company.com --username admin
provisioning oci login registry.io --password-stdin < token.txt
provisioning oci login registry.io --token-file ~/.provisioning/tokens/registry
```plaintext

**Options**:

- `--username <user>`: Username (default: `_token`)
- `--password-stdin`: Read password from stdin
- `--token-file <path>`: Read token from file

**Note**: Credentials are stored in Docker config (`~/.docker/config.json`)

---

### Logout from Registry

**Remove stored credentials**

```bash
provisioning oci logout <registry>

# Example:
provisioning oci logout harbor.company.com
```plaintext

---

### Delete Extension

**Remove extension from registry**

```bash
provisioning oci delete <artifact>:<version> [OPTIONS]

# Examples:
provisioning oci delete kubernetes:1.27.0
provisioning oci delete redis:6.0.0 --force  # Skip confirmation
```plaintext

**Options**:

- `--force`: Skip confirmation prompt
- `--registry <endpoint>`: Target registry
- `--namespace <name>`: Target namespace

**Warning**: This operation is irreversible. Use with caution.

---

### Copy Extension

**Copy extension between registries**

```bash
provisioning oci copy <source> <destination> [OPTIONS]

# Examples:
# Copy between namespaces in same registry
provisioning oci copy \
  localhost:5000/test/kubernetes:1.28.0 \
  localhost:5000/production/kubernetes:1.28.0

# Copy between different registries
provisioning oci copy \
  localhost:5000/provisioning-extensions/kubernetes:1.28.0 \
  harbor.company.com/provisioning/kubernetes:1.28.0
```plaintext

---

### Show OCI Configuration

**Display current OCI settings**

```bash
provisioning oci config

# Output:
{
  tool: "oras"
  registry: "localhost:5000"
  namespace: {
    extensions: "provisioning-extensions"
    platform: "provisioning-platform"
  }
  cache_dir: "~/.provisioning/oci-cache"
  tls_enabled: false
}
```plaintext

---

## Dependency Management

### Dependency Configuration

Dependencies are configured in `workspace/config/provisioning.yaml`:

```yaml
dependencies:
  # Core provisioning system
  core:
    source: "oci://harbor.company.com/provisioning-core:v3.5.0"

  # Extensions (providers, taskservs, clusters)
  extensions:
    source_type: "oci"

    oci:
      registry: "localhost:5000"
      namespace: "provisioning-extensions"
      tls_enabled: false
      auth_token_path: "~/.provisioning/tokens/oci"

    modules:
      providers:
        - "oci://localhost:5000/provisioning-extensions/aws:2.0.0"
        - "oci://localhost:5000/provisioning-extensions/upcloud:1.5.0"

      taskservs:
        - "oci://localhost:5000/provisioning-extensions/kubernetes:1.28.0"
        - "oci://localhost:5000/provisioning-extensions/containerd:1.7.0"
        - "oci://localhost:5000/provisioning-extensions/etcd:3.5.0"

      clusters:
        - "oci://localhost:5000/provisioning-extensions/buildkit:0.12.0"

  # Platform services
  platform:
    source_type: "oci"
    oci:
      registry: "harbor.company.com"
      namespace: "provisioning-platform"
```plaintext

### Resolve Dependencies

```bash
# Resolve and install all configured dependencies
provisioning dep resolve

# Dry-run (show what would be installed)
provisioning dep resolve --dry-run

# Resolve with specific version constraints
provisioning dep resolve --update  # Update to latest versions
```plaintext

### Check for Updates

```bash
# Check all dependencies for updates
provisioning dep check-updates

# Output:
┬─────────────┬─────────┬────────┬──────────────────┐
│ name        │ current │ latest │ update_available │
├─────────────┼─────────┼────────┼──────────────────┤
│ kubernetes  │ 1.28.0  │ 1.29.0 │ true             │
│ containerd  │ 1.7.0   │ 1.7.0  │ false            │
│ etcd        │ 3.5.0   │ 3.5.1  │ true             │
└─────────────┴─────────┴────────┴──────────────────┘
```plaintext

### Update Dependency

```bash
# Update specific extension to latest version
provisioning dep update kubernetes

# Update to specific version
provisioning dep update kubernetes --version 1.29.0
```plaintext

### Dependency Tree

```bash
# Show dependency tree for extension
provisioning dep tree kubernetes

# Output:
kubernetes:1.28.0
├── containerd:1.7.0
│   └── runc:1.1.0
├── etcd:3.5.0
└── kubectl:1.28.0
```plaintext

### Validate Dependencies

```bash
# Validate dependency graph (check for cycles, conflicts)
provisioning dep validate

# Validate specific extension
provisioning dep validate kubernetes
```plaintext

---

## Extension Development

### Create New Extension

```bash
# Generate extension from template
provisioning generate extension taskserv redis

# Directory structure created:
# extensions/taskservs/redis/
# ├── schemas/
# │   ├── manifest.toml
# │   ├── main.ncl
# │   ├── version.ncl
# │   └── dependencies.ncl
# ├── scripts/
# │   ├── install.nu
# │   ├── check.nu
# │   └── uninstall.nu
# ├── templates/
# ├── docs/
# │   └── README.md
# ├── tests/
# └── manifest.yaml
```plaintext

### Extension Manifest

Edit `manifest.yaml`:

```yaml
name: redis
type: taskserv
version: 1.0.0
description: Redis in-memory data structure store
author: Your Name
license: MIT
homepage: https://redis.io
repository: https://gitea.example.com/provisioning-extensions/redis

dependencies:
  os: ">=1.0.0"  # Required OS taskserv

tags:
  - database
  - cache
  - key-value

platforms:
  - linux/amd64
  - linux/arm64

min_provisioning_version: "3.0.0"
```plaintext

### Test Extension Locally

```bash
# Load extension from local path
provisioning module load taskserv workspace_dev redis --source local

# Test installation
provisioning taskserv create redis --infra test-env --check

# Run tests
provisioning test extension redis
```plaintext

### Validate Extension

```bash
# Validate extension structure
provisioning oci package validate ./extensions/taskservs/redis

# Output:
✓ Extension structure valid
Warnings:
  - Missing docs/README.md (recommended)
```plaintext

### Package Extension

```bash
# Package as OCI artifact
provisioning oci package ./extensions/taskservs/redis

# Output: redis-1.0.0.tar.gz

# Inspect package
provisioning oci inspect-artifact redis-1.0.0.tar.gz
```plaintext

### Publish Extension

```bash
# Login to registry (one-time)
provisioning oci login localhost:5000

# Publish extension
provisioning oci push ./extensions/taskservs/redis redis 1.0.0

# Verify publication
provisioning oci tags redis

# Share with team
echo "Published: oci://localhost:5000/provisioning-extensions/redis:1.0.0"
```plaintext

---

## Registry Setup

### Local Registry (Development)

**Using Zot (lightweight)**:

```bash
# Start Zot registry
provisioning oci-registry start

# Configuration:
# - Endpoint: localhost:5000
# - Storage: ~/.provisioning/oci-registry/
# - No authentication
# - TLS disabled

# Stop registry
provisioning oci-registry stop

# Check status
provisioning oci-registry status
```plaintext

**Manual Zot Setup**:

```bash
# Install Zot
brew install project-zot/tap/zot

# Create config
cat > zot-config.json <<EOF
{
  "storage": {
    "rootDirectory": "/tmp/zot"
  },
  "http": {
    "address": "0.0.0.0",
    "port": "5000"
  },
  "log": {
    "level": "info"
  }
}
EOF

# Run Zot
zot serve zot-config.json
```plaintext

---

### Remote Registry (Production)

**Using Harbor**:

1. **Deploy Harbor**:

   ```bash
   # Using Docker Compose
   wget https://github.com/goharbor/harbor/releases/download/v2.9.0/harbor-offline-installer-v2.9.0.tgz
   tar xvf harbor-offline-installer-v2.9.0.tgz
   cd harbor
   ./install.sh

1. Configure Workspace:

   # workspace/config/provisioning.yaml
   dependencies:
     registry:
       type: "oci"
       oci:
         endpoint: "https://harbor.company.com"
         namespaces:
           extensions: "provisioning/extensions"
           platform: "provisioning/platform"
         tls_enabled: true
         auth_token_path: "~/.provisioning/tokens/harbor"
   

2. Login:

   provisioning oci login harbor.company.com --username admin
   

---

Troubleshooting

No OCI Tool Found

Error: "No OCI tool found. Install oras, crane, or skopeo"

Solution:

# Install ORAS (recommended)
brew install oras

# Or install Crane
go install github.com/google/go-containerregistry/cmd/crane@latest

# Or install Skopeo
brew install skopeo
```plaintext

---

### Connection Refused

**Error**: "Connection refused to localhost:5000"

**Solution**:

```bash
# Check if registry is running
curl http://localhost:5000/v2/_catalog

# Start local registry if not running
provisioning oci-registry start
```plaintext

---

### TLS Certificate Error

**Error**: "x509: certificate signed by unknown authority"

**Solution**:

```bash
# For development, use --insecure flag
provisioning oci pull kubernetes:1.28.0 --insecure

# For production, configure TLS properly in workspace config:
# dependencies:
#   extensions:
#     oci:
#       tls_enabled: true
#       # Add CA certificate to system trust store
```plaintext

---

### Authentication Failed

**Error**: "unauthorized: authentication required"

**Solution**:

```bash
# Login to registry
provisioning oci login localhost:5000

# Or provide auth token in config:
# dependencies:
#   extensions:
#     oci:
#       auth_token_path: "~/.provisioning/tokens/oci"
```plaintext

---

### Extension Not Found

**Error**: "Dependency not found: kubernetes"

**Solutions**:

1. **Check registry endpoint**:

   ```bash
   provisioning oci config

1. List available extensions:

   provisioning oci list
   

2. Check namespace:

   provisioning oci list --namespace provisioning-extensions
   

3. Verify extension exists:

   provisioning oci tags kubernetes
   

---

Dependency Resolution Failed

Error: "Circular dependency detected"

Solution:

# Validate dependency graph
provisioning dep validate kubernetes

# Check dependency tree
provisioning dep tree kubernetes

# Fix circular dependencies in extension manifests
```plaintext

---

## Best Practices

### Version Pinning

✅ **DO**: Pin to specific versions in production

```yaml
modules:
  taskservs:
    - "oci://registry/kubernetes:1.28.0"  # Specific version
```plaintext

❌ **DON'T**: Use `latest` tag in production

```yaml
modules:
  taskservs:
    - "oci://registry/kubernetes:latest"  # Unpredictable
```plaintext

---

### Semantic Versioning

✅ **DO**: Follow semver (MAJOR.MINOR.PATCH)

- `1.0.0` → `1.0.1`: Backward-compatible bug fix
- `1.0.0` → `1.1.0`: Backward-compatible new feature
- `1.0.0` → `2.0.0`: Breaking change

❌ **DON'T**: Use arbitrary version numbers

- `v1`, `version-2`, `latest-stable`

---

### Dependency Management

✅ **DO**: Specify version constraints

```yaml
dependencies:
  containerd: ">=1.7.0"
  etcd: "^3.5.0"  # 3.5.x compatible
```plaintext

❌ **DON'T**: Leave dependencies unversioned

```yaml
dependencies:
  containerd: "*"  # Too permissive
```plaintext

---

### Security

✅ **DO**:

- Use TLS for remote registries
- Rotate authentication tokens regularly
- Scan images for vulnerabilities (Harbor)
- Sign artifacts (cosign)

❌ **DON'T**:

- Use `--insecure` in production
- Store passwords in config files
- Skip certificate verification

---

## Related Documentation

- [Multi-Repository Architecture](../architecture/MULTI_REPO_ARCHITECTURE.md) - Overall architecture
- [Extension Development Guide](extension-development.md) - Create extensions
- [Dependency Resolution](dependency-resolution.md) - How dependencies work
- OCI Client Library - Low-level API

---

**Maintained By**: Documentation Team
**Last Updated**: 2025-10-06
**Next Review**: 2026-01-06