prvng_core/nulib/lib_provisioning/extensions

Extension Loading System

Version: 1.0.0 Status: Implemented Date: 2025-10-06

Overview

A comprehensive extension loading mechanism with OCI registry support, lazy loading, caching, and version resolution. Supports loading extensions from multiple sources: OCI registries, Gitea repositories, and local filesystems.

Architecture

Extension Loading System
├── OCI Client (oci/client.nu)
│   ├── Artifact pull/push operations
│   ├── Registry authentication
│   └── Manifest management
├── Cache System (cache.nu)
│   ├── Local artifact caching
│   ├── Cache index management
│   └── Automatic pruning
├── Loader (loader_oci.nu)
│   ├── Multi-source loading
│   ├── Lazy loading
│   └── Automatic source detection
├── Version Resolution (versions.nu)
│   ├── Semver parsing and comparison
│   ├── Constraint satisfaction (^, ~, ranges)
│   └── OCI tag resolution
├── Discovery (discovery.nu)
│   ├── Multi-source discovery
│   ├── Extension search
│   └── Metadata extraction
└── CLI Commands (commands.nu)
    ├── Load, search, list
    ├── Cache management
    └── Publishing

Features

1. Multi-Source Support

Load extensions from:

• OCI Registry: Container artifact registry (localhost:5000 by default)
• Gitea: Git repository hosting (planned)
• Local: Filesystem paths

2. Lazy Loading

Extensions are loaded on-demand:

1. Check if already in memory → return
2. Check cache → load from cache
3. Determine source (auto-detect or explicit)
4. Download from source
5. Cache locally
6. Load into memory

3. OCI Registry Integration

Full OCI artifact support:

• Pull artifacts with authentication
• Push extensions to registry
• List and search artifacts
• Version tag management
• Manifest metadata extraction

4. Caching System

Intelligent local caching:

• Cache directory: ~/.provisioning/cache/extensions/{type}/{name}/{version}/
• Cache index: JSON-based index for fast lookups
• Automatic pruning: Remove old cached versions
• Statistics: Track cache size and usage

5. Version Resolution

Semver-compliant version resolution:

• Exact: 1.2.3 → exactly version 1.2.3
• Caret: ^1.2.0 → >=1.2.0 <2.0.0 (compatible)
• Tilde: ~1.2.0 → >=1.2.0 <1.3.0 (approximately)
• Range: 1.2.0-1.5.0 → between versions
• Latest: * or latest → highest version

6. Discovery & Search

Multi-source extension discovery:

• Discover all extensions across sources
• Search by name or type
• Filter by extension type (provider, taskserv, cluster)
• Get available versions

Configuration

OCI Registry Configuration

Add to workspace config (workspace/config/local-overrides.toml):

[oci]
registry = "localhost:5000"
namespace = "provisioning-extensions"
auth_token_path = "~/.provisioning/oci-token"
insecure = false
timeout = 300
retry_count = 3

[extensions]
source_type = "auto"  # auto, oci, gitea, local

Environment Variables

• PROVISIONING_OCI_REGISTRY: Override OCI registry
• PROVISIONING_OCI_NAMESPACE: Override namespace
• PROVISIONING_EXTENSIONS_PATH: Additional extension paths

CLI Usage

Load Extension

# Load latest from auto-detected source
provisioning ext load kubernetes

# Load specific version from OCI
provisioning ext load kubernetes --version 1.28.0 --source oci

# Force reload
provisioning ext load kubernetes --force

# Load provider
provisioning ext load aws --type provider

Search Extensions

# Search all sources
provisioning ext search kubernetes

# Search OCI registry only
provisioning ext search kubernetes --source oci

# Search local only
provisioning ext search kube --source local

List Extensions

# List all extensions
provisioning ext list

# Filter by type
provisioning ext list --type taskserv

# JSON output
provisioning ext list --format json

# List from specific source
provisioning ext list --source oci

Extension Information

# Show extension info
provisioning ext info kubernetes

# Show specific version
provisioning ext info kubernetes --version 1.28.0

# Show versions
provisioning ext versions kubernetes

Cache Management

# List cached extensions
provisioning ext cache list

# Show cache statistics
provisioning ext cache stats

# Clear cache for specific extension
provisioning ext cache clear --type taskserv --name kubernetes

# Clear all cache
provisioning ext cache clear --all

# Prune old entries (older than 30 days)
provisioning ext cache prune --days 30

Pull to Cache

# Pull without loading
provisioning ext pull kubernetes --version 1.28.0

# Pull from specific source
provisioning ext pull redis --source oci

Publishing

# Publish to OCI registry
provisioning ext publish ./my-extension --version 1.0.0

# Publish to specific registry
provisioning ext publish ./my-extension \
  --version 1.0.0 \
  --registry localhost:5000 \
  --namespace my-namespace

# Force overwrite existing
provisioning ext publish ./my-extension --version 1.0.0 --force

Discovery

# Discover all extensions
provisioning ext discover

# Filter by type
provisioning ext discover --type taskserv

# Force refresh
provisioning ext discover --refresh

Test OCI Connection

# Test OCI registry connectivity
provisioning ext test-oci

Publishing Tool Usage

The standalone publishing tool provides additional commands:

# Publish extension
nu provisioning/tools/publish_extension.nu ./my-extension --version 1.0.0

# Dry run (validate without publishing)
nu provisioning/tools/publish_extension.nu ./my-extension --version 1.0.0 --dry-run

# List published extensions
nu provisioning/tools/publish_extension.nu list

# Show extension info
nu provisioning/tools/publish_extension.nu info kubernetes 1.28.0

# Delete extension
nu provisioning/tools/publish_extension.nu delete kubernetes 1.28.0 --force

Extension Structure

Required Files

my-extension/
├── extension.yaml       # Manifest (required)
├── nickel/                # Nickel schemas (optional)
│   ├── my-extension.ncl
│   └── nickel.mod
├── scripts/            # Scripts (optional)
│   └── install.nu
├── templates/          # Templates (optional)
│   └── config.yaml.j2
└── docs/              # Documentation (optional)
    └── README.md

Extension Manifest (extension.yaml)

extension:
  name: my-extension
  version: 1.0.0
  type: taskserv  # provider, taskserv, cluster
  description: My awesome extension
  author: Your Name <you@example.com>

  requires:
    - docker
    - kubernetes

  dependencies:
    - containerd
    - etcd

  metadata:
    homepage: https://example.com
    repository: https://github.com/user/extension
    license: MIT

API Reference

OCI Client (oci/client.nu)

Function	Description
oci-pull-artifact	Pull artifact from OCI registry
oci-push-artifact	Push artifact to OCI registry
oci-list-artifacts	List all artifacts in registry
oci-get-artifact-tags	Get tags for artifact
oci-get-artifact-manifest	Get manifest for artifact
oci-artifact-exists	Check if artifact exists
oci-delete-artifact	Delete artifact from registry
is-oci-available	Check OCI registry availability
test-oci-connection	Test connection and auth

Cache System (cache.nu)

Function	Description
get-from-cache	Get extension from cache
save-oci-to-cache	Save OCI artifact to cache
save-gitea-to-cache	Save Gitea artifact to cache
remove-from-cache	Remove from cache
clear-cache	Clear entire cache or specific type
list-cached	List cached extensions
get-cache-stats	Get cache statistics
prune-cache	Remove old cache entries

Loader (loader_oci.nu)

Function	Description
load-extension	Load extension from any source

Version Resolution (versions.nu)

Function	Description
resolve-version	Resolve version from spec
resolve-oci-version	Resolve from OCI tags
is-semver	Check if valid semver
compare-semver	Compare two versions
sort-by-semver	Sort versions
get-latest-version	Get latest from list
satisfies-constraint	Check constraint satisfaction

Discovery (discovery.nu)

Function	Description
discover-oci-extensions	Discover OCI extensions
discover-local-extensions	Discover local extensions
discover-all-extensions	Discover from all sources
search-extensions	Search extensions
list-extensions	List with formatting
get-extension-versions	Get available versions
get-oci-extension-metadata	Get OCI metadata

Testing

Run the test suite:

# Run all tests
nu provisioning/core/nulib/lib_provisioning/extensions/tests/run_all_tests.nu

# Run specific test suite
nu provisioning/core/nulib/lib_provisioning/extensions/tests/run_all_tests.nu --suite oci
nu provisioning/core/nulib/lib_provisioning/extensions/tests/run_all_tests.nu --suite cache
nu provisioning/core/nulib/lib_provisioning/extensions/tests/run_all_tests.nu --suite versions
nu provisioning/core/nulib/lib_provisioning/extensions/tests/run_all_tests.nu --suite discovery

# Run individual test
nu provisioning/core/nulib/lib_provisioning/extensions/tests/test_oci_client.nu
nu provisioning/core/nulib/lib_provisioning/extensions/tests/test_cache.nu
nu provisioning/core/nulib/lib_provisioning/extensions/tests/test_versions.nu
nu provisioning/core/nulib/lib_provisioning/extensions/tests/test_discovery.nu

Integration Examples

Example 1: Load Taskserv from OCI

use lib_provisioning/extensions/loader_oci.nu load-extension

let result = (load-extension "taskserv" "kubernetes" "^1.28.0" --source-type "oci")

if $result.success {
    print $"Loaded kubernetes:($result.version) from ($result.source)"
} else {
    print $"Failed: ($result.error)"
}

Example 2: Discover and Cache All Extensions

use lib_provisioning/extensions/discovery.nu discover-all-extensions
use lib_provisioning/extensions/loader_oci.nu load-extension

let extensions = (discover-all-extensions --include-oci)

for ext in $extensions {
    print $"Caching ($ext.name):($ext.latest)..."
    load-extension $ext.type $ext.name $ext.latest
}

Example 3: Version Resolution

use lib_provisioning/extensions/versions.nu resolve-oci-version

let version = (resolve-oci-version "taskserv" "kubernetes" "^1.28.0")
print $"Resolved to: ($version)"

Troubleshooting

OCI Registry Not Reachable

# Test connection
provisioning ext test-oci

# Check config
provisioning env | grep OCI

# Verify registry is running
curl http://localhost:5000/v2/

Extension Not Found

# Search all sources
provisioning ext search <name>

# Check specific source
provisioning ext list --source oci
provisioning ext list --source local

# Discover with refresh
provisioning ext discover --refresh

Cache Issues

# Check cache stats
provisioning ext cache stats

# Clear and rebuild
provisioning ext cache clear --all

# Prune old entries
provisioning ext cache prune --days 7

Version Resolution Issues

# Check available versions
provisioning ext versions <extension-name>

# Try explicit version
provisioning ext load <name> --version 1.2.3

# Force reload
provisioning ext load <name> --force

Performance Considerations

• Lazy Loading: Extensions loaded on-demand, not at startup
• Caching: Downloaded artifacts cached locally for fast access
• Parallel Discovery: Multiple sources discovered concurrently
• Index-Based Lookup: Cache index for O(1) lookups

Security

• Token-Based Auth: OCI registry authentication via tokens
• Manifest Validation: Extension structure validated before loading
• Permission Checks: Extension permission policies enforced
• Secure Defaults: HTTPS for registries (HTTP only for localhost)

Future Enhancements

• Gitea source implementation
• Digital signature verification
• Multi-registry support
• Extension dependency resolution
• Automatic updates
• Extension sandboxing
• WebAssembly extensions
• Extension marketplace UI

Contributing

See main project contributing guidelines. Extension system follows:

• Nushell idiomatic patterns
• PAP (Project Architecture Principles)
• Nickel idiomatic patterns for schemas

License

Same as main project.