prvng_core/nulib/lib_provisioning/extensions/README.md

# Extension Loading System

**Version**: 1.0.0
**Status**: Implemented
**Date**: 2025-10-06

## Overview

A comprehensive extension loading mechanism with OCI registry support, lazy loading, caching, and version resolution. Supports loading extensions from multiple sources: OCI registries, Gitea repositories, and local filesystems.

## Architecture

```
Extension Loading System
├── OCI Client (oci/client.nu)
│   ├── Artifact pull/push operations
│   ├── Registry authentication
│   └── Manifest management
├── Cache System (cache.nu)
│   ├── Local artifact caching
│   ├── Cache index management
│   └── Automatic pruning
├── Loader (loader_oci.nu)
│   ├── Multi-source loading
│   ├── Lazy loading
│   └── Automatic source detection
├── Version Resolution (versions.nu)
│   ├── Semver parsing and comparison
│   ├── Constraint satisfaction (^, ~, ranges)
│   └── OCI tag resolution
├── Discovery (discovery.nu)
│   ├── Multi-source discovery
│   ├── Extension search
│   └── Metadata extraction
└── CLI Commands (commands.nu)
    ├── Load, search, list
    ├── Cache management
    └── Publishing
```

## Features

### 1. Multi-Source Support

Load extensions from:
- **OCI Registry**: Container artifact registry (localhost:5000 by default)
- **Gitea**: Git repository hosting (planned)
- **Local**: Filesystem paths

### 2. Lazy Loading

Extensions are loaded on-demand:
1. Check if already in memory → return
2. Check cache → load from cache
3. Determine source (auto-detect or explicit)
4. Download from source
5. Cache locally
6. Load into memory

### 3. OCI Registry Integration

Full OCI artifact support:
- Pull artifacts with authentication
- Push extensions to registry
- List and search artifacts
- Version tag management
- Manifest metadata extraction

### 4. Caching System

Intelligent local caching:
- Cache directory: `~/.provisioning/cache/extensions/{type}/{name}/{version}/`
- Cache index: JSON-based index for fast lookups
- Automatic pruning: Remove old cached versions
- Statistics: Track cache size and usage

### 5. Version Resolution

Semver-compliant version resolution:
- **Exact**: `1.2.3` → exactly version 1.2.3
- **Caret**: `^1.2.0` → >=1.2.0 <2.0.0 (compatible)
- **Tilde**: `~1.2.0` → >=1.2.0 <1.3.0 (approximately)
- **Range**: `1.2.0-1.5.0` → between versions
- **Latest**: `*` or `latest` → highest version

### 6. Discovery & Search

Multi-source extension discovery:
- Discover all extensions across sources
- Search by name or type
- Filter by extension type (provider, taskserv, cluster)
- Get available versions

## Configuration

### OCI Registry Configuration

Add to workspace config (`workspace/config/local-overrides.toml`):

```toml
[oci]
registry = "localhost:5000"
namespace = "provisioning-extensions"
auth_token_path = "~/.provisioning/oci-token"
insecure = false
timeout = 300
retry_count = 3

[extensions]
source_type = "auto"  # auto, oci, gitea, local
```

### Environment Variables

- `PROVISIONING_OCI_REGISTRY`: Override OCI registry
- `PROVISIONING_OCI_NAMESPACE`: Override namespace
- `PROVISIONING_EXTENSIONS_PATH`: Additional extension paths

## CLI Usage

### Load Extension

```bash
# Load latest from auto-detected source
provisioning ext load kubernetes

# Load specific version from OCI
provisioning ext load kubernetes --version 1.28.0 --source oci

# Force reload
provisioning ext load kubernetes --force

# Load provider
provisioning ext load aws --type provider
```

### Search Extensions

```bash
# Search all sources
provisioning ext search kubernetes

# Search OCI registry only
provisioning ext search kubernetes --source oci

# Search local only
provisioning ext search kube --source local
```

### List Extensions

```bash
# List all extensions
provisioning ext list

# Filter by type
provisioning ext list --type taskserv

# JSON output
provisioning ext list --format json

# List from specific source
provisioning ext list --source oci
```

### Extension Information

```bash
# Show extension info
provisioning ext info kubernetes

# Show specific version
provisioning ext info kubernetes --version 1.28.0

# Show versions
provisioning ext versions kubernetes
```

### Cache Management

```bash
# List cached extensions
provisioning ext cache list

# Show cache statistics
provisioning ext cache stats

# Clear cache for specific extension
provisioning ext cache clear --type taskserv --name kubernetes

# Clear all cache
provisioning ext cache clear --all

# Prune old entries (older than 30 days)
provisioning ext cache prune --days 30
```

### Pull to Cache

```bash
# Pull without loading
provisioning ext pull kubernetes --version 1.28.0

# Pull from specific source
provisioning ext pull redis --source oci
```

### Publishing

```bash
# Publish to OCI registry
provisioning ext publish ./my-extension --version 1.0.0

# Publish to specific registry
provisioning ext publish ./my-extension \
  --version 1.0.0 \
  --registry localhost:5000 \
  --namespace my-namespace

# Force overwrite existing
provisioning ext publish ./my-extension --version 1.0.0 --force
```

### Discovery

```bash
# Discover all extensions
provisioning ext discover

# Filter by type
provisioning ext discover --type taskserv

# Force refresh
provisioning ext discover --refresh
```

### Test OCI Connection

```bash
# Test OCI registry connectivity
provisioning ext test-oci
```

## Publishing Tool Usage

The standalone publishing tool provides additional commands:

```bash
# Publish extension
nu provisioning/tools/publish_extension.nu ./my-extension --version 1.0.0

# Dry run (validate without publishing)
nu provisioning/tools/publish_extension.nu ./my-extension --version 1.0.0 --dry-run

# List published extensions
nu provisioning/tools/publish_extension.nu list

# Show extension info
nu provisioning/tools/publish_extension.nu info kubernetes 1.28.0

# Delete extension
nu provisioning/tools/publish_extension.nu delete kubernetes 1.28.0 --force
```

## Extension Structure

### Required Files

```
my-extension/
├── extension.yaml       # Manifest (required)
├── kcl/                # KCL schemas (optional)
│   ├── my-extension.k
│   └── kcl.mod
├── scripts/            # Scripts (optional)
│   └── install.nu
├── templates/          # Templates (optional)
│   └── config.yaml.j2
└── docs/              # Documentation (optional)
    └── README.md
```

### Extension Manifest (extension.yaml)

```yaml
extension:
  name: my-extension
  version: 1.0.0
  type: taskserv  # provider, taskserv, cluster
  description: My awesome extension
  author: Your Name <you@example.com>

  requires:
    - docker
    - kubernetes

  dependencies:
    - containerd
    - etcd

  metadata:
    homepage: https://example.com
    repository: https://github.com/user/extension
    license: MIT
```

## API Reference

### OCI Client (oci/client.nu)

| Function | Description |
|----------|-------------|
| `oci-pull-artifact` | Pull artifact from OCI registry |
| `oci-push-artifact` | Push artifact to OCI registry |
| `oci-list-artifacts` | List all artifacts in registry |
| `oci-get-artifact-tags` | Get tags for artifact |
| `oci-get-artifact-manifest` | Get manifest for artifact |
| `oci-artifact-exists` | Check if artifact exists |
| `oci-delete-artifact` | Delete artifact from registry |
| `is-oci-available` | Check OCI registry availability |
| `test-oci-connection` | Test connection and auth |

### Cache System (cache.nu)

| Function | Description |
|----------|-------------|
| `get-from-cache` | Get extension from cache |
| `save-oci-to-cache` | Save OCI artifact to cache |
| `save-gitea-to-cache` | Save Gitea artifact to cache |
| `remove-from-cache` | Remove from cache |
| `clear-cache` | Clear entire cache or specific type |
| `list-cached` | List cached extensions |
| `get-cache-stats` | Get cache statistics |
| `prune-cache` | Remove old cache entries |

### Loader (loader_oci.nu)

| Function | Description |
|----------|-------------|
| `load-extension` | Load extension from any source |

### Version Resolution (versions.nu)

| Function | Description |
|----------|-------------|
| `resolve-version` | Resolve version from spec |
| `resolve-oci-version` | Resolve from OCI tags |
| `is-semver` | Check if valid semver |
| `compare-semver` | Compare two versions |
| `sort-by-semver` | Sort versions |
| `get-latest-version` | Get latest from list |
| `satisfies-constraint` | Check constraint satisfaction |

### Discovery (discovery.nu)

| Function | Description |
|----------|-------------|
| `discover-oci-extensions` | Discover OCI extensions |
| `discover-local-extensions` | Discover local extensions |
| `discover-all-extensions` | Discover from all sources |
| `search-extensions` | Search extensions |
| `list-extensions` | List with formatting |
| `get-extension-versions` | Get available versions |
| `get-oci-extension-metadata` | Get OCI metadata |

## Testing

Run the test suite:

```bash
# Run all tests
nu provisioning/core/nulib/lib_provisioning/extensions/tests/run_all_tests.nu

# Run specific test suite
nu provisioning/core/nulib/lib_provisioning/extensions/tests/run_all_tests.nu --suite oci
nu provisioning/core/nulib/lib_provisioning/extensions/tests/run_all_tests.nu --suite cache
nu provisioning/core/nulib/lib_provisioning/extensions/tests/run_all_tests.nu --suite versions
nu provisioning/core/nulib/lib_provisioning/extensions/tests/run_all_tests.nu --suite discovery

# Run individual test
nu provisioning/core/nulib/lib_provisioning/extensions/tests/test_oci_client.nu
nu provisioning/core/nulib/lib_provisioning/extensions/tests/test_cache.nu
nu provisioning/core/nulib/lib_provisioning/extensions/tests/test_versions.nu
nu provisioning/core/nulib/lib_provisioning/extensions/tests/test_discovery.nu
```

## Integration Examples

### Example 1: Load Taskserv from OCI

```nushell
use lib_provisioning/extensions/loader_oci.nu load-extension

let result = (load-extension "taskserv" "kubernetes" "^1.28.0" --source-type "oci")

if $result.success {
    print $"Loaded kubernetes:($result.version) from ($result.source)"
} else {
    print $"Failed: ($result.error)"
}
```

### Example 2: Discover and Cache All Extensions

```nushell
use lib_provisioning/extensions/discovery.nu discover-all-extensions
use lib_provisioning/extensions/loader_oci.nu load-extension

let extensions = (discover-all-extensions --include-oci)

for ext in $extensions {
    print $"Caching ($ext.name):($ext.latest)..."
    load-extension $ext.type $ext.name $ext.latest
}
```

### Example 3: Version Resolution

```nushell
use lib_provisioning/extensions/versions.nu resolve-oci-version

let version = (resolve-oci-version "taskserv" "kubernetes" "^1.28.0")
print $"Resolved to: ($version)"
```

## Troubleshooting

### OCI Registry Not Reachable

```bash
# Test connection
provisioning ext test-oci

# Check config
provisioning env | grep OCI

# Verify registry is running
curl http://localhost:5000/v2/
```

### Extension Not Found

```bash
# Search all sources
provisioning ext search <name>

# Check specific source
provisioning ext list --source oci
provisioning ext list --source local

# Discover with refresh
provisioning ext discover --refresh
```

### Cache Issues

```bash
# Check cache stats
provisioning ext cache stats

# Clear and rebuild
provisioning ext cache clear --all

# Prune old entries
provisioning ext cache prune --days 7
```

### Version Resolution Issues

```bash
# Check available versions
provisioning ext versions <extension-name>

# Try explicit version
provisioning ext load <name> --version 1.2.3

# Force reload
provisioning ext load <name> --force
```

## Performance Considerations

- **Lazy Loading**: Extensions loaded on-demand, not at startup
- **Caching**: Downloaded artifacts cached locally for fast access
- **Parallel Discovery**: Multiple sources discovered concurrently
- **Index-Based Lookup**: Cache index for O(1) lookups

## Security

- **Token-Based Auth**: OCI registry authentication via tokens
- **Manifest Validation**: Extension structure validated before loading
- **Permission Checks**: Extension permission policies enforced
- **Secure Defaults**: HTTPS for registries (HTTP only for localhost)

## Future Enhancements

- [ ] Gitea source implementation
- [ ] Digital signature verification
- [ ] Multi-registry support
- [ ] Extension dependency resolution
- [ ] Automatic updates
- [ ] Extension sandboxing
- [ ] WebAssembly extensions
- [ ] Extension marketplace UI

## Contributing

See main project contributing guidelines. Extension system follows:
- Nushell idiomatic patterns
- PAP (Project Architecture Principles)
- KCL idiomatic patterns for schemas

## License

Same as main project.