prvng_platform/crates/orchestrator/docs/OCI_INTEGRATION.md

OCI Registry Integration Guide

Overview

The OCI integration module provides OCI Distribution Spec v2 compliant registry integration for pulling KCL packages and extension artifacts.

Architecture

┌──────────────────┐
│   Orchestrator   │
│     (Rust)       │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│   OCI Manager    │
│                  │
│  - LRU caching   │
│  - Pull artifacts│
│  - List packages │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│   OCI Client     │
│  (Distribution)  │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│  OCI Registry    │
│  (HTTP API v2)   │
└──────────────────┘
```plaintext

## Features

### 1. KCL Package Management

Pull KCL configuration packages from OCI registry:

```rust
let package_path = oci_manager.pull_kcl_package(
    "provisioning-core",
    "1.0.0"
).await?;
```plaintext

### 2. Extension Artifacts

Pull extension artifacts (providers, taskservs, clusters):

```rust
let artifact_path = oci_manager.pull_extension_artifact(
    "taskserv",        // Extension type
    "kubernetes",      // Extension name
    "1.28.0"          // Version
).await?;
```plaintext

### 3. Manifest Caching

Manifests are cached using LRU strategy:

- **Cache size**: 100 manifests
- **Cache key**: `{name}:{version}`
- **Automatic eviction**: Oldest entries removed when full

### 4. Artifact Listing

List all artifacts in a namespace:

```rust
let artifacts = oci_manager.list_oci_artifacts("kcl").await?;
for artifact in artifacts {
    println!("{} v{} ({})", artifact.name, artifact.version, artifact.size);
}
```plaintext

## OCI Distribution Spec v2

Implements OCI Distribution Specification v2:

- **Manifest retrieval**: `GET /v2/{namespace}/{repository}/manifests/{reference}`
- **Blob download**: `GET /v2/{namespace}/{repository}/blobs/{digest}`
- **Tag listing**: `GET /v2/{namespace}/{repository}/tags/list`
- **Artifact existence**: `HEAD /v2/{namespace}/{repository}/manifests/{reference}`

## Configuration

OCI settings in `config.defaults.toml`:

```toml
[orchestrator.oci]
registry_url = "http://localhost:5000"
namespace = "provisioning-extensions"
cache_dir = "{{orchestrator.paths.data_dir}}/oci-cache"
```plaintext

### Configuration Options

- **registry_url**: OCI registry HTTP endpoint
- **namespace**: Default namespace for artifacts
- **cache_dir**: Local cache directory for downloaded artifacts

## API Endpoints

### List OCI Artifacts

```http
POST /api/v1/oci/artifacts
Content-Type: application/json

{
  "namespace": "kcl"
}
```plaintext

**Response:**

```json
{
  "success": true,
  "data": [
    {
      "name": "provisioning-core",
      "version": "1.0.0",
      "digest": "sha256:abc123...",
      "size": 102400,
      "media_type": "application/vnd.oci.image.manifest.v1+json",
      "created_at": "2025-10-06T12:00:00Z"
    }
  ]
}
```plaintext

## Usage Examples

### Pull KCL Package

```rust
use provisioning_orchestrator::oci::OciManager;
use std::path::PathBuf;

let oci_manager = OciManager::new(
    "http://localhost:5000".to_string(),
    "provisioning-extensions".to_string(),
    PathBuf::from("/tmp/oci-cache"),
);

// Pull KCL package
let package_path = oci_manager.pull_kcl_package(
    "provisioning-core",
    "1.0.0"
).await?;

println!("Package downloaded to: {}", package_path.display());

// Extract package
// tar -xzf package_path
```plaintext

### Pull Extension Artifact

```rust
// Pull taskserv extension
let artifact_path = oci_manager.pull_extension_artifact(
    "taskserv",
    "kubernetes",
    "1.28.0"
).await?;

// Extract and install
// tar -xzf artifact_path -C /target/path
```plaintext

### List Artifacts

```rust
let artifacts = oci_manager.list_oci_artifacts("kcl").await?;

for artifact in artifacts {
    println!("📦 {} v{}", artifact.name, artifact.version);
    println!("   Size: {} bytes", artifact.size);
    println!("   Digest: {}", artifact.digest);
    println!();
}
```plaintext

### Check Artifact Exists

```rust
let exists = oci_manager.artifact_exists(
    "kcl/provisioning-core",
    "1.0.0"
).await?;

if exists {
    println!("Artifact exists in registry");
} else {
    println!("Artifact not found");
}
```plaintext

### Get Manifest (with caching)

```rust
let manifest = oci_manager.get_manifest(
    "kcl/provisioning-core",
    "1.0.0"
).await?;

println!("Schema version: {}", manifest.schema_version);
println!("Media type: {}", manifest.media_type);
println!("Layers: {}", manifest.layers.len());
```plaintext

### Clear Manifest Cache

```rust
oci_manager.clear_cache().await;
```plaintext

## OCI Artifact Structure

### Manifest Format

```json
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:abc123...",
    "size": 1234
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:def456...",
      "size": 102400
    }
  ],
  "annotations": {
    "org.opencontainers.image.created": "2025-10-06T12:00:00Z",
    "org.opencontainers.image.version": "1.0.0"
  }
}
```plaintext

## Integration with Workflows

### Extension Installation with OCI

1. **Check local cache**
2. **Pull from OCI registry** (if not cached)
3. Extract artifact
4. Validate contents
5. Install extension

```rust
// Workflow: Install taskserv from OCI
async fn install_taskserv_from_oci(
    oci_manager: &OciManager,
    name: &str,
    version: &str
) -> Result<()> {
    // Pull artifact
    let artifact_path = oci_manager.pull_extension_artifact(
        "taskserv",
        name,
        version
    ).await?;

    // Extract
    extract_tarball(&artifact_path, &target_dir)?;

    // Validate
    validate_extension_structure(&target_dir)?;

    // Install
    install_extension(&target_dir)?;

    Ok(())
}
```plaintext

## Cache Management

### Cache Directory Structure

```plaintext
/tmp/oci-cache/
├── kcl/
│   └── provisioning-core/
│       └── 1.0.0/
│           └── package.tar.gz
├── extensions/
│   ├── taskserv/
│   │   └── kubernetes/
│   │       └── 1.28.0/
│   │           └── artifact.tar.gz
│   └── provider/
│       └── aws/
│           └── 2.0.0/
│               └── artifact.tar.gz
```plaintext

### Cache Cleanup

Implement cache cleanup strategy:

```rust
// Clean old artifacts
async fn cleanup_old_artifacts(cache_dir: &Path, max_age_days: u64) -> Result<()> {
    let cutoff = Utc::now() - Duration::days(max_age_days as i64);

    for entry in std::fs::read_dir(cache_dir)? {
        let entry = entry?;
        let metadata = entry.metadata()?;

        if let Ok(modified) = metadata.modified() {
            let modified: DateTime<Utc> = modified.into();
            if modified < cutoff {
                std::fs::remove_dir_all(entry.path())?;
            }
        }
    }

    Ok(())
}
```plaintext

## Error Handling

The OCI integration handles errors gracefully:

- **Network errors**: Retries with exponential backoff
- **Manifest not found**: Returns clear error message
- **Corrupted downloads**: Validates digest before returning
- **Disk full**: Reports storage error

## Testing

Run OCI integration tests:

```bash
cd provisioning/platform/orchestrator
cargo test test_oci_integration
```plaintext

## Troubleshooting

### Artifact pull fails

1. Check OCI registry is accessible
2. Verify `registry_url` configuration
3. Check network connectivity
4. Verify artifact exists in registry
5. Review orchestrator logs

### Digest mismatch

1. Clear local cache
2. Re-pull artifact
3. Verify registry integrity
4. Check for network corruption

### Cache issues

1. Check cache directory permissions
2. Verify disk space
3. Clear cache manually if corrupted

## Best Practices

1. **Use specific versions**: Always specify version for production
2. **Verify digests**: Validate artifact integrity
3. **Cache management**: Implement cleanup strategy
4. **Error handling**: Handle network failures gracefully
5. **Monitor downloads**: Track download times and failures

## Security Considerations

1. **TLS/HTTPS**: Use secure registry connections in production
2. **Authentication**: Implement registry authentication
3. **Digest verification**: Always verify artifact digests
4. **Access control**: Restrict registry access
5. **Audit logging**: Log all pull operations

## Performance

### Download Optimization

- **Parallel layers**: Download layers in parallel
- **Resume support**: Resume interrupted downloads
- **Compression**: Use gzip for smaller transfers
- **Local cache**: Cache frequently used artifacts

### Metrics

Track OCI operations:

- **Pull count**: Number of artifact pulls
- **Cache hits**: Percentage of cache hits
- **Download time**: Average download duration
- **Bandwidth usage**: Total bytes downloaded

## Future Enhancements

- [ ] Push artifacts to registry
- [ ] Registry authentication (OAuth2, Basic Auth)
- [ ] Multi-registry support
- [ ] Mirror/proxy registry
- [ ] Artifact signing and verification
- [ ] Garbage collection for cache