prvng_platform/orchestrator/docs/OCI_INTEGRATION.md

OCI Registry Integration Guide

Overview

The OCI integration module provides OCI Distribution Spec v2 compliant registry integration for pulling KCL packages and extension artifacts.

Architecture

┌──────────────────┐
│   Orchestrator   │
│     (Rust)       │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│   OCI Manager    │
│                  │
│  - LRU caching   │
│  - Pull artifacts│
│  - List packages │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│   OCI Client     │
│  (Distribution)  │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│  OCI Registry    │
│  (HTTP API v2)   │
└──────────────────┘

Features

1. KCL Package Management

Pull KCL configuration packages from OCI registry:

let package_path = oci_manager.pull_kcl_package(
    "provisioning-core",
    "1.0.0"
).await?;

2. Extension Artifacts

Pull extension artifacts (providers, taskservs, clusters):

let artifact_path = oci_manager.pull_extension_artifact(
    "taskserv",        // Extension type
    "kubernetes",      // Extension name
    "1.28.0"          // Version
).await?;

3. Manifest Caching

Manifests are cached using LRU strategy:

• Cache size: 100 manifests
• Cache key: {name}:{version}
• Automatic eviction: Oldest entries removed when full

4. Artifact Listing

List all artifacts in a namespace:

let artifacts = oci_manager.list_oci_artifacts("kcl").await?;
for artifact in artifacts {
    println!("{} v{} ({})", artifact.name, artifact.version, artifact.size);
}

OCI Distribution Spec v2

Implements OCI Distribution Specification v2:

• Manifest retrieval: GET /v2/{namespace}/{repository}/manifests/{reference}
• Blob download: GET /v2/{namespace}/{repository}/blobs/{digest}
• Tag listing: GET /v2/{namespace}/{repository}/tags/list
• Artifact existence: HEAD /v2/{namespace}/{repository}/manifests/{reference}

Configuration

OCI settings in config.defaults.toml:

[orchestrator.oci]
registry_url = "http://localhost:5000"
namespace = "provisioning-extensions"
cache_dir = "{{orchestrator.paths.data_dir}}/oci-cache"

Configuration Options

• registry_url: OCI registry HTTP endpoint
• namespace: Default namespace for artifacts
• cache_dir: Local cache directory for downloaded artifacts

API Endpoints

List OCI Artifacts

POST /api/v1/oci/artifacts
Content-Type: application/json

{
  "namespace": "kcl"
}

Response:

{
  "success": true,
  "data": [
    {
      "name": "provisioning-core",
      "version": "1.0.0",
      "digest": "sha256:abc123...",
      "size": 102400,
      "media_type": "application/vnd.oci.image.manifest.v1+json",
      "created_at": "2025-10-06T12:00:00Z"
    }
  ]
}

Usage Examples

Pull KCL Package

use provisioning_orchestrator::oci::OciManager;
use std::path::PathBuf;

let oci_manager = OciManager::new(
    "http://localhost:5000".to_string(),
    "provisioning-extensions".to_string(),
    PathBuf::from("/tmp/oci-cache"),
);

// Pull KCL package
let package_path = oci_manager.pull_kcl_package(
    "provisioning-core",
    "1.0.0"
).await?;

println!("Package downloaded to: {}", package_path.display());

// Extract package
// tar -xzf package_path

Pull Extension Artifact

// Pull taskserv extension
let artifact_path = oci_manager.pull_extension_artifact(
    "taskserv",
    "kubernetes",
    "1.28.0"
).await?;

// Extract and install
// tar -xzf artifact_path -C /target/path

List Artifacts

let artifacts = oci_manager.list_oci_artifacts("kcl").await?;

for artifact in artifacts {
    println!("📦 {} v{}", artifact.name, artifact.version);
    println!("   Size: {} bytes", artifact.size);
    println!("   Digest: {}", artifact.digest);
    println!();
}

Check Artifact Exists

let exists = oci_manager.artifact_exists(
    "kcl/provisioning-core",
    "1.0.0"
).await?;

if exists {
    println!("Artifact exists in registry");
} else {
    println!("Artifact not found");
}

Get Manifest (with caching)

let manifest = oci_manager.get_manifest(
    "kcl/provisioning-core",
    "1.0.0"
).await?;

println!("Schema version: {}", manifest.schema_version);
println!("Media type: {}", manifest.media_type);
println!("Layers: {}", manifest.layers.len());

Clear Manifest Cache

oci_manager.clear_cache().await;

OCI Artifact Structure

Manifest Format

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:abc123...",
    "size": 1234
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:def456...",
      "size": 102400
    }
  ],
  "annotations": {
    "org.opencontainers.image.created": "2025-10-06T12:00:00Z",
    "org.opencontainers.image.version": "1.0.0"
  }
}

Integration with Workflows

Extension Installation with OCI

1. Check local cache
2. Pull from OCI registry (if not cached)
3. Extract artifact
4. Validate contents
5. Install extension

// Workflow: Install taskserv from OCI
async fn install_taskserv_from_oci(
    oci_manager: &OciManager,
    name: &str,
    version: &str
) -> Result<()> {
    // Pull artifact
    let artifact_path = oci_manager.pull_extension_artifact(
        "taskserv",
        name,
        version
    ).await?;

    // Extract
    extract_tarball(&artifact_path, &target_dir)?;

    // Validate
    validate_extension_structure(&target_dir)?;

    // Install
    install_extension(&target_dir)?;

    Ok(())
}

Cache Management

Cache Directory Structure

/tmp/oci-cache/
├── kcl/
│   └── provisioning-core/
│       └── 1.0.0/
│           └── package.tar.gz
├── extensions/
│   ├── taskserv/
│   │   └── kubernetes/
│   │       └── 1.28.0/
│   │           └── artifact.tar.gz
│   └── provider/
│       └── aws/
│           └── 2.0.0/
│               └── artifact.tar.gz

Cache Cleanup

Implement cache cleanup strategy:

// Clean old artifacts
async fn cleanup_old_artifacts(cache_dir: &Path, max_age_days: u64) -> Result<()> {
    let cutoff = Utc::now() - Duration::days(max_age_days as i64);

    for entry in std::fs::read_dir(cache_dir)? {
        let entry = entry?;
        let metadata = entry.metadata()?;

        if let Ok(modified) = metadata.modified() {
            let modified: DateTime<Utc> = modified.into();
            if modified < cutoff {
                std::fs::remove_dir_all(entry.path())?;
            }
        }
    }

    Ok(())
}

Error Handling

The OCI integration handles errors gracefully:

• Network errors: Retries with exponential backoff
• Manifest not found: Returns clear error message
• Corrupted downloads: Validates digest before returning
• Disk full: Reports storage error

Testing

Run OCI integration tests:

cd provisioning/platform/orchestrator
cargo test test_oci_integration

Troubleshooting

Artifact pull fails

1. Check OCI registry is accessible
2. Verify registry_url configuration
3. Check network connectivity
4. Verify artifact exists in registry
5. Review orchestrator logs

Digest mismatch

1. Clear local cache
2. Re-pull artifact
3. Verify registry integrity
4. Check for network corruption

Cache issues

1. Check cache directory permissions
2. Verify disk space
3. Clear cache manually if corrupted

Best Practices

1. Use specific versions: Always specify version for production
2. Verify digests: Validate artifact integrity
3. Cache management: Implement cleanup strategy
4. Error handling: Handle network failures gracefully
5. Monitor downloads: Track download times and failures

Security Considerations

1. TLS/HTTPS: Use secure registry connections in production
2. Authentication: Implement registry authentication
3. Digest verification: Always verify artifact digests
4. Access control: Restrict registry access
5. Audit logging: Log all pull operations

Performance

Download Optimization

• Parallel layers: Download layers in parallel
• Resume support: Resume interrupted downloads
• Compression: Use gzip for smaller transfers
• Local cache: Cache frequently used artifacts

Metrics

Track OCI operations:

• Pull count: Number of artifact pulls
• Cache hits: Percentage of cache hits
• Download time: Average download duration
• Bandwidth usage: Total bytes downloaded

Future Enhancements

• Push artifacts to registry
• Registry authentication (OAuth2, Basic Auth)
• Multi-registry support
• Mirror/proxy registry
• Artifact signing and verification
• Garbage collection for cache