jesus/provisioning

Fork 0

Jesús Pérez 17ef93ed23

chore: fix docs after fences fix

2026-01-14 04:53:58 +00:00

17 KiB

Raw Blame History

Provider Distribution Guide

Strategic Guide for Provider Management and Distribution

This guide explains the two complementary approaches for managing providers in the provisioning system and when to use each.

Overview
Module-Loader Approach
Provider Packs Approach
Comparison Matrix
Recommended Hybrid Workflow
Command Reference
Real-World Scenarios
Best Practices

Overview

The provisioning system supports two complementary approaches for provider management:

Module-Loader: Symlink-based local development with dynamic discovery
Provider Packs: Versioned, distributable artifacts for production

Both approaches work seamlessly together and serve different phases of the development lifecycle.

Module-Loader Approach

Purpose

Fast, local development with direct access to provider source code.

How It Works

# Install provider for infrastructure (creates symlinks)
provisioning providers install upcloud wuji

# Internal Process:
# 1. Discovers provider in extensions/providers/upcloud/
# 2. Creates symlink: workspace/infra/wuji/.nickel-modules/upcloud_prov -> extensions/providers/upcloud/nickel/
# 3. Updates workspace/infra/wuji/manifest.toml with local path dependency
# 4. Updates workspace/infra/wuji/providers.manifest.yaml

Key Features

✅ Instant Changes: Edit code in extensions/providers/, immediately available in infrastructure ✅ Auto-Discovery: Automatically finds all providers in extensions/ ✅ Simple Commands: providers install/remove/list/validate ✅ Easy Debugging: Direct access to source code ✅ No Packaging: Skip build/package step during development

Best Use Cases

🔧 Active Development: Writing new provider features
🧪 Testing: Rapid iteration and testing cycles
🏠 Local Infrastructure: Single machine or small team
📝 Debugging: Need to modify and test provider code
🎓 Learning: Understanding how providers work

Example Workflow

# 1. List available providers
provisioning providers list

# 2. Install provider for infrastructure
provisioning providers install upcloud wuji

# 3. Verify installation
provisioning providers validate wuji

# 4. Edit provider code
vim extensions/providers/upcloud/nickel/server_upcloud.ncl

# 5. Test changes immediately (no repackaging!)
cd workspace/infra/wuji
nickel export main.ncl

# 6. Remove when done
provisioning providers remove upcloud wuji

File Structure

extensions/providers/upcloud/
├── nickel/
│   ├── manifest.toml
│   ├── server_upcloud.ncl
│   └── network_upcloud.ncl
└── README.md

workspace/infra/wuji/
├── .nickel-modules/
│   └── upcloud_prov -> ../../../../extensions/providers/upcloud/nickel/  # Symlink
├── manifest.toml        # Updated with local path dependency
├── providers.manifest.yaml  # Tracks installed providers
└── schemas/
    └── servers.ncl

Provider Packs Approach

Purpose

Create versioned, distributable artifacts for production deployments and team collaboration.

How It Works

# Package providers into distributable artifacts
export PROVISIONING=/Users/Akasha/project-provisioning/provisioning
./provisioning/core/cli/pack providers

# Internal Process:
# 1. Enters each provider's nickel/ directory
# 2. Runs: nickel export . --format json (generates JSON for distribution)
# 3. Creates: upcloud_prov_0.0.1.tar
# 4. Generates metadata: distribution/registry/upcloud_prov.json

Key Features

✅ Versioned Artifacts: Immutable, reproducible packages ✅ Portable: Share across teams and environments ✅ Registry Publishing: Push to artifact registries ✅ Metadata: Version, maintainer, license information ✅ Production-Ready: What you package is what you deploy

Best Use Cases

🚀 Production Deployments: Stable, tested provider versions
📦 Distribution: Share across teams or organizations
🔄 CI/CD Pipelines: Automated build and deploy
📊 Version Control: Track provider versions explicitly
🌐 Registry Publishing: Publish to artifact registries
🔒 Compliance: Immutable artifacts for auditing

Example Workflow

# Set environment variable
export PROVISIONING=/Users/Akasha/project-provisioning/provisioning

# 1. Package all providers
./provisioning/core/cli/pack providers

# Output:
# ✅ Creates: distribution/packages/upcloud_prov_0.0.1.tar
# ✅ Creates: distribution/packages/aws_prov_0.0.1.tar
# ✅ Creates: distribution/packages/local_prov_0.0.1.tar
# ✅ Metadata: distribution/registry/*.json

# 2. List packaged modules
./provisioning/core/cli/pack list

# 3. Package only core schemas
./provisioning/core/cli/pack core

# 4. Clean old packages (keep latest 3 versions)
./provisioning/core/cli/pack clean --keep-latest 3

# 5. Upload to registry (your implementation)
# rsync distribution/packages/*.tar repo.jesusperez.pro:/registry/

File Structure

provisioning/
├── distribution/
│   ├── packages/
│   │   ├── provisioning_0.0.1.tar       # Core schemas
│   │   ├── upcloud_prov_0.0.1.tar       # Provider packages
│   │   ├── aws_prov_0.0.1.tar
│   │   └── local_prov_0.0.1.tar
│   └── registry/
│       ├── provisioning_core.json       # Metadata
│       ├── upcloud_prov.json
│       ├── aws_prov.json
│       └── local_prov.json
└── extensions/providers/                # Source code

Package Metadata Example

{
  "name": "upcloud_prov",
  "version": "0.0.1",
  "package_file": "/path/to/upcloud_prov_0.0.1.tar",
  "created": "2025-09-29 20:47:21",
  "maintainer": "JesusPerezLorenzo",
  "repository": "https://repo.jesusperez.pro/provisioning",
  "license": "MIT",
  "homepage": "https://github.com/jesusperezlorenzo/provisioning"
}

Comparison Matrix

Feature	Module-Loader	Provider Packs
Speed	⚡ Instant (symlinks)	📦 Requires packaging
Versioning	❌ No explicit versions	✅ Semantic versioning
Portability	❌ Local filesystem only	✅ Distributable archives
Development	✅ Excellent (live reload)	⚠️ Need repackage cycle
Production	⚠️ Mutable source	✅ Immutable artifacts
Discovery	✅ Auto-discovery	⚠️ Manual tracking
Team Sharing	⚠️ Git repository only	✅ Registry + Git
Debugging	✅ Direct source access	❌ Need to unpack
Rollback	⚠️ Git revert	✅ Version pinning
Compliance	❌ Hard to audit	✅ Signed artifacts
Setup Time	⚡ Seconds	⏱️ Minutes
CI/CD	⚠️ Not ideal	✅ Perfect

Recommended Hybrid Workflow

Development Phase

# 1. Start with module-loader for development
provisioning providers list
provisioning providers install upcloud wuji

# 2. Develop and iterate quickly
vim extensions/providers/upcloud/nickel/server_upcloud.ncl
# Test immediately - no packaging needed

# 3. Validate before release
provisioning providers validate wuji
nickel export workspace/infra/wuji/main.ncl

Release Phase

# 4. Create release packages
export PROVISIONING=/Users/Akasha/project-provisioning/provisioning
./provisioning/core/cli/pack providers

# 5. Verify packages
./provisioning/core/cli/pack list

# 6. Tag release
git tag v0.0.2
git push origin v0.0.2

# 7. Publish to registry (your workflow)
rsync distribution/packages/*.tar user@repo.jesusperez.pro:/registry/v0.0.2/

Production Deployment

# 8. Download specific version from registry
wget https://repo.jesusperez.pro/registry/v0.0.2/upcloud_prov_0.0.2.tar

# 9. Extract and install
tar -xf upcloud_prov_0.0.2.tar -C infrastructure/providers/

# 10. Use in production infrastructure
# (Configure manifest.toml to point to extracted package)

Command Reference

Module-Loader Commands

# List all available providers
provisioning providers list [--kcl] [--format table|json|yaml]

# Show provider information
provisioning providers info <provider> [--kcl]

# Install provider for infrastructure
provisioning providers install <provider> <infra> [--version 0.0.1]

# Remove provider from infrastructure
provisioning providers remove <provider> <infra> [--force]

# List installed providers
provisioning providers installed <infra> [--format table|json|yaml]

# Validate provider installation
provisioning providers validate <infra>

# Sync KCL dependencies
./provisioning/core/cli/module-loader sync-kcl <infra>

Provider Pack Commands

# Set environment variable (required)
export PROVISIONING=/path/to/provisioning

# Package core provisioning schemas
./provisioning/core/cli/pack core [--output dir] [--version 0.0.1]

# Package single provider
./provisioning/core/cli/pack provider <name> [--output dir] [--version 0.0.1]

# Package all providers
./provisioning/core/cli/pack providers [--output dir]

# List all packages
./provisioning/core/cli/pack list [--format table|json|yaml]

# Clean old packages
./provisioning/core/cli/pack clean [--keep-latest 3] [--dry-run]

Real-World Scenarios

Scenario 1: Solo Developer - Local Infrastructure

Situation: Working alone on local infrastructure projects

Recommendation: Module-Loader only

# Simple and fast
providers install upcloud homelab
providers install aws cloud-backup
# Edit and test freely

Why: No need for versioning, packaging overhead unnecessary.

Scenario 2: Small Team - Shared Development

Situation: 2-5 developers sharing code via Git

Recommendation: Module-Loader + Git

# Each developer
git clone repo
providers install upcloud project-x
# Make changes, commit to Git
git commit -m "Add upcloud GPU support"
git push
# Others pull changes
git pull
# Changes immediately available via symlinks

Why: Git provides version control, symlinks provide instant updates.

Scenario 3: Medium Team - Multiple Projects

Situation: 10+ developers, multiple infrastructure projects

Recommendation: Hybrid (Module-Loader dev + Provider Packs releases)

# Development (team member)
providers install upcloud staging-env
# Make changes...

# Release (release engineer)
pack providers                    # Create v0.2.0
git tag v0.2.0
# Upload to internal registry

# Other projects
# Download upcloud_prov_0.2.0.tar
# Use stable, tested version

Why: Developers iterate fast, other teams use stable versions.

Scenario 4: Enterprise - Production Infrastructure

Situation: Critical production systems, compliance requirements

Recommendation: Provider Packs only

# CI/CD Pipeline
pack providers                    # Build artifacts
# Run tests on packages
# Sign packages
# Publish to artifact registry

# Production Deployment
# Download signed upcloud_prov_1.0.0.tar
# Verify signature
# Deploy immutable artifact
# Document exact versions for compliance

Why: Immutability, auditability, and rollback capabilities required.

Scenario 5: Open Source - Public Distribution

Situation: Sharing providers with community

Recommendation: Provider Packs + Registry

# Maintainer
pack providers
# Create release on GitHub
gh release create v1.0.0 distribution/packages/*.tar

# Community User
# Download from GitHub releases
wget https://github.com/project/releases/v1.0.0/upcloud_prov_1.0.0.tar
# Extract and use

Why: Easy distribution, versioning, and downloading for users.

Best Practices

For Development

Use Module-Loader by default
- Fast iteration is crucial during development
- Symlinks allow immediate testing
Keep providers.manifest.yaml in Git
- Documents which providers are used
- Team members can sync easily

Validate before committing

providers validate wuji
nickel eval defs/servers.ncl

For Releases

Version Everything
- Use semantic versioning (0.1.0, 0.2.0, 1.0.0)
- Update version in kcl.mod before packing

Create Packs for Releases

pack providers --version 0.2.0
git tag v0.2.0

Test Packs Before Publishing
- Extract and test packages
- Verify metadata is correct

For Production

Pin Versions
- Use exact versions in production kcl.mod
- Never use "latest" or symlinks
Maintain Artifact Registry
- Store all production versions
- Keep old versions for rollback
Document Deployments
- Record which versions deployed when
- Maintain change log

For CI/CD

Automate Pack Creation

# .github/workflows/release.yml
- name: Pack Providers
  run: |
    export PROVISIONING=$GITHUB_WORKSPACE/provisioning
    ./provisioning/core/cli/pack providers

Run Tests on Packs
- Extract packages
- Run validation tests
- Ensure they work in isolation
Publish Automatically
- Upload to artifact registry on tag
- Update package index

Migration Path

From Module-Loader to Packs

When you're ready to move to production:

# 1. Clean up development setup
providers remove upcloud wuji

# 2. Create release pack
pack providers --version 1.0.0

# 3. Extract pack in infrastructure
cd workspace/infra/wuji
tar -xf ../../../distribution/packages/upcloud_prov_1.0.0.tar vendor/

# 4. Update kcl.mod to use vendored path
# Change from: upcloud_prov = { path = "./.kcl-modules/upcloud_prov" }
# To: upcloud_prov = { path = "./vendor/upcloud_prov", version = "1.0.0" }

# 5. Test
nickel eval defs/servers.ncl

From Packs Back to Module-Loader

When you need to debug or develop:

# 1. Remove vendored version
rm -rf workspace/infra/wuji/vendor/upcloud_prov

# 2. Install via module-loader
providers install upcloud wuji

# 3. Make changes in extensions/providers/upcloud/kcl/

# 4. Test immediately
cd workspace/infra/wuji
nickel eval defs/servers.ncl

Configuration

Environment Variables

# Required for pack commands
export PROVISIONING=/path/to/provisioning

# Alternative
export PROVISIONING_CONFIG=/path/to/provisioning

Config Files

Distribution settings in provisioning/config/config.defaults.toml:

[distribution]
pack_path = "{{paths.base}}/distribution/packages"
registry_path = "{{paths.base}}/distribution/registry"
cache_path = "{{paths.base}}/distribution/cache"
registry_type = "local"

[distribution.metadata]
maintainer = "JesusPerezLorenzo"
repository = "https://repo.jesusperez.pro/provisioning"
license = "MIT"
homepage = "https://github.com/jesusperezlorenzo/provisioning"

[kcl]
core_module = "{{paths.base}}/kcl"
core_version = "0.0.1"
core_package_name = "provisioning_core"
use_module_loader = true
modules_dir = ".kcl-modules"

Troubleshooting

Module-Loader Issues

Problem: Provider not found after install

# Check provider exists
providers list | grep upcloud

# Validate installation
providers validate wuji

# Check symlink
ls -la workspace/infra/wuji/.kcl-modules/

Problem: Changes not reflected

# Verify symlink is correct
readlink workspace/infra/wuji/.kcl-modules/upcloud_prov

# Should point to extensions/providers/upcloud/kcl/

Provider Pack Issues

Problem: No .tar file created

# Check KCL version (need 0.11.3+)
kcl version

# Check kcl.mod exists
ls extensions/providers/upcloud/kcl/kcl.mod

Problem: PROVISIONING environment variable not set

# Set it
export PROVISIONING=/Users/Akasha/project-provisioning/provisioning

# Or add to shell profile
echo 'export PROVISIONING=/path/to/provisioning' >> ~/.zshrc

Conclusion

Both approaches are valuable and complementary:

Module-Loader: Development velocity, rapid iteration
Provider Packs: Production stability, version control

Default Strategy:

Use Module-Loader for day-to-day development
Create Provider Packs for releases and production
Both systems work seamlessly together

The system is designed for flexibility - choose the right tool for your current phase of work!

Additional Resources

Module-Loader Implementation
KCL Packaging Implementation
[Providers CLI](.provisioning providers)
Pack CLI
KCL Documentation

Document Version: 1.0.0 Last Updated: 2025-09-29 Maintained by: JesusPerezLorenzo

17 KiB Raw Blame History

Provider Distribution Guide

Table of Contents

Overview

Module-Loader Approach

Purpose

How It Works

Key Features

Best Use Cases

Example Workflow

File Structure

Provider Packs Approach

Purpose

How It Works

Key Features

Best Use Cases

Example Workflow

File Structure

Package Metadata Example

Comparison Matrix

Recommended Hybrid Workflow

Development Phase

Release Phase

Production Deployment

Command Reference

Module-Loader Commands

Provider Pack Commands

Real-World Scenarios

Scenario 1: Solo Developer - Local Infrastructure

Scenario 2: Small Team - Shared Development

Scenario 3: Medium Team - Multiple Projects

Scenario 4: Enterprise - Production Infrastructure

Scenario 5: Open Source - Public Distribution

Best Practices

For Development

For Releases

For Production

For CI/CD

Migration Path

From Module-Loader to Packs

From Packs Back to Module-Loader

Configuration

Environment Variables

Config Files

Troubleshooting

Module-Loader Issues

Provider Pack Issues

Conclusion

Additional Resources

17 KiB

Raw Blame History