# Provider Distribution Guide\n\n**Strategic Guide for Provider Management and Distribution**\n\nThis guide explains the two complementary approaches for managing providers in the provisioning system and when to use each.\n\n---\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Module-Loader Approach](#module-loader-approach)\n- [Provider Packs Approach](#provider-packs-approach)\n- [Comparison Matrix](#comparison-matrix)\n- [Recommended Hybrid Workflow](#recommended-hybrid-workflow)\n- [Command Reference](#command-reference)\n- [Real-World Scenarios](#real-world-scenarios)\n- [Best Practices](#best-practices)\n\n---\n\n## Overview\n\nThe provisioning system supports **two complementary approaches** for provider management:\n\n1. **Module-Loader**: Symlink-based local development with dynamic discovery\n2. **Provider Packs**: Versioned, distributable artifacts for production\n\nBoth approaches work seamlessly together and serve different phases of the development lifecycle.\n\n---\n\n## Module-Loader Approach\n\n### Purpose\n\nFast, local development with direct access to provider source code.\n\n### How It Works\n\n```{$detected_lang}\n# Install provider for infrastructure (creates symlinks)\nprovisioning providers install upcloud wuji\n\n# Internal Process:\n# 1. Discovers provider in extensions/providers/upcloud/\n# 2. Creates symlink: workspace/infra/wuji/.nickel-modules/upcloud_prov -> extensions/providers/upcloud/nickel/\n# 3. Updates workspace/infra/wuji/manifest.toml with local path dependency\n# 4. Updates workspace/infra/wuji/providers.manifest.yaml\n```\n\n### Key Features\n\n✅ **Instant Changes**: Edit code in `extensions/providers/`, immediately available in infrastructure\n✅ **Auto-Discovery**: Automatically finds all providers in extensions/\n✅ **Simple Commands**: `providers install/remove/list/validate`\n✅ **Easy Debugging**: Direct access to source code\n✅ **No Packaging**: Skip build/package step during development\n\n### Best Use Cases\n\n- 🔧 **Active Development**: Writing new provider features\n- 🧪 **Testing**: Rapid iteration and testing cycles\n- 🏠 **Local Infrastructure**: Single machine or small team\n- 📝 **Debugging**: Need to modify and test provider code\n- 🎓 **Learning**: Understanding how providers work\n\n### Example Workflow\n\n```{$detected_lang}\n# 1. List available providers\nprovisioning providers list\n\n# 2. Install provider for infrastructure\nprovisioning providers install upcloud wuji\n\n# 3. Verify installation\nprovisioning providers validate wuji\n\n# 4. Edit provider code\nvim extensions/providers/upcloud/nickel/server_upcloud.ncl\n\n# 5. Test changes immediately (no repackaging!)\ncd workspace/infra/wuji\nnickel export main.ncl\n\n# 6. Remove when done\nprovisioning providers remove upcloud wuji\n```\n\n### File Structure\n\n```{$detected_lang}\nextensions/providers/upcloud/\n├── nickel/\n│   ├── manifest.toml\n│   ├── server_upcloud.ncl\n│   └── network_upcloud.ncl\n└── README.md\n\nworkspace/infra/wuji/\n├── .nickel-modules/\n│   └── upcloud_prov -> ../../../../extensions/providers/upcloud/nickel/  # Symlink\n├── manifest.toml        # Updated with local path dependency\n├── providers.manifest.yaml  # Tracks installed providers\n└── schemas/\n    └── servers.ncl\n```\n\n---\n\n## Provider Packs Approach\n\n### Purpose\n\nCreate versioned, distributable artifacts for production deployments and team collaboration.\n\n### How It Works\n\n```{$detected_lang}\n# Package providers into distributable artifacts\nexport PROVISIONING=/Users/Akasha/project-provisioning/provisioning\n./provisioning/core/cli/pack providers\n\n# Internal Process:\n# 1. Enters each provider's nickel/ directory\n# 2. Runs: nickel export . --format json (generates JSON for distribution)\n# 3. Creates: upcloud_prov_0.0.1.tar\n# 4. Generates metadata: distribution/registry/upcloud_prov.json\n```\n\n### Key Features\n\n✅ **Versioned Artifacts**: Immutable, reproducible packages\n✅ **Portable**: Share across teams and environments\n✅ **Registry Publishing**: Push to artifact registries\n✅ **Metadata**: Version, maintainer, license information\n✅ **Production-Ready**: What you package is what you deploy\n\n### Best Use Cases\n\n- 🚀 **Production Deployments**: Stable, tested provider versions\n- 📦 **Distribution**: Share across teams or organizations\n- 🔄 **CI/CD Pipelines**: Automated build and deploy\n- 📊 **Version Control**: Track provider versions explicitly\n- 🌐 **Registry Publishing**: Publish to artifact registries\n- 🔒 **Compliance**: Immutable artifacts for auditing\n\n### Example Workflow\n\n```{$detected_lang}\n# Set environment variable\nexport PROVISIONING=/Users/Akasha/project-provisioning/provisioning\n\n# 1. Package all providers\n./provisioning/core/cli/pack providers\n\n# Output:\n# ✅ Creates: distribution/packages/upcloud_prov_0.0.1.tar\n# ✅ Creates: distribution/packages/aws_prov_0.0.1.tar\n# ✅ Creates: distribution/packages/local_prov_0.0.1.tar\n# ✅ Metadata: distribution/registry/*.json\n\n# 2. List packaged modules\n./provisioning/core/cli/pack list\n\n# 3. Package only core schemas\n./provisioning/core/cli/pack core\n\n# 4. Clean old packages (keep latest 3 versions)\n./provisioning/core/cli/pack clean --keep-latest 3\n\n# 5. Upload to registry (your implementation)\n# rsync distribution/packages/*.tar repo.jesusperez.pro:/registry/\n```\n\n### File Structure\n\n```{$detected_lang}\nprovisioning/\n├── distribution/\n│   ├── packages/\n│   │   ├── provisioning_0.0.1.tar       # Core schemas\n│   │   ├── upcloud_prov_0.0.1.tar       # Provider packages\n│   │   ├── aws_prov_0.0.1.tar\n│   │   └── local_prov_0.0.1.tar\n│   └── registry/\n│       ├── provisioning_core.json       # Metadata\n│       ├── upcloud_prov.json\n│       ├── aws_prov.json\n│       └── local_prov.json\n└── extensions/providers/                # Source code\n```\n\n### Package Metadata Example\n\n```{$detected_lang}\n{\n  "name": "upcloud_prov",\n  "version": "0.0.1",\n  "package_file": "/path/to/upcloud_prov_0.0.1.tar",\n  "created": "2025-09-29 20:47:21",\n  "maintainer": "JesusPerezLorenzo",\n  "repository": "https://repo.jesusperez.pro/provisioning",\n  "license": "MIT",\n  "homepage": "https://github.com/jesusperezlorenzo/provisioning"\n}\n```\n\n---\n\n## Comparison Matrix\n\n| Feature | Module-Loader | Provider Packs |\n| --------- | -------------- | ---------------- |\n| **Speed** | ⚡ Instant (symlinks) | 📦 Requires packaging |\n| **Versioning** | ❌ No explicit versions | ✅ Semantic versioning |\n| **Portability** | ❌ Local filesystem only | ✅ Distributable archives |\n| **Development** | ✅ Excellent (live reload) | ⚠️ Need repackage cycle |\n| **Production** | ⚠️ Mutable source | ✅ Immutable artifacts |\n| **Discovery** | ✅ Auto-discovery | ⚠️ Manual tracking |\n| **Team Sharing** | ⚠️ Git repository only | ✅ Registry + Git |\n| **Debugging** | ✅ Direct source access | ❌ Need to unpack |\n| **Rollback** | ⚠️ Git revert | ✅ Version pinning |\n| **Compliance** | ❌ Hard to audit | ✅ Signed artifacts |\n| **Setup Time** | ⚡ Seconds | ⏱️ Minutes |\n| **CI/CD** | ⚠️ Not ideal | ✅ Perfect |\n\n---\n\n## Recommended Hybrid Workflow\n\n### Development Phase\n\n```{$detected_lang}\n# 1. Start with module-loader for development\nprovisioning providers list\nprovisioning providers install upcloud wuji\n\n# 2. Develop and iterate quickly\nvim extensions/providers/upcloud/nickel/server_upcloud.ncl\n# Test immediately - no packaging needed\n\n# 3. Validate before release\nprovisioning providers validate wuji\nnickel export workspace/infra/wuji/main.ncl\n```\n\n### Release Phase\n\n```{$detected_lang}\n# 4. Create release packages\nexport PROVISIONING=/Users/Akasha/project-provisioning/provisioning\n./provisioning/core/cli/pack providers\n\n# 5. Verify packages\n./provisioning/core/cli/pack list\n\n# 6. Tag release\ngit tag v0.0.2\ngit push origin v0.0.2\n\n# 7. Publish to registry (your workflow)\nrsync distribution/packages/*.tar user@repo.jesusperez.pro:/registry/v0.0.2/\n```\n\n### Production Deployment\n\n```{$detected_lang}\n# 8. Download specific version from registry\nwget https://repo.jesusperez.pro/registry/v0.0.2/upcloud_prov_0.0.2.tar\n\n# 9. Extract and install\ntar -xf upcloud_prov_0.0.2.tar -C infrastructure/providers/\n\n# 10. Use in production infrastructure\n# (Configure manifest.toml to point to extracted package)\n```\n\n---\n\n## Command Reference\n\n### Module-Loader Commands\n\n```{$detected_lang}\n# List all available providers\nprovisioning providers list [--kcl] [--format table|json|yaml]\n\n# Show provider information\nprovisioning providers info <provider> [--kcl]\n\n# Install provider for infrastructure\nprovisioning providers install <provider> <infra> [--version 0.0.1]\n\n# Remove provider from infrastructure\nprovisioning providers remove <provider> <infra> [--force]\n\n# List installed providers\nprovisioning providers installed <infra> [--format table|json|yaml]\n\n# Validate provider installation\nprovisioning providers validate <infra>\n\n# Sync KCL dependencies\n./provisioning/core/cli/module-loader sync-kcl <infra>\n```\n\n### Provider Pack Commands\n\n```{$detected_lang}\n# Set environment variable (required)\nexport PROVISIONING=/path/to/provisioning\n\n# Package core provisioning schemas\n./provisioning/core/cli/pack core [--output dir] [--version 0.0.1]\n\n# Package single provider\n./provisioning/core/cli/pack provider <name> [--output dir] [--version 0.0.1]\n\n# Package all providers\n./provisioning/core/cli/pack providers [--output dir]\n\n# List all packages\n./provisioning/core/cli/pack list [--format table|json|yaml]\n\n# Clean old packages\n./provisioning/core/cli/pack clean [--keep-latest 3] [--dry-run]\n```\n\n---\n\n## Real-World Scenarios\n\n### Scenario 1: Solo Developer - Local Infrastructure\n\n**Situation**: Working alone on local infrastructure projects\n\n**Recommendation**: Module-Loader only\n\n```{$detected_lang}\n# Simple and fast\nproviders install upcloud homelab\nproviders install aws cloud-backup\n# Edit and test freely\n```\n\n**Why**: No need for versioning, packaging overhead unnecessary.\n\n---\n\n### Scenario 2: Small Team - Shared Development\n\n**Situation**: 2-5 developers sharing code via Git\n\n**Recommendation**: Module-Loader + Git\n\n```{$detected_lang}\n# Each developer\ngit clone repo\nproviders install upcloud project-x\n# Make changes, commit to Git\ngit commit -m "Add upcloud GPU support"\ngit push\n# Others pull changes\ngit pull\n# Changes immediately available via symlinks\n```\n\n**Why**: Git provides version control, symlinks provide instant updates.\n\n---\n\n### Scenario 3: Medium Team - Multiple Projects\n\n**Situation**: 10+ developers, multiple infrastructure projects\n\n**Recommendation**: Hybrid (Module-Loader dev + Provider Packs releases)\n\n```{$detected_lang}\n# Development (team member)\nproviders install upcloud staging-env\n# Make changes...\n\n# Release (release engineer)\npack providers                    # Create v0.2.0\ngit tag v0.2.0\n# Upload to internal registry\n\n# Other projects\n# Download upcloud_prov_0.2.0.tar\n# Use stable, tested version\n```\n\n**Why**: Developers iterate fast, other teams use stable versions.\n\n---\n\n### Scenario 4: Enterprise - Production Infrastructure\n\n**Situation**: Critical production systems, compliance requirements\n\n**Recommendation**: Provider Packs only\n\n```{$detected_lang}\n# CI/CD Pipeline\npack providers                    # Build artifacts\n# Run tests on packages\n# Sign packages\n# Publish to artifact registry\n\n# Production Deployment\n# Download signed upcloud_prov_1.0.0.tar\n# Verify signature\n# Deploy immutable artifact\n# Document exact versions for compliance\n```\n\n**Why**: Immutability, auditability, and rollback capabilities required.\n\n---\n\n### Scenario 5: Open Source - Public Distribution\n\n**Situation**: Sharing providers with community\n\n**Recommendation**: Provider Packs + Registry\n\n```{$detected_lang}\n# Maintainer\npack providers\n# Create release on GitHub\ngh release create v1.0.0 distribution/packages/*.tar\n\n# Community User\n# Download from GitHub releases\nwget https://github.com/project/releases/v1.0.0/upcloud_prov_1.0.0.tar\n# Extract and use\n```\n\n**Why**: Easy distribution, versioning, and downloading for users.\n\n---\n\n## Best Practices\n\n### For Development\n\n1. **Use Module-Loader by default**\n   - Fast iteration is crucial during development\n   - Symlinks allow immediate testing\n\n2. **Keep providers.manifest.yaml in Git**\n   - Documents which providers are used\n   - Team members can sync easily\n\n3. **Validate before committing**\n\n   ```bash\n   providers validate wuji\n   nickel eval defs/servers.ncl\n   ```\n\n### For Releases\n\n1. **Version Everything**\n   - Use semantic versioning (0.1.0, 0.2.0, 1.0.0)\n   - Update version in kcl.mod before packing\n\n2. **Create Packs for Releases**\n\n   ```bash\n   pack providers --version 0.2.0\n   git tag v0.2.0\n   ```\n\n3. **Test Packs Before Publishing**\n   - Extract and test packages\n   - Verify metadata is correct\n\n### For Production\n\n1. **Pin Versions**\n   - Use exact versions in production kcl.mod\n   - Never use "latest" or symlinks\n\n2. **Maintain Artifact Registry**\n   - Store all production versions\n   - Keep old versions for rollback\n\n3. **Document Deployments**\n   - Record which versions deployed when\n   - Maintain change log\n\n### For CI/CD\n\n1. **Automate Pack Creation**\n\n   ```yaml\n   # .github/workflows/release.yml\n   - name: Pack Providers\n     run: |\n       export PROVISIONING=$GITHUB_WORKSPACE/provisioning\n       ./provisioning/core/cli/pack providers\n   ```\n\n2. **Run Tests on Packs**\n   - Extract packages\n   - Run validation tests\n   - Ensure they work in isolation\n\n3. **Publish Automatically**\n   - Upload to artifact registry on tag\n   - Update package index\n\n---\n\n## Migration Path\n\n### From Module-Loader to Packs\n\nWhen you're ready to move to production:\n\n```{$detected_lang}\n# 1. Clean up development setup\nproviders remove upcloud wuji\n\n# 2. Create release pack\npack providers --version 1.0.0\n\n# 3. Extract pack in infrastructure\ncd workspace/infra/wuji\ntar -xf ../../../distribution/packages/upcloud_prov_1.0.0.tar vendor/\n\n# 4. Update kcl.mod to use vendored path\n# Change from: upcloud_prov = { path = "./.kcl-modules/upcloud_prov" }\n# To: upcloud_prov = { path = "./vendor/upcloud_prov", version = "1.0.0" }\n\n# 5. Test\nnickel eval defs/servers.ncl\n```\n\n### From Packs Back to Module-Loader\n\nWhen you need to debug or develop:\n\n```{$detected_lang}\n# 1. Remove vendored version\nrm -rf workspace/infra/wuji/vendor/upcloud_prov\n\n# 2. Install via module-loader\nproviders install upcloud wuji\n\n# 3. Make changes in extensions/providers/upcloud/kcl/\n\n# 4. Test immediately\ncd workspace/infra/wuji\nnickel eval defs/servers.ncl\n```\n\n---\n\n## Configuration\n\n### Environment Variables\n\n```{$detected_lang}\n# Required for pack commands\nexport PROVISIONING=/path/to/provisioning\n\n# Alternative\nexport PROVISIONING_CONFIG=/path/to/provisioning\n```\n\n### Config Files\n\nDistribution settings in `provisioning/config/config.defaults.toml`:\n\n```{$detected_lang}\n[distribution]\npack_path = "{{paths.base}}/distribution/packages"\nregistry_path = "{{paths.base}}/distribution/registry"\ncache_path = "{{paths.base}}/distribution/cache"\nregistry_type = "local"\n\n[distribution.metadata]\nmaintainer = "JesusPerezLorenzo"\nrepository = "https://repo.jesusperez.pro/provisioning"\nlicense = "MIT"\nhomepage = "https://github.com/jesusperezlorenzo/provisioning"\n\n[kcl]\ncore_module = "{{paths.base}}/kcl"\ncore_version = "0.0.1"\ncore_package_name = "provisioning_core"\nuse_module_loader = true\nmodules_dir = ".kcl-modules"\n```\n\n---\n\n## Troubleshooting\n\n### Module-Loader Issues\n\n**Problem**: Provider not found after install\n\n```{$detected_lang}\n# Check provider exists\nproviders list | grep upcloud\n\n# Validate installation\nproviders validate wuji\n\n# Check symlink\nls -la workspace/infra/wuji/.kcl-modules/\n```\n\n**Problem**: Changes not reflected\n\n```{$detected_lang}\n# Verify symlink is correct\nreadlink workspace/infra/wuji/.kcl-modules/upcloud_prov\n\n# Should point to extensions/providers/upcloud/kcl/\n```\n\n### Provider Pack Issues\n\n**Problem**: No .tar file created\n\n```{$detected_lang}\n# Check KCL version (need 0.11.3+)\nkcl version\n\n# Check kcl.mod exists\nls extensions/providers/upcloud/kcl/kcl.mod\n```\n\n**Problem**: PROVISIONING environment variable not set\n\n```{$detected_lang}\n# Set it\nexport PROVISIONING=/Users/Akasha/project-provisioning/provisioning\n\n# Or add to shell profile\necho 'export PROVISIONING=/path/to/provisioning' >> ~/.zshrc\n```\n\n---\n\n## Conclusion\n\n**Both approaches are valuable and complementary:**\n\n- **Module-Loader**: Development velocity, rapid iteration\n- **Provider Packs**: Production stability, version control\n\n**Default Strategy:**\n\n- Use **Module-Loader** for day-to-day development\n- Create **Provider Packs** for releases and production\n- Both systems work seamlessly together\n\n**The system is designed for flexibility** - choose the right tool for your current phase of work!\n\n---\n\n## Additional Resources\n\n- [Module-Loader Implementation](../provisioning/core/nulib/lib_provisioning/kcl_module_loader.nu)\n- [KCL Packaging Implementation](../provisioning/core/nulib/lib_provisioning/kcl_packaging.nu)\n- [Providers CLI](.provisioning providers)\n- [Pack CLI](../provisioning/core/cli/pack)\n- [KCL Documentation](https://kcl-lang.io/)\n\n---\n\n**Document Version**: 1.0.0\n**Last Updated**: 2025-09-29\n**Maintained by**: JesusPerezLorenzo
