provisioning/docs/book/development/project-structure.html
Jesús Pérez 44648e3206
chore: complete nickel migration and consolidate legacy configs
- Remove KCL ecosystem (~220 files deleted)
- Migrate all infrastructure to Nickel schema system
- Consolidate documentation: legacy docs → provisioning/docs/src/
- Add CI/CD workflows (.github/) and Rust build config (.cargo/)
- Update core system for Nickel schema parsing
- Update README.md and CHANGES.md for v5.0.0 release
- Fix pre-commit hooks: end-of-file, trailing-whitespace
- Breaking changes: KCL workspaces require migration
- Migration bridge available in docs/src/development/
2026-01-08 09:55:37 +00:00

630 lines
24 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE HTML>
<html lang="en" class="ayu sidebar-visible" dir="ltr">
<head>
<!-- Book generated using mdBook -->
<meta charset="UTF-8">
<title>Project Structure - Provisioning Platform Documentation</title>
<!-- Custom HTML head -->
<meta name="description" content="Complete documentation for the Provisioning Platform - Infrastructure automation with Nushell, KCL, and Rust">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#ffffff">
<link rel="icon" href="../favicon.svg">
<link rel="shortcut icon" href="../favicon.png">
<link rel="stylesheet" href="../css/variables.css">
<link rel="stylesheet" href="../css/general.css">
<link rel="stylesheet" href="../css/chrome.css">
<link rel="stylesheet" href="../css/print.css" media="print">
<!-- Fonts -->
<link rel="stylesheet" href="../FontAwesome/css/font-awesome.css">
<link rel="stylesheet" href="../fonts/fonts.css">
<!-- Highlight.js Stylesheets -->
<link rel="stylesheet" id="highlight-css" href="../highlight.css">
<link rel="stylesheet" id="tomorrow-night-css" href="../tomorrow-night.css">
<link rel="stylesheet" id="ayu-highlight-css" href="../ayu-highlight.css">
<!-- Custom theme stylesheets -->
<!-- Provide site root and default themes to javascript -->
<script>
const path_to_root = "../";
const default_light_theme = "ayu";
const default_dark_theme = "navy";
</script>
<!-- Start loading toc.js asap -->
<script src="../toc.js"></script>
</head>
<body>
<div id="mdbook-help-container">
<div id="mdbook-help-popup">
<h2 class="mdbook-help-title">Keyboard shortcuts</h2>
<div>
<p>Press <kbd></kbd> or <kbd></kbd> to navigate between chapters</p>
<p>Press <kbd>S</kbd> or <kbd>/</kbd> to search in the book</p>
<p>Press <kbd>?</kbd> to show this help</p>
<p>Press <kbd>Esc</kbd> to hide this help</p>
</div>
</div>
</div>
<div id="body-container">
<!-- Work around some values being stored in localStorage wrapped in quotes -->
<script>
try {
let theme = localStorage.getItem('mdbook-theme');
let sidebar = localStorage.getItem('mdbook-sidebar');
if (theme.startsWith('"') && theme.endsWith('"')) {
localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));
}
if (sidebar.startsWith('"') && sidebar.endsWith('"')) {
localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));
}
} catch (e) { }
</script>
<!-- Set the theme before any content is loaded, prevents flash -->
<script>
const default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? default_dark_theme : default_light_theme;
let theme;
try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }
if (theme === null || theme === undefined) { theme = default_theme; }
const html = document.documentElement;
html.classList.remove('ayu')
html.classList.add(theme);
html.classList.add("js");
</script>
<input type="checkbox" id="sidebar-toggle-anchor" class="hidden">
<!-- Hide / unhide sidebar before it is displayed -->
<script>
let sidebar = null;
const sidebar_toggle = document.getElementById("sidebar-toggle-anchor");
if (document.body.clientWidth >= 1080) {
try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }
sidebar = sidebar || 'visible';
} else {
sidebar = 'hidden';
}
sidebar_toggle.checked = sidebar === 'visible';
html.classList.remove('sidebar-visible');
html.classList.add("sidebar-" + sidebar);
</script>
<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<!-- populated by js -->
<mdbook-sidebar-scrollbox class="sidebar-scrollbox"></mdbook-sidebar-scrollbox>
<noscript>
<iframe class="sidebar-iframe-outer" src="../toc.html"></iframe>
</noscript>
<div id="sidebar-resize-handle" class="sidebar-resize-handle">
<div class="sidebar-resize-indicator"></div>
</div>
</nav>
<div id="page-wrapper" class="page-wrapper">
<div class="page">
<div id="menu-bar-hover-placeholder"></div>
<div id="menu-bar" class="menu-bar sticky">
<div class="left-buttons">
<label id="sidebar-toggle" class="icon-button" for="sidebar-toggle-anchor" title="Toggle Table of Contents" aria-label="Toggle Table of Contents" aria-controls="sidebar">
<i class="fa fa-bars"></i>
</label>
<button id="theme-toggle" class="icon-button" type="button" title="Change theme" aria-label="Change theme" aria-haspopup="true" aria-expanded="false" aria-controls="theme-list">
<i class="fa fa-paint-brush"></i>
</button>
<ul id="theme-list" class="theme-popup" aria-label="Themes" role="menu">
<li role="none"><button role="menuitem" class="theme" id="default_theme">Auto</button></li>
<li role="none"><button role="menuitem" class="theme" id="light">Light</button></li>
<li role="none"><button role="menuitem" class="theme" id="rust">Rust</button></li>
<li role="none"><button role="menuitem" class="theme" id="coal">Coal</button></li>
<li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li>
<li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li>
</ul>
<button id="search-toggle" class="icon-button" type="button" title="Search (`/`)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="/ s" aria-controls="searchbar">
<i class="fa fa-search"></i>
</button>
</div>
<h1 class="menu-title">Provisioning Platform Documentation</h1>
<div class="right-buttons">
<a href="../print.html" title="Print this book" aria-label="Print this book">
<i id="print-button" class="fa fa-print"></i>
</a>
<a href="https://github.com/provisioning/provisioning-platform" title="Git repository" aria-label="Git repository">
<i id="git-repository-button" class="fa fa-github"></i>
</a>
<a href="https://github.com/provisioning/provisioning-platform/edit/main/provisioning/docs/src/development/project-structure.md" title="Suggest an edit" aria-label="Suggest an edit">
<i id="git-edit-button" class="fa fa-edit"></i>
</a>
</div>
</div>
<div id="search-wrapper" class="hidden">
<form id="searchbar-outer" class="searchbar-outer">
<input type="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header">
</form>
<div id="searchresults-outer" class="searchresults-outer hidden">
<div id="searchresults-header" class="searchresults-header"></div>
<ul id="searchresults">
</ul>
</div>
</div>
<!-- Apply ARIA attributes after the sidebar and the sidebar toggle button are added to the DOM -->
<script>
document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');
document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');
Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {
link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);
});
</script>
<div id="content" class="content">
<main>
<h1 id="project-structure-guide"><a class="header" href="#project-structure-guide">Project Structure Guide</a></h1>
<p>This document provides a comprehensive overview of the provisioning projects structure after the major reorganization, explaining both the new development-focused organization and the preserved existing functionality.</p>
<h2 id="table-of-contents"><a class="header" href="#table-of-contents">Table of Contents</a></h2>
<ol>
<li><a href="#overview">Overview</a></li>
<li><a href="#new-structure-vs-legacy">New Structure vs Legacy</a></li>
<li><a href="#core-directories">Core Directories</a></li>
<li><a href="#development-workspace">Development Workspace</a></li>
<li><a href="#file-naming-conventions">File Naming Conventions</a></li>
<li><a href="#navigation-guide">Navigation Guide</a></li>
<li><a href="#migration-path">Migration Path</a></li>
</ol>
<h2 id="overview"><a class="header" href="#overview">Overview</a></h2>
<p>The provisioning project has been restructured to support a dual-organization approach:</p>
<ul>
<li><strong><code>src/</code></strong>: Development-focused structure with build tools, distribution system, and core components</li>
<li><strong>Legacy directories</strong>: Preserved in their original locations for backward compatibility</li>
<li><strong><code>workspace/</code></strong>: Development workspace with tools and runtime management</li>
</ul>
<p>This reorganization enables efficient development workflows while maintaining full backward compatibility with existing deployments.</p>
<h2 id="new-structure-vs-legacy"><a class="header" href="#new-structure-vs-legacy">New Structure vs Legacy</a></h2>
<h3 id="new-development-structure-src"><a class="header" href="#new-development-structure-src">New Development Structure (<code>/src/</code>)</a></h3>
<pre><code class="language-plaintext">src/
├── config/ # System configuration
├── control-center/ # Control center application
├── control-center-ui/ # Web UI for control center
├── core/ # Core system libraries
├── docs/ # Documentation (new)
├── extensions/ # Extension framework
├── generators/ # Code generation tools
├── kcl/ # KCL configuration language files
├── orchestrator/ # Hybrid Rust/Nushell orchestrator
├── platform/ # Platform-specific code
├── provisioning/ # Main provisioning
├── templates/ # Template files
├── tools/ # Build and development tools
└── utils/ # Utility scripts
```plaintext
### Legacy Structure (Preserved)
```plaintext
repo-cnz/
├── cluster/ # Cluster configurations (preserved)
├── core/ # Core system (preserved)
├── generate/ # Generation scripts (preserved)
├── kcl/ # KCL files (preserved)
├── klab/ # Development lab (preserved)
├── nushell-plugins/ # Plugin development (preserved)
├── providers/ # Cloud providers (preserved)
├── taskservs/ # Task services (preserved)
└── templates/ # Template files (preserved)
```plaintext
### Development Workspace (`/workspace/`)
```plaintext
workspace/
├── config/ # Development configuration
├── extensions/ # Extension development
├── infra/ # Development infrastructure
├── lib/ # Workspace libraries
├── runtime/ # Runtime data
└── tools/ # Workspace management tools
```plaintext
## Core Directories
### `/src/core/` - Core Development Libraries
**Purpose**: Development-focused core libraries and entry points
**Key Files**:
- `nulib/provisioning` - Main CLI entry point (symlinks to legacy location)
- `nulib/lib_provisioning/` - Core provisioning libraries
- `nulib/workflows/` - Workflow management (orchestrator integration)
**Relationship to Legacy**: Preserves original `core/` functionality while adding development enhancements
### `/src/tools/` - Build and Development Tools
**Purpose**: Complete build system for the provisioning project
**Key Components**:
```plaintext
tools/
├── build/ # Build tools
│ ├── compile-platform.nu # Platform-specific compilation
│ ├── bundle-core.nu # Core library bundling
│ ├── validate-kcl.nu # KCL validation
│ ├── clean-build.nu # Build cleanup
│ └── test-distribution.nu # Distribution testing
├── distribution/ # Distribution tools
│ ├── generate-distribution.nu # Main distribution generator
│ ├── prepare-platform-dist.nu # Platform-specific distribution
│ ├── prepare-core-dist.nu # Core distribution
│ ├── create-installer.nu # Installer creation
│ └── generate-docs.nu # Documentation generation
├── package/ # Packaging tools
│ ├── package-binaries.nu # Binary packaging
│ ├── build-containers.nu # Container image building
│ ├── create-tarball.nu # Archive creation
│ └── validate-package.nu # Package validation
├── release/ # Release management
│ ├── create-release.nu # Release creation
│ ├── upload-artifacts.nu # Artifact upload
│ ├── rollback-release.nu # Release rollback
│ ├── notify-users.nu # Release notifications
│ └── update-registry.nu # Package registry updates
└── Makefile # Main build system (40+ targets)
```plaintext
### `/src/orchestrator/` - Hybrid Orchestrator
**Purpose**: Rust/Nushell hybrid orchestrator for solving deep call stack limitations
**Key Components**:
- `src/` - Rust orchestrator implementation
- `scripts/` - Orchestrator management scripts
- `data/` - File-based task queue and persistence
**Integration**: Provides REST API and workflow management while preserving all Nushell business logic
### `/src/provisioning/` - Enhanced Provisioning
**Purpose**: Enhanced version of the main provisioning with additional features
**Key Features**:
- Batch workflow system (v3.1.0)
- Provider-agnostic design
- Configuration-driven architecture (v2.0.0)
### `/workspace/` - Development Workspace
**Purpose**: Complete development environment with tools and runtime management
**Key Components**:
- `tools/workspace.nu` - Unified workspace management interface
- `lib/path-resolver.nu` - Smart path resolution system
- `config/` - Environment-specific development configurations
- `extensions/` - Extension development templates and examples
- `infra/` - Development infrastructure examples
- `runtime/` - Isolated runtime data per user
## Development Workspace
### Workspace Management
The workspace provides a sophisticated development environment:
**Initialization**:
```bash
cd workspace/tools
nu workspace.nu init --user-name developer --infra-name my-infra
```plaintext
**Health Monitoring**:
```bash
nu workspace.nu health --detailed --fix-issues
```plaintext
**Path Resolution**:
```nushell
use lib/path-resolver.nu
let config = (path-resolver resolve_config "user" --workspace-user "john")
```plaintext
### Extension Development
The workspace provides templates for developing:
- **Providers**: Custom cloud provider implementations
- **Task Services**: Infrastructure service components
- **Clusters**: Complete deployment solutions
Templates are available in `workspace/extensions/{type}/template/`
### Configuration Hierarchy
The workspace implements a sophisticated configuration cascade:
1. Workspace user configuration (`workspace/config/{user}.toml`)
2. Environment-specific defaults (`workspace/config/{env}-defaults.toml`)
3. Workspace defaults (`workspace/config/dev-defaults.toml`)
4. Core system defaults (`config.defaults.toml`)
## File Naming Conventions
### Nushell Files (`.nu`)
- **Commands**: `kebab-case` - `create-server.nu`, `validate-config.nu`
- **Modules**: `snake_case` - `lib_provisioning`, `path_resolver`
- **Scripts**: `kebab-case` - `workspace-health.nu`, `runtime-manager.nu`
### Configuration Files
- **TOML**: `kebab-case.toml` - `config-defaults.toml`, `user-settings.toml`
- **Environment**: `{env}-defaults.toml` - `dev-defaults.toml`, `prod-defaults.toml`
- **Examples**: `*.toml.example` - `local-overrides.toml.example`
### KCL Files (`.k`)
- **Schemas**: `PascalCase` types - `ServerConfig`, `WorkflowDefinition`
- **Files**: `kebab-case.k` - `server-config.k`, `workflow-schema.k`
- **Modules**: `kcl.mod` - Module definition files
### Build and Distribution
- **Scripts**: `kebab-case.nu` - `compile-platform.nu`, `generate-distribution.nu`
- **Makefiles**: `Makefile` - Standard naming
- **Archives**: `{project}-{version}-{platform}-{variant}.{ext}`
## Navigation Guide
### Finding Components
**Core System Entry Points**:
```bash
# Main CLI (development version)
/src/core/nulib/provisioning
# Legacy CLI (production version)
/core/nulib/provisioning
# Workspace management
/workspace/tools/workspace.nu
```plaintext
**Build System**:
```bash
# Main build system
cd /src/tools &amp;&amp; make help
# Quick development build
make dev-build
# Complete distribution
make all
```plaintext
**Configuration Files**:
```bash
# System defaults
/config.defaults.toml
# User configuration (workspace)
/workspace/config/{user}.toml
# Environment-specific
/workspace/config/{env}-defaults.toml
```plaintext
**Extension Development**:
```bash
# Provider template
/workspace/extensions/providers/template/
# Task service template
/workspace/extensions/taskservs/template/
# Cluster template
/workspace/extensions/clusters/template/
```plaintext
### Common Workflows
**1. Development Setup**:
```bash
# Initialize workspace
cd workspace/tools
nu workspace.nu init --user-name $USER
# Check health
nu workspace.nu health --detailed
```plaintext
**2. Building Distribution**:
```bash
# Complete build
cd src/tools
make all
# Platform-specific build
make linux
make macos
make windows
```plaintext
**3. Extension Development**:
```bash
# Create new provider
cp -r workspace/extensions/providers/template workspace/extensions/providers/my-provider
# Test extension
nu workspace/extensions/providers/my-provider/nulib/provider.nu test
```plaintext
### Legacy Compatibility
**Existing Commands Still Work**:
```bash
# All existing commands preserved
./core/nulib/provisioning server create
./core/nulib/provisioning taskserv install kubernetes
./core/nulib/provisioning cluster create buildkit
```plaintext
**Configuration Migration**:
- ENV variables still supported as fallbacks
- New configuration system provides better defaults
- Migration tools available in `src/tools/migration/`
## Migration Path
### For Users
**No Changes Required**:
- All existing commands continue to work
- Configuration files remain compatible
- Existing infrastructure deployments unaffected
**Optional Enhancements**:
- Migrate to new configuration system for better defaults
- Use workspace for development environments
- Leverage new build system for custom distributions
### For Developers
**Development Environment**:
1. Initialize development workspace: `nu workspace/tools/workspace.nu init`
2. Use new build system: `cd src/tools &amp;&amp; make dev-build`
3. Leverage extension templates for custom development
**Build System**:
1. Use new Makefile for comprehensive build management
2. Leverage distribution tools for packaging
3. Use release management for version control
**Orchestrator Integration**:
1. Start orchestrator for workflow management: `cd src/orchestrator &amp;&amp; ./scripts/start-orchestrator.nu`
2. Use workflow APIs for complex operations
3. Leverage batch operations for efficiency
### Migration Tools
**Available Migration Scripts**:
- `src/tools/migration/config-migration.nu` - Configuration migration
- `src/tools/migration/workspace-setup.nu` - Workspace initialization
- `src/tools/migration/path-resolver.nu` - Path resolution migration
**Validation Tools**:
- `src/tools/validation/system-health.nu` - System health validation
- `src/tools/validation/compatibility-check.nu` - Compatibility verification
- `src/tools/validation/migration-status.nu` - Migration status tracking
## Architecture Benefits
### Development Efficiency
- **Build System**: Comprehensive 40+ target Makefile system
- **Workspace Isolation**: Per-user development environments
- **Extension Framework**: Template-based extension development
### Production Reliability
- **Backward Compatibility**: All existing functionality preserved
- **Configuration Migration**: Gradual migration from ENV to config-driven
- **Orchestrator Architecture**: Hybrid Rust/Nushell for performance and flexibility
- **Workflow Management**: Batch operations with rollback capabilities
### Maintenance Benefits
- **Clean Separation**: Development tools separate from production code
- **Organized Structure**: Logical grouping of related functionality
- **Documentation**: Comprehensive documentation and examples
- **Testing Framework**: Built-in testing and validation tools
This structure represents a significant evolution in the project's organization while maintaining complete backward compatibility and providing powerful new development capabilities.
</code></pre>
</main>
<nav class="nav-wrapper" aria-label="Page navigation">
<!-- Mobile navigation buttons -->
<a rel="prev" href="../development/taskserv-quick-guide.html" class="mobile-nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left">
<i class="fa fa-angle-left"></i>
</a>
<a rel="next prefetch" href="../development/provider-agnostic-architecture.html" class="mobile-nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right">
<i class="fa fa-angle-right"></i>
</a>
<div style="clear: both"></div>
</nav>
</div>
</div>
<nav class="nav-wide-wrapper" aria-label="Page navigation">
<a rel="prev" href="../development/taskserv-quick-guide.html" class="nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left">
<i class="fa fa-angle-left"></i>
</a>
<a rel="next prefetch" href="../development/provider-agnostic-architecture.html" class="nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right">
<i class="fa fa-angle-right"></i>
</a>
</nav>
</div>
<script>
window.playground_copyable = true;
</script>
<script src="../elasticlunr.min.js"></script>
<script src="../mark.min.js"></script>
<script src="../searcher.js"></script>
<script src="../clipboard.min.js"></script>
<script src="../highlight.js"></script>
<script src="../book.js"></script>
<!-- Custom JS scripts -->
</div>
</body>
</html>