44648e3206
- Remove KCL ecosystem (~220 files deleted) - Migrate all infrastructure to Nickel schema system - Consolidate documentation: legacy docs → provisioning/docs/src/ - Add CI/CD workflows (.github/) and Rust build config (.cargo/) - Update core system for Nickel schema parsing - Update README.md and CHANGES.md for v5.0.0 release - Fix pre-commit hooks: end-of-file, trailing-whitespace - Breaking changes: KCL workspaces require migration - Migration bridge available in docs/src/development/
410 lines
13 KiB
Markdown
410 lines
13 KiB
Markdown
# Project Structure Guide
|
|
|
|
This document provides a comprehensive overview of the provisioning project's structure after the major reorganization, explaining both the new development-focused organization and the preserved existing functionality.
|
|
|
|
## Table of Contents
|
|
|
|
1. [Overview](#overview)
|
|
2. [New Structure vs Legacy](#new-structure-vs-legacy)
|
|
3. [Core Directories](#core-directories)
|
|
4. [Development Workspace](#development-workspace)
|
|
5. [File Naming Conventions](#file-naming-conventions)
|
|
6. [Navigation Guide](#navigation-guide)
|
|
7. [Migration Path](#migration-path)
|
|
|
|
## Overview
|
|
|
|
The provisioning project has been restructured to support a dual-organization approach:
|
|
|
|
- **`src/`**: Development-focused structure with build tools, distribution system, and core components
|
|
- **Legacy directories**: Preserved in their original locations for backward compatibility
|
|
- **`workspace/`**: Development workspace with tools and runtime management
|
|
|
|
This reorganization enables efficient development workflows while maintaining full backward compatibility with existing deployments.
|
|
|
|
## New Structure vs Legacy
|
|
|
|
### New Development Structure (`/src/`)
|
|
|
|
```plaintext
|
|
src/
|
|
├── config/ # System configuration
|
|
├── control-center/ # Control center application
|
|
├── control-center-ui/ # Web UI for control center
|
|
├── core/ # Core system libraries
|
|
├── docs/ # Documentation (new)
|
|
├── extensions/ # Extension framework
|
|
├── generators/ # Code generation tools
|
|
├── schemas/ # Nickel configuration schemas (migrated from kcl/)
|
|
├── orchestrator/ # Hybrid Rust/Nushell orchestrator
|
|
├── platform/ # Platform-specific code
|
|
├── provisioning/ # Main provisioning
|
|
├── templates/ # Template files
|
|
├── tools/ # Build and development tools
|
|
└── utils/ # Utility scripts
|
|
```plaintext
|
|
|
|
### Legacy Structure (Preserved)
|
|
|
|
```plaintext
|
|
repo-cnz/
|
|
├── cluster/ # Cluster configurations (preserved)
|
|
├── core/ # Core system (preserved)
|
|
├── generate/ # Generation scripts (preserved)
|
|
├── schemas/ # Nickel schemas (migrated from kcl/)
|
|
├── klab/ # Development lab (preserved)
|
|
├── nushell-plugins/ # Plugin development (preserved)
|
|
├── providers/ # Cloud providers (preserved)
|
|
├── taskservs/ # Task services (preserved)
|
|
└── templates/ # Template files (preserved)
|
|
```plaintext
|
|
|
|
### Development Workspace (`/workspace/`)
|
|
|
|
```plaintext
|
|
workspace/
|
|
├── config/ # Development configuration
|
|
├── extensions/ # Extension development
|
|
├── infra/ # Development infrastructure
|
|
├── lib/ # Workspace libraries
|
|
├── runtime/ # Runtime data
|
|
└── tools/ # Workspace management tools
|
|
```plaintext
|
|
|
|
## Core Directories
|
|
|
|
### `/src/core/` - Core Development Libraries
|
|
|
|
**Purpose**: Development-focused core libraries and entry points
|
|
|
|
**Key Files**:
|
|
|
|
- `nulib/provisioning` - Main CLI entry point (symlinks to legacy location)
|
|
- `nulib/lib_provisioning/` - Core provisioning libraries
|
|
- `nulib/workflows/` - Workflow management (orchestrator integration)
|
|
|
|
**Relationship to Legacy**: Preserves original `core/` functionality while adding development enhancements
|
|
|
|
### `/src/tools/` - Build and Development Tools
|
|
|
|
**Purpose**: Complete build system for the provisioning project
|
|
|
|
**Key Components**:
|
|
|
|
```plaintext
|
|
tools/
|
|
├── build/ # Build tools
|
|
│ ├── compile-platform.nu # Platform-specific compilation
|
|
│ ├── bundle-core.nu # Core library bundling
|
|
│ ├── validate-nickel.nu # Nickel schema validation
|
|
│ ├── clean-build.nu # Build cleanup
|
|
│ └── test-distribution.nu # Distribution testing
|
|
├── distribution/ # Distribution tools
|
|
│ ├── generate-distribution.nu # Main distribution generator
|
|
│ ├── prepare-platform-dist.nu # Platform-specific distribution
|
|
│ ├── prepare-core-dist.nu # Core distribution
|
|
│ ├── create-installer.nu # Installer creation
|
|
│ └── generate-docs.nu # Documentation generation
|
|
├── package/ # Packaging tools
|
|
│ ├── package-binaries.nu # Binary packaging
|
|
│ ├── build-containers.nu # Container image building
|
|
│ ├── create-tarball.nu # Archive creation
|
|
│ └── validate-package.nu # Package validation
|
|
├── release/ # Release management
|
|
│ ├── create-release.nu # Release creation
|
|
│ ├── upload-artifacts.nu # Artifact upload
|
|
│ ├── rollback-release.nu # Release rollback
|
|
│ ├── notify-users.nu # Release notifications
|
|
│ └── update-registry.nu # Package registry updates
|
|
└── Makefile # Main build system (40+ targets)
|
|
```plaintext
|
|
|
|
### `/src/orchestrator/` - Hybrid Orchestrator
|
|
|
|
**Purpose**: Rust/Nushell hybrid orchestrator for solving deep call stack limitations
|
|
|
|
**Key Components**:
|
|
|
|
- `src/` - Rust orchestrator implementation
|
|
- `scripts/` - Orchestrator management scripts
|
|
- `data/` - File-based task queue and persistence
|
|
|
|
**Integration**: Provides REST API and workflow management while preserving all Nushell business logic
|
|
|
|
### `/src/provisioning/` - Enhanced Provisioning
|
|
|
|
**Purpose**: Enhanced version of the main provisioning with additional features
|
|
|
|
**Key Features**:
|
|
|
|
- Batch workflow system (v3.1.0)
|
|
- Provider-agnostic design
|
|
- Configuration-driven architecture (v2.0.0)
|
|
|
|
### `/workspace/` - Development Workspace
|
|
|
|
**Purpose**: Complete development environment with tools and runtime management
|
|
|
|
**Key Components**:
|
|
|
|
- `tools/workspace.nu` - Unified workspace management interface
|
|
- `lib/path-resolver.nu` - Smart path resolution system
|
|
- `config/` - Environment-specific development configurations
|
|
- `extensions/` - Extension development templates and examples
|
|
- `infra/` - Development infrastructure examples
|
|
- `runtime/` - Isolated runtime data per user
|
|
|
|
## Development Workspace
|
|
|
|
### Workspace Management
|
|
|
|
The workspace provides a sophisticated development environment:
|
|
|
|
**Initialization**:
|
|
|
|
```bash
|
|
cd workspace/tools
|
|
nu workspace.nu init --user-name developer --infra-name my-infra
|
|
```plaintext
|
|
|
|
**Health Monitoring**:
|
|
|
|
```bash
|
|
nu workspace.nu health --detailed --fix-issues
|
|
```plaintext
|
|
|
|
**Path Resolution**:
|
|
|
|
```nushell
|
|
use lib/path-resolver.nu
|
|
let config = (path-resolver resolve_config "user" --workspace-user "john")
|
|
```plaintext
|
|
|
|
### Extension Development
|
|
|
|
The workspace provides templates for developing:
|
|
|
|
- **Providers**: Custom cloud provider implementations
|
|
- **Task Services**: Infrastructure service components
|
|
- **Clusters**: Complete deployment solutions
|
|
|
|
Templates are available in `workspace/extensions/{type}/template/`
|
|
|
|
### Configuration Hierarchy
|
|
|
|
The workspace implements a sophisticated configuration cascade:
|
|
|
|
1. Workspace user configuration (`workspace/config/{user}.toml`)
|
|
2. Environment-specific defaults (`workspace/config/{env}-defaults.toml`)
|
|
3. Workspace defaults (`workspace/config/dev-defaults.toml`)
|
|
4. Core system defaults (`config.defaults.toml`)
|
|
|
|
## File Naming Conventions
|
|
|
|
### Nushell Files (`.nu`)
|
|
|
|
- **Commands**: `kebab-case` - `create-server.nu`, `validate-config.nu`
|
|
- **Modules**: `snake_case` - `lib_provisioning`, `path_resolver`
|
|
- **Scripts**: `kebab-case` - `workspace-health.nu`, `runtime-manager.nu`
|
|
|
|
### Configuration Files
|
|
|
|
- **TOML**: `kebab-case.toml` - `config-defaults.toml`, `user-settings.toml`
|
|
- **Environment**: `{env}-defaults.toml` - `dev-defaults.toml`, `prod-defaults.toml`
|
|
- **Examples**: `*.toml.example` - `local-overrides.toml.example`
|
|
|
|
### Nickel Files (`.ncl`)
|
|
|
|
- **Schemas**: `kebab-case.ncl` - `server-config.ncl`, `workflow-schema.ncl`
|
|
- **Configuration**: `manifest.toml` - Package metadata
|
|
- **Structure**: Organized in `schemas/` directories per extension
|
|
|
|
### Build and Distribution
|
|
|
|
- **Scripts**: `kebab-case.nu` - `compile-platform.nu`, `generate-distribution.nu`
|
|
- **Makefiles**: `Makefile` - Standard naming
|
|
- **Archives**: `{project}-{version}-{platform}-{variant}.{ext}`
|
|
|
|
## Navigation Guide
|
|
|
|
### Finding Components
|
|
|
|
**Core System Entry Points**:
|
|
|
|
```bash
|
|
# Main CLI (development version)
|
|
/src/core/nulib/provisioning
|
|
|
|
# Legacy CLI (production version)
|
|
/core/nulib/provisioning
|
|
|
|
# Workspace management
|
|
/workspace/tools/workspace.nu
|
|
```plaintext
|
|
|
|
**Build System**:
|
|
|
|
```bash
|
|
# Main build system
|
|
cd /src/tools && make help
|
|
|
|
# Quick development build
|
|
make dev-build
|
|
|
|
# Complete distribution
|
|
make all
|
|
```plaintext
|
|
|
|
**Configuration Files**:
|
|
|
|
```bash
|
|
# System defaults
|
|
/config.defaults.toml
|
|
|
|
# User configuration (workspace)
|
|
/workspace/config/{user}.toml
|
|
|
|
# Environment-specific
|
|
/workspace/config/{env}-defaults.toml
|
|
```plaintext
|
|
|
|
**Extension Development**:
|
|
|
|
```bash
|
|
# Provider template
|
|
/workspace/extensions/providers/template/
|
|
|
|
# Task service template
|
|
/workspace/extensions/taskservs/template/
|
|
|
|
# Cluster template
|
|
/workspace/extensions/clusters/template/
|
|
```plaintext
|
|
|
|
### Common Workflows
|
|
|
|
**1. Development Setup**:
|
|
|
|
```bash
|
|
# Initialize workspace
|
|
cd workspace/tools
|
|
nu workspace.nu init --user-name $USER
|
|
|
|
# Check health
|
|
nu workspace.nu health --detailed
|
|
```plaintext
|
|
|
|
**2. Building Distribution**:
|
|
|
|
```bash
|
|
# Complete build
|
|
cd src/tools
|
|
make all
|
|
|
|
# Platform-specific build
|
|
make linux
|
|
make macos
|
|
make windows
|
|
```plaintext
|
|
|
|
**3. Extension Development**:
|
|
|
|
```bash
|
|
# Create new provider
|
|
cp -r workspace/extensions/providers/template workspace/extensions/providers/my-provider
|
|
|
|
# Test extension
|
|
nu workspace/extensions/providers/my-provider/nulib/provider.nu test
|
|
```plaintext
|
|
|
|
### Legacy Compatibility
|
|
|
|
**Existing Commands Still Work**:
|
|
|
|
```bash
|
|
# All existing commands preserved
|
|
./core/nulib/provisioning server create
|
|
./core/nulib/provisioning taskserv install kubernetes
|
|
./core/nulib/provisioning cluster create buildkit
|
|
```plaintext
|
|
|
|
**Configuration Migration**:
|
|
|
|
- ENV variables still supported as fallbacks
|
|
- New configuration system provides better defaults
|
|
- Migration tools available in `src/tools/migration/`
|
|
|
|
## Migration Path
|
|
|
|
### For Users
|
|
|
|
**No Changes Required**:
|
|
|
|
- All existing commands continue to work
|
|
- Configuration files remain compatible
|
|
- Existing infrastructure deployments unaffected
|
|
|
|
**Optional Enhancements**:
|
|
|
|
- Migrate to new configuration system for better defaults
|
|
- Use workspace for development environments
|
|
- Leverage new build system for custom distributions
|
|
|
|
### For Developers
|
|
|
|
**Development Environment**:
|
|
|
|
1. Initialize development workspace: `nu workspace/tools/workspace.nu init`
|
|
2. Use new build system: `cd src/tools && make dev-build`
|
|
3. Leverage extension templates for custom development
|
|
|
|
**Build System**:
|
|
|
|
1. Use new Makefile for comprehensive build management
|
|
2. Leverage distribution tools for packaging
|
|
3. Use release management for version control
|
|
|
|
**Orchestrator Integration**:
|
|
|
|
1. Start orchestrator for workflow management: `cd src/orchestrator && ./scripts/start-orchestrator.nu`
|
|
2. Use workflow APIs for complex operations
|
|
3. Leverage batch operations for efficiency
|
|
|
|
### Migration Tools
|
|
|
|
**Available Migration Scripts**:
|
|
|
|
- `src/tools/migration/config-migration.nu` - Configuration migration
|
|
- `src/tools/migration/workspace-setup.nu` - Workspace initialization
|
|
- `src/tools/migration/path-resolver.nu` - Path resolution migration
|
|
|
|
**Validation Tools**:
|
|
|
|
- `src/tools/validation/system-health.nu` - System health validation
|
|
- `src/tools/validation/compatibility-check.nu` - Compatibility verification
|
|
- `src/tools/validation/migration-status.nu` - Migration status tracking
|
|
|
|
## Architecture Benefits
|
|
|
|
### Development Efficiency
|
|
|
|
- **Build System**: Comprehensive 40+ target Makefile system
|
|
- **Workspace Isolation**: Per-user development environments
|
|
- **Extension Framework**: Template-based extension development
|
|
|
|
### Production Reliability
|
|
|
|
- **Backward Compatibility**: All existing functionality preserved
|
|
- **Configuration Migration**: Gradual migration from ENV to config-driven
|
|
- **Orchestrator Architecture**: Hybrid Rust/Nushell for performance and flexibility
|
|
- **Workflow Management**: Batch operations with rollback capabilities
|
|
|
|
### Maintenance Benefits
|
|
|
|
- **Clean Separation**: Development tools separate from production code
|
|
- **Organized Structure**: Logical grouping of related functionality
|
|
- **Documentation**: Comprehensive documentation and examples
|
|
- **Testing Framework**: Built-in testing and validation tools
|
|
|
|
This structure represents a significant evolution in the project's organization while maintaining complete backward compatibility and providing powerful new development capabilities.
|