--- # Core Engine The **Core Engine** is the foundational component of the [Provisioning project](https://repo.jesusperez.pro/jesus/provisioning), providing the unified CLI interface, core Nushell libraries, and essential utility scripts. Built on **Nushell** and **KCL**, it serves as the primary entry point for all infrastructure operations. ## Overview The Core Engine provides: - **Unified CLI Interface** - Single command-line tool for all infrastructure operations - **Core Libraries** - Reusable Nushell modules for configuration, validation, and utilities - **Provider Abstraction** - Cloud-agnostic interface supporting UpCloud, AWS, and local providers - **Workflow Integration** - Commands for submitting and managing workflows (executed by the orchestrator) - **Configuration System** - Hierarchical, config-driven architecture with 476+ configuration accessors ## Project Structure ``` provisioning/core/ ├── cli/ # Command-line interface │ └── provisioning # Main CLI entry point (211 lines, 84% reduction) ├── nulib/ # Core Nushell libraries │ ├── lib_provisioning/ # Core provisioning libraries │ │ ├── config/ # Configuration loading and management │ │ ├── utils/ # Utility functions (SSH, validation, logging) │ │ ├── providers/ # Provider abstraction layer │ │ ├── secrets/ # Secrets management (SOPS integration) │ │ ├── workspace/ # Workspace management │ │ └── infra_validator/ # Infrastructure validation engine │ ├── main_provisioning/ # CLI command handlers │ │ ├── flags.nu # Centralized flag handling │ │ ├── dispatcher.nu # Command routing (80+ shortcuts) │ │ ├── help_system.nu # Categorized help system │ │ └── commands/ # Domain-focused command modules │ ├── servers/ # Server management modules │ ├── taskservs/ # Task service modules │ ├── clusters/ # Cluster management modules │ └── workflows/ # Workflow orchestration modules ├── scripts/ # Utility scripts │ └── test/ # Test automation └── resources/ # Images and logos ``` ## Installation ### Prerequisites - **Nushell 0.107.1+** - Primary shell and scripting environment - **KCL 0.11.2+** - Configuration language for infrastructure definitions - **SOPS 3.10.2+** - Secrets management (optional but recommended) - **Age 1.2.1+** - Encryption tool for secrets (optional) ### Adding to PATH To use the CLI globally, add it to your PATH: ```bash # Create symbolic link ln -sf "$(pwd)/provisioning/core/cli/provisioning" /usr/local/bin/provisioning # Or add to PATH in your shell config (~/.bashrc, ~/.zshrc, etc.) export PATH="$PATH:/path/to/project-provisioning/provisioning/core/cli" ``` Verify installation: ```bash provisioning version provisioning help ``` ## Quick Start ### Basic Commands ```bash # Show help and available commands provisioning help # Show environment and configuration provisioning env provisioning allenv # Validate configuration provisioning validate config # List available providers provisioning providers # Show system information provisioning nuinfo ``` ### Infrastructure Operations ```bash # Create new infrastructure workspace provisioning workspace init my-project # Create servers provisioning server create --check # Install task services (infrastructure components) provisioning taskserv create kubernetes # Create complete cluster provisioning cluster create my-cluster # SSH into server provisioning server ssh hostname-01 ``` ### Quick Reference For fastest command reference: ```bash provisioning sc ``` For complete guides: ```bash provisioning guide from-scratch # Complete deployment guide provisioning guide quickstart # Command shortcuts reference provisioning guide customize # Customization patterns ``` ## Core Libraries ### Configuration System (`lib_provisioning/config/`) Hierarchical configuration loading with 476+ config accessors: - **Defaults** → **User** → **Project** → **Infrastructure** → **Environment** → **Runtime** - Replaced 200+ ENV variables with config-driven architecture - Variable interpolation: `{{paths.base}}`, `{{env.HOME}}`, `{{now.date}}` ```nushell use lib_provisioning/config # Get configuration value let value = config get "servers.default_plan" # Load workspace config let ws_config = config load-workspace "my-project" ``` ### Provider Abstraction (`lib_provisioning/providers/`) Unified interface for cloud providers: ```nushell use lib_provisioning/providers # Get provider interface let provider = providers get "upcloud" # Create server using provider $provider | invoke "create_server" $server_config ``` ### Utilities (`lib_provisioning/utils/`) Common utility functions: - **SSH Operations** - `utils/ssh.nu` - **Validation** - `utils/validation.nu` - **Logging** - `utils/logging.nu` - **Error Handling** - `utils/error.nu` - **File Operations** - `utils/files.nu` - **Version Management** - `utils/version_manager.nu` ### Workflow Orchestration (`workflows/`) Batch operations with dependency resolution: ```bash # Submit batch workflow provisioning batch submit workflows/example.k # Monitor workflow progress provisioning batch monitor # List all workflows provisioning workflow list # Get workflow status provisioning workflow status ``` ## CLI Architecture ### Modular Design The CLI uses a domain-driven architecture: - **Infrastructure** - Servers, task services, clusters - **Orchestration** - Workflows, batch operations, orchestrator management - **Development** - Modules, layers, versioning, packaging - **Workspace** - Workspace management, templates - **Configuration** - Settings, environment, validation - **Utilities** - SSH, SOPS, cache, plugins ### Command Shortcuts 80+ shortcuts for improved productivity: | Full Command | Shortcuts | Description | |--------------|-----------|-------------| | `server` | `s` | Server operations | | `taskserv` | `t`, `task` | Task service operations | | `cluster` | `cl` | Cluster operations | | `workspace` | `ws` | Workspace management | | `workflow` | `wf`, `flow` | Workflow operations | | `batch` | `bat` | Batch operations | | `module` | `mod` | Module management | | `generate` | `g`, `gen` | Code generation | | `validate` | `val` | Validation operations | See complete reference: `provisioning sc` or `provisioning guide quickstart` ### Bi-directional Help Help works in both directions: ```bash provisioning help workspace # ✅ provisioning workspace help # ✅ Same result provisioning ws help # ✅ Shortcut also works ``` ## Configuration ### Configuration Files With the new structure (2025-09-26): - **System Defaults**: `provisioning/config/config.defaults.toml` - **User Config**: `workspace/config/local-overrides.toml` - **Environment Configs**: `workspace/config/{dev,test,prod}-defaults.toml` - **Infrastructure Configs**: `workspace/infra/{name}/config.toml` ### Environment Variables Legacy ENV variables still supported during transition: ```bash # Show current environment provisioning env # Show all configuration and environment provisioning allenv # Use specific environment PROVISIONING_ENV=prod provisioning server list ``` ### Debug Flags ```bash # Enable debug mode provisioning --debug # Check mode (dry-run, no changes) provisioning --check server create # Auto-confirm operations provisioning --yes cluster delete # Specify infrastructure provisioning --infra my-project server list ``` ## Design Principles ### Modularity Clean separation between libraries, CLI, and extensions: - **Core Engine** (`core/`) - CLI, libraries, scripts - **Extensions** (`extensions/`) - Providers, task services, clusters - **Platform Services** (`platform/`) - Orchestrator, control center, API gateway ### Extensibility Plugin architecture for extending functionality: - Provider plugins for cloud integrations - Task service definitions for infrastructure components - Cluster templates for complete deployments - Workflow templates for automation ### Consistency Unified API patterns across all components: - Standardized command structure - Common flag handling - Consistent error messages - Predictable output formats ### Performance Optimized for large-scale infrastructure operations: - Token-optimized agents (85-90% efficiency) - Batch operations with parallel execution - Checkpoint-based recovery - Incremental state management ## Migration Strategy The project follows a three-phase migration: 1. **Phase 1** (Complete) - Created symbolic links to existing implementations 2. **Phase 2** (In Progress) - Gradual migration of components to new structure 3. **Phase 3** (Planned) - Consolidate and optimize unified core engine ## Dependencies ### Required - **Nushell 0.107.1+** - Shell and scripting language - **KCL 0.11.2+** - Configuration language ### Recommended - **SOPS 3.10.2+** - Secrets management - **Age 1.2.1+** - Encryption for secrets - **K9s 0.50.6+** - Kubernetes management interface ### Optional - **nu_plugin_tera** - Template rendering - **nu_plugin_kcl** - KCL integration (CLI `kcl` is required, plugin optional) ## Documentation ### User Guides - **Quick Start**: `provisioning guide from-scratch` - **Command Reference**: `provisioning sc` - **Update Guide**: `provisioning guide update` - **Customization**: `provisioning guide customize` ### Architecture Documentation - **CLI Architecture**: `docs/architecture/ADR-006-provisioning-cli-refactoring.md` - **Configuration System**: See `.claude/features/configuration-system.md` - **Batch Workflows**: See `.claude/features/batch-workflow-system.md` - **Orchestrator**: See `.claude/features/orchestrator-architecture.md` ### API Documentation - **REST API**: See `docs/api/` (when orchestrator is running) - **Nushell Modules**: See inline documentation in `nulib/` modules ## Testing ### Run Tests ```bash # Run all tests nu provisioning/core/scripts/test/test_all.nu # Run specific test group nu provisioning/core/scripts/test/test_config.nu nu provisioning/core/scripts/test/test_cli.nu ``` ### Test Coverage The core engine includes comprehensive test coverage: - Configuration loading and validation - Provider interface contracts - CLI command routing - Workflow orchestration - Error handling ## Contributing When contributing to the Core Engine: 1. **Follow Nushell Patterns** - See `.claude/best_nushell_code.md` 2. **Use Modular Design** - Domain-focused modules in appropriate directories 3. **Add Tests** - Include tests for new functionality 4. **Update Documentation** - Keep README and inline docs current 5. **Follow PAP** - Project Architecture Principles (see `CLAUDE.md`) ## Troubleshooting ### Common Issues **Missing environment variables:** ```bash provisioning env # Check current configuration provisioning validate config # Validate configuration files ``` **KCL compilation errors:** ```bash kcl fmt .k # Format KCL file kcl run .k # Test KCL file ``` **Provider authentication:** ```bash provisioning providers # List available providers provisioning show settings # View provider configuration ``` ### Debug Mode Enable verbose logging: ```bash provisioning --debug ``` ### Getting Help ```bash provisioning help # Show main help provisioning help # Category-specific help provisioning help # Command-specific help provisioning guide list # List all guides ``` ## Version Information Check system versions: ```bash provisioning version # Show all versions provisioning nuinfo # Nushell information ``` ## License See project root LICENSE file. --- **Maintained By**: Architecture Team **Last Updated**: 2025-10-07