commit ce2fce28986bfacfd78626d68b5c20dcc171e807 Author: Jesús Pérez Date: Tue Oct 7 05:51:13 2025 +0100 init repo diff --git a/README.md b/README.md new file mode 100644 index 0000000..4b869ca --- /dev/null +++ b/README.md @@ -0,0 +1,455 @@ +

+ Provisioning Logo +

+

+ Provisioning +

+ +--- + +# Core Engine + +The **Core Engine** is the foundation of the Provisioning platform—a modular, high-performance infrastructure automation system built on **Nushell** and **KCL**. It provides unified CLI tools, core libraries, and extensible architecture for managing cloud infrastructure, Kubernetes clusters, and infrastructure-as-code workflows. + +## Overview + +The Core Engine serves as the central orchestration layer, providing: + +- **Unified CLI Interface** - Single command-line interface for all infrastructure operations +- **Core Libraries** - Reusable Nushell modules for configuration, validation, deployment, and workflow management +- **Provider Abstraction** - Cloud-agnostic interface supporting UpCloud, AWS, and local providers +- **Workflow Orchestration** - Batch operations, dependency resolution, and state management +- **Configuration System** - Hierarchical, config-driven architecture with 476+ configuration accessors + +## Project Structure + +``` +provisioning/core/ +├── cli/ # Command-line interface +│ └── provisioning # Main CLI entry point (211 lines, 84% reduction) +├── nulib/ # Core Nushell libraries +│ ├── lib_provisioning/ # Core provisioning libraries +│ │ ├── config/ # Configuration loading and management +│ │ ├── utils/ # Utility functions (SSH, validation, logging) +│ │ ├── providers/ # Provider abstraction layer +│ │ ├── secrets/ # Secrets management (SOPS integration) +│ │ ├── workspace/ # Workspace management +│ │ └── infra_validator/ # Infrastructure validation engine +│ ├── main_provisioning/ # CLI command handlers +│ │ ├── flags.nu # Centralized flag handling +│ │ ├── dispatcher.nu # Command routing (80+ shortcuts) +│ │ ├── help_system.nu # Categorized help system +│ │ └── commands/ # Domain-focused command modules +│ ├── servers/ # Server management modules +│ ├── taskservs/ # Task service modules +│ ├── clusters/ # Cluster management modules +│ └── workflows/ # Workflow orchestration modules +├── scripts/ # Utility scripts +│ └── test/ # Test automation +└── resources/ # Images and logos +``` + +## Installation + +### Prerequisites + +- **Nushell 0.107.1+** - Primary shell and scripting environment +- **KCL 0.11.2+** - Configuration language for infrastructure definitions +- **SOPS 3.10.2+** - Secrets management (optional but recommended) +- **Age 1.2.1+** - Encryption tool for secrets (optional) + +### Adding to PATH + +To use the CLI globally, add it to your PATH: + +```bash +# Create symbolic link +ln -sf "$(pwd)/provisioning/core/cli/provisioning" /usr/local/bin/provisioning + +# Or add to PATH in your shell config (~/.bashrc, ~/.zshrc, etc.) +export PATH="$PATH:/path/to/project-provisioning/provisioning/core/cli" +``` + +Verify installation: + +```bash +provisioning version +provisioning help +``` + +## Quick Start + +### Basic Commands + +```bash +# Show help and available commands +provisioning help + +# Show environment and configuration +provisioning env +provisioning allenv + +# Validate configuration +provisioning validate config + +# List available providers +provisioning providers + +# Show system information +provisioning nuinfo +``` + +### Infrastructure Operations + +```bash +# Create new infrastructure workspace +provisioning workspace init my-project + +# Create servers +provisioning server create --check + +# Install task services (infrastructure components) +provisioning taskserv create kubernetes + +# Create complete cluster +provisioning cluster create my-cluster + +# SSH into server +provisioning server ssh hostname-01 +``` + +### Quick Reference + +For fastest command reference: + +```bash +provisioning sc +``` + +For complete guides: + +```bash +provisioning guide from-scratch # Complete deployment guide +provisioning guide quickstart # Command shortcuts reference +provisioning guide customize # Customization patterns +``` + +## Core Libraries + +### Configuration System (`lib_provisioning/config/`) + +Hierarchical configuration loading with 476+ config accessors: + +- **Defaults** → **User** → **Project** → **Infrastructure** → **Environment** → **Runtime** +- Replaced 200+ ENV variables with config-driven architecture +- Variable interpolation: `{{paths.base}}`, `{{env.HOME}}`, `{{now.date}}` + +```nushell +use lib_provisioning/config + +# Get configuration value +let value = config get "servers.default_plan" + +# Load workspace config +let ws_config = config load-workspace "my-project" +``` + +### Provider Abstraction (`lib_provisioning/providers/`) + +Unified interface for cloud providers: + +```nushell +use lib_provisioning/providers + +# Get provider interface +let provider = providers get "upcloud" + +# Create server using provider +$provider | invoke "create_server" $server_config +``` + +### Utilities (`lib_provisioning/utils/`) + +Common utility functions: + +- **SSH Operations** - `utils/ssh.nu` +- **Validation** - `utils/validation.nu` +- **Logging** - `utils/logging.nu` +- **Error Handling** - `utils/error.nu` +- **File Operations** - `utils/files.nu` +- **Version Management** - `utils/version_manager.nu` + +### Workflow Orchestration (`workflows/`) + +Batch operations with dependency resolution: + +```bash +# Submit batch workflow +provisioning batch submit workflows/example.k + +# Monitor workflow progress +provisioning batch monitor + +# List all workflows +provisioning workflow list + +# Get workflow status +provisioning workflow status +``` + +## CLI Architecture + +### Modular Design + +The CLI uses a domain-driven architecture: + +- **Infrastructure** - Servers, task services, clusters +- **Orchestration** - Workflows, batch operations, orchestrator management +- **Development** - Modules, layers, versioning, packaging +- **Workspace** - Workspace management, templates +- **Configuration** - Settings, environment, validation +- **Utilities** - SSH, SOPS, cache, plugins + +### Command Shortcuts + +80+ shortcuts for improved productivity: + +| Full Command | Shortcuts | Description | +|--------------|-----------|-------------| +| `server` | `s` | Server operations | +| `taskserv` | `t`, `task` | Task service operations | +| `cluster` | `cl` | Cluster operations | +| `workspace` | `ws` | Workspace management | +| `workflow` | `wf`, `flow` | Workflow operations | +| `batch` | `bat` | Batch operations | +| `module` | `mod` | Module management | +| `generate` | `g`, `gen` | Code generation | +| `validate` | `val` | Validation operations | + +See complete reference: `provisioning sc` or `provisioning guide quickstart` + +### Bi-directional Help + +Help works in both directions: + +```bash +provisioning help workspace # ✅ +provisioning workspace help # ✅ Same result +provisioning ws help # ✅ Shortcut also works +``` + +## Configuration + +### Configuration Files + +With the new structure (2025-09-26): + +- **System Defaults**: `provisioning/config/config.defaults.toml` +- **User Config**: `workspace/config/local-overrides.toml` +- **Environment Configs**: `workspace/config/{dev,test,prod}-defaults.toml` +- **Infrastructure Configs**: `workspace/infra/{name}/config.toml` + +### Environment Variables + +Legacy ENV variables still supported during transition: + +```bash +# Show current environment +provisioning env + +# Show all configuration and environment +provisioning allenv + +# Use specific environment +PROVISIONING_ENV=prod provisioning server list +``` + +### Debug Flags + +```bash +# Enable debug mode +provisioning --debug + +# Check mode (dry-run, no changes) +provisioning --check server create + +# Auto-confirm operations +provisioning --yes cluster delete + +# Specify infrastructure +provisioning --infra my-project server list +``` + +## Design Principles + +### Modularity + +Clean separation between libraries, CLI, and extensions: + +- **Core Engine** (`core/`) - CLI, libraries, scripts +- **Extensions** (`extensions/`) - Providers, task services, clusters +- **Platform Services** (`platform/`) - Orchestrator, control center, API gateway + +### Extensibility + +Plugin architecture for extending functionality: + +- Provider plugins for cloud integrations +- Task service definitions for infrastructure components +- Cluster templates for complete deployments +- Workflow templates for automation + +### Consistency + +Unified API patterns across all components: + +- Standardized command structure +- Common flag handling +- Consistent error messages +- Predictable output formats + +### Performance + +Optimized for large-scale infrastructure operations: + +- Token-optimized agents (85-90% efficiency) +- Batch operations with parallel execution +- Checkpoint-based recovery +- Incremental state management + +## Migration Strategy + +The project follows a three-phase migration: + +1. **Phase 1** (Complete) - Created symbolic links to existing implementations +2. **Phase 2** (In Progress) - Gradual migration of components to new structure +3. **Phase 3** (Planned) - Consolidate and optimize unified core engine + +## Dependencies + +### Required + +- **Nushell 0.107.1+** - Shell and scripting language +- **KCL 0.11.2+** - Configuration language + +### Recommended + +- **SOPS 3.10.2+** - Secrets management +- **Age 1.2.1+** - Encryption for secrets +- **K9s 0.50.6+** - Kubernetes management interface + +### Optional + +- **nu_plugin_tera** - Template rendering +- **nu_plugin_kcl** - KCL integration (CLI `kcl` is required, plugin optional) + +## Documentation + +### User Guides + +- **Quick Start**: `provisioning guide from-scratch` +- **Command Reference**: `provisioning sc` +- **Update Guide**: `provisioning guide update` +- **Customization**: `provisioning guide customize` + +### Architecture Documentation + +- **CLI Architecture**: `docs/architecture/ADR-006-provisioning-cli-refactoring.md` +- **Configuration System**: See `.claude/features/configuration-system.md` +- **Batch Workflows**: See `.claude/features/batch-workflow-system.md` +- **Orchestrator**: See `.claude/features/orchestrator-architecture.md` + +### API Documentation + +- **REST API**: See `docs/api/` (when orchestrator is running) +- **Nushell Modules**: See inline documentation in `nulib/` modules + +## Testing + +### Run Tests + +```bash +# Run all tests +nu provisioning/core/scripts/test/test_all.nu + +# Run specific test group +nu provisioning/core/scripts/test/test_config.nu +nu provisioning/core/scripts/test/test_cli.nu +``` + +### Test Coverage + +The core engine includes comprehensive test coverage: + +- Configuration loading and validation +- Provider interface contracts +- CLI command routing +- Workflow orchestration +- Error handling + +## Contributing + +When contributing to the Core Engine: + +1. **Follow Nushell Patterns** - See `.claude/best_nushell_code.md` +2. **Use Modular Design** - Domain-focused modules in appropriate directories +3. **Add Tests** - Include tests for new functionality +4. **Update Documentation** - Keep README and inline docs current +5. **Follow PAP** - Project Architecture Principles (see `CLAUDE.md`) + +## Troubleshooting + +### Common Issues + +**Missing environment variables:** +```bash +provisioning env # Check current configuration +provisioning validate config # Validate configuration files +``` + +**KCL compilation errors:** +```bash +kcl fmt .k # Format KCL file +kcl run .k # Test KCL file +``` + +**Provider authentication:** +```bash +provisioning providers # List available providers +provisioning show settings # View provider configuration +``` + +### Debug Mode + +Enable verbose logging: + +```bash +provisioning --debug +``` + +### Getting Help + +```bash +provisioning help # Show main help +provisioning help # Category-specific help +provisioning help # Command-specific help +provisioning guide list # List all guides +``` + +## Version Information + +Check system versions: + +```bash +provisioning version # Show all versions +provisioning nuinfo # Nushell information +``` + +## License + +See project root LICENSE file. + +--- + +**Maintained By**: Architecture Team +**Last Updated**: 2025-10-07