From 8affca20aaf22f5fdafe3f4c29af684f5cfe13fd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jesu=CC=81s=20Pe=CC=81rez?= Date: Thu, 9 Jul 2026 20:53:34 +0100 Subject: [PATCH] init repo --- README.md | 566 ++++++++++++++++++++++++++++++++ resources/logo-text.svg | 149 +++++++++ resources/provisioning_logo.svg | 161 +++++++++ 3 files changed, 876 insertions(+) create mode 100644 README.md create mode 100644 resources/logo-text.svg create mode 100644 resources/provisioning_logo.svg diff --git a/README.md b/README.md new file mode 100644 index 0000000..4a62ac8 --- /dev/null +++ b/README.md @@ -0,0 +1,566 @@ +

+ Provisioning Logo +

+

+ Provisioning +

+ +--- + +# Core Engine + +The **Core Engine** is the foundational component of the [Provisioning project](https://repo.jesusperez.pro/provisioning/provisioning), +providing the unified CLI interface, core Nushell libraries, and essential utility scripts. +Built on **Nushell** and **Nickel**, it serves as the primary entry point for all infrastructure operations. + +## Overview + +The Core Engine provides: + +- **Unified CLI Interface** - Single command-line tool for all infrastructure operations +- **Core Libraries** - Reusable Nushell modules for configuration, validation, and utilities +- **Provider Abstraction** - Cloud-agnostic interface supporting UpCloud, AWS, and local providers +- **Workflow Integration** - Commands for submitting and managing workflows (executed by the orchestrator) +- **Configuration System** - Hierarchical, config-driven architecture with 476+ configuration accessors + +## Project Structure + +```text +provisioning/core/ +├── cli/ # Command-line interface +│ ├── provisioning # Main bash wrapper (command-registry cache aware) +│ ├── tty-dispatch.sh # TTY-safe dispatcher (replaces shlib) +│ ├── tty-filter.sh # TTY command filter +│ └── tty-commands.conf # TTY command manifest +├── nulib/ # Core Nushell libraries +│ ├── commands-registry.ncl # Command catalog (Nickel → JSON cache) +│ ├── lib_provisioning/ # Core provisioning libraries +│ │ ├── config/ # Hierarchical loader, cache, DAG loader +│ │ ├── platform/ # Service manager, startup, bootstrap, health +│ │ ├── utils/ # SSH, logging, nickel_processor, path-utils +│ │ ├── plugins/ # auth, kms, orchestrator, secretumvault +│ │ ├── providers/ # Provider registry and loader +│ │ ├── workspace/ # Workspace config, verification, enforcement +│ │ └── infra_validator/ # Schema-aware validation engine +│ ├── main_provisioning/ # CLI command handlers +│ │ ├── dispatcher.nu # Command routing (80+ shortcuts) +│ │ ├── dag.nu # `dag show/validate/export` +│ │ ├── components.nu # Components + capabilities queries +│ │ ├── workflow.nu # Workflow engine (topo sort, NATS events) +│ │ ├── bootstrap.nu # Platform bootstrap +│ │ ├── cluster-deploy.nu # Component/taskserv dispatch +│ │ ├── ontoref-queries.nu # on+re-aware CLI queries +│ │ └── commands/ # Domain-focused command modules +│ ├── components/ # Component dispatch module (NEW) +│ ├── images/ # Golden image lifecycle (create/list/update/watch) +│ ├── servers/ # Server management modules +│ ├── taskservs/ # Task service modules (+ dag-executor) +│ ├── clusters/ # Cluster management modules +│ ├── workflows/ # Workflow orchestration modules +│ ├── workspace/ # Workspace state + sync +│ └── scripts/ # In-repo nushell scripts (query-*, validate-*) +├── scripts/ # Utility scripts (refactor, deploy, manage-ports) +└── services/ # Service definitions +``` + +## Installation + +### Prerequisites + +- **Nushell 0.112.2** - Primary shell and scripting environment +- **Nickel 1.15.1+** - Configuration language for infrastructure definitions +- **SOPS 3.10.2+** - Secrets management (optional but recommended) +- **Age 1.2.1+** - Encryption tool for secrets (optional) + +### Adding to PATH + +Recommended installation uses a symlink plus the `prvng` shell alias: + +```bash +# Symlink the bash wrapper into ~/.local/bin +ln -sf "$(pwd)/provisioning/core/cli/provisioning" "$HOME/.local/bin/provisioning" + +# Optional shell alias (add to ~/.bashrc / ~/.zshrc) +alias prvng='provisioning' +``` + +Verify installation: + +```text +provisioning version +provisioning help +prvng s list # alias + single-char shortcut +``` + +## Quick Start + +### Basic Commands + +```bash +# Show help and available commands +provisioning help + +# Show environment and configuration +provisioning env +provisioning allenv + +# Validate configuration +provisioning validate config + +# List available providers +provisioning providers + +# Show system information +provisioning nuinfo +``` + +### Infrastructure Operations + +```bash +# Create new infrastructure workspace +provisioning workspace init my-project + +# Create servers +provisioning server create --check + +# Install task services (infrastructure components) +provisioning taskserv create kubernetes + +# Create complete cluster +provisioning cluster create my-cluster + +# SSH into server +provisioning server ssh hostname-01 +``` + +### DAG, Components & Workflows + +```bash +# Inspect workspace DAG composition (nodes, edges, health gates) +provisioning dag show --infra wuji +provisioning dag validate --infra wuji +provisioning dag export --infra wuji --format dot + +# Components and extension capabilities +provisioning component list +provisioning component info postgresql +provisioning extensions capabilities +provisioning extensions graph + +# Workflows (topological scheduling + NATS events) +provisioning workflow list +provisioning workflow run deploy-services --infra libre-daoshi +provisioning workflow status +``` + +### Command Registry & Fast Path + +Every `prvng`/`provisioning` invocation validates the command against a JSON cache +rebuilt from `nulib/commands-registry.ncl` whenever the source is newer. Single-char +aliases (`s`, `t`, `c`, `e`, `w`, `j`, `b`, `o`, `a`) are expanded in the bash wrapper +before dispatch. Adding a new top-level command requires a registry entry **plus** a +dispatch case in `cli/provisioning` — see `nulib/main_provisioning/ADDING_COMMANDS.md`. + +### Quick Reference + +For fastest command reference: + +```text +provisioning sc +``` + +For complete guides: + +```text +provisioning guide from-scratch # Complete deployment guide +provisioning guide quickstart # Command shortcuts reference +provisioning guide customize # Customization patterns +``` + +## Core Libraries + +### Configuration System (`lib_provisioning/config/`) + +Hierarchical configuration loading with 476+ config accessors: + +- **Defaults** → **User** → **Project** → **Infrastructure** → **Environment** → **Runtime** +- Replaced 200+ ENV variables with config-driven architecture +- Variable interpolation: `{{paths.base}}`, `{{env.HOME}}`, `{{now.date}}` + +```nushell +use lib_provisioning/config + +# Get configuration value +let value = config get "servers.default_plan" + +# Load workspace config +let ws_config = config load-workspace "my-project" +``` + +### Provider Abstraction (`lib_provisioning/providers/`) + +Unified interface for cloud providers: + +```nushell +use lib_provisioning/providers + +# Get provider interface +let provider = providers get "upcloud" + +# Create server using provider +$provider | invoke "create_server" $server_config +``` + +### Utilities (`lib_provisioning/utils/`) + +Common utility functions: + +- **SSH Operations** - `utils/ssh.nu` +- **Validation** - `utils/validation.nu` +- **Logging** - `utils/logging.nu` +- **Error Handling** - `utils/error.nu` +- **File Operations** - `utils/files.nu` +- **Version Management** - `utils/version_manager.nu` + +### Workflow Orchestration (`workflows/`) + +Batch operations with dependency resolution: + +```bash +# Submit batch workflow +provisioning batch submit workflows/example.ncl + +# Monitor workflow progress +provisioning batch monitor + +# List all workflows +provisioning workflow list + +# Get workflow status +provisioning workflow status +``` + +## Internationalization (i18n) + +### Fluent-based Localization + +The help system supports multiple languages using the Fluent catalog format: + +```bash +# Automatic locale detection from LANG environment variable +export LANG=es_ES.UTF-8 +provisioning help # Shows Spanish help if es-ES catalog exists + +# Falls back to en-US if translation not available +export LANG=fr_FR.UTF-8 +provisioning help # Shows French help if fr-FR exists, otherwise English +``` + +**Catalog Structure**: + +```text +provisioning/locales/ +├── en-US/ +│ └── help.ftl # English help strings +├── es-ES/ +│ └── help.ftl # Spanish help strings +└── de-DE/ + └── help.ftl # German help strings +``` + +**Supported Locales**: en-US (default), with framework ready for es-ES, fr-FR, de-DE, etc. + +--- + +## CLI Architecture + +### Modular Design + +The CLI uses a domain-driven architecture: + +- **Infrastructure** - Servers, task services, clusters +- **Orchestration** - Workflows, batch operations, orchestrator management +- **Development** - Modules, layers, versioning, packaging +- **Workspace** - Workspace management, templates +- **Configuration** - Settings, environment, validation +- **Utilities** - SSH, SOPS, cache, plugins + +### Command Shortcuts + +80+ shortcuts for improved productivity: + +| Full Command | Shortcuts | Description | +| ------------ | --------- | ----------- | +| `server` | `s` | Server operations | +| `taskserv` | `t`, `task` | Task service operations | +| `cluster` | `cl` | Cluster operations | +| `workspace` | `ws` | Workspace management | +| `workflow` | `wf`, `flow` | Workflow operations | +| `batch` | `bat` | Batch operations | +| `module` | `mod` | Module management | +| `generate` | `g`, `gen` | Code generation | +| `validate` | `val` | Validation operations | + +See complete reference: `provisioning sc` or `provisioning guide quickstart` + +### Bi-directional Help + +Help works in both directions: + +```text +provisioning help workspace # ✅ +provisioning workspace help # ✅ Same result +provisioning ws help # ✅ Shortcut also works +``` + +## Configuration + +### Configuration Files + +With the new structure (2025-09-26): + +- **System Defaults**: `provisioning/config/config.defaults.toml` +- **User Config**: `workspace/config/local-overrides.toml` +- **Environment Configs**: `workspace/config/{dev,test,prod}-defaults.toml` +- **Infrastructure Configs**: `workspace/infra/{name}/config.toml` + +### Environment Variables + +Legacy ENV variables still supported during transition: + +```bash +# Show current environment +provisioning env + +# Show all configuration and environment +provisioning allenv + +# Use specific environment +PROVISIONING_ENV=prod provisioning server list +``` + +### Debug Flags + +```bash +# Enable debug mode +provisioning --debug + +# Check mode (dry-run, no changes) +provisioning --check server create + +# Auto-confirm operations +provisioning --yes cluster delete + +# Specify infrastructure +provisioning --infra my-project server list +``` + +## Design Principles + +### Modularity + +Clean separation between libraries, CLI, and extensions: + +- **Core Engine** (`core/`) - CLI, libraries, scripts +- **Extensions** (`extensions/`) - Providers, task services, clusters +- **Platform Services** (`platform/`) - Orchestrator, control center, API gateway + +### Extensibility + +Plugin architecture for extending functionality: + +- Provider plugins for cloud integrations +- Task service definitions for infrastructure components +- Cluster templates for complete deployments +- Workflow templates for automation + +### Consistency + +Unified API patterns across all components: + +- Standardized command structure +- Common flag handling +- Consistent error messages +- Predictable output formats + +### Performance + +Optimized for large-scale infrastructure operations: + +- Token-optimized agents (85-90% efficiency) +- Batch operations with parallel execution +- Checkpoint-based recovery +- Incremental state management + +## Migration Strategy + +The project follows a three-phase migration: + +1. **Phase 1** (Complete) - Created symbolic links to existing implementations +2. **Phase 2** (In Progress) - Gradual migration of components to new structure +3. **Phase 3** (Planned) - Consolidate and optimize unified core engine + +## Dependencies + +### Required + +- **Nushell 0.112.2** - Shell and scripting language +- **Nickel 1.15.1+** - Configuration language + +### Recommended + +- **SOPS 3.10.2+** - Secrets management +- **Age 1.2.1+** - Encryption for secrets +- **K9s 0.50.6+** - Kubernetes management interface + +### Optional + +- **nu_plugin_tera** - Template rendering +- **Nickel Language** - Native Nickel support via CLI (no plugin required) + +## Documentation + +### User Guides + +- **Quick Start**: `provisioning guide from-scratch` +- **Command Reference**: `provisioning sc` +- **Update Guide**: `provisioning guide update` +- **Customization**: `provisioning guide customize` + +### Architecture Documentation + +- **CLI Architecture**: `../docs/src/architecture/adr/ADR-006-provisioning-cli-refactoring.md` +- **Configuration System**: `../docs/src/infrastructure/configuration-system.md` +- **Batch Workflows**: `../docs/src/infrastructure/batch-workflow-system.md` +- **Orchestrator**: `../docs/src/operations/orchestrator-system.md` + +### API Documentation + +- **REST API**: See `../docs/src/api-reference/` (when orchestrator is running) +- **Nushell Modules**: See inline documentation in `nulib/` modules + +## Testing + +### Run Tests + +```bash +# Run all tests +nu provisioning/core/scripts/test/test_all.nu + +# Run specific test group +nu provisioning/core/scripts/test/test_config.nu +nu provisioning/core/scripts/test/test_cli.nu +``` + +### Test Coverage + +The core engine includes comprehensive test coverage: + +- Configuration loading and validation +- Provider interface contracts +- CLI command routing +- Workflow orchestration +- Error handling + +## Contributing + +When contributing to the Core Engine: + +1. **Follow Nushell Patterns** - See `.claude/best_nushell_code.md` +2. **Use Modular Design** - Domain-focused modules in appropriate directories +3. **Add Tests** - Include tests for new functionality +4. **Update Documentation** - Keep README and inline docs current +5. **Follow PAP** - Project Architecture Principles (see `CLAUDE.md`) + +## Troubleshooting + +### Common Issues + +**Missing environment variables:** + +```text +provisioning env # Check current configuration +provisioning validate config # Validate configuration files +``` + +**Nickel schema errors:** + +```text +nickel fmt .ncl # Format Nickel file +nickel eval .ncl # Evaluate Nickel schema +nickel typecheck .ncl # Type check schema +``` + +**Provider authentication:** + +```text +provisioning providers # List available providers +provisioning show settings # View provider configuration +``` + +### Debug Mode + +Enable verbose logging: + +```text +provisioning --debug +``` + +### Getting Help + +```text +provisioning help # Show main help +provisioning help # Category-specific help +provisioning help # Command-specific help +provisioning guide list # List all guides +``` + +## Version Information + +Check system versions: + +```text +provisioning version # Show all versions +provisioning nuinfo # Nushell information +``` + +## License + +See project root LICENSE file. + +--- + +## Recent Updates + +### 2026-04-17 - DAG architecture, commands-registry, Nushell 0.110/0.112 refactor + +- **Unified Component Architecture**: `components/`, `workflow.nu`, and `components.nu` + implement the libre-daoshi unified model (ComponentDef, WorkflowDef, capabilities). + See `memory/unified_component_arch.md` and ADRs 020/021. +- **Three-layer DAG**: `dag.nu` + `lib_provisioning/config/loader/dag.nu` add + `dag show/validate/export` backed by `schemas/lib/dag/*.ncl`; orchestrator emits + NATS events via `WorkspaceComposition::into_workflow`. +- **Commands-registry cache**: `nulib/commands-registry.ncl` feeds the bash wrapper's + `_validate_command`; fast-path single-char alias expansion avoids cold Nu startup. +- **Platform service manager**: new `platform/service-manager.nu`, `platform/startup.nu`, + and `bootstrap.nu` consolidate autostart, health checks, and lifecycle. +- **Nushell 0.112.2 compliance**: `scripts/refactor-try-catch*.nu` drove the + migration — no `try/catch`, no bash redirections, all external commands prefixed. +- **TTY stack**: `shlib/*-tty.sh` removed; replaced by `cli/tty-dispatch.sh`, + `cli/tty-filter.sh`, and `cli/tty-commands.conf`. +- **New domain modules**: `images/` (golden image lifecycle), `workspace/state.nu` + + `workspace/sync.nu`, `main_provisioning/ontoref-queries.nu`, `main_provisioning/fip.nu`, + `main_provisioning/state.nu`, `main_provisioning/extensions.nu`. +- **Config loader overhaul**: `loader/core.nu` slimmed (−759 lines of legacy paths), + `cache/core.nu` refactored, `loaders/file_loader.nu` removed. + +### 2026-01-14 - Terminology Migration & i18n + +- **Cluster → Taskserv**: Complete refactoring across nulib/ modules +- **Fluent i18n System**: Automatic locale detection with en-US fallback +- Enhanced ANSI output formatting for improved CLI readability + +--- + +**Maintained By**: Core Team +**Last Updated**: 2026-04-17 diff --git a/resources/logo-text.svg b/resources/logo-text.svg new file mode 100644 index 0000000..bef17d6 --- /dev/null +++ b/resources/logo-text.svg @@ -0,0 +1,149 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/resources/provisioning_logo.svg b/resources/provisioning_logo.svg new file mode 100644 index 0000000..d8320ac --- /dev/null +++ b/resources/provisioning_logo.svg @@ -0,0 +1,161 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +