894046ef5a
- DAG architecture: `dag show/validate/export` (nulib/main_provisioning/dag.nu),
config loader (lib_provisioning/config/loader/dag.nu), taskserv dag-executor.
Backed by schemas/lib/dag/*.ncl; orchestrator emits NATS events via
WorkspaceComposition::into_workflow. See ADR-020, ADR-021.
- Unified Component Architecture: components/mod.nu, main_provisioning/
{components,workflow,extensions,ontoref-queries}.nu. Full workflow engine with
topological sort and NATS subject emission. Blocks A-H complete (libre-daoshi).
- Commands-registry: nulib/commands-registry.ncl (Nickel source, 314 lines) +
JSON cache at ~/.cache/provisioning/commands-registry.json rebuilt on source
change. cli/provisioning fast-path alias expansion avoids cold Nu startup.
ADDING_COMMANDS.md documents new-command workflow.
- Platform service manager: service-manager.nu (+573), startup.nu (+611),
service-check.nu (+255); autostart/bootstrap/health/target refactored.
- Nushell 0.112.2 migration: removed all try/catch and bash redirections;
external commands prefixed with ^; type signatures enforced. Driven by
scripts/refactor-try-catch{,-simplified}.nu.
- TTY stack: removed shlib/*-tty.sh; replaced by cli/tty-dispatch.sh,
tty-filter.sh, tty-commands.conf.
- New domain modules: images/ (golden image lifecycle), workspace/{state,sync}.nu,
main_provisioning/{bootstrap,cluster-deploy,fip,state}.nu, commands/{state,
build,integrations/auth,utilities/alias}.nu, platform.nu expanded (+874).
- Config loader overhaul: loader/core.nu slimmed (-759), cache/core.nu
refactored (-454), removed legacy loaders/file_loader.nu (-330).
- Thirteen new provisioning-<domain>.nu top-level modules for bash dispatcher.
- Tests: test_workspace_state.nu (+351); updates to test_oci_registry,
test_services.
- README + CHANGELOG updated.
566 lines
17 KiB
Markdown
566 lines
17 KiB
Markdown
<p align="center">
|
||
<img src="https://repo.jesusperez.pro/jesus/provisioning/media/branch/main/resources/provisioning_logo.svg" alt="Provisioning Logo" width="300"/>
|
||
</p>
|
||
<p align="center">
|
||
<img src="https://repo.jesusperez.pro/jesus/provisioning/media/branch/main/resources/logo-text.svg" alt="Provisioning" width="500"/>
|
||
</p>
|
||
|
||
---
|
||
|
||
# Core Engine
|
||
|
||
The **Core Engine** is the foundational component of the [Provisioning project](https://repo.jesusperez.pro/jesus/provisioning),
|
||
providing the unified CLI interface, core Nushell libraries, and essential utility scripts.
|
||
Built on **Nushell** and **Nickel**, it serves as the primary entry point for all infrastructure operations.
|
||
|
||
## Overview
|
||
|
||
The Core Engine provides:
|
||
|
||
- **Unified CLI Interface** - Single command-line tool for all infrastructure operations
|
||
- **Core Libraries** - Reusable Nushell modules for configuration, validation, and utilities
|
||
- **Provider Abstraction** - Cloud-agnostic interface supporting UpCloud, AWS, and local providers
|
||
- **Workflow Integration** - Commands for submitting and managing workflows (executed by the orchestrator)
|
||
- **Configuration System** - Hierarchical, config-driven architecture with 476+ configuration accessors
|
||
|
||
## Project Structure
|
||
|
||
```text
|
||
provisioning/core/
|
||
├── cli/ # Command-line interface
|
||
│ ├── provisioning # Main bash wrapper (command-registry cache aware)
|
||
│ ├── tty-dispatch.sh # TTY-safe dispatcher (replaces shlib)
|
||
│ ├── tty-filter.sh # TTY command filter
|
||
│ └── tty-commands.conf # TTY command manifest
|
||
├── nulib/ # Core Nushell libraries
|
||
│ ├── commands-registry.ncl # Command catalog (Nickel → JSON cache)
|
||
│ ├── lib_provisioning/ # Core provisioning libraries
|
||
│ │ ├── config/ # Hierarchical loader, cache, DAG loader
|
||
│ │ ├── platform/ # Service manager, startup, bootstrap, health
|
||
│ │ ├── utils/ # SSH, logging, nickel_processor, path-utils
|
||
│ │ ├── plugins/ # auth, kms, orchestrator, secretumvault
|
||
│ │ ├── providers/ # Provider registry and loader
|
||
│ │ ├── workspace/ # Workspace config, verification, enforcement
|
||
│ │ └── infra_validator/ # Schema-aware validation engine
|
||
│ ├── main_provisioning/ # CLI command handlers
|
||
│ │ ├── dispatcher.nu # Command routing (80+ shortcuts)
|
||
│ │ ├── dag.nu # `dag show/validate/export`
|
||
│ │ ├── components.nu # Components + capabilities queries
|
||
│ │ ├── workflow.nu # Workflow engine (topo sort, NATS events)
|
||
│ │ ├── bootstrap.nu # Platform bootstrap
|
||
│ │ ├── cluster-deploy.nu # Component/taskserv dispatch
|
||
│ │ ├── ontoref-queries.nu # on+re-aware CLI queries
|
||
│ │ └── commands/ # Domain-focused command modules
|
||
│ ├── components/ # Component dispatch module (NEW)
|
||
│ ├── images/ # Golden image lifecycle (create/list/update/watch)
|
||
│ ├── servers/ # Server management modules
|
||
│ ├── taskservs/ # Task service modules (+ dag-executor)
|
||
│ ├── clusters/ # Cluster management modules
|
||
│ ├── workflows/ # Workflow orchestration modules
|
||
│ ├── workspace/ # Workspace state + sync
|
||
│ └── scripts/ # In-repo nushell scripts (query-*, validate-*)
|
||
├── scripts/ # Utility scripts (refactor, deploy, manage-ports)
|
||
└── services/ # Service definitions
|
||
```
|
||
|
||
## Installation
|
||
|
||
### Prerequisites
|
||
|
||
- **Nushell 0.112.2** - Primary shell and scripting environment
|
||
- **Nickel 1.15.1+** - Configuration language for infrastructure definitions
|
||
- **SOPS 3.10.2+** - Secrets management (optional but recommended)
|
||
- **Age 1.2.1+** - Encryption tool for secrets (optional)
|
||
|
||
### Adding to PATH
|
||
|
||
Recommended installation uses a symlink plus the `prvng` shell alias:
|
||
|
||
```bash
|
||
# Symlink the bash wrapper into ~/.local/bin
|
||
ln -sf "$(pwd)/provisioning/core/cli/provisioning" "$HOME/.local/bin/provisioning"
|
||
|
||
# Optional shell alias (add to ~/.bashrc / ~/.zshrc)
|
||
alias prvng='provisioning'
|
||
```
|
||
|
||
Verify installation:
|
||
|
||
```text
|
||
provisioning version
|
||
provisioning help
|
||
prvng s list # alias + single-char shortcut
|
||
```
|
||
|
||
## Quick Start
|
||
|
||
### Basic Commands
|
||
|
||
```bash
|
||
# Show help and available commands
|
||
provisioning help
|
||
|
||
# Show environment and configuration
|
||
provisioning env
|
||
provisioning allenv
|
||
|
||
# Validate configuration
|
||
provisioning validate config
|
||
|
||
# List available providers
|
||
provisioning providers
|
||
|
||
# Show system information
|
||
provisioning nuinfo
|
||
```
|
||
|
||
### Infrastructure Operations
|
||
|
||
```bash
|
||
# Create new infrastructure workspace
|
||
provisioning workspace init my-project
|
||
|
||
# Create servers
|
||
provisioning server create --check
|
||
|
||
# Install task services (infrastructure components)
|
||
provisioning taskserv create kubernetes
|
||
|
||
# Create complete cluster
|
||
provisioning cluster create my-cluster
|
||
|
||
# SSH into server
|
||
provisioning server ssh hostname-01
|
||
```
|
||
|
||
### DAG, Components & Workflows
|
||
|
||
```bash
|
||
# Inspect workspace DAG composition (nodes, edges, health gates)
|
||
provisioning dag show --infra wuji
|
||
provisioning dag validate --infra wuji
|
||
provisioning dag export --infra wuji --format dot
|
||
|
||
# Components and extension capabilities
|
||
provisioning component list
|
||
provisioning component info postgresql
|
||
provisioning extensions capabilities
|
||
provisioning extensions graph
|
||
|
||
# Workflows (topological scheduling + NATS events)
|
||
provisioning workflow list
|
||
provisioning workflow run deploy-services --infra libre-daoshi
|
||
provisioning workflow status <id>
|
||
```
|
||
|
||
### Command Registry & Fast Path
|
||
|
||
Every `prvng`/`provisioning` invocation validates the command against a JSON cache
|
||
rebuilt from `nulib/commands-registry.ncl` whenever the source is newer. Single-char
|
||
aliases (`s`, `t`, `c`, `e`, `w`, `j`, `b`, `o`, `a`) are expanded in the bash wrapper
|
||
before dispatch. Adding a new top-level command requires a registry entry **plus** a
|
||
dispatch case in `cli/provisioning` — see `nulib/main_provisioning/ADDING_COMMANDS.md`.
|
||
|
||
### Quick Reference
|
||
|
||
For fastest command reference:
|
||
|
||
```text
|
||
provisioning sc
|
||
```
|
||
|
||
For complete guides:
|
||
|
||
```text
|
||
provisioning guide from-scratch # Complete deployment guide
|
||
provisioning guide quickstart # Command shortcuts reference
|
||
provisioning guide customize # Customization patterns
|
||
```
|
||
|
||
## Core Libraries
|
||
|
||
### Configuration System (`lib_provisioning/config/`)
|
||
|
||
Hierarchical configuration loading with 476+ config accessors:
|
||
|
||
- **Defaults** → **User** → **Project** → **Infrastructure** → **Environment** → **Runtime**
|
||
- Replaced 200+ ENV variables with config-driven architecture
|
||
- Variable interpolation: `{{paths.base}}`, `{{env.HOME}}`, `{{now.date}}`
|
||
|
||
```nushell
|
||
use lib_provisioning/config
|
||
|
||
# Get configuration value
|
||
let value = config get "servers.default_plan"
|
||
|
||
# Load workspace config
|
||
let ws_config = config load-workspace "my-project"
|
||
```
|
||
|
||
### Provider Abstraction (`lib_provisioning/providers/`)
|
||
|
||
Unified interface for cloud providers:
|
||
|
||
```nushell
|
||
use lib_provisioning/providers
|
||
|
||
# Get provider interface
|
||
let provider = providers get "upcloud"
|
||
|
||
# Create server using provider
|
||
$provider | invoke "create_server" $server_config
|
||
```
|
||
|
||
### Utilities (`lib_provisioning/utils/`)
|
||
|
||
Common utility functions:
|
||
|
||
- **SSH Operations** - `utils/ssh.nu`
|
||
- **Validation** - `utils/validation.nu`
|
||
- **Logging** - `utils/logging.nu`
|
||
- **Error Handling** - `utils/error.nu`
|
||
- **File Operations** - `utils/files.nu`
|
||
- **Version Management** - `utils/version_manager.nu`
|
||
|
||
### Workflow Orchestration (`workflows/`)
|
||
|
||
Batch operations with dependency resolution:
|
||
|
||
```bash
|
||
# Submit batch workflow
|
||
provisioning batch submit workflows/example.ncl
|
||
|
||
# Monitor workflow progress
|
||
provisioning batch monitor <workflow-id>
|
||
|
||
# List all workflows
|
||
provisioning workflow list
|
||
|
||
# Get workflow status
|
||
provisioning workflow status <id>
|
||
```
|
||
|
||
## Internationalization (i18n)
|
||
|
||
### Fluent-based Localization
|
||
|
||
The help system supports multiple languages using the Fluent catalog format:
|
||
|
||
```bash
|
||
# Automatic locale detection from LANG environment variable
|
||
export LANG=es_ES.UTF-8
|
||
provisioning help # Shows Spanish help if es-ES catalog exists
|
||
|
||
# Falls back to en-US if translation not available
|
||
export LANG=fr_FR.UTF-8
|
||
provisioning help # Shows French help if fr-FR exists, otherwise English
|
||
```
|
||
|
||
**Catalog Structure**:
|
||
|
||
```text
|
||
provisioning/locales/
|
||
├── en-US/
|
||
│ └── help.ftl # English help strings
|
||
├── es-ES/
|
||
│ └── help.ftl # Spanish help strings
|
||
└── de-DE/
|
||
└── help.ftl # German help strings
|
||
```
|
||
|
||
**Supported Locales**: en-US (default), with framework ready for es-ES, fr-FR, de-DE, etc.
|
||
|
||
---
|
||
|
||
## CLI Architecture
|
||
|
||
### Modular Design
|
||
|
||
The CLI uses a domain-driven architecture:
|
||
|
||
- **Infrastructure** - Servers, task services, clusters
|
||
- **Orchestration** - Workflows, batch operations, orchestrator management
|
||
- **Development** - Modules, layers, versioning, packaging
|
||
- **Workspace** - Workspace management, templates
|
||
- **Configuration** - Settings, environment, validation
|
||
- **Utilities** - SSH, SOPS, cache, plugins
|
||
|
||
### Command Shortcuts
|
||
|
||
80+ shortcuts for improved productivity:
|
||
|
||
| Full Command | Shortcuts | Description |
|
||
| ------------ | --------- | ----------- |
|
||
| `server` | `s` | Server operations |
|
||
| `taskserv` | `t`, `task` | Task service operations |
|
||
| `cluster` | `cl` | Cluster operations |
|
||
| `workspace` | `ws` | Workspace management |
|
||
| `workflow` | `wf`, `flow` | Workflow operations |
|
||
| `batch` | `bat` | Batch operations |
|
||
| `module` | `mod` | Module management |
|
||
| `generate` | `g`, `gen` | Code generation |
|
||
| `validate` | `val` | Validation operations |
|
||
|
||
See complete reference: `provisioning sc` or `provisioning guide quickstart`
|
||
|
||
### Bi-directional Help
|
||
|
||
Help works in both directions:
|
||
|
||
```text
|
||
provisioning help workspace # ✅
|
||
provisioning workspace help # ✅ Same result
|
||
provisioning ws help # ✅ Shortcut also works
|
||
```
|
||
|
||
## Configuration
|
||
|
||
### Configuration Files
|
||
|
||
With the new structure (2025-09-26):
|
||
|
||
- **System Defaults**: `provisioning/config/config.defaults.toml`
|
||
- **User Config**: `workspace/config/local-overrides.toml`
|
||
- **Environment Configs**: `workspace/config/{dev,test,prod}-defaults.toml`
|
||
- **Infrastructure Configs**: `workspace/infra/{name}/config.toml`
|
||
|
||
### Environment Variables
|
||
|
||
Legacy ENV variables still supported during transition:
|
||
|
||
```bash
|
||
# Show current environment
|
||
provisioning env
|
||
|
||
# Show all configuration and environment
|
||
provisioning allenv
|
||
|
||
# Use specific environment
|
||
PROVISIONING_ENV=prod provisioning server list
|
||
```
|
||
|
||
### Debug Flags
|
||
|
||
```bash
|
||
# Enable debug mode
|
||
provisioning --debug <command>
|
||
|
||
# Check mode (dry-run, no changes)
|
||
provisioning --check server create
|
||
|
||
# Auto-confirm operations
|
||
provisioning --yes cluster delete
|
||
|
||
# Specify infrastructure
|
||
provisioning --infra my-project server list
|
||
```
|
||
|
||
## Design Principles
|
||
|
||
### Modularity
|
||
|
||
Clean separation between libraries, CLI, and extensions:
|
||
|
||
- **Core Engine** (`core/`) - CLI, libraries, scripts
|
||
- **Extensions** (`extensions/`) - Providers, task services, clusters
|
||
- **Platform Services** (`platform/`) - Orchestrator, control center, API gateway
|
||
|
||
### Extensibility
|
||
|
||
Plugin architecture for extending functionality:
|
||
|
||
- Provider plugins for cloud integrations
|
||
- Task service definitions for infrastructure components
|
||
- Cluster templates for complete deployments
|
||
- Workflow templates for automation
|
||
|
||
### Consistency
|
||
|
||
Unified API patterns across all components:
|
||
|
||
- Standardized command structure
|
||
- Common flag handling
|
||
- Consistent error messages
|
||
- Predictable output formats
|
||
|
||
### Performance
|
||
|
||
Optimized for large-scale infrastructure operations:
|
||
|
||
- Token-optimized agents (85-90% efficiency)
|
||
- Batch operations with parallel execution
|
||
- Checkpoint-based recovery
|
||
- Incremental state management
|
||
|
||
## Migration Strategy
|
||
|
||
The project follows a three-phase migration:
|
||
|
||
1. **Phase 1** (Complete) - Created symbolic links to existing implementations
|
||
2. **Phase 2** (In Progress) - Gradual migration of components to new structure
|
||
3. **Phase 3** (Planned) - Consolidate and optimize unified core engine
|
||
|
||
## Dependencies
|
||
|
||
### Required
|
||
|
||
- **Nushell 0.112.2** - Shell and scripting language
|
||
- **Nickel 1.15.1+** - Configuration language
|
||
|
||
### Recommended
|
||
|
||
- **SOPS 3.10.2+** - Secrets management
|
||
- **Age 1.2.1+** - Encryption for secrets
|
||
- **K9s 0.50.6+** - Kubernetes management interface
|
||
|
||
### Optional
|
||
|
||
- **nu_plugin_tera** - Template rendering
|
||
- **Nickel Language** - Native Nickel support via CLI (no plugin required)
|
||
|
||
## Documentation
|
||
|
||
### User Guides
|
||
|
||
- **Quick Start**: `provisioning guide from-scratch`
|
||
- **Command Reference**: `provisioning sc`
|
||
- **Update Guide**: `provisioning guide update`
|
||
- **Customization**: `provisioning guide customize`
|
||
|
||
### Architecture Documentation
|
||
|
||
- **CLI Architecture**: `../docs/src/architecture/adr/ADR-006-provisioning-cli-refactoring.md`
|
||
- **Configuration System**: `../docs/src/infrastructure/configuration-system.md`
|
||
- **Batch Workflows**: `../docs/src/infrastructure/batch-workflow-system.md`
|
||
- **Orchestrator**: `../docs/src/operations/orchestrator-system.md`
|
||
|
||
### API Documentation
|
||
|
||
- **REST API**: See `../docs/src/api-reference/` (when orchestrator is running)
|
||
- **Nushell Modules**: See inline documentation in `nulib/` modules
|
||
|
||
## Testing
|
||
|
||
### Run Tests
|
||
|
||
```bash
|
||
# Run all tests
|
||
nu provisioning/core/scripts/test/test_all.nu
|
||
|
||
# Run specific test group
|
||
nu provisioning/core/scripts/test/test_config.nu
|
||
nu provisioning/core/scripts/test/test_cli.nu
|
||
```
|
||
|
||
### Test Coverage
|
||
|
||
The core engine includes comprehensive test coverage:
|
||
|
||
- Configuration loading and validation
|
||
- Provider interface contracts
|
||
- CLI command routing
|
||
- Workflow orchestration
|
||
- Error handling
|
||
|
||
## Contributing
|
||
|
||
When contributing to the Core Engine:
|
||
|
||
1. **Follow Nushell Patterns** - See `.claude/best_nushell_code.md`
|
||
2. **Use Modular Design** - Domain-focused modules in appropriate directories
|
||
3. **Add Tests** - Include tests for new functionality
|
||
4. **Update Documentation** - Keep README and inline docs current
|
||
5. **Follow PAP** - Project Architecture Principles (see `CLAUDE.md`)
|
||
|
||
## Troubleshooting
|
||
|
||
### Common Issues
|
||
|
||
**Missing environment variables:**
|
||
|
||
```text
|
||
provisioning env # Check current configuration
|
||
provisioning validate config # Validate configuration files
|
||
```
|
||
|
||
**Nickel schema errors:**
|
||
|
||
```text
|
||
nickel fmt <file>.ncl # Format Nickel file
|
||
nickel eval <file>.ncl # Evaluate Nickel schema
|
||
nickel typecheck <file>.ncl # Type check schema
|
||
```
|
||
|
||
**Provider authentication:**
|
||
|
||
```text
|
||
provisioning providers # List available providers
|
||
provisioning show settings # View provider configuration
|
||
```
|
||
|
||
### Debug Mode
|
||
|
||
Enable verbose logging:
|
||
|
||
```text
|
||
provisioning --debug <command>
|
||
```
|
||
|
||
### Getting Help
|
||
|
||
```text
|
||
provisioning help # Show main help
|
||
provisioning help <category> # Category-specific help
|
||
provisioning <command> help # Command-specific help
|
||
provisioning guide list # List all guides
|
||
```
|
||
|
||
## Version Information
|
||
|
||
Check system versions:
|
||
|
||
```text
|
||
provisioning version # Show all versions
|
||
provisioning nuinfo # Nushell information
|
||
```
|
||
|
||
## License
|
||
|
||
See project root LICENSE file.
|
||
|
||
---
|
||
|
||
## Recent Updates
|
||
|
||
### 2026-04-17 - DAG architecture, commands-registry, Nushell 0.110/0.112 refactor
|
||
|
||
- **Unified Component Architecture**: `components/`, `workflow.nu`, and `components.nu`
|
||
implement the libre-daoshi unified model (ComponentDef, WorkflowDef, capabilities).
|
||
See `memory/unified_component_arch.md` and ADRs 020/021.
|
||
- **Three-layer DAG**: `dag.nu` + `lib_provisioning/config/loader/dag.nu` add
|
||
`dag show/validate/export` backed by `schemas/lib/dag/*.ncl`; orchestrator emits
|
||
NATS events via `WorkspaceComposition::into_workflow`.
|
||
- **Commands-registry cache**: `nulib/commands-registry.ncl` feeds the bash wrapper's
|
||
`_validate_command`; fast-path single-char alias expansion avoids cold Nu startup.
|
||
- **Platform service manager**: new `platform/service-manager.nu`, `platform/startup.nu`,
|
||
and `bootstrap.nu` consolidate autostart, health checks, and lifecycle.
|
||
- **Nushell 0.112.2 compliance**: `scripts/refactor-try-catch*.nu` drove the
|
||
migration — no `try/catch`, no bash redirections, all external commands prefixed.
|
||
- **TTY stack**: `shlib/*-tty.sh` removed; replaced by `cli/tty-dispatch.sh`,
|
||
`cli/tty-filter.sh`, and `cli/tty-commands.conf`.
|
||
- **New domain modules**: `images/` (golden image lifecycle), `workspace/state.nu` +
|
||
`workspace/sync.nu`, `main_provisioning/ontoref-queries.nu`, `main_provisioning/fip.nu`,
|
||
`main_provisioning/state.nu`, `main_provisioning/extensions.nu`.
|
||
- **Config loader overhaul**: `loader/core.nu` slimmed (−759 lines of legacy paths),
|
||
`cache/core.nu` refactored, `loaders/file_loader.nu` removed.
|
||
|
||
### 2026-01-14 - Terminology Migration & i18n
|
||
|
||
- **Cluster → Taskserv**: Complete refactoring across nulib/ modules
|
||
- **Fluent i18n System**: Automatic locale detection with en-US fallback
|
||
- Enhanced ANSI output formatting for improved CLI readability
|
||
|
||
---
|
||
|
||
**Maintained By**: Core Team
|
||
**Last Updated**: 2026-04-17
|