prvng_core/README.md

<p align="center">
  <img src="https://repo.jesusperez.pro/jesus/provisioning/media/branch/main/resources/provisioning_logo.svg" alt="Provisioning Logo" width="300"/>
</p>
<p align="center">
  <img src="https://repo.jesusperez.pro/jesus/provisioning/media/branch/main/resources/logo-text.svg" alt="Provisioning" width="500"/>
</p>

---

# Core Engine

The **Core Engine** is the foundational component of the [Provisioning project](https://repo.jesusperez.pro/jesus/provisioning),
providing the unified CLI interface, core Nushell libraries, and essential utility scripts.
Built on **Nushell** and **Nickel**, it serves as the primary entry point for all infrastructure operations.

## Overview

The Core Engine provides:

- **Unified CLI Interface** - Single command-line tool for all infrastructure operations
- **Core Libraries** - Reusable Nushell modules for configuration, validation, and utilities
- **Provider Abstraction** - Cloud-agnostic interface supporting UpCloud, AWS, and local providers
- **Workflow Integration** - Commands for submitting and managing workflows (executed by the orchestrator)
- **Configuration System** - Hierarchical, config-driven architecture with 476+ configuration accessors

## Project Structure

```text
provisioning/core/
├── cli/                    # Command-line interface
│   ├── provisioning        # Main bash wrapper (command-registry cache aware)
│   ├── tty-dispatch.sh     # TTY-safe dispatcher (replaces shlib)
│   ├── tty-filter.sh       # TTY command filter
│   └── tty-commands.conf   # TTY command manifest
├── nulib/                  # Core Nushell libraries
│   ├── commands-registry.ncl       # Command catalog (Nickel → JSON cache)
│   ├── lib_provisioning/           # Core provisioning libraries
│   │   ├── config/                 # Hierarchical loader, cache, DAG loader
│   │   ├── platform/               # Service manager, startup, bootstrap, health
│   │   ├── utils/                  # SSH, logging, nickel_processor, path-utils
│   │   ├── plugins/                # auth, kms, orchestrator, secretumvault
│   │   ├── providers/              # Provider registry and loader
│   │   ├── workspace/              # Workspace config, verification, enforcement
│   │   └── infra_validator/        # Schema-aware validation engine
│   ├── main_provisioning/          # CLI command handlers
│   │   ├── dispatcher.nu           # Command routing (80+ shortcuts)
│   │   ├── dag.nu                  # `dag show/validate/export`
│   │   ├── components.nu           # Components + capabilities queries
│   │   ├── workflow.nu             # Workflow engine (topo sort, NATS events)
│   │   ├── bootstrap.nu            # Platform bootstrap
│   │   ├── cluster-deploy.nu       # Component/taskserv dispatch
│   │   ├── ontoref-queries.nu      # on+re-aware CLI queries
│   │   └── commands/               # Domain-focused command modules
│   ├── components/         # Component dispatch module (NEW)
│   ├── images/             # Golden image lifecycle (create/list/update/watch)
│   ├── servers/            # Server management modules
│   ├── taskservs/          # Task service modules (+ dag-executor)
│   ├── clusters/           # Cluster management modules
│   ├── workflows/          # Workflow orchestration modules
│   ├── workspace/          # Workspace state + sync
│   └── scripts/            # In-repo nushell scripts (query-*, validate-*)
├── scripts/                # Utility scripts (refactor, deploy, manage-ports)
└── services/               # Service definitions
```

## Installation

### Prerequisites

- **Nushell 0.112.2** - Primary shell and scripting environment
- **Nickel 1.15.1+** - Configuration language for infrastructure definitions
- **SOPS 3.10.2+** - Secrets management (optional but recommended)
- **Age 1.2.1+** - Encryption tool for secrets (optional)

### Adding to PATH

Recommended installation uses a symlink plus the `prvng` shell alias:

```bash
# Symlink the bash wrapper into ~/.local/bin
ln -sf "$(pwd)/provisioning/core/cli/provisioning" "$HOME/.local/bin/provisioning"

# Optional shell alias (add to ~/.bashrc / ~/.zshrc)
alias prvng='provisioning'
```

Verify installation:

```text
provisioning version
provisioning help
prvng s list       # alias + single-char shortcut
```

## Quick Start

### Basic Commands

```bash
# Show help and available commands
provisioning help

# Show environment and configuration
provisioning env
provisioning allenv

# Validate configuration
provisioning validate config

# List available providers
provisioning providers

# Show system information
provisioning nuinfo
```

### Infrastructure Operations

```bash
# Create new infrastructure workspace
provisioning workspace init my-project

# Create servers
provisioning server create --check

# Install task services (infrastructure components)
provisioning taskserv create kubernetes

# Create complete cluster
provisioning cluster create my-cluster

# SSH into server
provisioning server ssh hostname-01
```

### DAG, Components & Workflows

```bash
# Inspect workspace DAG composition (nodes, edges, health gates)
provisioning dag show --infra wuji
provisioning dag validate --infra wuji
provisioning dag export --infra wuji --format dot

# Components and extension capabilities
provisioning component list
provisioning component info postgresql
provisioning extensions capabilities
provisioning extensions graph

# Workflows (topological scheduling + NATS events)
provisioning workflow list
provisioning workflow run deploy-services --infra libre-daoshi
provisioning workflow status <id>
```

### Command Registry & Fast Path

Every `prvng`/`provisioning` invocation validates the command against a JSON cache
rebuilt from `nulib/commands-registry.ncl` whenever the source is newer. Single-char
aliases (`s`, `t`, `c`, `e`, `w`, `j`, `b`, `o`, `a`) are expanded in the bash wrapper
before dispatch. Adding a new top-level command requires a registry entry **plus** a
dispatch case in `cli/provisioning` — see `nulib/main_provisioning/ADDING_COMMANDS.md`.

### Quick Reference

For fastest command reference:

```text
provisioning sc
```

For complete guides:

```text
provisioning guide from-scratch    # Complete deployment guide
provisioning guide quickstart      # Command shortcuts reference
provisioning guide customize       # Customization patterns
```

## Core Libraries

### Configuration System (`lib_provisioning/config/`)

Hierarchical configuration loading with 476+ config accessors:

- **Defaults** → **User** → **Project** → **Infrastructure** → **Environment** → **Runtime**
- Replaced 200+ ENV variables with config-driven architecture
- Variable interpolation: `{{paths.base}}`, `{{env.HOME}}`, `{{now.date}}`

```nushell
use lib_provisioning/config

# Get configuration value
let value = config get "servers.default_plan"

# Load workspace config
let ws_config = config load-workspace "my-project"
```

### Provider Abstraction (`lib_provisioning/providers/`)

Unified interface for cloud providers:

```nushell
use lib_provisioning/providers

# Get provider interface
let provider = providers get "upcloud"

# Create server using provider
$provider | invoke "create_server" $server_config
```

### Utilities (`lib_provisioning/utils/`)

Common utility functions:

- **SSH Operations** - `utils/ssh.nu`
- **Validation** - `utils/validation.nu`
- **Logging** - `utils/logging.nu`
- **Error Handling** - `utils/error.nu`
- **File Operations** - `utils/files.nu`
- **Version Management** - `utils/version_manager.nu`

### Workflow Orchestration (`workflows/`)

Batch operations with dependency resolution:

```bash
# Submit batch workflow
provisioning batch submit workflows/example.ncl

# Monitor workflow progress
provisioning batch monitor <workflow-id>

# List all workflows
provisioning workflow list

# Get workflow status
provisioning workflow status <id>
```

## Internationalization (i18n)

### Fluent-based Localization

The help system supports multiple languages using the Fluent catalog format:

```bash
# Automatic locale detection from LANG environment variable
export LANG=es_ES.UTF-8
provisioning help  # Shows Spanish help if es-ES catalog exists

# Falls back to en-US if translation not available
export LANG=fr_FR.UTF-8
provisioning help  # Shows French help if fr-FR exists, otherwise English
```

**Catalog Structure**:

```text
provisioning/locales/
├── en-US/
│   └── help.ftl          # English help strings
├── es-ES/
│   └── help.ftl          # Spanish help strings
└── de-DE/
    └── help.ftl          # German help strings
```

**Supported Locales**: en-US (default), with framework ready for es-ES, fr-FR, de-DE, etc.

---

## CLI Architecture

### Modular Design

The CLI uses a domain-driven architecture:

- **Infrastructure** - Servers, task services, clusters
- **Orchestration** - Workflows, batch operations, orchestrator management
- **Development** - Modules, layers, versioning, packaging
- **Workspace** - Workspace management, templates
- **Configuration** - Settings, environment, validation
- **Utilities** - SSH, SOPS, cache, plugins

### Command Shortcuts

80+ shortcuts for improved productivity:

| Full Command | Shortcuts | Description |
| ------------ | --------- | ----------- |
| `server` | `s` | Server operations |
| `taskserv` | `t`, `task` | Task service operations |
| `cluster` | `cl` | Cluster operations |
| `workspace` | `ws` | Workspace management |
| `workflow` | `wf`, `flow` | Workflow operations |
| `batch` | `bat` | Batch operations |
| `module` | `mod` | Module management |
| `generate` | `g`, `gen` | Code generation |
| `validate` | `val` | Validation operations |

See complete reference: `provisioning sc` or `provisioning guide quickstart`

### Bi-directional Help

Help works in both directions:

```text
provisioning help workspace  # ✅
provisioning workspace help  # ✅ Same result
provisioning ws help         # ✅ Shortcut also works
```

## Configuration

### Configuration Files

With the new structure (2025-09-26):

- **System Defaults**: `provisioning/config/config.defaults.toml`
- **User Config**: `workspace/config/local-overrides.toml`
- **Environment Configs**: `workspace/config/{dev,test,prod}-defaults.toml`
- **Infrastructure Configs**: `workspace/infra/{name}/config.toml`

### Environment Variables

Legacy ENV variables still supported during transition:

```bash
# Show current environment
provisioning env

# Show all configuration and environment
provisioning allenv

# Use specific environment
PROVISIONING_ENV=prod provisioning server list
```

### Debug Flags

```bash
# Enable debug mode
provisioning --debug <command>

# Check mode (dry-run, no changes)
provisioning --check server create

# Auto-confirm operations
provisioning --yes cluster delete

# Specify infrastructure
provisioning --infra my-project server list
```

## Design Principles

### Modularity

Clean separation between libraries, CLI, and extensions:

- **Core Engine** (`core/`) - CLI, libraries, scripts
- **Extensions** (`extensions/`) - Providers, task services, clusters
- **Platform Services** (`platform/`) - Orchestrator, control center, API gateway

### Extensibility

Plugin architecture for extending functionality:

- Provider plugins for cloud integrations
- Task service definitions for infrastructure components
- Cluster templates for complete deployments
- Workflow templates for automation

### Consistency

Unified API patterns across all components:

- Standardized command structure
- Common flag handling
- Consistent error messages
- Predictable output formats

### Performance

Optimized for large-scale infrastructure operations:

- Token-optimized agents (85-90% efficiency)
- Batch operations with parallel execution
- Checkpoint-based recovery
- Incremental state management

## Migration Strategy

The project follows a three-phase migration:

1. **Phase 1** (Complete) - Created symbolic links to existing implementations
2. **Phase 2** (In Progress) - Gradual migration of components to new structure
3. **Phase 3** (Planned) - Consolidate and optimize unified core engine

## Dependencies

### Required

- **Nushell 0.112.2** - Shell and scripting language
- **Nickel 1.15.1+** - Configuration language

### Recommended

- **SOPS 3.10.2+** - Secrets management
- **Age 1.2.1+** - Encryption for secrets
- **K9s 0.50.6+** - Kubernetes management interface

### Optional

- **nu_plugin_tera** - Template rendering
- **Nickel Language** - Native Nickel support via CLI (no plugin required)

## Documentation

### User Guides

- **Quick Start**: `provisioning guide from-scratch`
- **Command Reference**: `provisioning sc`
- **Update Guide**: `provisioning guide update`
- **Customization**: `provisioning guide customize`

### Architecture Documentation

- **CLI Architecture**: `../docs/src/architecture/adr/ADR-006-provisioning-cli-refactoring.md`
- **Configuration System**: `../docs/src/infrastructure/configuration-system.md`
- **Batch Workflows**: `../docs/src/infrastructure/batch-workflow-system.md`
- **Orchestrator**: `../docs/src/operations/orchestrator-system.md`

### API Documentation

- **REST API**: See `../docs/src/api-reference/` (when orchestrator is running)
- **Nushell Modules**: See inline documentation in `nulib/` modules

## Testing

### Run Tests

```bash
# Run all tests
nu provisioning/core/scripts/test/test_all.nu

# Run specific test group
nu provisioning/core/scripts/test/test_config.nu
nu provisioning/core/scripts/test/test_cli.nu
```

### Test Coverage

The core engine includes comprehensive test coverage:

- Configuration loading and validation
- Provider interface contracts
- CLI command routing
- Workflow orchestration
- Error handling

## Contributing

When contributing to the Core Engine:

1. **Follow Nushell Patterns** - See `.claude/best_nushell_code.md`
2. **Use Modular Design** - Domain-focused modules in appropriate directories
3. **Add Tests** - Include tests for new functionality
4. **Update Documentation** - Keep README and inline docs current
5. **Follow PAP** - Project Architecture Principles (see `CLAUDE.md`)

## Troubleshooting

### Common Issues

**Missing environment variables:**

```text
provisioning env          # Check current configuration
provisioning validate config  # Validate configuration files
```

**Nickel schema errors:**

```text
nickel fmt <file>.ncl          # Format Nickel file
nickel eval <file>.ncl         # Evaluate Nickel schema
nickel typecheck <file>.ncl    # Type check schema
```

**Provider authentication:**

```text
provisioning providers    # List available providers
provisioning show settings # View provider configuration
```

### Debug Mode

Enable verbose logging:

```text
provisioning --debug <command>
```

### Getting Help

```text
provisioning help              # Show main help
provisioning help <category>   # Category-specific help
provisioning <command> help    # Command-specific help
provisioning guide list        # List all guides
```

## Version Information

Check system versions:

```text
provisioning version           # Show all versions
provisioning nuinfo            # Nushell information
```

## License

See project root LICENSE file.

---

## Recent Updates

### 2026-04-17 - DAG architecture, commands-registry, Nushell 0.110/0.112 refactor

- **Unified Component Architecture**: `components/`, `workflow.nu`, and `components.nu`
  implement the libre-daoshi unified model (ComponentDef, WorkflowDef, capabilities).
  See `memory/unified_component_arch.md` and ADRs 020/021.
- **Three-layer DAG**: `dag.nu` + `lib_provisioning/config/loader/dag.nu` add
  `dag show/validate/export` backed by `schemas/lib/dag/*.ncl`; orchestrator emits
  NATS events via `WorkspaceComposition::into_workflow`.
- **Commands-registry cache**: `nulib/commands-registry.ncl` feeds the bash wrapper's
  `_validate_command`; fast-path single-char alias expansion avoids cold Nu startup.
- **Platform service manager**: new `platform/service-manager.nu`, `platform/startup.nu`,
  and `bootstrap.nu` consolidate autostart, health checks, and lifecycle.
- **Nushell 0.112.2 compliance**: `scripts/refactor-try-catch*.nu` drove the
  migration — no `try/catch`, no bash redirections, all external commands prefixed.
- **TTY stack**: `shlib/*-tty.sh` removed; replaced by `cli/tty-dispatch.sh`,
  `cli/tty-filter.sh`, and `cli/tty-commands.conf`.
- **New domain modules**: `images/` (golden image lifecycle), `workspace/state.nu` +
  `workspace/sync.nu`, `main_provisioning/ontoref-queries.nu`, `main_provisioning/fip.nu`,
  `main_provisioning/state.nu`, `main_provisioning/extensions.nu`.
- **Config loader overhaul**: `loader/core.nu` slimmed (−759 lines of legacy paths),
  `cache/core.nu` refactored, `loaders/file_loader.nu` removed.

### 2026-01-14 - Terminology Migration & i18n

- **Cluster → Taskserv**: Complete refactoring across nulib/ modules
- **Fluent i18n System**: Automatic locale detection with en-US fallback
- Enhanced ANSI output formatting for improved CLI readability

---

**Maintained By**: Core Team
**Last Updated**: 2026-04-17