init repo
This commit is contained in:
Jesús Pérez
commit
ce2fce2898
455
README.md
Normal file
455
README.md
Normal file
@ -0,0 +1,455 @@
|
||||
<p align="center">
|
||||
<img src="resources/provisioning_logo.svg" alt="Provisioning Logo" width="300"/>
|
||||
</p>
|
||||
<p align="center">
|
||||
<img src="resources/logo-text.svg" alt="Provisioning" width="500"/>
|
||||
</p>
|
||||
|
||||
---
|
||||
|
||||
# Core Engine
|
||||
|
||||
The **Core Engine** is the foundation of the Provisioning platform—a modular, high-performance infrastructure automation system built on **Nushell** and **KCL**. It provides unified CLI tools, core libraries, and extensible architecture for managing cloud infrastructure, Kubernetes clusters, and infrastructure-as-code workflows.
|
||||
|
||||
## Overview
|
||||
|
||||
The Core Engine serves as the central orchestration layer, providing:
|
||||
|
||||
- **Unified CLI Interface** - Single command-line interface for all infrastructure operations
|
||||
- **Core Libraries** - Reusable Nushell modules for configuration, validation, deployment, and workflow management
|
||||
- **Provider Abstraction** - Cloud-agnostic interface supporting UpCloud, AWS, and local providers
|
||||
- **Workflow Orchestration** - Batch operations, dependency resolution, and state management
|
||||
- **Configuration System** - Hierarchical, config-driven architecture with 476+ configuration accessors
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
provisioning/core/
|
||||
├── cli/ # Command-line interface
|
||||
│ └── provisioning # Main CLI entry point (211 lines, 84% reduction)
|
||||
├── nulib/ # Core Nushell libraries
|
||||
│ ├── lib_provisioning/ # Core provisioning libraries
|
||||
│ │ ├── config/ # Configuration loading and management
|
||||
│ │ ├── utils/ # Utility functions (SSH, validation, logging)
|
||||
│ │ ├── providers/ # Provider abstraction layer
|
||||
│ │ ├── secrets/ # Secrets management (SOPS integration)
|
||||
│ │ ├── workspace/ # Workspace management
|
||||
│ │ └── infra_validator/ # Infrastructure validation engine
|
||||
│ ├── main_provisioning/ # CLI command handlers
|
||||
│ │ ├── flags.nu # Centralized flag handling
|
||||
│ │ ├── dispatcher.nu # Command routing (80+ shortcuts)
|
||||
│ │ ├── help_system.nu # Categorized help system
|
||||
│ │ └── commands/ # Domain-focused command modules
|
||||
│ ├── servers/ # Server management modules
|
||||
│ ├── taskservs/ # Task service modules
|
||||
│ ├── clusters/ # Cluster management modules
|
||||
│ └── workflows/ # Workflow orchestration modules
|
||||
├── scripts/ # Utility scripts
|
||||
│ └── test/ # Test automation
|
||||
└── resources/ # Images and logos
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **Nushell 0.107.1+** - Primary shell and scripting environment
|
||||
- **KCL 0.11.2+** - Configuration language for infrastructure definitions
|
||||
- **SOPS 3.10.2+** - Secrets management (optional but recommended)
|
||||
- **Age 1.2.1+** - Encryption tool for secrets (optional)
|
||||
|
||||
### Adding to PATH
|
||||
|
||||
To use the CLI globally, add it to your PATH:
|
||||
|
||||
```bash
|
||||
# Create symbolic link
|
||||
ln -sf "$(pwd)/provisioning/core/cli/provisioning" /usr/local/bin/provisioning
|
||||
|
||||
# Or add to PATH in your shell config (~/.bashrc, ~/.zshrc, etc.)
|
||||
export PATH="$PATH:/path/to/project-provisioning/provisioning/core/cli"
|
||||
```
|
||||
|
||||
Verify installation:
|
||||
|
||||
```bash
|
||||
provisioning version
|
||||
provisioning help
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Basic Commands
|
||||
|
||||
```bash
|
||||
# Show help and available commands
|
||||
provisioning help
|
||||
|
||||
# Show environment and configuration
|
||||
provisioning env
|
||||
provisioning allenv
|
||||
|
||||
# Validate configuration
|
||||
provisioning validate config
|
||||
|
||||
# List available providers
|
||||
provisioning providers
|
||||
|
||||
# Show system information
|
||||
provisioning nuinfo
|
||||
```
|
||||
|
||||
### Infrastructure Operations
|
||||
|
||||
```bash
|
||||
# Create new infrastructure workspace
|
||||
provisioning workspace init my-project
|
||||
|
||||
# Create servers
|
||||
provisioning server create --check
|
||||
|
||||
# Install task services (infrastructure components)
|
||||
provisioning taskserv create kubernetes
|
||||
|
||||
# Create complete cluster
|
||||
provisioning cluster create my-cluster
|
||||
|
||||
# SSH into server
|
||||
provisioning server ssh hostname-01
|
||||
```
|
||||
|
||||
### Quick Reference
|
||||
|
||||
For fastest command reference:
|
||||
|
||||
```bash
|
||||
provisioning sc
|
||||
```
|
||||
|
||||
For complete guides:
|
||||
|
||||
```bash
|
||||
provisioning guide from-scratch # Complete deployment guide
|
||||
provisioning guide quickstart # Command shortcuts reference
|
||||
provisioning guide customize # Customization patterns
|
||||
```
|
||||
|
||||
## Core Libraries
|
||||
|
||||
### Configuration System (`lib_provisioning/config/`)
|
||||
|
||||
Hierarchical configuration loading with 476+ config accessors:
|
||||
|
||||
- **Defaults** → **User** → **Project** → **Infrastructure** → **Environment** → **Runtime**
|
||||
- Replaced 200+ ENV variables with config-driven architecture
|
||||
- Variable interpolation: `{{paths.base}}`, `{{env.HOME}}`, `{{now.date}}`
|
||||
|
||||
```nushell
|
||||
use lib_provisioning/config
|
||||
|
||||
# Get configuration value
|
||||
let value = config get "servers.default_plan"
|
||||
|
||||
# Load workspace config
|
||||
let ws_config = config load-workspace "my-project"
|
||||
```
|
||||
|
||||
### Provider Abstraction (`lib_provisioning/providers/`)
|
||||
|
||||
Unified interface for cloud providers:
|
||||
|
||||
```nushell
|
||||
use lib_provisioning/providers
|
||||
|
||||
# Get provider interface
|
||||
let provider = providers get "upcloud"
|
||||
|
||||
# Create server using provider
|
||||
$provider | invoke "create_server" $server_config
|
||||
```
|
||||
|
||||
### Utilities (`lib_provisioning/utils/`)
|
||||
|
||||
Common utility functions:
|
||||
|
||||
- **SSH Operations** - `utils/ssh.nu`
|
||||
- **Validation** - `utils/validation.nu`
|
||||
- **Logging** - `utils/logging.nu`
|
||||
- **Error Handling** - `utils/error.nu`
|
||||
- **File Operations** - `utils/files.nu`
|
||||
- **Version Management** - `utils/version_manager.nu`
|
||||
|
||||
### Workflow Orchestration (`workflows/`)
|
||||
|
||||
Batch operations with dependency resolution:
|
||||
|
||||
```bash
|
||||
# Submit batch workflow
|
||||
provisioning batch submit workflows/example.k
|
||||
|
||||
# Monitor workflow progress
|
||||
provisioning batch monitor <workflow-id>
|
||||
|
||||
# List all workflows
|
||||
provisioning workflow list
|
||||
|
||||
# Get workflow status
|
||||
provisioning workflow status <id>
|
||||
```
|
||||
|
||||
## CLI Architecture
|
||||
|
||||
### Modular Design
|
||||
|
||||
The CLI uses a domain-driven architecture:
|
||||
|
||||
- **Infrastructure** - Servers, task services, clusters
|
||||
- **Orchestration** - Workflows, batch operations, orchestrator management
|
||||
- **Development** - Modules, layers, versioning, packaging
|
||||
- **Workspace** - Workspace management, templates
|
||||
- **Configuration** - Settings, environment, validation
|
||||
- **Utilities** - SSH, SOPS, cache, plugins
|
||||
|
||||
### Command Shortcuts
|
||||
|
||||
80+ shortcuts for improved productivity:
|
||||
|
||||
| Full Command | Shortcuts | Description |
|
||||
|--------------|-----------|-------------|
|
||||
| `server` | `s` | Server operations |
|
||||
| `taskserv` | `t`, `task` | Task service operations |
|
||||
| `cluster` | `cl` | Cluster operations |
|
||||
| `workspace` | `ws` | Workspace management |
|
||||
| `workflow` | `wf`, `flow` | Workflow operations |
|
||||
| `batch` | `bat` | Batch operations |
|
||||
| `module` | `mod` | Module management |
|
||||
| `generate` | `g`, `gen` | Code generation |
|
||||
| `validate` | `val` | Validation operations |
|
||||
|
||||
See complete reference: `provisioning sc` or `provisioning guide quickstart`
|
||||
|
||||
### Bi-directional Help
|
||||
|
||||
Help works in both directions:
|
||||
|
||||
```bash
|
||||
provisioning help workspace # ✅
|
||||
provisioning workspace help # ✅ Same result
|
||||
provisioning ws help # ✅ Shortcut also works
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Configuration Files
|
||||
|
||||
With the new structure (2025-09-26):
|
||||
|
||||
- **System Defaults**: `provisioning/config/config.defaults.toml`
|
||||
- **User Config**: `workspace/config/local-overrides.toml`
|
||||
- **Environment Configs**: `workspace/config/{dev,test,prod}-defaults.toml`
|
||||
- **Infrastructure Configs**: `workspace/infra/{name}/config.toml`
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Legacy ENV variables still supported during transition:
|
||||
|
||||
```bash
|
||||
# Show current environment
|
||||
provisioning env
|
||||
|
||||
# Show all configuration and environment
|
||||
provisioning allenv
|
||||
|
||||
# Use specific environment
|
||||
PROVISIONING_ENV=prod provisioning server list
|
||||
```
|
||||
|
||||
### Debug Flags
|
||||
|
||||
```bash
|
||||
# Enable debug mode
|
||||
provisioning --debug <command>
|
||||
|
||||
# Check mode (dry-run, no changes)
|
||||
provisioning --check server create
|
||||
|
||||
# Auto-confirm operations
|
||||
provisioning --yes cluster delete
|
||||
|
||||
# Specify infrastructure
|
||||
provisioning --infra my-project server list
|
||||
```
|
||||
|
||||
## Design Principles
|
||||
|
||||
### Modularity
|
||||
|
||||
Clean separation between libraries, CLI, and extensions:
|
||||
|
||||
- **Core Engine** (`core/`) - CLI, libraries, scripts
|
||||
- **Extensions** (`extensions/`) - Providers, task services, clusters
|
||||
- **Platform Services** (`platform/`) - Orchestrator, control center, API gateway
|
||||
|
||||
### Extensibility
|
||||
|
||||
Plugin architecture for extending functionality:
|
||||
|
||||
- Provider plugins for cloud integrations
|
||||
- Task service definitions for infrastructure components
|
||||
- Cluster templates for complete deployments
|
||||
- Workflow templates for automation
|
||||
|
||||
### Consistency
|
||||
|
||||
Unified API patterns across all components:
|
||||
|
||||
- Standardized command structure
|
||||
- Common flag handling
|
||||
- Consistent error messages
|
||||
- Predictable output formats
|
||||
|
||||
### Performance
|
||||
|
||||
Optimized for large-scale infrastructure operations:
|
||||
|
||||
- Token-optimized agents (85-90% efficiency)
|
||||
- Batch operations with parallel execution
|
||||
- Checkpoint-based recovery
|
||||
- Incremental state management
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
The project follows a three-phase migration:
|
||||
|
||||
1. **Phase 1** (Complete) - Created symbolic links to existing implementations
|
||||
2. **Phase 2** (In Progress) - Gradual migration of components to new structure
|
||||
3. **Phase 3** (Planned) - Consolidate and optimize unified core engine
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Required
|
||||
|
||||
- **Nushell 0.107.1+** - Shell and scripting language
|
||||
- **KCL 0.11.2+** - Configuration language
|
||||
|
||||
### Recommended
|
||||
|
||||
- **SOPS 3.10.2+** - Secrets management
|
||||
- **Age 1.2.1+** - Encryption for secrets
|
||||
- **K9s 0.50.6+** - Kubernetes management interface
|
||||
|
||||
### Optional
|
||||
|
||||
- **nu_plugin_tera** - Template rendering
|
||||
- **nu_plugin_kcl** - KCL integration (CLI `kcl` is required, plugin optional)
|
||||
|
||||
## Documentation
|
||||
|
||||
### User Guides
|
||||
|
||||
- **Quick Start**: `provisioning guide from-scratch`
|
||||
- **Command Reference**: `provisioning sc`
|
||||
- **Update Guide**: `provisioning guide update`
|
||||
- **Customization**: `provisioning guide customize`
|
||||
|
||||
### Architecture Documentation
|
||||
|
||||
- **CLI Architecture**: `docs/architecture/ADR-006-provisioning-cli-refactoring.md`
|
||||
- **Configuration System**: See `.claude/features/configuration-system.md`
|
||||
- **Batch Workflows**: See `.claude/features/batch-workflow-system.md`
|
||||
- **Orchestrator**: See `.claude/features/orchestrator-architecture.md`
|
||||
|
||||
### API Documentation
|
||||
|
||||
- **REST API**: See `docs/api/` (when orchestrator is running)
|
||||
- **Nushell Modules**: See inline documentation in `nulib/` modules
|
||||
|
||||
## Testing
|
||||
|
||||
### Run Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
nu provisioning/core/scripts/test/test_all.nu
|
||||
|
||||
# Run specific test group
|
||||
nu provisioning/core/scripts/test/test_config.nu
|
||||
nu provisioning/core/scripts/test/test_cli.nu
|
||||
```
|
||||
|
||||
### Test Coverage
|
||||
|
||||
The core engine includes comprehensive test coverage:
|
||||
|
||||
- Configuration loading and validation
|
||||
- Provider interface contracts
|
||||
- CLI command routing
|
||||
- Workflow orchestration
|
||||
- Error handling
|
||||
|
||||
## Contributing
|
||||
|
||||
When contributing to the Core Engine:
|
||||
|
||||
1. **Follow Nushell Patterns** - See `.claude/best_nushell_code.md`
|
||||
2. **Use Modular Design** - Domain-focused modules in appropriate directories
|
||||
3. **Add Tests** - Include tests for new functionality
|
||||
4. **Update Documentation** - Keep README and inline docs current
|
||||
5. **Follow PAP** - Project Architecture Principles (see `CLAUDE.md`)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Missing environment variables:**
|
||||
```bash
|
||||
provisioning env # Check current configuration
|
||||
provisioning validate config # Validate configuration files
|
||||
```
|
||||
|
||||
**KCL compilation errors:**
|
||||
```bash
|
||||
kcl fmt <file>.k # Format KCL file
|
||||
kcl run <file>.k # Test KCL file
|
||||
```
|
||||
|
||||
**Provider authentication:**
|
||||
```bash
|
||||
provisioning providers # List available providers
|
||||
provisioning show settings # View provider configuration
|
||||
```
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable verbose logging:
|
||||
|
||||
```bash
|
||||
provisioning --debug <command>
|
||||
```
|
||||
|
||||
### Getting Help
|
||||
|
||||
```bash
|
||||
provisioning help # Show main help
|
||||
provisioning help <category> # Category-specific help
|
||||
provisioning <command> help # Command-specific help
|
||||
provisioning guide list # List all guides
|
||||
```
|
||||
|
||||
## Version Information
|
||||
|
||||
Check system versions:
|
||||
|
||||
```bash
|
||||
provisioning version # Show all versions
|
||||
provisioning nuinfo # Nushell information
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
See project root LICENSE file.
|
||||
|
||||
---
|
||||
|
||||
**Maintained By**: Architecture Team
|
||||
**Last Updated**: 2025-10-07
|
||||
Loading…
x
Reference in New Issue
Block a user