Provisioning Platform Documentation

Last Updated: 2025-10-06

Welcome to the comprehensive documentation for the Provisioning Platform - a modern, cloud-native infrastructure automation system built with Nushell, KCL, and Rust.

Quick Navigation

🚀 Getting Started

📚 User Guides

🏗️ Architecture

📋 Architecture Decision Records (ADRs)

🔌 API Documentation

🛠️ Development

🐛 Troubleshooting

📖 How-To Guides

🔐 Configuration

📦 Quick References

Documentation Structure

docs/
├── README.md (this file)          # Documentation hub
├── architecture/                  # System architecture
│   ├── ADR/                       # Architecture Decision Records
│   ├── design-principles.md
│   ├── integration-patterns.md
│   └── system-overview.md
├── user/                          # User guides
│   ├── getting-started.md
│   ├── cli-reference.md
│   ├── installation-guide.md
│   └── troubleshooting-guide.md
├── api/                           # API documentation
│   ├── rest-api.md
│   ├── websocket.md
│   └── extensions.md
├── development/                   # Developer guides
│   ├── README.md
│   ├── implementation-guide.md
│   └── kcl/                       # KCL documentation
├── guides/                        # How-to guides
│   ├── from-scratch.md
│   ├── update-infrastructure.md
│   └── customize-infrastructure.md
├── configuration/                 # Configuration docs
│   └── workspace-config-architecture.md
├── troubleshooting/               # Troubleshooting
│   └── CTRL-C_SUDO_HANDLING.md
└── quick-reference/               # Quick refs
    └── SUDO_PASSWORD_HANDLING.md

Key Concepts

Infrastructure as Code (IaC)

The provisioning platform uses declarative configuration to manage infrastructure. Instead of manually creating resources, you define what you want in KCL configuration files, and the system makes it happen.

Mode-Based Architecture

The system supports four operational modes:

• Solo: Single developer local development
• Multi-user: Team collaboration with shared services
• CI/CD: Automated pipeline execution
• Enterprise: Production deployment with strict compliance

Extension System

Extensibility through:

• Providers: Cloud platform integrations (AWS, UpCloud, Local)
• Task Services: Infrastructure components (Kubernetes, databases, etc.)
• Clusters: Complete deployment configurations

OCI-Native Distribution

Extensions and packages distributed as OCI artifacts, enabling:

• Industry-standard packaging
• Efficient caching and bandwidth
• Version pinning and rollback
• Air-gapped deployments

Documentation by Role

For New Users

1. Start with Installation Guide
2. Read Getting Started
3. Follow From Scratch Guide
4. Reference Quickstart Cheatsheet

For Developers

1. Review System Overview
2. Study Design Principles
3. Read relevant ADRs
4. Follow Development Guide
5. Reference KCL Quick Reference

For Operators

1. Understand Mode System
2. Learn Service Management
3. Review Infrastructure Management
4. Study OCI Registry

For Architects

1. Read System Overview
2. Study all ADRs
3. Review Integration Patterns
4. Understand Multi-Repo Architecture

System Capabilities

✅ Infrastructure Automation

• Multi-cloud support (AWS, UpCloud, Local)
• Declarative configuration with KCL
• Automated dependency resolution
• Batch operations with rollback

✅ Workflow Orchestration

• Hybrid Rust/Nushell orchestration
• Checkpoint-based recovery
• Parallel execution with limits
• Real-time monitoring

✅ Test Environments

• Containerized testing
• Multi-node cluster simulation
• Topology templates
• Automated cleanup

✅ Mode-Based Operation

• Solo: Local development
• Multi-user: Team collaboration
• CI/CD: Automated pipelines
• Enterprise: Production deployment

✅ Extension Management

• OCI-native distribution
• Automatic dependency resolution
• Version management
• Local and remote sources

Key Achievements

🚀 Batch Workflow System (v3.1.0)

• Provider-agnostic batch operations
• Mixed provider support (UpCloud + AWS + local)
• Dependency resolution with soft/hard dependencies
• Real-time monitoring and rollback

🏗️ Hybrid Orchestrator (v3.0.0)

• Solves Nushell deep call stack limitations
• Preserves all business logic
• REST API for external integration
• Checkpoint-based state management

⚙️ Configuration System (v2.0.0)

• Migrated from ENV to config-driven
• Hierarchical configuration loading
• Variable interpolation
• True IaC without hardcoded fallbacks

🎯 Modular CLI (v3.2.0)

• 84% reduction in main file size
• Domain-driven handlers
• 80+ shortcuts
• Bi-directional help system

🧪 Test Environment Service (v3.4.0)

• Automated containerized testing
• Multi-node cluster topologies
• CI/CD integration ready
• Template-based configurations

🔄 Workspace Switching (v2.0.5)

• Centralized workspace management
• Single-command workspace switching
• Active workspace tracking
• User preference system

Technology Stack

Support

Getting Help

• Documentation: You’re reading it!
• Quick Reference: Run provisioning sc or provisioning guide quickstart
• Help System: Run provisioning help or provisioning <command> help
• Interactive Shell: Run provisioning nu for Nushell REPL

Reporting Issues

• Check Troubleshooting Guide
• Review FAQ
• Enable debug mode: provisioning --debug <command>
• Check logs: provisioning platform logs <service>

Contributing

This project welcomes contributions! See Development Guide for:

• Development setup
• Code style guidelines
• Testing requirements
• Pull request process

License

[Add license information]

Version History

Maintained By: Provisioning Team Last Review: 2025-10-06 Next Review: 2026-01-06

Provisioning Platform Glossary

Last Updated: 2025-10-10 Version: 1.0.0

This glossary defines key terminology used throughout the Provisioning Platform documentation. Terms are listed alphabetically with definitions, usage context, and cross-references to related documentation.

A

ADR (Architecture Decision Record)

Definition: Documentation of significant architectural decisions, including context, decision, and consequences.

Where Used:

• Architecture planning and review
• Technical decision-making process
• System design documentation

Related Concepts: Architecture, Design Patterns, Technical Debt

Examples:

• ADR-001: Project Structure
• ADR-006: CLI Refactoring
• ADR-009: Complete Security System

See Also: Architecture Documentation

Agent

Definition: A specialized, token-efficient component that performs a specific task in the system (e.g., Agent 1-16 in documentation generation).

Where Used:

• Documentation generation workflows
• Task orchestration
• Parallel processing patterns

Related Concepts: Orchestrator, Workflow, Task

See Also: Batch Workflow System

Anchor Link

Definition: An internal document link to a specific section within the same or different markdown file using the # symbol.

Where Used:

• Cross-referencing documentation sections
• Table of contents generation
• Navigation within long documents

Related Concepts: Internal Link, Cross-Reference, Documentation

Examples:

• [See Installation](#installation) - Same document
• [Configuration Guide](config.md#setup) - Different document

API Gateway

Definition: Platform service that provides unified REST API access to provisioning operations.

Where Used:

• External system integration
• Web Control Center backend
• MCP server communication

Related Concepts: REST API, Platform Service, Orchestrator

Location: provisioning/platform/api-gateway/

See Also: REST API Documentation

Auth (Authentication)

Definition: The process of verifying user identity using JWT tokens, MFA, and secure session management.

Where Used:

• User login flows
• API access control
• CLI session management

Related Concepts: Authorization, JWT, MFA, Security

See Also:

• Authentication Layer Guide
• Auth Quick Reference

Authorization

Definition: The process of determining user permissions using Cedar policy language.

Where Used:

• Access control decisions
• Resource permission checks
• Multi-tenant security

Related Concepts: Auth, Cedar, Policies, RBAC

See Also: Cedar Authorization Implementation

B

Batch Operation

Definition: A collection of related infrastructure operations executed as a single workflow unit.

Where Used:

• Multi-server deployments
• Cluster creation
• Bulk taskserv installation

Related Concepts: Workflow, Operation, Orchestrator

Commands:

provisioning batch submit workflow.k
provisioning batch list
provisioning batch status <id>

See Also: Batch Workflow System

Break-Glass

Definition: Emergency access mechanism requiring multi-party approval for critical operations.

Where Used:

• Emergency system access
• Incident response
• Security override scenarios

Related Concepts: Security, Compliance, Audit

Commands:

provisioning break-glass request "reason"
provisioning break-glass approve <id>

See Also: Break-Glass Training Guide

C

Cedar

Definition: Amazon’s policy language used for fine-grained authorization decisions.

Where Used:

• Authorization policies
• Access control rules
• Resource permissions

Related Concepts: Authorization, Policies, Security

See Also: Cedar Authorization Implementation

Checkpoint

Definition: A saved state of a workflow allowing resume from point of failure.

Where Used:

• Workflow recovery
• Long-running operations
• Batch processing

Related Concepts: Workflow, State Management, Recovery

See Also: Batch Workflow System

CLI (Command-Line Interface)

Definition: The provisioning command-line tool providing access to all platform operations.

Where Used:

• Daily operations
• Script automation
• CI/CD pipelines

Related Concepts: Command, Shortcut, Module

Location: provisioning/core/cli/provisioning

Examples:

provisioning server create
provisioning taskserv install kubernetes
provisioning workspace switch prod

See Also:

• CLI Architecture
• CLI Reference

Cluster

Definition: A complete, pre-configured deployment of multiple servers and taskservs working together.

Where Used:

• Kubernetes deployments
• Database clusters
• Complete infrastructure stacks

Related Concepts: Infrastructure, Server, Taskserv

Location: provisioning/extensions/clusters/{name}/

Commands:

provisioning cluster create <name>
provisioning cluster list
provisioning cluster delete <name>

See Also: Infrastructure Management

Compliance

Definition: System capabilities ensuring adherence to regulatory requirements (GDPR, SOC2, ISO 27001).

Where Used:

• Audit logging
• Data retention policies
• Incident response

Related Concepts: Audit, Security, GDPR

See Also: Compliance Implementation Summary

Config (Configuration)

Definition: System settings stored in TOML files with hierarchical loading and variable interpolation.

Where Used:

• System initialization
• User preferences
• Environment-specific settings

Related Concepts: Settings, Environment, Workspace

Files:

• provisioning/config/config.defaults.toml - System defaults
• workspace/config/local-overrides.toml - User settings

See Also: Configuration System

Control Center

Definition: Web-based UI for managing provisioning operations built with Ratatui/Crossterm.

Where Used:

• Visual infrastructure management
• Real-time monitoring
• Guided workflows

Related Concepts: UI, Platform Service, Orchestrator

Location: provisioning/platform/control-center/

See Also: Platform Services

CoreDNS

Definition: DNS server taskserv providing service discovery and DNS management.

Where Used:

• Kubernetes DNS
• Service discovery
• Internal DNS resolution

Related Concepts: Taskserv, Kubernetes, Networking

See Also:

• CoreDNS Guide
• CoreDNS Quick Reference

Cross-Reference

Definition: Links between related documentation sections or concepts.

Where Used:

• Documentation navigation
• Related topic discovery
• Learning path guidance

Related Concepts: Documentation, Navigation, See Also

Examples: “See Also” sections at the end of documentation pages

D

Dependency

Definition: A requirement that must be satisfied before installing or running a component.

Where Used:

• Taskserv installation order
• Version compatibility checks
• Cluster deployment sequencing

Related Concepts: Version, Taskserv, Workflow

Schema: provisioning/kcl/dependencies.k

See Also: KCL Dependency Patterns

Diagnostics

Definition: System health checking and troubleshooting assistance.

Where Used:

• System status verification
• Problem identification
• Guided troubleshooting

Related Concepts: Health Check, Monitoring, Troubleshooting

Commands:

provisioning status
provisioning diagnostics run

Dynamic Secrets

Definition: Temporary credentials generated on-demand with automatic expiration.

Where Used:

• AWS STS tokens
• SSH temporary keys
• Database credentials

Related Concepts: Security, KMS, Secrets Management

See Also:

• Dynamic Secrets Implementation
• Dynamic Secrets Quick Reference

E

Environment

Definition: A deployment context (dev, test, prod) with specific configuration overrides.

Where Used:

• Configuration loading
• Resource isolation
• Deployment targeting

Related Concepts: Config, Workspace, Infrastructure

Config Files: config.{dev,test,prod}.toml

Usage:

PROVISIONING_ENV=prod provisioning server list

Extension

Definition: A pluggable component adding functionality (provider, taskserv, cluster, or workflow).

Where Used:

• Custom cloud providers
• Third-party taskservs
• Custom deployment patterns

Related Concepts: Provider, Taskserv, Cluster, Workflow

Location: provisioning/extensions/{type}/{name}/

See Also: Extension Development

F

Feature

Definition: A major system capability documented in .claude/features/.

Where Used:

• Architecture documentation
• Feature planning
• System capabilities

Related Concepts: ADR, Architecture, System

Location: .claude/features/*.md

Examples:

• Batch Workflow System
• Orchestrator Architecture
• CLI Architecture

See Also: Features README

G

GDPR (General Data Protection Regulation)

Definition: EU data protection regulation compliance features in the platform.

Where Used:

• Data export requests
• Right to erasure
• Audit compliance

Related Concepts: Compliance, Audit, Security

Commands:

provisioning compliance gdpr export <user>
provisioning compliance gdpr delete <user>

See Also: Compliance Implementation

Glossary

Definition: This document - a comprehensive terminology reference for the platform.

Where Used:

• Learning the platform
• Understanding documentation
• Resolving terminology questions

Related Concepts: Documentation, Reference, Cross-Reference

Guide

Definition: Step-by-step walkthrough documentation for common workflows.

Where Used:

• Onboarding new users
• Learning workflows
• Reference implementation

Related Concepts: Documentation, Workflow, Tutorial

Commands:

provisioning guide from-scratch
provisioning guide update
provisioning guide customize

See Also: Guide System

H

Health Check

Definition: Automated verification that a component is running correctly.

Where Used:

• Taskserv validation
• System monitoring
• Dependency verification

Related Concepts: Diagnostics, Monitoring, Status

Example:

health_check = {
    endpoint = "http://localhost:6443/healthz"
    timeout = 30
    interval = 10
}

Hybrid Architecture

Definition: System design combining Rust orchestrator with Nushell business logic.

Where Used:

• Core platform architecture
• Performance optimization
• Call stack management

Related Concepts: Orchestrator, Architecture, Design

See Also:

• Orchestrator Architecture
• ADR-004: Hybrid Architecture

I

Infrastructure

Definition: A named collection of servers, configurations, and deployments managed as a unit.

Where Used:

• Environment isolation
• Resource organization
• Deployment targeting

Related Concepts: Workspace, Server, Environment

Location: workspace/infra/{name}/

Commands:

provisioning infra list
provisioning generate infra --new <name>

See Also: Infrastructure Management

Integration

Definition: Connection between platform components or external systems.

Where Used:

• API integration
• CI/CD pipelines
• External tool connectivity

Related Concepts: API, Extension, Platform

See Also:

• Integration Patterns
• Integration Examples

Internal Link

Definition: A markdown link to another documentation file or section within the platform docs.

Where Used:

• Cross-referencing documentation
• Navigation between topics
• Related content discovery

Related Concepts: Anchor Link, Cross-Reference, Documentation

Examples:

• [See Configuration](./configuration.md)
• [Architecture Overview](../architecture/README.md)

J

JWT (JSON Web Token)

Definition: Token-based authentication mechanism using RS256 signatures.

Where Used:

• User authentication
• API authorization
• Session management

Related Concepts: Auth, Security, Token

See Also: JWT Auth Implementation

K

KCL (KCL Configuration Language)

Definition: Declarative configuration language used for infrastructure definitions.

Where Used:

• Infrastructure schemas
• Workflow definitions
• Configuration validation

Related Concepts: Schema, Configuration, Validation

Version: 0.11.3+

Location: provisioning/kcl/*.k

See Also:

• KCL Idiomatic Patterns
• KCL Quick Reference

KMS (Key Management Service)

Definition: Encryption key management system supporting multiple backends (RustyVault, Age, AWS, Vault).

Where Used:

• Configuration encryption
• Secret management
• Data protection

Related Concepts: Security, Encryption, Secrets

See Also: RustyVault KMS Guide

Kubernetes

Definition: Container orchestration platform available as a taskserv.

Where Used:

• Container deployments
• Cluster management
• Production workloads

Related Concepts: Taskserv, Cluster, Container

Commands:

provisioning taskserv create kubernetes
provisioning test quick kubernetes

L

Layer

Definition: A level in the configuration hierarchy (Core → Workspace → Infrastructure).

Where Used:

• Configuration inheritance
• Customization patterns
• Settings override

Related Concepts: Config, Workspace, Infrastructure

See Also: Configuration System

M

MCP (Model Context Protocol)

Definition: AI-powered server providing intelligent configuration assistance.

Where Used:

• Configuration validation
• Troubleshooting guidance
• Documentation search

Related Concepts: Platform Service, AI, Guidance

Location: provisioning/platform/mcp-server/

See Also: Platform Services

MFA (Multi-Factor Authentication)

Definition: Additional authentication layer using TOTP or WebAuthn/FIDO2.

Where Used:

• Enhanced security
• Compliance requirements
• Production access

Related Concepts: Auth, Security, TOTP, WebAuthn

Commands:

provisioning mfa totp enroll
provisioning mfa webauthn enroll
provisioning mfa verify <code>

See Also: MFA Implementation Summary

Migration

Definition: Process of updating existing infrastructure or moving between system versions.

Where Used:

• System upgrades
• Configuration changes
• Infrastructure evolution

Related Concepts: Update, Upgrade, Version

See Also: Migration Guide

Module

Definition: A reusable component (provider, taskserv, cluster) loaded into a workspace.

Where Used:

• Extension management
• Workspace customization
• Component distribution

Related Concepts: Extension, Workspace, Package

Commands:

provisioning module discover provider
provisioning module load provider <ws> <name>
provisioning module list taskserv

See Also: Module System

N

Nushell

Definition: Primary shell and scripting language (v0.107.1) used throughout the platform.

Where Used:

• CLI implementation
• Automation scripts
• Business logic

Related Concepts: CLI, Script, Automation

Version: 0.107.1

See Also: Best Nushell Code

O

OCI (Open Container Initiative)

Definition: Standard format for packaging and distributing extensions.

Where Used:

• Extension distribution
• Package registry
• Version management

Related Concepts: Registry, Package, Distribution

See Also: OCI Registry Guide

Operation

Definition: A single infrastructure action (create server, install taskserv, etc.).

Where Used:

• Workflow steps
• Batch processing
• Orchestrator tasks

Related Concepts: Workflow, Task, Action

Orchestrator

Definition: Hybrid Rust/Nushell service coordinating complex infrastructure operations.

Where Used:

• Workflow execution
• Task coordination
• State management

Related Concepts: Hybrid Architecture, Workflow, Platform Service

Location: provisioning/platform/orchestrator/

Commands:

cd provisioning/platform/orchestrator
./scripts/start-orchestrator.nu --background

See Also: Orchestrator Architecture

P

PAP (Project Architecture Principles)

Definition: Core architectural rules and patterns that must be followed.

Where Used:

• Code review
• Architecture decisions
• Design validation

Related Concepts: Architecture, ADR, Best Practices

See Also: Architecture Overview

Platform Service

Definition: A core service providing platform-level functionality (Orchestrator, Control Center, MCP, API Gateway).

Where Used:

• System infrastructure
• Core capabilities
• Service integration

Related Concepts: Service, Architecture, Infrastructure

Location: provisioning/platform/{service}/

Plugin

Definition: Native Nushell plugin providing performance-optimized operations.

Where Used:

• Auth operations (10-50x faster)
• KMS encryption
• Orchestrator queries

Related Concepts: Nushell, Performance, Native

Commands:

provisioning plugin list
provisioning plugin install

See Also: Nushell Plugins Guide

Provider

Definition: Cloud platform integration (AWS, UpCloud, local) handling infrastructure provisioning.

Where Used:

• Server creation
• Resource management
• Cloud operations

Related Concepts: Extension, Infrastructure, Cloud

Location: provisioning/extensions/providers/{name}/

Examples: aws, upcloud, local

Commands:

provisioning module discover provider
provisioning providers list

See Also: Quick Provider Guide

Q

Quick Reference

Definition: Condensed command and configuration reference for rapid lookup.

Where Used:

• Daily operations
• Quick reminders
• Command syntax

Related Concepts: Guide, Documentation, Cheatsheet

Commands:

provisioning sc  # Fastest
provisioning guide quickstart

See Also: Quickstart Cheatsheet

R

RBAC (Role-Based Access Control)

Definition: Permission system with 5 roles (admin, operator, developer, viewer, auditor).

Where Used:

• User permissions
• Access control
• Security policies

Related Concepts: Authorization, Cedar, Security

Roles: Admin, Operator, Developer, Viewer, Auditor

Registry

Definition: OCI-compliant repository for storing and distributing extensions.

Where Used:

• Extension publishing
• Version management
• Package distribution

Related Concepts: OCI, Package, Distribution

See Also: OCI Registry Guide

REST API

Definition: HTTP endpoints exposing platform operations to external systems.

Where Used:

• External integration
• Web UI backend
• Programmatic access

Related Concepts: API, Integration, HTTP

Endpoint: http://localhost:9090

See Also: REST API Documentation

Rollback

Definition: Reverting a failed workflow or operation to previous stable state.

Where Used:

• Failure recovery
• Deployment safety
• State restoration

Related Concepts: Workflow, Checkpoint, Recovery

Commands:

provisioning batch rollback <workflow-id>

RustyVault

Definition: Rust-based secrets management backend for KMS.

Where Used:

• Key storage
• Secret encryption
• Configuration protection

Related Concepts: KMS, Security, Encryption

See Also: RustyVault KMS Guide

S

Schema

Definition: KCL type definition specifying structure and validation rules.

Where Used:

• Configuration validation
• Type safety
• Documentation

Related Concepts: KCL, Validation, Type

Example:

schema ServerConfig:
    hostname: str
    cores: int
    memory: int

    check:
        cores > 0, "Cores must be positive"

See Also: KCL Idiomatic Patterns

Secrets Management

Definition: System for secure storage and retrieval of sensitive data.

Where Used:

• Password storage
• API keys
• Certificates

Related Concepts: KMS, Security, Encryption

See Also: Dynamic Secrets Implementation

Security System

Definition: Comprehensive enterprise-grade security with 12 components (Auth, Cedar, MFA, KMS, Secrets, Compliance, etc.).

Where Used:

• User authentication
• Access control
• Data protection

Related Concepts: Auth, Authorization, MFA, KMS, Audit

See Also: Security System Implementation

Server

Definition: Virtual machine or physical host managed by the platform.

Where Used:

• Infrastructure provisioning
• Compute resources
• Deployment targets

Related Concepts: Infrastructure, Provider, Taskserv

Commands:

provisioning server create
provisioning server list
provisioning server ssh <hostname>

See Also: Infrastructure Management

Service

Definition: A running application or daemon (interchangeable with Taskserv in many contexts).

Where Used:

• Service management
• Application deployment
• System administration

Related Concepts: Taskserv, Daemon, Application

See Also: Service Management Guide

Shortcut

Definition: Abbreviated command alias for faster CLI operations.

Where Used:

• Daily operations
• Quick commands
• Productivity enhancement

Related Concepts: CLI, Command, Alias

Examples:

• provisioning s create → provisioning server create
• provisioning ws list → provisioning workspace list
• provisioning sc → Quick reference

See Also: CLI Architecture

SOPS (Secrets OPerationS)

Definition: Encryption tool for managing secrets in version control.

Where Used:

• Configuration encryption
• Secret management
• Secure storage

Related Concepts: Encryption, Security, Age

Version: 3.10.2

Commands:

provisioning sops edit <file>

SSH (Secure Shell)

Definition: Encrypted remote access protocol with temporal key support.

Where Used:

• Server administration
• Remote commands
• Secure file transfer

Related Concepts: Security, Server, Remote Access

Commands:

provisioning server ssh <hostname>
provisioning ssh connect <server>

See Also: SSH Temporal Keys User Guide

State Management

Definition: Tracking and persisting workflow execution state.

Where Used:

• Workflow recovery
• Progress tracking
• Failure handling

Related Concepts: Workflow, Checkpoint, Orchestrator

T

Task

Definition: A unit of work submitted to the orchestrator for execution.

Where Used:

• Workflow execution
• Job processing
• Operation tracking

Related Concepts: Operation, Workflow, Orchestrator

Taskserv

Definition: An installable infrastructure service (Kubernetes, PostgreSQL, Redis, etc.).

Where Used:

• Service installation
• Application deployment
• Infrastructure components

Related Concepts: Service, Extension, Package

Location: provisioning/extensions/taskservs/{category}/{name}/

Commands:

provisioning taskserv create <name>
provisioning taskserv list
provisioning test quick <taskserv>

See Also: Taskserv Developer Guide

Template

Definition: Parameterized configuration file supporting variable substitution.

Where Used:

• Configuration generation
• Infrastructure customization
• Deployment automation

Related Concepts: Config, Generation, Customization

Location: provisioning/templates/

Test Environment

Definition: Containerized isolated environment for testing taskservs and clusters.

Where Used:

• Development testing
• CI/CD integration
• Pre-deployment validation

Related Concepts: Container, Testing, Validation

Commands:

provisioning test quick <taskserv>
provisioning test env single <taskserv>
provisioning test env cluster <cluster>

See Also: Test Environment Service

Topology

Definition: Multi-node cluster configuration template (Kubernetes HA, etcd cluster, etc.).

Where Used:

• Cluster testing
• Multi-node deployments
• Production simulation

Related Concepts: Test Environment, Cluster, Configuration

Examples: kubernetes_3node, etcd_cluster, kubernetes_single

TOTP (Time-based One-Time Password)

Definition: MFA method generating time-sensitive codes.

Where Used:

• Two-factor authentication
• MFA enrollment
• Security enhancement

Related Concepts: MFA, Security, Auth

Commands:

provisioning mfa totp enroll
provisioning mfa totp verify <code>

Troubleshooting

Definition: System problem diagnosis and resolution guidance.

Where Used:

• Problem solving
• Error resolution
• System debugging

Related Concepts: Diagnostics, Guide, Support

See Also: Troubleshooting Guide

U

UI (User Interface)

Definition: Visual interface for platform operations (Control Center, Web UI).

Where Used:

• Visual management
• Guided workflows
• Monitoring dashboards

Related Concepts: Control Center, Platform Service, GUI

Update

Definition: Process of upgrading infrastructure components to newer versions.

Where Used:

• Version management
• Security patches
• Feature updates

Related Concepts: Version, Migration, Upgrade

Commands:

provisioning version check
provisioning version apply

See Also: Update Infrastructure Guide

V

Validation

Definition: Verification that configuration or infrastructure meets requirements.

Where Used:

• Configuration checks
• Schema validation
• Pre-deployment verification

Related Concepts: Schema, KCL, Check

Commands:

provisioning validate config
provisioning validate infrastructure

See Also: Config Validation

Version

Definition: Semantic version identifier for components and compatibility.

Where Used:

• Component versioning
• Compatibility checking
• Update management

Related Concepts: Update, Dependency, Compatibility

Commands:

provisioning version
provisioning version check
provisioning taskserv check-updates

W

WebAuthn

Definition: FIDO2-based passwordless authentication standard.

Where Used:

• Hardware key authentication
• Passwordless login
• Enhanced MFA

Related Concepts: MFA, Security, FIDO2

Commands:

provisioning mfa webauthn enroll
provisioning mfa webauthn verify

Workflow

Definition: A sequence of related operations with dependency management and state tracking.

Where Used:

• Complex deployments
• Multi-step operations
• Automated processes

Related Concepts: Batch Operation, Orchestrator, Task

Commands:

provisioning workflow list
provisioning workflow status <id>
provisioning workflow monitor <id>

See Also: Batch Workflow System

Workspace

Definition: An isolated environment containing infrastructure definitions and configuration.

Where Used:

• Project isolation
• Environment separation
• Team workspaces

Related Concepts: Infrastructure, Config, Environment

Location: workspace/{name}/

Commands:

provisioning workspace list
provisioning workspace switch <name>
provisioning workspace create <name>

See Also: Workspace Switching Guide

X-Z

YAML

Definition: Data serialization format used for Kubernetes manifests and configuration.

Where Used:

• Kubernetes deployments
• Configuration files
• Data interchange

Related Concepts: Config, Kubernetes, Data Format

Symbol and Acronym Index

Cross-Reference Map

By Topic Area

Infrastructure:

• Infrastructure, Server, Cluster, Provider, Taskserv, Module

Security:

• Auth, Authorization, JWT, MFA, TOTP, WebAuthn, Cedar, KMS, Secrets Management, RBAC, Break-Glass

Configuration:

• Config, KCL, Schema, Validation, Environment, Layer, Workspace

Workflow & Operations:

• Workflow, Batch Operation, Operation, Task, Orchestrator, Checkpoint, Rollback

Platform Services:

• Orchestrator, Control Center, MCP, API Gateway, Platform Service

Documentation:

• Glossary, Guide, ADR, Cross-Reference, Internal Link, Anchor Link

Development:

• Extension, Plugin, Template, Module, Integration

Testing:

• Test Environment, Topology, Validation, Health Check

Compliance:

• Compliance, GDPR, Audit, Security System

By User Journey

New User:

1. Glossary (this document)
2. Guide
3. Quick Reference
4. Workspace
5. Infrastructure
6. Server
7. Taskserv

Developer:

1. Extension
2. Provider
3. Taskserv
4. KCL
5. Schema
6. Template
7. Plugin

Operations:

1. Workflow
2. Orchestrator
3. Monitoring
4. Troubleshooting
5. Security
6. Compliance

Terminology Guidelines

Writing Style

Consistency: Use the same term throughout documentation (e.g., “Taskserv” not “task service” or “task-serv”)

Capitalization:

• Proper nouns and acronyms: CAPITALIZE (KCL, JWT, MFA)
• Generic terms: lowercase (server, cluster, workflow)
• Platform-specific terms: Title Case (Taskserv, Workspace, Orchestrator)

Pluralization:

• Taskservs (not taskservices)
• Workspaces (standard plural)
• Topologies (not topologys)

Avoiding Confusion

Contributing to the Glossary

Adding New Terms

1. Alphabetical placement in appropriate section

2. Include all standard sections:

   • Definition
   • Where Used
   • Related Concepts
   • Examples (if applicable)
   • Commands (if applicable)
   • See Also (links to docs)

3. Cross-reference in related terms

4. Update Symbol and Acronym Index if applicable

5. Update Cross-Reference Map

Updating Existing Terms

1. Verify changes don’t break cross-references
2. Update “Last Updated” date at top
3. Increment version if major changes
4. Review related terms for consistency

Version History

Maintained By: Documentation Team Review Cycle: Quarterly or when major features are added Feedback: Please report missing or unclear terms via issues

Prerequisites

Before installing the Provisioning Platform, ensure your system meets the following requirements.

Hardware Requirements

Minimum Requirements (Solo Mode)

• CPU: 2 cores
• RAM: 4GB
• Disk: 20GB available space
• Network: Internet connection for downloading dependencies

Recommended Requirements (Multi-User Mode)

• CPU: 4 cores
• RAM: 8GB
• Disk: 50GB available space
• Network: Reliable internet connection

Production Requirements (Enterprise Mode)

• CPU: 16 cores
• RAM: 32GB
• Disk: 500GB available space (SSD recommended)
• Network: High-bandwidth connection with static IP

Operating System

Supported Platforms

• macOS: 12.0 (Monterey) or later
• Linux:
  • Ubuntu 22.04 LTS or later
  • Fedora 38 or later
  • Debian 12 (Bookworm) or later
  • RHEL 9 or later

Platform-Specific Notes

macOS:

• Xcode Command Line Tools required
• Homebrew recommended for package management

Linux:

• systemd-based distribution recommended
• sudo access required for some operations

Required Software

Core Dependencies

Optional Dependencies

Installation Verification

Before proceeding, verify your system has the core dependencies installed:

Nushell

# Check Nushell version
nu --version

# Expected output: 0.107.1 or higher

KCL

# Check KCL version
kcl --version

# Expected output: 0.11.2 or higher

Docker

# Check Docker version
docker --version

# Check Docker is running
docker ps

# Expected: Docker version 20.10+ and connection successful

SOPS

# Check SOPS version
sops --version

# Expected output: 3.10.2 or higher

Age

# Check Age version
age --version

# Expected output: 1.2.1 or higher

Installing Missing Dependencies

macOS (using Homebrew)

# Install Homebrew if not already installed
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install Nushell
brew install nushell

# Install KCL
brew install kcl

# Install Docker Desktop
brew install --cask docker

# Install SOPS
brew install sops

# Install Age
brew install age

# Optional: Install extras
brew install k9s glow bat

Ubuntu/Debian

# Update package list
sudo apt update

# Install prerequisites
sudo apt install -y curl git build-essential

# Install Nushell (from GitHub releases)
curl -LO https://github.com/nushell/nushell/releases/download/0.107.1/nu-0.107.1-x86_64-linux-musl.tar.gz
tar xzf nu-0.107.1-x86_64-linux-musl.tar.gz
sudo mv nu /usr/local/bin/

# Install KCL
curl -LO https://github.com/kcl-lang/cli/releases/download/v0.11.2/kcl-v0.11.2-linux-amd64.tar.gz
tar xzf kcl-v0.11.2-linux-amd64.tar.gz
sudo mv kcl /usr/local/bin/

# Install Docker
sudo apt install -y docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

# Install SOPS
curl -LO https://github.com/getsops/sops/releases/download/v3.10.2/sops-v3.10.2.linux.amd64
chmod +x sops-v3.10.2.linux.amd64
sudo mv sops-v3.10.2.linux.amd64 /usr/local/bin/sops

# Install Age
sudo apt install -y age

Fedora/RHEL

# Install Nushell
sudo dnf install -y nushell

# Install KCL (from releases)
curl -LO https://github.com/kcl-lang/cli/releases/download/v0.11.2/kcl-v0.11.2-linux-amd64.tar.gz
tar xzf kcl-v0.11.2-linux-amd64.tar.gz
sudo mv kcl /usr/local/bin/

# Install Docker
sudo dnf install -y docker
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

# Install SOPS
sudo dnf install -y sops

# Install Age
sudo dnf install -y age

Network Requirements

Firewall Ports

If running platform services, ensure these ports are available:

External Connectivity

The platform requires outbound internet access to:

• Download dependencies and updates
• Pull container images
• Access cloud provider APIs (AWS, UpCloud)
• Fetch extension packages

Cloud Provider Credentials (Optional)

If you plan to use cloud providers, prepare credentials:

AWS

• AWS Access Key ID
• AWS Secret Access Key
• Configured via ~/.aws/credentials or environment variables

UpCloud

• UpCloud username
• UpCloud password
• Configured via environment variables or config files

Next Steps

Once all prerequisites are met, proceed to: → Installation

Installation

This guide walks you through installing the Provisioning Platform on your system.

Overview

The installation process involves:

1. Cloning the repository
2. Installing Nushell plugins
3. Setting up configuration
4. Initializing your first workspace

Estimated time: 15-20 minutes

Step 1: Clone the Repository

# Clone the repository
git clone https://github.com/provisioning/provisioning-platform.git
cd provisioning-platform

# Checkout the latest stable release (optional)
git checkout tags/v3.5.0

Step 2: Install Nushell Plugins

The platform uses several Nushell plugins for enhanced functionality.

Install nu_plugin_tera (Template Rendering)

# Install from crates.io
cargo install nu_plugin_tera

# Register with Nushell
nu -c "plugin add ~/.cargo/bin/nu_plugin_tera; plugin use tera"

Install nu_plugin_kcl (Optional, KCL Integration)

# Install from custom repository
cargo install --git https://repo.jesusperez.pro/jesus/nushell-plugins nu_plugin_kcl

# Register with Nushell
nu -c "plugin add ~/.cargo/bin/nu_plugin_kcl; plugin use kcl"

Verify Plugin Installation

# Start Nushell
nu

# List installed plugins
plugin list

# Expected output should include:
# - tera
# - kcl (if installed)

Step 3: Add CLI to PATH

Make the provisioning command available globally:

# Option 1: Symlink to /usr/local/bin (recommended)
sudo ln -s "$(pwd)/provisioning/core/cli/provisioning" /usr/local/bin/provisioning

# Option 2: Add to PATH in your shell profile
echo 'export PATH="$PATH:'"$(pwd)"'/provisioning/core/cli"' >> ~/.bashrc  # or ~/.zshrc
source ~/.bashrc  # or ~/.zshrc

# Verify installation
provisioning --version

Step 4: Generate Age Encryption Keys

Generate keys for encrypting sensitive configuration:

# Create Age key directory
mkdir -p ~/.config/provisioning/age

# Generate private key
age-keygen -o ~/.config/provisioning/age/private_key.txt

# Extract public key
age-keygen -y ~/.config/provisioning/age/private_key.txt > ~/.config/provisioning/age/public_key.txt

# Secure the keys
chmod 600 ~/.config/provisioning/age/private_key.txt
chmod 644 ~/.config/provisioning/age/public_key.txt

Step 5: Configure Environment

Set up basic environment variables:

# Create environment file
cat > ~/.provisioning/env << 'ENVEOF'
# Provisioning Environment Configuration
export PROVISIONING_ENV=dev
export PROVISIONING_PATH=$(pwd)
export PROVISIONING_KAGE=~/.config/provisioning/age
ENVEOF

# Source the environment
source ~/.provisioning/env

# Add to shell profile for persistence
echo 'source ~/.provisioning/env' >> ~/.bashrc  # or ~/.zshrc

Step 6: Initialize Workspace

Create your first workspace:

# Initialize a new workspace
provisioning workspace init my-first-workspace

# Expected output:
# ✓ Workspace 'my-first-workspace' created successfully
# ✓ Configuration template generated
# ✓ Workspace activated

# Verify workspace
provisioning workspace list

Step 7: Validate Installation

Run the installation verification:

# Check system configuration
provisioning validate config

# Check all dependencies
provisioning env

# View detailed environment
provisioning allenv

Expected output should show:

• ✅ All core dependencies installed
• ✅ Age keys configured
• ✅ Workspace initialized
• ✅ Configuration valid

Optional: Install Platform Services

If you plan to use platform services (orchestrator, control center, etc.):

# Build platform services
cd provisioning/platform

# Build orchestrator
cd orchestrator
cargo build --release
cd ..

# Build control center
cd control-center
cargo build --release
cd ..

# Build KMS service
cd kms-service
cargo build --release
cd ..

# Verify builds
ls */target/release/

Optional: Install Platform with Installer

Use the interactive installer for a guided setup:

# Build the installer
cd provisioning/platform/installer
cargo build --release

# Run interactive installer
./target/release/provisioning-installer

# Or headless installation
./target/release/provisioning-installer --headless --mode solo --yes

Troubleshooting

Nushell Plugin Not Found

If plugins aren’t recognized:

# Rebuild plugin registry
nu -c "plugin list; plugin use tera"

Permission Denied

If you encounter permission errors:

# Ensure proper ownership
sudo chown -R $USER:$USER ~/.config/provisioning

# Check PATH
echo $PATH | grep provisioning

Age Keys Not Found

If encryption fails:

# Verify keys exist
ls -la ~/.config/provisioning/age/

# Regenerate if needed
age-keygen -o ~/.config/provisioning/age/private_key.txt

Next Steps

Once installation is complete, proceed to: → First Deployment

Additional Resources

• Detailed Installation Guide
• Workspace Management
• Troubleshooting Guide

First Deployment

This guide walks you through deploying your first infrastructure using the Provisioning Platform.

Overview

In this chapter, you’ll:

1. Configure a simple infrastructure
2. Create your first server
3. Install a task service (Kubernetes)
4. Verify the deployment

Estimated time: 10-15 minutes

Step 1: Configure Infrastructure

Create a basic infrastructure configuration:

# Generate infrastructure template
provisioning generate infra --new my-infra

# This creates: workspace/infra/my-infra/
# - config.toml (infrastructure settings)
# - settings.k (KCL configuration)

Step 2: Edit Configuration

Edit the generated configuration:

# Edit with your preferred editor
$EDITOR workspace/infra/my-infra/settings.k

Example configuration:

import provisioning.settings as cfg

# Infrastructure settings
infra_settings = cfg.InfraSettings {
    name = "my-infra"
    provider = "local"  # Start with local provider
    environment = "development"
}

# Server configuration
servers = [
    {
        hostname = "dev-server-01"
        cores = 2
        memory = 4096  # MB
        disk = 50  # GB
    }
]

Step 3: Create Server (Check Mode)

First, run in check mode to see what would happen:

# Check mode - no actual changes
provisioning server create --infra my-infra --check

# Expected output:
# ✓ Validation passed
# ⚠ Check mode: No changes will be made
# 
# Would create:
# - Server: dev-server-01 (2 cores, 4GB RAM, 50GB disk)

Step 4: Create Server (Real)

If check mode looks good, create the server:

# Create server
provisioning server create --infra my-infra

# Expected output:
# ✓ Creating server: dev-server-01
# ✓ Server created successfully
# ✓ IP Address: 192.168.1.100
# ✓ SSH access: ssh user@192.168.1.100

Step 5: Verify Server

Check server status:

# List all servers
provisioning server list

# Get detailed server info
provisioning server info dev-server-01

# SSH to server (optional)
provisioning server ssh dev-server-01

Step 6: Install Kubernetes (Check Mode)

Install a task service on the server:

# Check mode first
provisioning taskserv create kubernetes --infra my-infra --check

# Expected output:
# ✓ Validation passed
# ⚠ Check mode: No changes will be made
#
# Would install:
# - Kubernetes v1.28.0
# - Required dependencies: containerd, etcd
# - On servers: dev-server-01

Step 7: Install Kubernetes (Real)

Proceed with installation:

# Install Kubernetes
provisioning taskserv create kubernetes --infra my-infra --wait

# This will:
# 1. Check dependencies
# 2. Install containerd
# 3. Install etcd
# 4. Install Kubernetes
# 5. Configure and start services

# Monitor progress
provisioning workflow monitor <task-id>

Step 8: Verify Installation

Check that Kubernetes is running:

# List installed task services
provisioning taskserv list --infra my-infra

# Check Kubernetes status
provisioning server ssh dev-server-01
kubectl get nodes  # On the server
exit

# Or remotely
provisioning server exec dev-server-01 -- kubectl get nodes

Common Deployment Patterns

Pattern 1: Multiple Servers

Create multiple servers at once:

servers = [
    {hostname = "web-01", cores = 2, memory = 4096},
    {hostname = "web-02", cores = 2, memory = 4096},
    {hostname = "db-01", cores = 4, memory = 8192}
]

provisioning server create --infra my-infra --servers web-01,web-02,db-01

Pattern 2: Server with Multiple Task Services

Install multiple services on one server:

provisioning taskserv create kubernetes,cilium,postgres --infra my-infra --servers web-01

Pattern 3: Complete Cluster

Deploy a complete cluster configuration:

provisioning cluster create buildkit --infra my-infra

Deployment Workflow

The typical deployment workflow:

# 1. Initialize workspace
provisioning workspace init production

# 2. Generate infrastructure
provisioning generate infra --new prod-infra

# 3. Configure (edit settings.k)
$EDITOR workspace/infra/prod-infra/settings.k

# 4. Validate configuration
provisioning validate config --infra prod-infra

# 5. Create servers (check mode)
provisioning server create --infra prod-infra --check

# 6. Create servers (real)
provisioning server create --infra prod-infra

# 7. Install task services
provisioning taskserv create kubernetes --infra prod-infra --wait

# 8. Deploy cluster (if needed)
provisioning cluster create my-cluster --infra prod-infra

# 9. Verify
provisioning server list
provisioning taskserv list

Troubleshooting

Server Creation Fails

# Check logs
provisioning server logs dev-server-01

# Try with debug mode
provisioning --debug server create --infra my-infra

Task Service Installation Fails

# Check task service logs
provisioning taskserv logs kubernetes

# Retry installation
provisioning taskserv create kubernetes --infra my-infra --force

SSH Connection Issues

# Verify SSH key
ls -la ~/.ssh/

# Test SSH manually
ssh -v user@<server-ip>

# Use provisioning SSH helper
provisioning server ssh dev-server-01 --debug

Next Steps

Now that you’ve completed your first deployment: → Verification - Verify your deployment is working correctly

Additional Resources

• Complete Deployment Guide
• Infrastructure Management
• Troubleshooting Guide

Verification

This guide helps you verify that your Provisioning Platform deployment is working correctly.

Overview

After completing your first deployment, verify:

1. System configuration
2. Server accessibility
3. Task service health
4. Platform services (if installed)

Step 1: Verify Configuration

Check that all configuration is valid:

# Validate all configuration
provisioning validate config

# Expected output:
# ✓ Configuration valid
# ✓ No errors found
# ✓ All required fields present

# Check environment variables
provisioning env

# View complete configuration
provisioning allenv

Step 2: Verify Servers

Check that servers are accessible and healthy:

# List all servers
provisioning server list

# Expected output:
# ┌───────────────┬──────────┬───────┬────────┬──────────────┬──────────┐
# │ Hostname      │ Provider │ Cores │ Memory │ IP Address   │ Status   │
# ├───────────────┼──────────┼───────┼────────┼──────────────┼──────────┤
# │ dev-server-01 │ local    │ 2     │ 4096   │ 192.168.1.100│ running  │
# └───────────────┴──────────┴───────┴────────┴──────────────┴──────────┘

# Check server details
provisioning server info dev-server-01

# Test SSH connectivity
provisioning server ssh dev-server-01 -- echo "SSH working"

Step 3: Verify Task Services

Check installed task services:

# List task services
provisioning taskserv list

# Expected output:
# ┌────────────┬─────────┬────────────────┬──────────┐
# │ Name       │ Version │ Server         │ Status   │
# ├────────────┼─────────┼────────────────┼──────────┤
# │ containerd │ 1.7.0   │ dev-server-01  │ running  │
# │ etcd       │ 3.5.0   │ dev-server-01  │ running  │
# │ kubernetes │ 1.28.0  │ dev-server-01  │ running  │
# └────────────┴─────────┴────────────────┴──────────┘

# Check specific task service
provisioning taskserv status kubernetes

# View task service logs
provisioning taskserv logs kubernetes --tail 50

Step 4: Verify Kubernetes (If Installed)

If you installed Kubernetes, verify it’s working:

# Check Kubernetes nodes
provisioning server ssh dev-server-01 -- kubectl get nodes

# Expected output:
# NAME            STATUS   ROLES           AGE   VERSION
# dev-server-01   Ready    control-plane   10m   v1.28.0

# Check Kubernetes pods
provisioning server ssh dev-server-01 -- kubectl get pods -A

# All pods should be Running or Completed

Step 5: Verify Platform Services (Optional)

If you installed platform services:

Orchestrator

# Check orchestrator health
curl http://localhost:8080/health

# Expected:
# {"status":"healthy","version":"0.1.0"}

# List tasks
curl http://localhost:8080/tasks

Control Center

# Check control center health
curl http://localhost:9090/health

# Test policy evaluation
curl -X POST http://localhost:9090/policies/evaluate \
  -H "Content-Type: application/json" \
  -d '{"principal":{"id":"test"},"action":{"id":"read"},"resource":{"id":"test"}}'

KMS Service

# Check KMS health
curl http://localhost:8082/api/v1/kms/health

# Test encryption
echo "test" | provisioning kms encrypt

Step 6: Run Health Checks

Run comprehensive health checks:

# Check all components
provisioning health check

# Expected output:
# ✓ Configuration: OK
# ✓ Servers: 1/1 healthy
# ✓ Task Services: 3/3 running
# ✓ Platform Services: 3/3 healthy
# ✓ Network Connectivity: OK
# ✓ Encryption Keys: OK

Step 7: Verify Workflows

If you used workflows:

# List all workflows
provisioning workflow list

# Check specific workflow
provisioning workflow status <workflow-id>

# View workflow stats
provisioning workflow stats

Common Verification Checks

DNS Resolution (If CoreDNS Installed)

# Test DNS resolution
dig @localhost test.provisioning.local

# Check CoreDNS status
provisioning server ssh dev-server-01 -- systemctl status coredns

Network Connectivity

# Test server-to-server connectivity
provisioning server ssh dev-server-01 -- ping -c 3 dev-server-02

# Check firewall rules
provisioning server ssh dev-server-01 -- sudo iptables -L

Storage and Resources

# Check disk usage
provisioning server ssh dev-server-01 -- df -h

# Check memory usage
provisioning server ssh dev-server-01 -- free -h

# Check CPU usage
provisioning server ssh dev-server-01 -- top -bn1 | head -20

Troubleshooting Failed Verifications

Configuration Validation Failed

# View detailed error
provisioning validate config --verbose

# Check specific infrastructure
provisioning validate config --infra my-infra

Server Unreachable

# Check server logs
provisioning server logs dev-server-01

# Try debug mode
provisioning --debug server ssh dev-server-01

Task Service Not Running

# Check service logs
provisioning taskserv logs kubernetes

# Restart service
provisioning taskserv restart kubernetes --infra my-infra

Platform Service Down

# Check service status
provisioning platform status orchestrator

# View service logs
provisioning platform logs orchestrator --tail 100

# Restart service
provisioning platform restart orchestrator

Performance Verification

Response Time Tests

# Measure server response time
time provisioning server info dev-server-01

# Measure task service response time
time provisioning taskserv list

# Measure workflow submission time
time provisioning workflow submit test-workflow.k

Resource Usage

# Check platform resource usage
docker stats  # If using Docker

# Check system resources
provisioning system resources

Security Verification

Encryption

# Verify encryption keys
ls -la ~/.config/provisioning/age/

# Test encryption/decryption
echo "test" | provisioning kms encrypt | provisioning kms decrypt

Authentication (If Enabled)

# Test login
provisioning login --username admin

# Verify token
provisioning whoami

# Test MFA (if enabled)
provisioning mfa verify <code>

Verification Checklist

Use this checklist to ensure everything is working:

• Configuration validation passes
• All servers are accessible via SSH
• All servers show “running” status
• All task services show “running” status
• Kubernetes nodes are “Ready” (if installed)
• Kubernetes pods are “Running” (if installed)
• Platform services respond to health checks
• Encryption/decryption works
• Workflows can be submitted and complete
• No errors in logs
• Resource usage is within expected limits

Next Steps

Once verification is complete:

• User Guide - Learn advanced features
• Quick Reference - Command shortcuts
• Infrastructure Management - Day-to-day operations
• Troubleshooting - Common issues and solutions

Additional Resources

• Complete From-Scratch Guide
• Service Management Guide
• Test Environment Guide

Congratulations! You’ve successfully deployed and verified your first Provisioning Platform infrastructure!

Overview

Quick Start

This guide has moved to a multi-chapter format for better readability.

📖 Navigate to Quick Start Guide

Please see the complete quick start guide here:

• Prerequisites - System requirements and setup
• Installation - Install provisioning platform
• First Deployment - Deploy your first infrastructure
• Verification - Verify your deployment

Quick Commands

# Check system status
provisioning status

# Get next step suggestions
provisioning next

# View interactive guide
provisioning guide from-scratch

For the complete step-by-step walkthrough, start with Prerequisites.

Command Reference

Complete command reference for the provisioning CLI.

📖 Service Management Guide

The primary command reference is now part of the Service Management Guide:

→ Service Management Guide - Complete CLI reference

This guide includes:

• All CLI commands and shortcuts
• Command syntax and examples
• Service lifecycle management
• Troubleshooting commands

Quick Reference

Essential Commands

# System status
provisioning status
provisioning health

# Server management
provisioning server create
provisioning server list
provisioning server ssh <hostname>

# Task services
provisioning taskserv create <service>
provisioning taskserv list

# Workspace management
provisioning workspace list
provisioning workspace switch <name>

# Get help
provisioning help
provisioning <command> help

Additional References

• Service Management Guide - Complete CLI reference
• Service Management Quick Reference - Quick lookup
• Quick Start Cheatsheet - All shortcuts
• Authentication Guide - Auth commands

For complete command documentation, see Service Management Guide.

Workspace Guide

Complete guide to workspace management in the provisioning platform.

📖 Workspace Switching Guide

The comprehensive workspace guide is available here:

→ Workspace Switching Guide - Complete workspace documentation

This guide covers:

• Workspace creation and initialization
• Switching between multiple workspaces
• User preferences and configuration
• Workspace registry management
• Backup and restore operations

Quick Start

# List all workspaces
provisioning workspace list

# Switch to a workspace
provisioning workspace switch <name>

# Create new workspace
provisioning workspace init <name>

# Show active workspace
provisioning workspace active

Additional Workspace Resources

• Workspace Switching Guide - Complete guide
• Workspace Configuration - Configuration commands
• Workspace Setup - Initial setup guide

For complete workspace documentation, see Workspace Switching Guide.

CoreDNS Integration Guide

Version: 1.0.0 Date: 2025-10-06 Author: CoreDNS Integration Agent

Table of Contents

1. Overview
2. Installation
3. Configuration
4. CLI Commands
5. Zone Management
6. Record Management
7. Docker Deployment
8. Integration
9. Troubleshooting
10. Advanced Topics

Overview

The CoreDNS integration provides comprehensive DNS management capabilities for the provisioning system. It supports:

• Local DNS service - Run CoreDNS as binary or Docker container
• Dynamic DNS updates - Automatic registration of infrastructure changes
• Multi-zone support - Manage multiple DNS zones
• Provider integration - Seamless integration with orchestrator
• REST API - Programmatic DNS management
• Docker deployment - Containerized CoreDNS with docker-compose

Key Features

✅ Automatic Server Registration - Servers automatically registered in DNS on creation ✅ Zone File Management - Create, update, and manage zone files programmatically ✅ Multiple Deployment Modes - Binary, Docker, remote, or hybrid ✅ Health Monitoring - Built-in health checks and metrics ✅ CLI Interface - Comprehensive command-line tools ✅ API Integration - REST API for external integration

Installation

Prerequisites

• Nushell 0.107+ - For CLI and scripts
• Docker (optional) - For containerized deployment
• dig (optional) - For DNS queries

Install CoreDNS Binary

# Install latest version
provisioning dns install

# Install specific version
provisioning dns install 1.11.1

# Check mode
provisioning dns install --check

The binary will be installed to ~/.provisioning/bin/coredns.

Verify Installation

# Check CoreDNS version
~/.provisioning/bin/coredns -version

# Verify installation
ls -lh ~/.provisioning/bin/coredns

Configuration

KCL Configuration Schema

Add CoreDNS configuration to your infrastructure config:

# In workspace/infra/{name}/config.k
import provisioning.coredns as dns

coredns_config: dns.CoreDNSConfig = {
    mode = "local"

    local = {
        enabled = True
        deployment_type = "binary"  # or "docker"
        binary_path = "~/.provisioning/bin/coredns"
        config_path = "~/.provisioning/coredns/Corefile"
        zones_path = "~/.provisioning/coredns/zones"
        port = 5353
        auto_start = True
        zones = ["provisioning.local", "workspace.local"]
    }

    dynamic_updates = {
        enabled = True
        api_endpoint = "http://localhost:9090/dns"
        auto_register_servers = True
        auto_unregister_servers = True
        ttl = 300
    }

    upstream = ["8.8.8.8", "1.1.1.1"]
    default_ttl = 3600
    enable_logging = True
    enable_metrics = True
    metrics_port = 9153
}

Configuration Modes

Local Mode (Binary)

Run CoreDNS as a local binary process:

coredns_config: CoreDNSConfig = {
    mode = "local"
    local = {
        deployment_type = "binary"
        auto_start = True
    }
}

Local Mode (Docker)

Run CoreDNS in Docker container:

coredns_config: CoreDNSConfig = {
    mode = "local"
    local = {
        deployment_type = "docker"
        docker = {
            image = "coredns/coredns:1.11.1"
            container_name = "provisioning-coredns"
            restart_policy = "unless-stopped"
        }
    }
}

Remote Mode

Connect to external CoreDNS service:

coredns_config: CoreDNSConfig = {
    mode = "remote"
    remote = {
        enabled = True
        endpoints = ["https://dns1.example.com", "https://dns2.example.com"]
        zones = ["production.local"]
        verify_tls = True
    }
}

Disabled Mode

Disable CoreDNS integration:

coredns_config: CoreDNSConfig = {
    mode = "disabled"
}

CLI Commands

Service Management

# Check status
provisioning dns status

# Start service
provisioning dns start

# Start in foreground (for debugging)
provisioning dns start --foreground

# Stop service
provisioning dns stop

# Restart service
provisioning dns restart

# Reload configuration (graceful)
provisioning dns reload

# View logs
provisioning dns logs

# Follow logs
provisioning dns logs --follow

# Show last 100 lines
provisioning dns logs --lines 100

Health & Monitoring

# Check health
provisioning dns health

# View configuration
provisioning dns config show

# Validate configuration
provisioning dns config validate

# Generate new Corefile
provisioning dns config generate

Zone Management

List Zones

# List all zones
provisioning dns zone list

Output:

DNS Zones
=========
  • provisioning.local ✓
  • workspace.local ✓

Create Zone

# Create new zone
provisioning dns zone create myapp.local

# Check mode
provisioning dns zone create myapp.local --check

Show Zone Details

# Show all records in zone
provisioning dns zone show provisioning.local

# JSON format
provisioning dns zone show provisioning.local --format json

# YAML format
provisioning dns zone show provisioning.local --format yaml

Delete Zone

# Delete zone (with confirmation)
provisioning dns zone delete myapp.local

# Force deletion (skip confirmation)
provisioning dns zone delete myapp.local --force

# Check mode
provisioning dns zone delete myapp.local --check

Record Management

Add Records

A Record (IPv4)

provisioning dns record add server-01 A 10.0.1.10

# With custom TTL
provisioning dns record add server-01 A 10.0.1.10 --ttl 600

# With comment
provisioning dns record add server-01 A 10.0.1.10 --comment "Web server"

# Different zone
provisioning dns record add server-01 A 10.0.1.10 --zone myapp.local

AAAA Record (IPv6)

provisioning dns record add server-01 AAAA 2001:db8::1

CNAME Record

provisioning dns record add web CNAME server-01.provisioning.local

MX Record

provisioning dns record add @ MX mail.example.com --priority 10

TXT Record

provisioning dns record add @ TXT "v=spf1 mx -all"

Remove Records

# Remove record
provisioning dns record remove server-01

# Different zone
provisioning dns record remove server-01 --zone myapp.local

# Check mode
provisioning dns record remove server-01 --check

Update Records

# Update record value
provisioning dns record update server-01 A 10.0.1.20

# With new TTL
provisioning dns record update server-01 A 10.0.1.20 --ttl 1800

List Records

# List all records in zone
provisioning dns record list

# Different zone
provisioning dns record list --zone myapp.local

# JSON format
provisioning dns record list --format json

# YAML format
provisioning dns record list --format yaml

Example Output:

DNS Records - Zone: provisioning.local

╭───┬──────────────┬──────┬─────────────┬─────╮
│ # │     name     │ type │    value    │ ttl │
├───┼──────────────┼──────┼─────────────┼─────┤
│ 0 │ server-01    │ A    │ 10.0.1.10   │ 300 │
│ 1 │ server-02    │ A    │ 10.0.1.11   │ 300 │
│ 2 │ db-01        │ A    │ 10.0.2.10   │ 300 │
│ 3 │ web          │ CNAME│ server-01   │ 300 │
╰───┴──────────────┴──────┴─────────────┴─────╯

Docker Deployment

Prerequisites

Ensure Docker and docker-compose are installed:

docker --version
docker-compose --version

Start CoreDNS in Docker

# Start CoreDNS container
provisioning dns docker start

# Check mode
provisioning dns docker start --check

Manage Docker Container

# Check status
provisioning dns docker status

# View logs
provisioning dns docker logs

# Follow logs
provisioning dns docker logs --follow

# Restart container
provisioning dns docker restart

# Stop container
provisioning dns docker stop

# Check health
provisioning dns docker health

Update Docker Image

# Pull latest image
provisioning dns docker pull

# Pull specific version
provisioning dns docker pull --version 1.11.1

# Update and restart
provisioning dns docker update

Remove Container

# Remove container (with confirmation)
provisioning dns docker remove

# Remove with volumes
provisioning dns docker remove --volumes

# Force remove (skip confirmation)
provisioning dns docker remove --force

# Check mode
provisioning dns docker remove --check

View Configuration

# Show docker-compose config
provisioning dns docker config

Integration

Automatic Server Registration

When dynamic DNS is enabled, servers are automatically registered:

# Create server (automatically registers in DNS)
provisioning server create web-01 --infra myapp

# Server gets DNS record: web-01.provisioning.local -> <server-ip>

Manual Registration

use lib_provisioning/coredns/integration.nu *

# Register server
register-server-in-dns "web-01" "10.0.1.10"

# Unregister server
unregister-server-from-dns "web-01"

# Bulk register
bulk-register-servers [
    {hostname: "web-01", ip: "10.0.1.10"}
    {hostname: "web-02", ip: "10.0.1.11"}
    {hostname: "db-01", ip: "10.0.2.10"}
]

Sync Infrastructure with DNS

# Sync all servers in infrastructure with DNS
provisioning dns sync myapp

# Check mode
provisioning dns sync myapp --check

Service Registration

use lib_provisioning/coredns/integration.nu *

# Register service
register-service-in-dns "api" "10.0.1.10"

# Unregister service
unregister-service-from-dns "api"

Query DNS

Using CLI

# Query A record
provisioning dns query server-01

# Query specific type
provisioning dns query server-01 --type AAAA

# Query different server
provisioning dns query server-01 --server 8.8.8.8 --port 53

# Query from local CoreDNS
provisioning dns query server-01 --server 127.0.0.1 --port 5353

Using dig

# Query from local CoreDNS
dig @127.0.0.1 -p 5353 server-01.provisioning.local

# Query CNAME
dig @127.0.0.1 -p 5353 web.provisioning.local CNAME

# Query MX
dig @127.0.0.1 -p 5353 example.com MX

Troubleshooting

CoreDNS Not Starting

Symptoms: dns start fails or service doesn’t respond

Solutions:

1. Check if port is in use:

   lsof -i :5353
   netstat -an | grep 5353
   

2. Validate Corefile:

   provisioning dns config validate
   

3. Check logs:

   provisioning dns logs
   tail -f ~/.provisioning/coredns/coredns.log
   

4. Verify binary exists:

   ls -lh ~/.provisioning/bin/coredns
   provisioning dns install
   

DNS Queries Not Working

Symptoms: dig returns SERVFAIL or timeout

Solutions:

1. Check CoreDNS is running:

   provisioning dns status
   provisioning dns health
   

2. Verify zone file exists:

   ls -lh ~/.provisioning/coredns/zones/
   cat ~/.provisioning/coredns/zones/provisioning.local.zone
   

3. Test with dig:

   dig @127.0.0.1 -p 5353 provisioning.local SOA
   

4. Check firewall:

   # macOS
   sudo pfctl -sr | grep 5353
   
   # Linux
   sudo iptables -L -n | grep 5353
   

Zone File Validation Errors

Symptoms: dns config validate shows errors

Solutions:

1. Backup zone file:

   cp ~/.provisioning/coredns/zones/provisioning.local.zone \
      ~/.provisioning/coredns/zones/provisioning.local.zone.backup
   

2. Regenerate zone:

   provisioning dns zone create provisioning.local --force
   

3. Check syntax manually:

   cat ~/.provisioning/coredns/zones/provisioning.local.zone
   

4. Increment serial:

   • Edit zone file manually
   • Increase serial number in SOA record

Docker Container Issues

Symptoms: Docker container won’t start or crashes

Solutions:

1. Check Docker logs:

   provisioning dns docker logs
   docker logs provisioning-coredns
   

2. Verify volumes exist:

   ls -lh ~/.provisioning/coredns/
   

3. Check container status:

   provisioning dns docker status
   docker ps -a | grep coredns
   

4. Recreate container:

   provisioning dns docker stop
   provisioning dns docker remove --volumes
   provisioning dns docker start
   

Dynamic Updates Not Working

Symptoms: Servers not auto-registered in DNS

Solutions:

1. Check if enabled:

   provisioning dns config show | grep -A 5 dynamic_updates
   

2. Verify orchestrator running:

   curl http://localhost:9090/health
   

3. Check logs for errors:

   provisioning dns logs | grep -i error
   

4. Test manual registration:

   use lib_provisioning/coredns/integration.nu *
   register-server-in-dns "test-server" "10.0.0.1"
   

Advanced Topics

Custom Corefile Plugins

Add custom plugins to Corefile:

use lib_provisioning/coredns/corefile.nu *

# Add plugin to zone
add-corefile-plugin \
    "~/.provisioning/coredns/Corefile" \
    "provisioning.local" \
    "cache 30"

Backup and Restore

# Backup configuration
tar czf coredns-backup.tar.gz ~/.provisioning/coredns/

# Restore configuration
tar xzf coredns-backup.tar.gz -C ~/

Zone File Backup

use lib_provisioning/coredns/zones.nu *

# Backup zone
backup-zone-file "provisioning.local"

# Creates: ~/.provisioning/coredns/zones/provisioning.local.zone.YYYYMMDD-HHMMSS.bak

Metrics and Monitoring

CoreDNS exposes Prometheus metrics on port 9153:

# View metrics
curl http://localhost:9153/metrics

# Common metrics:
# - coredns_dns_request_duration_seconds
# - coredns_dns_requests_total
# - coredns_dns_responses_total

Multi-Zone Setup

coredns_config: CoreDNSConfig = {
    local = {
        zones = [
            "provisioning.local",
            "workspace.local",
            "dev.local",
            "staging.local",
            "prod.local"
        ]
    }
}

Split-Horizon DNS

Configure different zones for internal/external:

coredns_config: CoreDNSConfig = {
    local = {
        zones = ["internal.local"]
        port = 5353
    }
    remote = {
        zones = ["external.com"]
        endpoints = ["https://dns.external.com"]
    }
}

Configuration Reference

CoreDNSConfig Fields

LocalCoreDNS Fields

DynamicDNS Fields

Examples

Complete Setup Example

# 1. Install CoreDNS
provisioning dns install

# 2. Generate configuration
provisioning dns config generate

# 3. Start service
provisioning dns start

# 4. Create custom zone
provisioning dns zone create myapp.local

# 5. Add DNS records
provisioning dns record add web-01 A 10.0.1.10
provisioning dns record add web-02 A 10.0.1.11
provisioning dns record add api CNAME web-01.myapp.local --zone myapp.local

# 6. Query records
provisioning dns query web-01 --server 127.0.0.1 --port 5353

# 7. Check status
provisioning dns status
provisioning dns health

Docker Deployment Example

# 1. Start CoreDNS in Docker
provisioning dns docker start

# 2. Check status
provisioning dns docker status

# 3. View logs
provisioning dns docker logs --follow

# 4. Add records (container must be running)
provisioning dns record add server-01 A 10.0.1.10

# 5. Query
dig @127.0.0.1 -p 5353 server-01.provisioning.local

# 6. Stop
provisioning dns docker stop

Best Practices

1. Use TTL wisely - Lower TTL (300s) for frequently changing records, higher (3600s) for stable
2. Enable logging - Essential for troubleshooting
3. Regular backups - Backup zone files before major changes
4. Validate before reload - Always run dns config validate before reloading
5. Monitor metrics - Track DNS query rates and error rates
6. Use comments - Add comments to records for documentation
7. Separate zones - Use different zones for different environments (dev, staging, prod)

See Also

• Architecture Documentation
• API Reference
• Orchestrator Integration
• KCL Schema Reference

Last Updated: 2025-10-06 Version: 1.0.0

Service Management Guide

Version: 1.0.0 Last Updated: 2025-10-06

Table of Contents

1. Overview
2. Service Architecture
3. Service Registry
4. Platform Commands
5. Service Commands
6. Deployment Modes
7. Health Monitoring
8. Dependency Management
9. Pre-flight Checks
10. Troubleshooting

Overview

The Service Management System provides comprehensive lifecycle management for all platform services (orchestrator, control-center, CoreDNS, Gitea, OCI registry, MCP server, API gateway).

Key Features

• Unified Service Management: Single interface for all services
• Automatic Dependency Resolution: Start services in correct order
• Health Monitoring: Continuous health checks with automatic recovery
• Multiple Deployment Modes: Binary, Docker, Docker Compose, Kubernetes, Remote
• Pre-flight Checks: Validate prerequisites before operations
• Service Registry: Centralized service configuration

Supported Services

Service Architecture

System Architecture

┌─────────────────────────────────────────┐
│         Service Management CLI          │
│  (platform/services commands)           │
└─────────────────┬───────────────────────┘
                  │
       ┌──────────┴──────────┐
       │                     │
       ▼                     ▼
┌──────────────┐    ┌───────────────┐
│   Manager    │    │   Lifecycle   │
│   (Core)     │    │   (Start/Stop)│
└──────┬───────┘    └───────┬───────┘
       │                    │
       ▼                    ▼
┌──────────────┐    ┌───────────────┐
│   Health     │    │  Dependencies │
│   (Checks)   │    │  (Resolution) │
└──────────────┘    └───────────────┘
       │                    │
       └────────┬───────────┘
                │
                ▼
       ┌────────────────┐
       │   Pre-flight   │
       │   (Validation) │
       └────────────────┘

Component Responsibilities

Manager (manager.nu)

• Service registry loading
• Service status tracking
• State persistence

Lifecycle (lifecycle.nu)

• Service start/stop operations
• Deployment mode handling
• Process management

Health (health.nu)

• Health check execution
• HTTP/TCP/Command/File checks
• Continuous monitoring

Dependencies (dependencies.nu)

• Dependency graph analysis
• Topological sorting
• Startup order calculation

Pre-flight (preflight.nu)

• Prerequisite validation
• Conflict detection
• Auto-start orchestration

Service Registry

Configuration File

Location: provisioning/config/services.toml

Service Definition Structure

[services.<service-name>]
name = "<service-name>"
type = "platform" | "infrastructure" | "utility"
category = "orchestration" | "auth" | "dns" | "git" | "registry" | "api" | "ui"
description = "Service description"
required_for = ["operation1", "operation2"]
dependencies = ["dependency1", "dependency2"]
conflicts = ["conflicting-service"]

[services.<service-name>.deployment]
mode = "binary" | "docker" | "docker-compose" | "kubernetes" | "remote"

# Mode-specific configuration
[services.<service-name>.deployment.binary]
binary_path = "/path/to/binary"
args = ["--arg1", "value1"]
working_dir = "/working/directory"
env = { KEY = "value" }

[services.<service-name>.health_check]
type = "http" | "tcp" | "command" | "file" | "none"
interval = 10
retries = 3
timeout = 5

[services.<service-name>.health_check.http]
endpoint = "http://localhost:9090/health"
expected_status = 200
method = "GET"

[services.<service-name>.startup]
auto_start = true
start_timeout = 30
start_order = 10
restart_on_failure = true
max_restarts = 3

Example: Orchestrator Service

[services.orchestrator]
name = "orchestrator"
type = "platform"
category = "orchestration"
description = "Rust-based orchestrator for workflow coordination"
required_for = ["server", "taskserv", "cluster", "workflow", "batch"]

[services.orchestrator.deployment]
mode = "binary"

[services.orchestrator.deployment.binary]
binary_path = "${HOME}/.provisioning/bin/provisioning-orchestrator"
args = ["--port", "8080", "--data-dir", "${HOME}/.provisioning/orchestrator/data"]

[services.orchestrator.health_check]
type = "http"

[services.orchestrator.health_check.http]
endpoint = "http://localhost:9090/health"
expected_status = 200

[services.orchestrator.startup]
auto_start = true
start_timeout = 30
start_order = 10

Platform Commands

Platform commands manage all services as a cohesive system.

Start Platform

Start all auto-start services or specific services:

# Start all auto-start services
provisioning platform start

# Start specific services (with dependencies)
provisioning platform start orchestrator control-center

# Force restart if already running
provisioning platform start --force orchestrator

Behavior:

1. Resolves dependencies
2. Calculates startup order (topological sort)
3. Starts services in correct order
4. Waits for health checks
5. Reports success/failure

Stop Platform

Stop all running services or specific services:

# Stop all running services
provisioning platform stop

# Stop specific services
provisioning platform stop orchestrator control-center

# Force stop (kill -9)
provisioning platform stop --force orchestrator

Behavior:

1. Checks for dependent services
2. Stops in reverse dependency order
3. Updates service state
4. Cleans up PID files

Restart Platform

Restart running services:

# Restart all running services
provisioning platform restart

# Restart specific services
provisioning platform restart orchestrator

Platform Status

Show status of all services:

provisioning platform status

Output:

Platform Services Status

Running: 3/7

=== ORCHESTRATION ===
  🟢 orchestrator - running (uptime: 3600s) ✅

=== UI ===
  🟢 control-center - running (uptime: 3550s) ✅

=== DNS ===
  ⚪ coredns - stopped ❓

=== GIT ===
  ⚪ gitea - stopped ❓

=== REGISTRY ===
  ⚪ oci-registry - stopped ❓

=== API ===
  🟢 mcp-server - running (uptime: 3540s) ✅
  ⚪ api-gateway - stopped ❓

Platform Health

Check health of all running services:

provisioning platform health

Output:

Platform Health Check

✅ orchestrator: Healthy - HTTP health check passed
✅ control-center: Healthy - HTTP status 200 matches expected
⚪ coredns: Not running
✅ mcp-server: Healthy - HTTP health check passed

Summary: 3 healthy, 0 unhealthy, 4 not running

Platform Logs

View service logs:

# View last 50 lines
provisioning platform logs orchestrator

# View last 100 lines
provisioning platform logs orchestrator --lines 100

# Follow logs in real-time
provisioning platform logs orchestrator --follow

Service Commands

Individual service management commands.

List Services

# List all services
provisioning services list

# List only running services
provisioning services list --running

# Filter by category
provisioning services list --category orchestration

Output:

name             type          category       status   deployment_mode  auto_start
orchestrator     platform      orchestration  running  binary          true
control-center   platform      ui             stopped  binary          false
coredns          infrastructure dns           stopped  docker          false

Service Status

Get detailed status of a service:

provisioning services status orchestrator

Output:

Service: orchestrator
Type: platform
Category: orchestration
Status: running
Deployment: binary
Health: healthy
Auto-start: true
PID: 12345
Uptime: 3600s
Dependencies: []

Start Service

# Start service (with pre-flight checks)
provisioning services start orchestrator

# Force start (skip checks)
provisioning services start orchestrator --force

Pre-flight Checks:

1. Validate prerequisites (binary exists, Docker running, etc.)
2. Check for conflicts
3. Verify dependencies are running
4. Auto-start dependencies if needed

Stop Service

# Stop service (with dependency check)
provisioning services stop orchestrator

# Force stop (ignore dependents)
provisioning services stop orchestrator --force

Restart Service

provisioning services restart orchestrator

Service Health

Check service health:

provisioning services health orchestrator

Output:

Service: orchestrator
Status: healthy
Healthy: true
Message: HTTP health check passed
Check type: http
Check duration: 15ms

Service Logs

# View logs
provisioning services logs orchestrator

# Follow logs
provisioning services logs orchestrator --follow

# Custom line count
provisioning services logs orchestrator --lines 200

Check Required Services

Check which services are required for an operation:

provisioning services check server

Output:

Operation: server
Required services: orchestrator
All running: true

Service Dependencies

View dependency graph:

# View all dependencies
provisioning services dependencies

# View specific service dependencies
provisioning services dependencies control-center

Validate Services

Validate all service configurations:

provisioning services validate

Output:

Total services: 7
Valid: 6
Invalid: 1

Invalid services:
  ❌ coredns:
    - Docker is not installed or not running

Readiness Report

Get platform readiness report:

provisioning services readiness

Output:

Platform Readiness Report

Total services: 7
Running: 3
Ready to start: 6

Services:
  🟢 orchestrator - platform - orchestration
  🟢 control-center - platform - ui
  🔴 coredns - infrastructure - dns
      Issues: 1
  🟡 gitea - infrastructure - git

Monitor Service

Continuous health monitoring:

# Monitor with default interval (30s)
provisioning services monitor orchestrator

# Custom interval
provisioning services monitor orchestrator --interval 10

Deployment Modes

Binary Deployment

Run services as native binaries.

Configuration:

[services.orchestrator.deployment]
mode = "binary"

[services.orchestrator.deployment.binary]
binary_path = "${HOME}/.provisioning/bin/provisioning-orchestrator"
args = ["--port", "8080"]
working_dir = "${HOME}/.provisioning/orchestrator"
env = { RUST_LOG = "info" }

Process Management:

• PID tracking in ~/.provisioning/services/pids/
• Log output to ~/.provisioning/services/logs/
• State tracking in ~/.provisioning/services/state/

Docker Deployment

Run services as Docker containers.

Configuration:

[services.coredns.deployment]
mode = "docker"

[services.coredns.deployment.docker]
image = "coredns/coredns:1.11.1"
container_name = "provisioning-coredns"
ports = ["5353:53/udp"]
volumes = ["${HOME}/.provisioning/coredns/Corefile:/Corefile:ro"]
restart_policy = "unless-stopped"

Prerequisites:

• Docker daemon running
• Docker CLI installed

Docker Compose Deployment

Run services via Docker Compose.

Configuration:

[services.platform.deployment]
mode = "docker-compose"

[services.platform.deployment.docker_compose]
compose_file = "${HOME}/.provisioning/platform/docker-compose.yaml"
service_name = "orchestrator"
project_name = "provisioning"

File: provisioning/platform/docker-compose.yaml

Kubernetes Deployment

Run services on Kubernetes.

Configuration:

[services.orchestrator.deployment]
mode = "kubernetes"

[services.orchestrator.deployment.kubernetes]
namespace = "provisioning"
deployment_name = "orchestrator"
manifests_path = "${HOME}/.provisioning/k8s/orchestrator/"

Prerequisites:

• kubectl installed and configured
• Kubernetes cluster accessible

Remote Deployment

Connect to remotely-running services.

Configuration:

[services.orchestrator.deployment]
mode = "remote"

[services.orchestrator.deployment.remote]
endpoint = "https://orchestrator.example.com"
tls_enabled = true
auth_token_path = "${HOME}/.provisioning/tokens/orchestrator.token"

Health Monitoring

Health Check Types

HTTP Health Check

[services.orchestrator.health_check]
type = "http"

[services.orchestrator.health_check.http]
endpoint = "http://localhost:9090/health"
expected_status = 200
method = "GET"

TCP Health Check

[services.coredns.health_check]
type = "tcp"

[services.coredns.health_check.tcp]
host = "localhost"
port = 5353

Command Health Check

[services.custom.health_check]
type = "command"

[services.custom.health_check.command]
command = "systemctl is-active myservice"
expected_exit_code = 0

File Health Check

[services.custom.health_check]
type = "file"

[services.custom.health_check.file]
path = "/var/run/myservice.pid"
must_exist = true

Health Check Configuration

• interval: Seconds between checks (default: 10)
• retries: Max retry attempts (default: 3)
• timeout: Check timeout in seconds (default: 5)

Continuous Monitoring

provisioning services monitor orchestrator --interval 30

Output:

Starting health monitoring for orchestrator (interval: 30s)
Press Ctrl+C to stop
2025-10-06 14:30:00 ✅ orchestrator: HTTP health check passed
2025-10-06 14:30:30 ✅ orchestrator: HTTP health check passed
2025-10-06 14:31:00 ✅ orchestrator: HTTP health check passed

Dependency Management

Dependency Graph

Services can depend on other services:

[services.control-center]
dependencies = ["orchestrator"]

[services.api-gateway]
dependencies = ["orchestrator", "control-center", "mcp-server"]

Startup Order

Services start in topological order:

orchestrator (order: 10)
  └─> control-center (order: 20)
       └─> api-gateway (order: 45)

Dependency Resolution

Automatic dependency resolution when starting services:

# Starting control-center automatically starts orchestrator first
provisioning services start control-center

Output:

Starting dependency: orchestrator
✅ Started orchestrator with PID 12345
Waiting for orchestrator to become healthy...
✅ Service orchestrator is healthy
Starting service: control-center
✅ Started control-center with PID 12346
✅ Service control-center is healthy

Conflicts

Services can conflict with each other:

[services.coredns]
conflicts = ["dnsmasq", "systemd-resolved"]

Attempting to start a conflicting service will fail:

provisioning services start coredns

Output:

❌ Pre-flight check failed: conflicts
Conflicting services running: dnsmasq

Reverse Dependencies

Check which services depend on a service:

provisioning services dependencies orchestrator

Output:

## orchestrator
- Type: platform
- Category: orchestration
- Required by:
  - control-center
  - mcp-server
  - api-gateway

Safe Stop

System prevents stopping services with running dependents:

provisioning services stop orchestrator

Output:

❌ Cannot stop orchestrator:
  Dependent services running: control-center, mcp-server, api-gateway
  Use --force to stop anyway

Pre-flight Checks

Purpose

Pre-flight checks ensure services can start successfully before attempting to start them.

Check Types

1. Prerequisites: Binary exists, Docker running, etc.
2. Conflicts: No conflicting services running
3. Dependencies: All dependencies available

Automatic Checks

Pre-flight checks run automatically when starting services:

provisioning services start orchestrator

Check Process:

Running pre-flight checks for orchestrator...
✅ Binary found: /Users/user/.provisioning/bin/provisioning-orchestrator
✅ No conflicts detected
✅ All dependencies available
Starting service: orchestrator

Manual Validation

Validate all services:

provisioning services validate

Validate specific service:

provisioning services status orchestrator

Auto-Start

Services with auto_start = true can be started automatically when needed:

# Orchestrator auto-starts if needed for server operations
provisioning server create

Output:

Starting required services...
✅ Orchestrator started
Creating server...

Troubleshooting

Service Won’t Start

Check prerequisites:

provisioning services validate
provisioning services status <service>

Common issues:

• Binary not found: Check binary_path in config
• Docker not running: Start Docker daemon
• Port already in use: Check for conflicting processes
• Dependencies not running: Start dependencies first

Service Health Check Failing

View health status:

provisioning services health <service>

Check logs:

provisioning services logs <service> --follow

Common issues:

• Service not fully initialized: Wait longer or increase start_timeout
• Wrong health check endpoint: Verify endpoint in config
• Network issues: Check firewall, port bindings

Dependency Issues

View dependency tree:

provisioning services dependencies <service>

Check dependency status:

provisioning services status <dependency>

Start with dependencies:

provisioning platform start <service>

Circular Dependencies

Validate dependency graph:

# This is done automatically but you can check manually
nu -c "use lib_provisioning/services/mod.nu *; validate-dependency-graph"

PID File Stale

If service reports running but isn’t:

# Manual cleanup
rm ~/.provisioning/services/pids/<service>.pid

# Force restart
provisioning services restart <service>

Port Conflicts

Find process using port:

lsof -i :9090

Kill conflicting process:

kill <PID>

Docker Issues

Check Docker status:

docker ps
docker info

View container logs:

docker logs provisioning-<service>

Restart Docker daemon:

# macOS
killall Docker && open /Applications/Docker.app

# Linux
systemctl restart docker

Service Logs

View recent logs:

tail -f ~/.provisioning/services/logs/<service>.log

Search logs:

grep "ERROR" ~/.provisioning/services/logs/<service>.log

Advanced Usage

Custom Service Registration

Add custom services by editing provisioning/config/services.toml.

Integration with Workflows

Services automatically start when required by workflows:

# Orchestrator starts automatically if not running
provisioning workflow submit my-workflow

CI/CD Integration

# GitLab CI
before_script:
  - provisioning platform start orchestrator
  - provisioning services health orchestrator

test:
  script:
    - provisioning test quick kubernetes

Monitoring Integration

Services can integrate with monitoring systems via health endpoints.

Related Documentation

• Orchestrator README
• Test Environment Guide
• Workflow Management

Maintained By: Platform Team Support: GitHub Issues

Service Management Quick Reference

Version: 1.0.0

Platform Commands (Manage All Services)

# Start all auto-start services
provisioning platform start

# Start specific services with dependencies
provisioning platform start control-center mcp-server

# Stop all running services
provisioning platform stop

# Stop specific services
provisioning platform stop orchestrator

# Restart services
provisioning platform restart

# Show platform status
provisioning platform status

# Check platform health
provisioning platform health

# View service logs
provisioning platform logs orchestrator --follow

Service Commands (Individual Services)

# List all services
provisioning services list

# List only running services
provisioning services list --running

# Filter by category
provisioning services list --category orchestration

# Service status
provisioning services status orchestrator

# Start service (with pre-flight checks)
provisioning services start orchestrator

# Force start (skip checks)
provisioning services start orchestrator --force

# Stop service
provisioning services stop orchestrator

# Force stop (ignore dependents)
provisioning services stop orchestrator --force

# Restart service
provisioning services restart orchestrator

# Check health
provisioning services health orchestrator

# View logs
provisioning services logs orchestrator --follow --lines 100

# Monitor health continuously
provisioning services monitor orchestrator --interval 30

Dependency & Validation

# View dependency graph
provisioning services dependencies

# View specific service dependencies
provisioning services dependencies control-center

# Validate all services
provisioning services validate

# Check readiness
provisioning services readiness

# Check required services for operation
provisioning services check server

Registered Services

Docker Compose

# Start all services
cd provisioning/platform
docker-compose up -d

# Start specific services
docker-compose up -d orchestrator control-center

# Check status
docker-compose ps

# View logs
docker-compose logs -f orchestrator

# Stop all services
docker-compose down

# Stop and remove volumes
docker-compose down -v

Service State Directories

~/.provisioning/services/
├── pids/          # Process ID files
├── state/         # Service state (JSON)
└── logs/          # Service logs

Health Check Endpoints

Common Workflows

Start Platform for Development

# Start core services
provisioning platform start orchestrator

# Check status
provisioning platform status

# Check health
provisioning platform health

Start Full Platform Stack

# Use Docker Compose
cd provisioning/platform
docker-compose up -d

# Verify
docker-compose ps
provisioning platform health

Debug Service Issues

# Check service status
provisioning services status <service>

# View logs
provisioning services logs <service> --follow

# Check health
provisioning services health <service>

# Validate prerequisites
provisioning services validate

# Restart service
provisioning services restart <service>

Safe Service Shutdown

# Check dependents
nu -c "use lib_provisioning/services/mod.nu *; can-stop-service orchestrator"

# Stop with dependency check
provisioning services stop orchestrator

# Force stop if needed
provisioning services stop orchestrator --force

Troubleshooting

Service Won’t Start

# 1. Check prerequisites
provisioning services validate

# 2. View detailed status
provisioning services status <service>

# 3. Check logs
provisioning services logs <service>

# 4. Verify binary/image exists
ls ~/.provisioning/bin/<service>
docker images | grep <service>

Health Check Failing

# Check endpoint manually
curl http://localhost:9090/health

# View health details
provisioning services health <service>

# Monitor continuously
provisioning services monitor <service> --interval 10

PID File Stale

# Remove stale PID file
rm ~/.provisioning/services/pids/<service>.pid

# Restart service
provisioning services restart <service>

Port Already in Use

# Find process using port
lsof -i :9090

# Kill process
kill <PID>

# Restart service
provisioning services start <service>

Integration with Operations

Server Operations

# Orchestrator auto-starts if needed
provisioning server create

# Manual check
provisioning services check server

Workflow Operations

# Orchestrator auto-starts
provisioning workflow submit my-workflow

# Check status
provisioning services status orchestrator

Test Operations

# Orchestrator required for test environments
provisioning test quick kubernetes

# Pre-flight check
provisioning services check test-env

Advanced Usage

Custom Service Startup Order

Services start based on:

1. Dependency order (topological sort)
2. start_order field (lower = earlier)

Auto-Start Configuration

Edit provisioning/config/services.toml:

[services.<service>.startup]
auto_start = true  # Enable auto-start
start_timeout = 30 # Timeout in seconds
start_order = 10   # Startup priority

Health Check Configuration

[services.<service>.health_check]
type = "http"      # http, tcp, command, file
interval = 10      # Seconds between checks
retries = 3        # Max retry attempts
timeout = 5        # Check timeout

[services.<service>.health_check.http]
endpoint = "http://localhost:9090/health"
expected_status = 200

Key Files

• Service Registry: provisioning/config/services.toml
• KCL Schema: provisioning/kcl/services.k
• Docker Compose: provisioning/platform/docker-compose.yaml
• User Guide: docs/user/SERVICE_MANAGEMENT_GUIDE.md

Getting Help

# View documentation
cat docs/user/SERVICE_MANAGEMENT_GUIDE.md | less

# Run verification
nu provisioning/core/nulib/tests/verify_services.nu

# Check readiness
provisioning services readiness

Quick Tip: Use --help flag with any command for detailed usage information.

Test Environment Guide

Version: 1.0.0 Date: 2025-10-06 Status: Production Ready

Overview

The Test Environment Service provides automated containerized testing for taskservs, servers, and multi-node clusters. Built into the orchestrator, it eliminates manual Docker management and provides realistic test scenarios.

Architecture

┌─────────────────────────────────────────────────┐
│         Orchestrator (port 8080)                │
│  ┌──────────────────────────────────────────┐  │
│  │  Test Orchestrator                       │  │
│  │  • Container Manager (Docker API)        │  │
│  │  • Network Isolation                     │  │
│  │  • Multi-node Topologies                 │  │
│  │  • Test Execution                        │  │
│  └──────────────────────────────────────────┘  │
└─────────────────────────────────────────────────┘
                      ↓
         ┌────────────────────────┐
         │   Docker Containers    │
         │  • Isolated Networks   │
         │  • Resource Limits     │
         │  • Volume Mounts       │
         └────────────────────────┘

Test Environment Types

1. Single Taskserv Test

Test individual taskserv in isolated container.

# Basic test
provisioning test env single kubernetes

# With resource limits
provisioning test env single redis --cpu 2000 --memory 4096

# Auto-start and cleanup
provisioning test quick postgres

2. Server Simulation

Simulate complete server with multiple taskservs.

# Server with taskservs
provisioning test env server web-01 [containerd kubernetes cilium]

# With infrastructure context
provisioning test env server db-01 [postgres redis] --infra prod-stack

3. Cluster Topology

Multi-node cluster simulation from templates.

# 3-node Kubernetes cluster
provisioning test topology load kubernetes_3node | test env cluster kubernetes --auto-start

# etcd cluster
provisioning test topology load etcd_cluster | test env cluster etcd

Quick Start

Prerequisites

1. Docker running:

   docker ps  # Should work without errors
   

2. Orchestrator running:

   cd provisioning/platform/orchestrator
   ./scripts/start-orchestrator.nu --background
   

Basic Workflow

# 1. Quick test (fastest)
provisioning test quick kubernetes

# 2. Or step-by-step
# Create environment
provisioning test env single kubernetes --auto-start

# List environments
provisioning test env list

# Check status
provisioning test env status <env-id>

# View logs
provisioning test env logs <env-id>

# Cleanup
provisioning test env cleanup <env-id>

Topology Templates

Available Templates

# List templates
provisioning test topology list

Using Templates

# Load and use template
provisioning test topology load kubernetes_3node | test env cluster kubernetes

# View template
provisioning test topology load etcd_cluster

Custom Topology

Create my-topology.toml:

[my_cluster]
name = "My Custom Cluster"
cluster_type = "custom"

[[my_cluster.nodes]]
name = "node-01"
role = "primary"
taskservs = ["postgres", "redis"]
[my_cluster.nodes.resources]
cpu_millicores = 2000
memory_mb = 4096

[[my_cluster.nodes]]
name = "node-02"
role = "replica"
taskservs = ["postgres"]
[my_cluster.nodes.resources]
cpu_millicores = 1000
memory_mb = 2048

[my_cluster.network]
subnet = "172.30.0.0/16"

Commands Reference

Environment Management

# Create from config
provisioning test env create <config>

# Single taskserv
provisioning test env single <taskserv> [--cpu N] [--memory MB]

# Server simulation
provisioning test env server <name> <taskservs> [--infra NAME]

# Cluster topology
provisioning test env cluster <type> <topology>

# List environments
provisioning test env list

# Get details
provisioning test env get <env-id>

# Show status
provisioning test env status <env-id>

Test Execution

# Run tests
provisioning test env run <env-id> [--tests [test1, test2]]

# View logs
provisioning test env logs <env-id>

# Cleanup
provisioning test env cleanup <env-id>

Quick Test

# One-command test (create, run, cleanup)
provisioning test quick <taskserv> [--infra NAME]

REST API

Create Environment

curl -X POST http://localhost:9090/test/environments/create \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "type": "single_taskserv",
      "taskserv": "kubernetes",
      "base_image": "ubuntu:22.04",
      "environment": {},
      "resources": {
        "cpu_millicores": 2000,
        "memory_mb": 4096
      }
    },
    "infra": "my-project",
    "auto_start": true,
    "auto_cleanup": false
  }'

List Environments

curl http://localhost:9090/test/environments

Run Tests

curl -X POST http://localhost:9090/test/environments/{id}/run \
  -H "Content-Type: application/json" \
  -d '{
    "tests": [],
    "timeout_seconds": 300
  }'

Cleanup

curl -X DELETE http://localhost:9090/test/environments/{id}

Use Cases

1. Taskserv Development

Test taskserv before deployment:

# Test new taskserv version
provisioning test env single my-taskserv --auto-start

# Check logs
provisioning test env logs <env-id>

2. Multi-Taskserv Integration

Test taskserv combinations:

# Test kubernetes + cilium + containerd
provisioning test env server k8s-test [kubernetes cilium containerd] --auto-start

3. Cluster Validation

Test cluster configurations:

# Test 3-node etcd cluster
provisioning test topology load etcd_cluster | test env cluster etcd --auto-start

4. CI/CD Integration

# .gitlab-ci.yml
test-taskserv:
  stage: test
  script:
    - provisioning test quick kubernetes
    - provisioning test quick redis
    - provisioning test quick postgres

Advanced Features

Resource Limits

# Custom CPU and memory
provisioning test env single postgres \
  --cpu 4000 \
  --memory 8192

Network Isolation

Each environment gets isolated network:

• Subnet: 172.20.0.0/16 (default)
• DNS enabled
• Container-to-container communication

Auto-Cleanup

# Auto-cleanup after tests
provisioning test env single redis --auto-start --auto-cleanup

Multiple Environments

Run tests in parallel:

# Create multiple environments
provisioning test env single kubernetes --auto-start &
provisioning test env single postgres --auto-start &
provisioning test env single redis --auto-start &

wait

# List all
provisioning test env list

Troubleshooting

Docker not running

Error: Failed to connect to Docker

Solution:

# Check Docker
docker ps

# Start Docker daemon
sudo systemctl start docker  # Linux
open -a Docker  # macOS

Orchestrator not running

Error: Connection refused (port 8080)

Solution:

cd provisioning/platform/orchestrator
./scripts/start-orchestrator.nu --background

Environment creation fails

Check logs:

provisioning test env logs <env-id>

Check Docker:

docker ps -a
docker logs <container-id>

Out of resources

Error: Cannot allocate memory

Solution:

# Cleanup old environments
provisioning test env list | each {|env| provisioning test env cleanup $env.id }

# Or cleanup Docker
docker system prune -af

Best Practices

1. Use Templates

Reuse topology templates instead of recreating:

provisioning test topology load kubernetes_3node | test env cluster kubernetes

2. Auto-Cleanup

Always use auto-cleanup in CI/CD:

provisioning test quick <taskserv>  # Includes auto-cleanup

3. Resource Planning

Adjust resources based on needs:

• Development: 1-2 cores, 2GB RAM
• Integration: 2-4 cores, 4-8GB RAM
• Production-like: 4+ cores, 8+ GB RAM

4. Parallel Testing

Run independent tests in parallel:

for taskserv in [kubernetes postgres redis] {
    provisioning test quick $taskserv &
}
wait

Configuration

Default Settings

• Base image: ubuntu:22.04
• CPU: 1000 millicores (1 core)
• Memory: 2048 MB (2GB)
• Network: 172.20.0.0/16

Custom Config

# Override defaults
provisioning test env single postgres \
  --base-image debian:12 \
  --cpu 2000 \
  --memory 4096

Related Documentation

• Test Environment API
• Topology Templates
• Orchestrator Guide
• Taskserv Development

Version History

Maintained By: Infrastructure Team

Test Environment Service - Guía Completa de Uso

Versión: 1.0.0 Fecha: 2025-10-06 Estado: Producción

Índice

1. Introducción
2. Requerimientos
3. Configuración Inicial
4. Guía de Uso Rápido
5. Tipos de Entornos
6. Comandos Detallados
7. Topologías y Templates
8. Casos de Uso Prácticos
9. Integración CI/CD
10. Troubleshooting

Introducción

El Test Environment Service es un sistema de testing containerizado integrado en el orquestador que permite probar:

• ✅ Taskservs individuales - Test aislado de un servicio
• ✅ Servidores completos - Simulación de servidor con múltiples taskservs
• ✅ Clusters multi-nodo - Topologías distribuidas (Kubernetes, etcd, etc.)

¿Por qué usar Test Environments?

• Sin gestión manual de Docker - Todo automatizado
• Entornos aislados - Redes dedicadas, sin interferencias
• Realista - Simula configuraciones de producción
• Rápido - Un comando para crear, probar y limpiar
• CI/CD Ready - Fácil integración en pipelines

Requerimientos

Obligatorios

1. Docker

Versión mínima: Docker 20.10+

# Verificar instalación
docker --version

# Verificar que funciona
docker ps

# Verificar recursos disponibles
docker info | grep -E "CPUs|Total Memory"

Instalación según OS:

macOS:

# Opción 1: Docker Desktop
brew install --cask docker

# Opción 2: OrbStack (más ligero)
brew install orbstack

Linux (Ubuntu/Debian):

# Instalar Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Añadir usuario al grupo docker
sudo usermod -aG docker $USER
newgrp docker

# Verificar
docker ps

Linux (Fedora):

sudo dnf install docker
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

2. Orchestrator

Puerto por defecto: 8080

# Verificar que el orquestador está corriendo
curl http://localhost:9090/health

# Si no está corriendo, iniciarlo
cd provisioning/platform/orchestrator
./scripts/start-orchestrator.nu --background

# Verificar logs
tail -f ./data/orchestrator.log

3. Nushell

Versión mínima: 0.107.1+

# Verificar versión
nu --version

Recursos Recomendados

Verificar recursos disponibles:

# En el sistema
docker info | grep -E "CPUs|Total Memory"

# Recursos usados actualmente
docker stats --no-stream

Opcional pero Recomendado

• jq - Para procesar JSON: brew install jq / apt install jq
• glow - Para visualizar docs: brew install glow
• k9s - Para gestionar K8s tests: brew install k9s

Configuración Inicial

1. Iniciar el Orquestador

# Navegar al directorio del orquestador
cd provisioning/platform/orchestrator

# Opción 1: Iniciar en background (recomendado)
./scripts/start-orchestrator.nu --background

# Opción 2: Iniciar en foreground (para debug)
cargo run --release

# Verificar que está corriendo
curl http://localhost:9090/health
# Respuesta esperada: {"success":true,"data":"Orchestrator is healthy"}

2. Verificar Docker

# Test básico de Docker
docker run --rm hello-world

# Verificar que hay imágenes base (se descargan automáticamente)
docker images | grep ubuntu

3. Configurar Variables de Entorno (opcional)

# Añadir a tu ~/.bashrc o ~/.zshrc
export PROVISIONING_ORCHESTRATOR="http://localhost:9090"
export PROVISIONING_PATH="/ruta/a/provisioning"

4. Verificar Instalación

# Test completo del sistema
provisioning test quick redis

# Debe mostrar:
# 🧪 Quick test for redis
# ✅ Environment ready, running tests...
# ✅ Quick test completed

Guía de Uso Rápido

Test Rápido (Recomendado para empezar)

# Un solo comando: crea, prueba, limpia
provisioning test quick <taskserv>

# Ejemplos
provisioning test quick kubernetes
provisioning test quick postgres
provisioning test quick redis

Flujo Completo Paso a Paso

# 1. Crear entorno
provisioning test env single kubernetes --auto-start

# Retorna: environment_id = "abc-123-def-456"

# 2. Listar entornos
provisioning test env list

# 3. Ver status
provisioning test env status abc-123-def-456

# 4. Ver logs
provisioning test env logs abc-123-def-456

# 5. Limpiar
provisioning test env cleanup abc-123-def-456

Con Auto-Cleanup

# Se limpia automáticamente al terminar
provisioning test env single redis \
  --auto-start \
  --auto-cleanup

Tipos de Entornos

1. Single Taskserv

Test de un solo taskserv en container aislado.

Cuándo usar:

• Desarrollo de nuevo taskserv
• Validación de configuración
• Debug de problemas específicos

Comando:

provisioning test env single <taskserv> [opciones]

# Opciones
--cpu <millicores>        # Default: 1000 (1 core)
--memory <MB>             # Default: 2048 (2GB)
--base-image <imagen>     # Default: ubuntu:22.04
--infra <nombre>          # Contexto de infraestructura
--auto-start              # Ejecutar tests automáticamente
--auto-cleanup            # Limpiar al terminar

Ejemplos:

# Test básico
provisioning test env single kubernetes

# Con más recursos
provisioning test env single postgres --cpu 4000 --memory 8192

# Test completo automatizado
provisioning test env single redis --auto-start --auto-cleanup

# Con contexto de infra
provisioning test env single cilium --infra prod-cluster

2. Server Simulation

Simula servidor completo con múltiples taskservs.

Cuándo usar:

• Test de integración entre taskservs
• Validar dependencias
• Simular servidor de producción

Comando:

provisioning test env server <nombre> <taskservs> [opciones]

# taskservs: lista entre corchetes [ts1 ts2 ts3]

Ejemplos:

# Server con stack de aplicación
provisioning test env server app-01 [containerd kubernetes cilium]

# Server de base de datos
provisioning test env server db-01 [postgres redis]

# Con auto-resolución de dependencias
provisioning test env server web-01 [kubernetes] --auto-start
# Automáticamente incluye: containerd, etcd (dependencias de k8s)

3. Cluster Topology

Cluster multi-nodo con topología definida.

Cuándo usar:

• Test de clusters distribuidos
• Validar HA (High Availability)
• Test de failover
• Simular producción real

Comando:

# Desde template predefinido
provisioning test topology load <template> | test env cluster <tipo> [opciones]

Ejemplos:

# Cluster Kubernetes 3 nodos (1 CP + 2 workers)
provisioning test topology load kubernetes_3node | \
  test env cluster kubernetes --auto-start

# Cluster etcd 3 miembros
provisioning test topology load etcd_cluster | \
  test env cluster etcd

# Cluster K8s single-node
provisioning test topology load kubernetes_single | \
  test env cluster kubernetes

Comandos Detallados

Gestión de Entornos

test env create

Crear entorno desde configuración custom.

provisioning test env create <config> [opciones]

# Opciones
--infra <nombre>      # Infraestructura context
--auto-start          # Iniciar tests automáticamente
--auto-cleanup        # Limpiar al finalizar

test env list

Listar todos los entornos activos.

provisioning test env list

# Salida ejemplo:
# id                    env_type          status    containers
# abc-123               single_taskserv   ready     1
# def-456               cluster_topology  running   3

test env get

Obtener detalles completos de un entorno.

provisioning test env get <env-id>

# Retorna JSON con:
# - Configuración completa
# - Estados de containers
# - IPs asignadas
# - Resultados de tests
# - Logs

test env status

Ver status resumido de un entorno.

provisioning test env status <env-id>

# Muestra:
# - ID y tipo
# - Status actual
# - Containers y sus IPs
# - Resultados de tests

test env run

Ejecutar tests en un entorno.

provisioning test env run <env-id> [opciones]

# Opciones
--tests [test1 test2]   # Tests específicos (default: todos)
--timeout <segundos>    # Timeout para tests

Ejemplo:

# Ejecutar todos los tests
provisioning test env run abc-123

# Tests específicos
provisioning test env run abc-123 --tests [connectivity health]

# Con timeout
provisioning test env run abc-123 --timeout 300

test env logs

Ver logs del entorno.

provisioning test env logs <env-id>

# Muestra:
# - Logs de creación
# - Logs de containers
# - Logs de tests
# - Errores si los hay

test env cleanup

Limpiar y destruir entorno.

provisioning test env cleanup <env-id>

# Elimina:
# - Containers
# - Red dedicada
# - Volúmenes
# - Estado del orquestador

Topologías

test topology list

Listar templates disponibles.

provisioning test topology list

# Salida:
# name
# kubernetes_3node
# kubernetes_single
# etcd_cluster
# containerd_test
# postgres_redis

test topology load

Cargar configuración de template.

provisioning test topology load <nombre>

# Retorna configuración JSON/TOML
# Se puede usar con pipe para crear cluster

Quick Test

test quick

Test rápido todo-en-uno.

provisioning test quick <taskserv> [opciones]

# Hace:
# 1. Crea entorno single taskserv
# 2. Ejecuta tests
# 3. Muestra resultados
# 4. Limpia automáticamente

# Opciones
--infra <nombre>   # Contexto de infraestructura

Ejemplos:

# Test rápido de kubernetes
provisioning test quick kubernetes

# Con contexto
provisioning test quick postgres --infra prod-db

Topologías y Templates

Templates Predefinidos

El sistema incluye 5 templates listos para usar:

1. kubernetes_3node - Cluster K8s HA

# Configuración:
# - 1 Control Plane: etcd, kubernetes, containerd (2 cores, 4GB)
# - 2 Workers: kubernetes, containerd, cilium (2 cores, 2GB cada uno)
# - Red: 172.20.0.0/16

# Uso:
provisioning test topology load kubernetes_3node | \
  test env cluster kubernetes --auto-start

2. kubernetes_single - K8s All-in-One

# Configuración:
# - 1 Nodo: etcd, kubernetes, containerd, cilium (4 cores, 8GB)
# - Red: 172.22.0.0/16

# Uso:
provisioning test topology load kubernetes_single | \
  test env cluster kubernetes

3. etcd_cluster - Cluster etcd

# Configuración:
# - 3 Miembros etcd (1 core, 1GB cada uno)
# - Red: 172.21.0.0/16
# - Cluster configurado automáticamente

# Uso:
provisioning test topology load etcd_cluster | \
  test env cluster etcd --auto-start

4. containerd_test - Containerd standalone

# Configuración:
# - 1 Nodo: containerd (1 core, 2GB)
# - Red: 172.23.0.0/16

# Uso:
provisioning test topology load containerd_test | \
  test env cluster containerd

5. postgres_redis - Stack de DBs

# Configuración:
# - 1 PostgreSQL: (2 cores, 4GB)
# - 1 Redis: (1 core, 1GB)
# - Red: 172.24.0.0/16

# Uso:
provisioning test topology load postgres_redis | \
  test env cluster databases --auto-start

Crear Template Custom

1. Crear archivo TOML:

# /path/to/my-topology.toml

[mi_cluster]
name = "Mi Cluster Custom"
description = "Descripción del cluster"
cluster_type = "custom"

[[mi_cluster.nodes]]
name = "node-01"
role = "primary"
taskservs = ["postgres", "redis"]
[mi_cluster.nodes.resources]
cpu_millicores = 2000
memory_mb = 4096
[mi_cluster.nodes.environment]
POSTGRES_PASSWORD = "secret"

[[mi_cluster.nodes]]
name = "node-02"
role = "replica"
taskservs = ["postgres"]
[mi_cluster.nodes.resources]
cpu_millicores = 1000
memory_mb = 2048

[mi_cluster.network]
subnet = "172.30.0.0/16"
dns_enabled = true

2. Copiar a config:

cp my-topology.toml provisioning/config/test-topologies.toml

3. Usar:

provisioning test topology load mi_cluster | \
  test env cluster custom --auto-start

Casos de Uso Prácticos

Desarrollo de Taskservs

Escenario: Desarrollando nuevo taskserv

# 1. Test inicial
provisioning test quick my-new-taskserv

# 2. Si falla, debug con logs
provisioning test env single my-new-taskserv --auto-start
ENV_ID=$(provisioning test env list | tail -1 | awk '{print $1}')
provisioning test env logs $ENV_ID

# 3. Iterar hasta que funcione

# 4. Cleanup
provisioning test env cleanup $ENV_ID

Validación Pre-Despliegue

Escenario: Validar taskserv antes de producción

# 1. Test con configuración de producción
provisioning test env single kubernetes \
  --cpu 4000 \
  --memory 8192 \
  --infra prod-cluster \
  --auto-start

# 2. Revisar resultados
provisioning test env status <env-id>

# 3. Si pasa, desplegar a producción
provisioning taskserv create kubernetes --infra prod-cluster

Test de Integración

Escenario: Validar stack completo

# Test server con stack de aplicación
provisioning test env server app-stack [nginx postgres redis] \
  --cpu 6000 \
  --memory 12288 \
  --auto-start \
  --auto-cleanup

# El sistema:
# 1. Resuelve dependencias automáticamente
# 2. Crea containers con recursos especificados
# 3. Configura red aislada
# 4. Ejecuta tests de integración
# 5. Limpia todo al terminar

Test de Clusters HA

Escenario: Validar cluster Kubernetes

# 1. Crear cluster 3-nodos
provisioning test topology load kubernetes_3node | \
  test env cluster kubernetes --auto-start

# 2. Obtener env-id
ENV_ID=$(provisioning test env list | grep kubernetes | awk '{print $1}')

# 3. Ver status del cluster
provisioning test env status $ENV_ID

# 4. Ejecutar tests específicos
provisioning test env run $ENV_ID --tests [cluster-health node-ready]

# 5. Logs si hay problemas
provisioning test env logs $ENV_ID

# 6. Cleanup
provisioning test env cleanup $ENV_ID

Troubleshooting de Producción

Escenario: Reproducir issue de producción

# 1. Crear entorno idéntico a producción
# Copiar config de prod a topology custom

# 2. Cargar y ejecutar
provisioning test topology load prod-replica | \
  test env cluster app --auto-start

# 3. Reproducir el issue

# 4. Debug con logs detallados
provisioning test env logs <env-id>

# 5. Fix y re-test

# 6. Cleanup
provisioning test env cleanup <env-id>

Integración CI/CD

GitLab CI

# .gitlab-ci.yml

stages:
  - test
  - deploy

variables:
  ORCHESTRATOR_URL: "http://orchestrator:9090"

# Test stage
test-taskservs:
  stage: test
  image: nushell:latest
  services:
    - docker:dind
  before_script:
    - cd provisioning/platform/orchestrator
    - ./scripts/start-orchestrator.nu --background
    - sleep 5  # Wait for orchestrator
  script:
    # Quick tests
    - provisioning test quick kubernetes
    - provisioning test quick postgres
    - provisioning test quick redis
    # Cluster test
    - provisioning test topology load kubernetes_3node | test env cluster kubernetes --auto-start --auto-cleanup
  after_script:
    # Cleanup any remaining environments
    - provisioning test env list | tail -n +2 | awk '{print $1}' | xargs -I {} provisioning test env cleanup {}

# Integration test
test-integration:
  stage: test
  script:
    - provisioning test env server app-stack [nginx postgres redis] --auto-start --auto-cleanup

# Deploy only if tests pass
deploy-production:
  stage: deploy
  script:
    - provisioning taskserv create kubernetes --infra production
  only:
    - main
  dependencies:
    - test-taskservs
    - test-integration

GitHub Actions

# .github/workflows/test.yml

name: Test Infrastructure

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test-taskservs:
    runs-on: ubuntu-latest

    services:
      docker:
        image: docker:dind

    steps:
      - uses: actions/checkout@v3

      - name: Setup Nushell
        run: |
          cargo install nu

      - name: Start Orchestrator
        run: |
          cd provisioning/platform/orchestrator
          cargo build --release
          ./target/release/provisioning-orchestrator &
          sleep 5
          curl http://localhost:9090/health

      - name: Run Quick Tests
        run: |
          provisioning test quick kubernetes
          provisioning test quick postgres
          provisioning test quick redis

      - name: Run Cluster Test
        run: |
          provisioning test topology load kubernetes_3node | \
            test env cluster kubernetes --auto-start --auto-cleanup

      - name: Cleanup
        if: always()
        run: |
          for env in $(provisioning test env list | tail -n +2 | awk '{print $1}'); do
            provisioning test env cleanup $env
          done

Jenkins Pipeline

// Jenkinsfile

pipeline {
    agent any

    environment {
        ORCHESTRATOR_URL = 'http://localhost:9090'
    }

    stages {
        stage('Setup') {
            steps {
                sh '''
                    cd provisioning/platform/orchestrator
                    ./scripts/start-orchestrator.nu --background
                    sleep 5
                '''
            }
        }

        stage('Quick Tests') {
            parallel {
                stage('Kubernetes') {
                    steps {
                        sh 'provisioning test quick kubernetes'
                    }
                }
                stage('PostgreSQL') {
                    steps {
                        sh 'provisioning test quick postgres'
                    }
                }
                stage('Redis') {
                    steps {
                        sh 'provisioning test quick redis'
                    }
                }
            }
        }

        stage('Integration Test') {
            steps {
                sh '''
                    provisioning test env server app-stack [nginx postgres redis] \
                      --auto-start --auto-cleanup
                '''
            }
        }

        stage('Cluster Test') {
            steps {
                sh '''
                    provisioning test topology load kubernetes_3node | \
                      test env cluster kubernetes --auto-start --auto-cleanup
                '''
            }
        }
    }

    post {
        always {
            sh '''
                # Cleanup all test environments
                provisioning test env list | tail -n +2 | awk '{print $1}' | \
                  xargs -I {} provisioning test env cleanup {}
            '''
        }
    }
}

Troubleshooting

Problemas Comunes

1. “Failed to connect to Docker”

Error:

Error: Failed to connect to Docker daemon

Solución:

# Verificar que Docker está corriendo
docker ps

# Si no funciona, iniciar Docker
# macOS
open -a Docker

# Linux
sudo systemctl start docker

# Verificar que tu usuario está en el grupo docker
groups | grep docker
sudo usermod -aG docker $USER
newgrp docker

2. “Connection refused (port 8080)”

Error:

Error: Connection refused

Solución:

# Verificar orquestador
curl http://localhost:9090/health

# Si no responde, iniciar
cd provisioning/platform/orchestrator
./scripts/start-orchestrator.nu --background

# Verificar logs
tail -f ./data/orchestrator.log

# Verificar que el puerto no está ocupado
lsof -i :9090

3. “Out of memory / resources”

Error:

Error: Cannot allocate memory

Solución:

# Verificar recursos disponibles
docker info | grep -E "CPUs|Total Memory"
docker stats --no-stream

# Limpiar containers antiguos
docker container prune -f

# Limpiar imágenes no usadas
docker image prune -a -f

# Limpiar todo el sistema
docker system prune -af --volumes

# Ajustar límites de Docker (Docker Desktop)
# Settings → Resources → Aumentar Memory/CPU

4. “Network already exists”

Error:

Error: Network test-net-xxx already exists

Solución:

# Listar redes
docker network ls | grep test

# Eliminar red específica
docker network rm test-net-xxx

# Eliminar todas las redes de test
docker network ls | grep test | awk '{print $1}' | xargs docker network rm

5. “Image pull failed”

Error:

Error: Failed to pull image ubuntu:22.04

Solución:

# Verificar conexión a internet
ping docker.io

# Pull manual
docker pull ubuntu:22.04

# Si persiste, usar mirror
# Editar /etc/docker/daemon.json
{
  "registry-mirrors": ["https://mirror.gcr.io"]
}

# Reiniciar Docker
sudo systemctl restart docker

6. “Environment not found”

Error:

Error: Environment abc-123 not found

Solución:

# Listar entornos activos
provisioning test env list

# Verificar logs del orquestador
tail -f provisioning/platform/orchestrator/data/orchestrator.log

# Reiniciar orquestador si es necesario
cd provisioning/platform/orchestrator
./scripts/start-orchestrator.nu --stop
./scripts/start-orchestrator.nu --background

Debug Avanzado

Ver logs de container específico

# 1. Obtener environment
provisioning test env get <env-id>

# 2. Copiar container_id del output

# 3. Ver logs del container
docker logs <container-id>

# 4. Ver logs en tiempo real
docker logs -f <container-id>

Ejecutar comandos dentro del container

# Obtener container ID
CONTAINER_ID=$(provisioning test env get <env-id> | jq -r '.containers[0].container_id')

# Entrar al container
docker exec -it $CONTAINER_ID bash

# O ejecutar comando directo
docker exec $CONTAINER_ID ps aux
docker exec $CONTAINER_ID cat /etc/os-release

Inspeccionar red

# Obtener network ID
NETWORK_ID=$(provisioning test env get <env-id> | jq -r '.network_id')

# Inspeccionar red
docker network inspect $NETWORK_ID

# Ver containers conectados
docker network inspect $NETWORK_ID | jq '.[0].Containers'

Verificar recursos del container

# Stats de un container
docker stats <container-id> --no-stream

# Stats de todos los containers de test
docker stats $(docker ps --filter "label=type=test_container" -q) --no-stream

Mejores Prácticas

1. Siempre usar Auto-Cleanup en CI/CD

# ✅ Bueno
provisioning test quick kubernetes

# ✅ Bueno
provisioning test env single postgres --auto-start --auto-cleanup

# ❌ Malo (deja basura si falla el pipeline)
provisioning test env single postgres --auto-start

2. Ajustar Recursos según Necesidad

# Development: recursos mínimos
provisioning test env single redis --cpu 500 --memory 512

# Integration: recursos medios
provisioning test env single postgres --cpu 2000 --memory 4096

# Production-like: recursos completos
provisioning test env single kubernetes --cpu 4000 --memory 8192

3. Usar Templates para Clusters

# ✅ Bueno: reutilizable, documentado
provisioning test topology load kubernetes_3node | test env cluster kubernetes

# ❌ Malo: configuración manual, propenso a errores
# Crear config manual cada vez

4. Nombrar Entornos Descriptivamente

# Al crear custom configs, usar nombres claros
{
  "type": "server_simulation",
  "server_name": "prod-db-replica-test",  # ✅ Descriptivo
  ...
}

5. Limpiar Regularmente

# Script de limpieza (añadir a cron)
#!/usr/bin/env nu

# Limpiar entornos viejos (>1 hora)
provisioning test env list |
  where created_at < (date now | date subtract 1hr) |
  each {|env| provisioning test env cleanup $env.id }

# Limpiar Docker
docker system prune -f

Referencia Rápida

Comandos Esenciales

# Quick test
provisioning test quick <taskserv>

# Single taskserv
provisioning test env single <taskserv> [--auto-start] [--auto-cleanup]

# Server simulation
provisioning test env server <name> [taskservs]

# Cluster from template
provisioning test topology load <template> | test env cluster <type>

# List & manage
provisioning test env list
provisioning test env status <id>
provisioning test env logs <id>
provisioning test env cleanup <id>

REST API

# Create
curl -X POST http://localhost:9090/test/environments/create \
  -H "Content-Type: application/json" \
  -d @config.json

# List
curl http://localhost:9090/test/environments

# Status
curl http://localhost:9090/test/environments/{id}

# Run tests
curl -X POST http://localhost:9090/test/environments/{id}/run

# Logs
curl http://localhost:9090/test/environments/{id}/logs

# Cleanup
curl -X DELETE http://localhost:9090/test/environments/{id}

Recursos Adicionales

• Documentación de Arquitectura: docs/architecture/test-environment-architecture.md
• API Reference: docs/api/test-environment-api.md
• Topologías: provisioning/config/test-topologies.toml
• Código Fuente: provisioning/platform/orchestrator/src/test_*.rs

Soporte

Issues: https://github.com/tu-org/provisioning/issues Documentación: provisioning help test Logs: provisioning/platform/orchestrator/data/orchestrator.log

Versión del documento: 1.0.0 Última actualización: 2025-10-06

Troubleshooting Guide

This comprehensive troubleshooting guide helps you diagnose and resolve common issues with Infrastructure Automation.

What You’ll Learn

• Common issues and their solutions
• Diagnostic commands and techniques
• Error message interpretation
• Performance optimization
• Recovery procedures
• Prevention strategies

General Troubleshooting Approach

1. Identify the Problem

# Check overall system status
provisioning env
provisioning validate config

# Check specific component status
provisioning show servers --infra my-infra
provisioning taskserv list --infra my-infra --installed

2. Gather Information

# Enable debug mode for detailed output
provisioning --debug <command>

# Check logs and errors
provisioning show logs --infra my-infra

3. Use Diagnostic Commands

# Validate configuration
provisioning validate config --detailed

# Test connectivity
provisioning provider test aws
provisioning network test --infra my-infra

Installation and Setup Issues

Issue: Installation Fails

Symptoms:

• Installation script errors
• Missing dependencies
• Permission denied errors

Diagnosis:

# Check system requirements
uname -a
df -h
whoami

# Check permissions
ls -la /usr/local/
sudo -l

Solutions:

Permission Issues

# Run installer with sudo
sudo ./install-provisioning

# Or install to user directory
./install-provisioning --prefix=$HOME/provisioning
export PATH="$HOME/provisioning/bin:$PATH"

Missing Dependencies

# Ubuntu/Debian
sudo apt update
sudo apt install -y curl wget tar build-essential

# RHEL/CentOS
sudo dnf install -y curl wget tar gcc make

Architecture Issues

# Check architecture
uname -m

# Download correct architecture package
# x86_64: Intel/AMD 64-bit
# arm64: ARM 64-bit (Apple Silicon)
wget https://releases.example.com/provisioning-linux-x86_64.tar.gz

Issue: Command Not Found

Symptoms:

bash: provisioning: command not found

Diagnosis:

# Check if provisioning is installed
which provisioning
ls -la /usr/local/bin/provisioning

# Check PATH
echo $PATH

Solutions:

# Add to PATH
export PATH="/usr/local/bin:$PATH"

# Make permanent (add to shell profile)
echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

# Create symlink if missing
sudo ln -sf /usr/local/provisioning/core/nulib/provisioning /usr/local/bin/provisioning

Issue: Nushell Plugin Errors

Symptoms:

Plugin not found: nu_plugin_kcl
Plugin registration failed

Diagnosis:

# Check Nushell version
nu --version

# Check KCL installation (required for nu_plugin_kcl)
kcl version

# Check plugin registration
nu -c "version | get installed_plugins"

Solutions:

# Install KCL CLI (required for nu_plugin_kcl)
# Download from: https://github.com/kcl-lang/cli/releases

# Re-register plugins
nu -c "plugin add /usr/local/provisioning/plugins/nu_plugin_kcl"
nu -c "plugin add /usr/local/provisioning/plugins/nu_plugin_tera"

# Restart Nushell after plugin registration

Configuration Issues

Issue: Configuration Not Found

Symptoms:

Configuration file not found
Failed to load configuration

Diagnosis:

# Check configuration file locations
provisioning env | grep config

# Check if files exist
ls -la ~/.config/provisioning/
ls -la /usr/local/provisioning/config.defaults.toml

Solutions:

# Initialize user configuration
provisioning init config

# Create missing directories
mkdir -p ~/.config/provisioning

# Copy template
cp /usr/local/provisioning/config-examples/config.user.toml ~/.config/provisioning/config.toml

# Verify configuration
provisioning validate config

Issue: Configuration Validation Errors

Symptoms:

Configuration validation failed
Invalid configuration value
Missing required field

Diagnosis:

# Detailed validation
provisioning validate config --detailed

# Check specific sections
provisioning config show --section paths
provisioning config show --section providers

Solutions:

Path Configuration Issues

# Check base path exists
ls -la /path/to/provisioning

# Update configuration
nano ~/.config/provisioning/config.toml

# Fix paths section
[paths]
base = "/correct/path/to/provisioning"

Provider Configuration Issues

# Test provider connectivity
provisioning provider test aws

# Check credentials
aws configure list  # For AWS
upcloud-cli config  # For UpCloud

# Update provider configuration
[providers.aws]
interface = "CLI"  # or "API"

Issue: Interpolation Failures

Symptoms:

Interpolation pattern not resolved: {{env.VARIABLE}}
Template rendering failed

Diagnosis:

# Test interpolation
provisioning validate interpolation test

# Check environment variables
env | grep VARIABLE

# Debug interpolation
provisioning --debug validate interpolation validate

Solutions:

# Set missing environment variables
export MISSING_VARIABLE="value"

# Use fallback values in configuration
config_value = "{{env.VARIABLE || 'default_value'}}"

# Check interpolation syntax
# Correct: {{env.HOME}}
# Incorrect: ${HOME} or $HOME

Server Management Issues

Issue: Server Creation Fails

Symptoms:

Failed to create server
Provider API error
Insufficient quota

Diagnosis:

# Check provider status
provisioning provider status aws

# Test connectivity
ping api.provider.com
curl -I https://api.provider.com

# Check quota
provisioning provider quota --infra my-infra

# Debug server creation
provisioning --debug server create web-01 --infra my-infra --check

Solutions:

API Authentication Issues

# AWS
aws configure list
aws sts get-caller-identity

# UpCloud
upcloud-cli account show

# Update credentials
aws configure  # For AWS
export UPCLOUD_USERNAME="your-username"
export UPCLOUD_PASSWORD="your-password"

Quota/Limit Issues

# Check current usage
provisioning show costs --infra my-infra

# Request quota increase from provider
# Or reduce resource requirements

# Use smaller instance types
# Reduce number of servers

Network/Connectivity Issues

# Test network connectivity
curl -v https://api.aws.amazon.com
curl -v https://api.upcloud.com

# Check DNS resolution
nslookup api.aws.amazon.com

# Check firewall rules
# Ensure outbound HTTPS (port 443) is allowed

Issue: SSH Access Fails

Symptoms:

Connection refused
Permission denied
Host key verification failed

Diagnosis:

# Check server status
provisioning server list --infra my-infra

# Test SSH manually
ssh -v user@server-ip

# Check SSH configuration
provisioning show servers web-01 --infra my-infra

Solutions:

Connection Issues

# Wait for server to be fully ready
provisioning server list --infra my-infra --status

# Check security groups/firewall
# Ensure SSH (port 22) is allowed

# Use correct IP address
provisioning show servers web-01 --infra my-infra | grep ip

Authentication Issues

# Check SSH key
ls -la ~/.ssh/
ssh-add -l

# Generate new key if needed
ssh-keygen -t ed25519 -f ~/.ssh/provisioning_key

# Use specific key
provisioning server ssh web-01 --key ~/.ssh/provisioning_key --infra my-infra

Host Key Issues

# Remove old host key
ssh-keygen -R server-ip

# Accept new host key
ssh -o StrictHostKeyChecking=accept-new user@server-ip

Task Service Issues

Issue: Service Installation Fails

Symptoms:

Service installation failed
Package not found
Dependency conflicts

Diagnosis:

# Check service prerequisites
provisioning taskserv check kubernetes --infra my-infra

# Debug installation
provisioning --debug taskserv create kubernetes --infra my-infra --check

# Check server resources
provisioning server ssh web-01 --command "free -h && df -h" --infra my-infra

Solutions:

Resource Issues

# Check available resources
provisioning server ssh web-01 --command "
    echo 'Memory:' && free -h
    echo 'Disk:' && df -h
    echo 'CPU:' && nproc
" --infra my-infra

# Upgrade server if needed
provisioning server resize web-01 --plan larger-plan --infra my-infra

Package Repository Issues

# Update package lists
provisioning server ssh web-01 --command "
    sudo apt update && sudo apt upgrade -y
" --infra my-infra

# Check repository connectivity
provisioning server ssh web-01 --command "
    curl -I https://download.docker.com/linux/ubuntu/
" --infra my-infra

Dependency Issues

# Install missing dependencies
provisioning taskserv create containerd --infra my-infra

# Then install dependent service
provisioning taskserv create kubernetes --infra my-infra

Issue: Service Not Running

Symptoms:

Service status: failed
Service not responding
Health check failures

Diagnosis:

# Check service status
provisioning taskserv status kubernetes --infra my-infra

# Check service logs
provisioning taskserv logs kubernetes --infra my-infra

# SSH and check manually
provisioning server ssh web-01 --command "
    sudo systemctl status kubernetes
    sudo journalctl -u kubernetes --no-pager -n 50
" --infra my-infra

Solutions:

Configuration Issues

# Reconfigure service
provisioning taskserv configure kubernetes --infra my-infra

# Reset to defaults
provisioning taskserv reset kubernetes --infra my-infra

Port Conflicts

# Check port usage
provisioning server ssh web-01 --command "
    sudo netstat -tulpn | grep :6443
    sudo ss -tulpn | grep :6443
" --infra my-infra

# Change port configuration or stop conflicting service

Permission Issues

# Fix permissions
provisioning server ssh web-01 --command "
    sudo chown -R kubernetes:kubernetes /var/lib/kubernetes
    sudo chmod 600 /etc/kubernetes/admin.conf
" --infra my-infra

Cluster Management Issues

Issue: Cluster Deployment Fails

Symptoms:

Cluster deployment failed
Pod creation errors
Service unavailable

Diagnosis:

# Check cluster status
provisioning cluster status web-cluster --infra my-infra

# Check Kubernetes cluster
provisioning server ssh master-01 --command "
    kubectl get nodes
    kubectl get pods --all-namespaces
" --infra my-infra

# Check cluster logs
provisioning cluster logs web-cluster --infra my-infra

Solutions:

Node Issues

# Check node status
provisioning server ssh master-01 --command "
    kubectl describe nodes
" --infra my-infra

# Drain and rejoin problematic nodes
provisioning server ssh master-01 --command "
    kubectl drain worker-01 --ignore-daemonsets
    kubectl delete node worker-01
" --infra my-infra

# Rejoin node
provisioning taskserv configure kubernetes --infra my-infra --servers worker-01

Resource Constraints

# Check resource usage
provisioning server ssh master-01 --command "
    kubectl top nodes
    kubectl top pods --all-namespaces
" --infra my-infra

# Scale down or add more nodes
provisioning cluster scale web-cluster --replicas 3 --infra my-infra
provisioning server create worker-04 --infra my-infra

Network Issues

# Check network plugin
provisioning server ssh master-01 --command "
    kubectl get pods -n kube-system | grep cilium
" --infra my-infra

# Restart network plugin
provisioning taskserv restart cilium --infra my-infra

Performance Issues

Issue: Slow Operations

Symptoms:

• Commands take very long to complete
• Timeouts during operations
• High CPU/memory usage

Diagnosis:

# Check system resources
top
htop
free -h
df -h

# Check network latency
ping api.aws.amazon.com
traceroute api.aws.amazon.com

# Profile command execution
time provisioning server list --infra my-infra

Solutions:

Local System Issues

# Close unnecessary applications
# Upgrade system resources
# Use SSD storage if available

# Increase timeout values
export PROVISIONING_TIMEOUT=600  # 10 minutes

Network Issues

# Use region closer to your location
[providers.aws]
region = "us-west-1"  # Closer region

# Enable connection pooling/caching
[cache]
enabled = true

Large Infrastructure Issues

# Use parallel operations
provisioning server create --infra my-infra --parallel 4

# Filter results
provisioning server list --infra my-infra --filter "status == 'running'"

Issue: High Memory Usage

Symptoms:

• System becomes unresponsive
• Out of memory errors
• Swap usage high

Diagnosis:

# Check memory usage
free -h
ps aux --sort=-%mem | head

# Check for memory leaks
valgrind provisioning server list --infra my-infra

Solutions:

# Increase system memory
# Close other applications
# Use streaming operations for large datasets

# Enable garbage collection
export PROVISIONING_GC_ENABLED=true

# Reduce concurrent operations
export PROVISIONING_MAX_PARALLEL=2

Network and Connectivity Issues

Issue: API Connectivity Problems

Symptoms:

Connection timeout
DNS resolution failed
SSL certificate errors

Diagnosis:

# Test basic connectivity
ping 8.8.8.8
curl -I https://api.aws.amazon.com
nslookup api.upcloud.com

# Check SSL certificates
openssl s_client -connect api.aws.amazon.com:443 -servername api.aws.amazon.com

Solutions:

DNS Issues

# Use alternative DNS
echo 'nameserver 8.8.8.8' | sudo tee /etc/resolv.conf

# Clear DNS cache
sudo systemctl restart systemd-resolved  # Ubuntu
sudo dscacheutil -flushcache             # macOS

Proxy/Firewall Issues

# Configure proxy if needed
export HTTP_PROXY=http://proxy.company.com:9090
export HTTPS_PROXY=http://proxy.company.com:9090

# Check firewall rules
sudo ufw status  # Ubuntu
sudo firewall-cmd --list-all  # RHEL/CentOS

Certificate Issues

# Update CA certificates
sudo apt update && sudo apt install ca-certificates  # Ubuntu
brew install ca-certificates                         # macOS

# Skip SSL verification (temporary)
export PROVISIONING_SKIP_SSL_VERIFY=true

Security and Encryption Issues

Issue: SOPS Decryption Fails

Symptoms:

SOPS decryption failed
Age key not found
Invalid key format

Diagnosis:

# Check SOPS configuration
provisioning sops config

# Test SOPS manually
sops -d encrypted-file.k

# Check Age keys
ls -la ~/.config/sops/age/keys.txt
age-keygen -y ~/.config/sops/age/keys.txt

Solutions:

Missing Keys

# Generate new Age key
age-keygen -o ~/.config/sops/age/keys.txt

# Update SOPS configuration
provisioning sops config --key-file ~/.config/sops/age/keys.txt

Key Permissions

# Fix key file permissions
chmod 600 ~/.config/sops/age/keys.txt
chown $(whoami) ~/.config/sops/age/keys.txt

Configuration Issues

# Update SOPS configuration in ~/.config/provisioning/config.toml
[sops]
use_sops = true
key_search_paths = [
    "~/.config/sops/age/keys.txt",
    "/path/to/your/key.txt"
]

Issue: Access Denied Errors

Symptoms:

Permission denied
Access denied
Insufficient privileges

Diagnosis:

# Check user permissions
id
groups

# Check file permissions
ls -la ~/.config/provisioning/
ls -la /usr/local/provisioning/

# Test with sudo
sudo provisioning env

Solutions:

# Fix file ownership
sudo chown -R $(whoami):$(whoami) ~/.config/provisioning/

# Fix permissions
chmod -R 755 ~/.config/provisioning/
chmod 600 ~/.config/provisioning/config.toml

# Add user to required groups
sudo usermod -a -G docker $(whoami)  # For Docker access

Data and Storage Issues

Issue: Disk Space Problems

Symptoms:

No space left on device
Write failed
Disk full

Diagnosis:

# Check disk usage
df -h
du -sh ~/.config/provisioning/
du -sh /usr/local/provisioning/

# Find large files
find /usr/local/provisioning -type f -size +100M

Solutions:

# Clean up cache files
rm -rf ~/.config/provisioning/cache/*
rm -rf /usr/local/provisioning/.cache/*

# Clean up logs
find /usr/local/provisioning -name "*.log" -mtime +30 -delete

# Clean up temporary files
rm -rf /tmp/provisioning-*

# Compress old backups
gzip ~/.config/provisioning/backups/*.yaml

Recovery Procedures

Configuration Recovery

# Restore from backup
provisioning config restore --backup latest

# Reset to defaults
provisioning config reset

# Recreate configuration
provisioning init config --force

Infrastructure Recovery

# Check infrastructure status
provisioning show servers --infra my-infra

# Recover failed servers
provisioning server create failed-server --infra my-infra

# Restore from backup
provisioning restore --backup latest --infra my-infra

Service Recovery

# Restart failed services
provisioning taskserv restart kubernetes --infra my-infra

# Reinstall corrupted services
provisioning taskserv delete kubernetes --infra my-infra
provisioning taskserv create kubernetes --infra my-infra

Prevention Strategies

Regular Maintenance

# Weekly maintenance script
#!/bin/bash

# Update system
provisioning update --check

# Validate configuration
provisioning validate config

# Check for service updates
provisioning taskserv check-updates

# Clean up old files
provisioning cleanup --older-than 30d

# Create backup
provisioning backup create --name "weekly-$(date +%Y%m%d)"

Monitoring Setup

# Set up health monitoring
#!/bin/bash

# Check system health every hour
0 * * * * /usr/local/bin/provisioning health check || echo "Health check failed" | mail -s "Provisioning Alert" admin@company.com

# Weekly cost reports
0 9 * * 1 /usr/local/bin/provisioning show costs --all | mail -s "Weekly Cost Report" finance@company.com

Best Practices

1. Configuration Management

   • Version control all configuration files
   • Use check mode before applying changes
   • Regular validation and testing

2. Security

   • Regular key rotation
   • Principle of least privilege
   • Audit logs review

3. Backup Strategy

   • Automated daily backups
   • Test restore procedures
   • Off-site backup storage

4. Documentation

   • Document custom configurations
   • Keep troubleshooting logs
   • Share knowledge with team

Getting Additional Help

Debug Information Collection

#!/bin/bash
# Collect debug information

echo "Collecting provisioning debug information..."

mkdir -p /tmp/provisioning-debug
cd /tmp/provisioning-debug

# System information
uname -a > system-info.txt
free -h >> system-info.txt
df -h >> system-info.txt

# Provisioning information
provisioning --version > provisioning-info.txt
provisioning env >> provisioning-info.txt
provisioning validate config --detailed > config-validation.txt 2>&1

# Configuration files
cp ~/.config/provisioning/config.toml user-config.toml 2>/dev/null || echo "No user config" > user-config.toml

# Logs
provisioning show logs > system-logs.txt 2>&1

# Create archive
cd /tmp
tar czf provisioning-debug-$(date +%Y%m%d_%H%M%S).tar.gz provisioning-debug/

echo "Debug information collected in: provisioning-debug-*.tar.gz"

Support Channels

1. Built-in Help

   provisioning help
   provisioning help <command>
   

2. Documentation

   • User guides in docs/user/
   • CLI reference: docs/user/cli-reference.md
   • Configuration guide: docs/user/configuration.md

3. Community Resources

   • Project repository issues
   • Community forums
   • Documentation wiki

4. Enterprise Support

   • Professional services
   • Priority support
   • Custom development

Remember: When reporting issues, always include the debug information collected above and specific error messages.

Authentication Layer Implementation Guide

Version: 1.0.0 Date: 2025-10-09 Status: Production Ready

Overview

A comprehensive authentication layer has been integrated into the provisioning system to secure sensitive operations. The system uses nu_plugin_auth for JWT authentication with MFA support, providing enterprise-grade security with graceful user experience.

Key Features

✅ JWT Authentication

• RS256 asymmetric signing
• Access tokens (15min) + refresh tokens (7d)
• OS keyring storage (macOS Keychain, Windows Credential Manager, Linux Secret Service)

✅ MFA Support

• TOTP (Google Authenticator, Authy)
• WebAuthn/FIDO2 (YubiKey, Touch ID)
• Required for production and destructive operations

✅ Security Policies

• Production environment: Requires authentication + MFA
• Destructive operations: Requires authentication + MFA (delete, destroy)
• Development/test: Requires authentication, allows skip with flag
• Check mode: Always bypasses authentication (dry-run operations)

✅ Audit Logging

• All authenticated operations logged
• User, timestamp, operation details
• MFA verification status
• JSON format for easy parsing

✅ User-Friendly Error Messages

• Clear instructions for login/MFA
• Distinct error types (platform auth vs provider auth)
• Helpful guidance for setup

Quick Start

1. Login to Platform

# Interactive login (password prompt)
provisioning auth login <username>

# Save credentials to keyring
provisioning auth login <username> --save

# Custom control center URL
provisioning auth login admin --url http://control.example.com:9080

2. Enroll MFA (First Time)

# Enroll TOTP (Google Authenticator)
provisioning auth mfa enroll totp

# Scan QR code with authenticator app
# Or enter secret manually

3. Verify MFA (For Sensitive Operations)

# Get 6-digit code from authenticator app
provisioning auth mfa verify --code 123456

4. Check Authentication Status

# View current authentication status
provisioning auth status

# Verify token is valid
provisioning auth verify

Protected Operations

Server Operations

# ✅ CREATE - Requires auth (prod: +MFA)
provisioning server create web-01                    # Auth required
provisioning server create web-01 --check            # Auth skipped (check mode)

# ❌ DELETE - Requires auth + MFA
provisioning server delete web-01                    # Auth + MFA required
provisioning server delete web-01 --check            # Auth skipped (check mode)

# 📖 READ - No auth required
provisioning server list                             # No auth required
provisioning server ssh web-01                       # No auth required

Task Service Operations

# ✅ CREATE - Requires auth (prod: +MFA)
provisioning taskserv create kubernetes              # Auth required
provisioning taskserv create kubernetes --check      # Auth skipped

# ❌ DELETE - Requires auth + MFA
provisioning taskserv delete kubernetes              # Auth + MFA required

# 📖 READ - No auth required
provisioning taskserv list                           # No auth required

Cluster Operations

# ✅ CREATE - Requires auth (prod: +MFA)
provisioning cluster create buildkit                 # Auth required
provisioning cluster create buildkit --check         # Auth skipped

# ❌ DELETE - Requires auth + MFA
provisioning cluster delete buildkit                 # Auth + MFA required

Batch Workflows

# ✅ SUBMIT - Requires auth (prod: +MFA)
provisioning batch submit workflow.k                 # Auth required
provisioning batch submit workflow.k --skip-auth     # Auth skipped (if allowed)

# 📖 READ - No auth required
provisioning batch list                              # No auth required
provisioning batch status <task-id>                  # No auth required

Configuration

Security Settings (config.defaults.toml)

[security]
require_auth = true  # Enable authentication system
require_mfa_for_production = true  # MFA for prod environment
require_mfa_for_destructive = true  # MFA for delete operations
auth_timeout = 3600  # Token timeout (1 hour)
audit_log_path = "{{paths.base}}/logs/audit.log"

[security.bypass]
allow_skip_auth = false  # Allow PROVISIONING_SKIP_AUTH env var

[plugins]
auth_enabled = true  # Enable nu_plugin_auth

[platform.control_center]
url = "http://localhost:9080"  # Control center URL

Environment-Specific Configuration

# Development
[environments.dev]
security.bypass.allow_skip_auth = true  # Allow auth bypass in dev

# Production
[environments.prod]
security.bypass.allow_skip_auth = false  # Never allow bypass
security.require_mfa_for_production = true

Authentication Bypass (Dev/Test Only)

Environment Variable Method

# Export environment variable (dev/test only)
export PROVISIONING_SKIP_AUTH=true

# Run operations without authentication
provisioning server create web-01

# Unset when done
unset PROVISIONING_SKIP_AUTH

Per-Command Flag

# Some commands support --skip-auth flag
provisioning batch submit workflow.k --skip-auth

Check Mode (Always Bypasses Auth)

# Check mode is always allowed without auth
provisioning server create web-01 --check
provisioning taskserv create kubernetes --check

⚠️ WARNING: Auth bypass should ONLY be used in development/testing environments. Production systems should have security.bypass.allow_skip_auth = false.

Error Messages

Not Authenticated

❌ Authentication Required

Operation: server create web-01
You must be logged in to perform this operation.

To login:
   provisioning auth login <username>

Note: Your credentials will be securely stored in the system keyring.

Solution: Run provisioning auth login <username>

MFA Required

❌ MFA Verification Required

Operation: server delete web-01
Reason: destructive operation (delete/destroy)

To verify MFA:
   1. Get code from your authenticator app
   2. Run: provisioning auth mfa verify --code <6-digit-code>

Don't have MFA set up?
   Run: provisioning auth mfa enroll totp

Solution: Run provisioning auth mfa verify --code 123456

Token Expired

❌ Authentication Required

Operation: server create web-02
You must be logged in to perform this operation.

Error: Token verification failed

Solution: Token expired, re-login with provisioning auth login <username>

Audit Logging

All authenticated operations are logged to the audit log file with the following information:

{
  "timestamp": "2025-10-09 14:32:15",
  "user": "admin",
  "operation": "server_create",
  "details": {
    "hostname": "web-01",
    "infra": "production",
    "environment": "prod",
    "orchestrated": false
  },
  "mfa_verified": true
}

Viewing Audit Logs

# View raw audit log
cat provisioning/logs/audit.log

# Filter by user
cat provisioning/logs/audit.log | jq '. | select(.user == "admin")'

# Filter by operation type
cat provisioning/logs/audit.log | jq '. | select(.operation == "server_create")'

# Filter by date
cat provisioning/logs/audit.log | jq '. | select(.timestamp | startswith("2025-10-09"))'

Integration with Control Center

The authentication system integrates with the provisioning platform’s control center REST API:

• POST /api/auth/login - Login with credentials
• POST /api/auth/logout - Revoke tokens
• POST /api/auth/verify - Verify token validity
• GET /api/auth/sessions - List active sessions
• POST /api/mfa/enroll - Enroll MFA device
• POST /api/mfa/verify - Verify MFA code

Starting Control Center

# Start control center (required for authentication)
cd provisioning/platform/control-center
cargo run --release

Or use the orchestrator which includes control center:

cd provisioning/platform/orchestrator
./scripts/start-orchestrator.nu --background

Testing Authentication

Manual Testing

# 1. Start control center
cd provisioning/platform/control-center
cargo run --release &

# 2. Login
provisioning auth login admin

# 3. Try creating server (should succeed if authenticated)
provisioning server create test-server --check

# 4. Logout
provisioning auth logout

# 5. Try creating server (should fail - not authenticated)
provisioning server create test-server --check

Automated Testing

# Run authentication tests
nu provisioning/core/nulib/lib_provisioning/plugins/auth_test.nu

Troubleshooting

Plugin Not Available

Error: Authentication plugin not available

Solution:

1. Check plugin is built: ls provisioning/core/plugins/nushell-plugins/nu_plugin_auth/target/release/
2. Register plugin: plugin add target/release/nu_plugin_auth
3. Use plugin: plugin use auth
4. Verify: which auth

Control Center Not Running

Error: Cannot connect to control center

Solution:

1. Start control center: cd provisioning/platform/control-center && cargo run --release
2. Or use orchestrator: cd provisioning/platform/orchestrator && ./scripts/start-orchestrator.nu --background
3. Check URL is correct in config: provisioning config get platform.control_center.url

MFA Not Working

Error: Invalid MFA code

Solutions:

• Ensure time is synchronized (TOTP codes are time-based)
• Code expires every 30 seconds, get fresh code
• Verify you’re using the correct authenticator app entry
• Re-enroll if needed: provisioning auth mfa enroll totp

Keyring Access Issues

Error: Keyring storage unavailable

macOS: Grant Keychain access to Terminal/iTerm2 in System Preferences → Security & Privacy

Linux: Ensure gnome-keyring or kwallet is running

Windows: Check Windows Credential Manager is accessible

Architecture

Authentication Flow

┌─────────────┐
│ User Command│
└──────┬──────┘
       │
       ▼
┌─────────────────────────────────┐
│ Infrastructure Command Handler  │
│ (infrastructure.nu)             │
└──────┬──────────────────────────┘
       │
       ▼
┌─────────────────────────────────┐
│ Auth Check                       │
│ - Determine operation type       │
│ - Check if auth required         │
│ - Check environment (prod/dev)   │
└──────┬──────────────────────────┘
       │
       ▼
┌─────────────────────────────────┐
│ Auth Plugin Wrapper              │
│ (auth.nu)                        │
│ - Call plugin or HTTP fallback   │
│ - Verify token validity          │
│ - Check MFA if required          │
└──────┬──────────────────────────┘
       │
       ▼
┌─────────────────────────────────┐
│ nu_plugin_auth                   │
│ - JWT verification (RS256)       │
│ - Keyring token storage          │
│ - MFA verification               │
└──────┬──────────────────────────┘
       │
       ▼
┌─────────────────────────────────┐
│ Control Center API               │
│ - /api/auth/verify               │
│ - /api/mfa/verify                │
└──────┬──────────────────────────┘
       │
       ▼
┌─────────────────────────────────┐
│ Operation Execution              │
│ (servers/create.nu, etc.)        │
└──────┬──────────────────────────┘
       │
       ▼
┌─────────────────────────────────┐
│ Audit Logging                    │
│ - Log to audit.log               │
│ - Include user, timestamp, MFA   │
└─────────────────────────────────┘

File Structure

provisioning/
├── config/
│   └── config.defaults.toml           # Security configuration
├── core/nulib/
│   ├── lib_provisioning/plugins/
│   │   └── auth.nu                    # Auth wrapper (550 lines)
│   ├── servers/
│   │   └── create.nu                  # Server ops with auth
│   ├── workflows/
│   │   └── batch.nu                   # Batch workflows with auth
│   └── main_provisioning/commands/
│       └── infrastructure.nu          # Infrastructure commands with auth
├── core/plugins/nushell-plugins/
│   └── nu_plugin_auth/                # Native Rust plugin
│       ├── src/
│       │   ├── main.rs                # Plugin implementation
│       │   └── helpers.rs             # Helper functions
│       └── README.md                  # Plugin documentation
├── platform/control-center/           # Control Center (Rust)
│   └── src/auth/                      # JWT auth implementation
└── logs/
    └── audit.log                       # Audit trail

Related Documentation

• Security System Overview: docs/architecture/ADR-009-security-system-complete.md
• JWT Authentication: docs/architecture/JWT_AUTH_IMPLEMENTATION.md
• MFA Implementation: docs/architecture/MFA_IMPLEMENTATION_SUMMARY.md
• Plugin README: provisioning/core/plugins/nushell-plugins/nu_plugin_auth/README.md
• Control Center: provisioning/platform/control-center/README.md

Summary of Changes

Best Practices

For Users

1. Always login: Keep your session active to avoid interruptions
2. Use keyring: Save credentials with --save flag for persistence
3. Enable MFA: Use MFA for production operations
4. Check mode first: Always test with --check before actual operations
5. Monitor audit logs: Review audit logs regularly for security

For Developers

1. Check auth early: Verify authentication before expensive operations
2. Log operations: Always log authenticated operations for audit
3. Clear error messages: Provide helpful guidance for auth failures
4. Respect check mode: Always skip auth in check/dry-run mode
5. Test both paths: Test with and without authentication

For Operators

1. Production hardening: Set allow_skip_auth = false in production
2. MFA enforcement: Require MFA for all production environments
3. Monitor audit logs: Set up log monitoring and alerts
4. Token rotation: Configure short token timeouts (15min default)
5. Backup authentication: Ensure multiple admins have MFA enrolled

License

MIT License - See LICENSE file for details

Last Updated: 2025-10-09 Maintained By: Security Team

Authentication Quick Reference

Version: 1.0.0 Last Updated: 2025-10-09

Quick Commands

Login

provisioning auth login <username>              # Interactive password
provisioning auth login <username> --save       # Save to keyring

MFA

provisioning auth mfa enroll totp               # Enroll TOTP
provisioning auth mfa verify --code 123456      # Verify code

Status

provisioning auth status                        # Show auth status
provisioning auth verify                        # Verify token

Logout

provisioning auth logout                        # Logout current session
provisioning auth logout --all                  # Logout all sessions

Protected Operations

Bypass Authentication (Dev/Test Only)

Environment Variable

export PROVISIONING_SKIP_AUTH=true
provisioning server create test
unset PROVISIONING_SKIP_AUTH

Check Mode (Always Allowed)

provisioning server create prod --check
provisioning taskserv delete k8s --check

Config Flag

[security.bypass]
allow_skip_auth = true  # Only in dev/test

Configuration

Security Settings

[security]
require_auth = true
require_mfa_for_production = true
require_mfa_for_destructive = true
auth_timeout = 3600

[security.bypass]
allow_skip_auth = false  # true in dev only

[plugins]
auth_enabled = true

[platform.control_center]
url = "http://localhost:3000"

Error Messages

Not Authenticated

❌ Authentication Required
Operation: server create web-01
To login: provisioning auth login <username>

Fix: provisioning auth login <username>

MFA Required

❌ MFA Verification Required
Operation: server delete web-01
Reason: destructive operation

Fix: provisioning auth mfa verify --code <code>

Token Expired

Error: Token verification failed

Fix: Re-login: provisioning auth login <username>

Troubleshooting

Audit Logs

# View audit log
cat provisioning/logs/audit.log

# Filter by user
cat provisioning/logs/audit.log | jq '. | select(.user == "admin")'

# Filter by operation
cat provisioning/logs/audit.log | jq '. | select(.operation == "server_create")'

CI/CD Integration

Option 1: Skip Auth (Dev/Test Only)

export PROVISIONING_SKIP_AUTH=true
provisioning server create ci-server

Option 2: Check Mode

provisioning server create ci-server --check

Option 3: Service Account (Future)

export PROVISIONING_AUTH_TOKEN="<token>"
provisioning server create ci-server

Performance

Related Docs

• Full Guide: docs/user/AUTHENTICATION_LAYER_GUIDE.md
• Implementation: AUTHENTICATION_LAYER_IMPLEMENTATION_SUMMARY.md
• Security ADR: docs/architecture/ADR-009-security-system-complete.md

Quick Help: provisioning help auth or provisioning auth --help

Configuration Encryption Guide

Version: 1.0.0 Last Updated: 2025-10-08 Status: Production Ready

Overview

The Provisioning Platform includes a comprehensive configuration encryption system that provides:

• Transparent Encryption/Decryption: Configs are automatically decrypted on load
• Multiple KMS Backends: Age, AWS KMS, HashiCorp Vault, Cosmian KMS
• Memory-Only Decryption: Secrets never written to disk in plaintext
• SOPS Integration: Industry-standard encryption with SOPS
• Sensitive Data Detection: Automatic scanning for unencrypted sensitive data

Table of Contents

1. Prerequisites
2. Quick Start
3. Configuration Encryption
4. KMS Backends
5. CLI Commands
6. Integration with Config Loader
7. Best Practices
8. Troubleshooting

Prerequisites

Required Tools

1. SOPS (v3.10.2+)

   # macOS
   brew install sops
   
   # Linux
   wget https://github.com/mozilla/sops/releases/download/v3.10.2/sops-v3.10.2.linux.amd64
   sudo mv sops-v3.10.2.linux.amd64 /usr/local/bin/sops
   sudo chmod +x /usr/local/bin/sops
   

2. Age (for Age backend - recommended)

   # macOS
   brew install age
   
   # Linux
   apt install age
   

3. AWS CLI (for AWS KMS backend - optional)

   brew install awscli
   

Verify Installation

# Check SOPS
sops --version

# Check Age
age --version

# Check AWS CLI (optional)
aws --version

Quick Start

1. Initialize Encryption

Generate Age keys and create SOPS configuration:

provisioning config init-encryption --kms age

This will:

• Generate Age key pair in ~/.config/sops/age/keys.txt
• Display your public key (recipient)
• Create .sops.yaml in your project

2. Set Environment Variables

Add to your shell profile (~/.zshrc or ~/.bashrc):

# Age encryption
export SOPS_AGE_RECIPIENTS="age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p"
export PROVISIONING_KAGE="$HOME/.config/sops/age/keys.txt"

Replace the recipient with your actual public key.

3. Validate Setup

provisioning config validate-encryption

Expected output:

✅ Encryption configuration is valid
   SOPS installed: true
   Age backend: true
   KMS enabled: false
   Errors: 0
   Warnings: 0

4. Encrypt Your First Config

# Create a config with sensitive data
cat > workspace/config/secure.yaml <<EOF
database:
  host: localhost
  password: supersecret123
  api_key: key_abc123
EOF

# Encrypt it
provisioning config encrypt workspace/config/secure.yaml --in-place

# Verify it's encrypted
provisioning config is-encrypted workspace/config/secure.yaml

Configuration Encryption

File Naming Conventions

Encrypted files should follow these patterns:

• *.enc.yaml - Encrypted YAML files
• *.enc.yml - Encrypted YAML files (alternative)
• *.enc.toml - Encrypted TOML files
• secure.yaml - Files in workspace/config/

The .sops.yaml configuration automatically applies encryption rules based on file paths.

Encrypt a Configuration File

Basic Encryption

# Encrypt and create new file
provisioning config encrypt secrets.yaml

# Output: secrets.yaml.enc

In-Place Encryption

# Encrypt and replace original
provisioning config encrypt secrets.yaml --in-place

Specify Output Path

# Encrypt to specific location
provisioning config encrypt secrets.yaml --output workspace/config/secure.enc.yaml

Choose KMS Backend

# Use Age (default)
provisioning config encrypt secrets.yaml --kms age

# Use AWS KMS
provisioning config encrypt secrets.yaml --kms aws-kms

# Use Vault
provisioning config encrypt secrets.yaml --kms vault

Decrypt a Configuration File

# Decrypt to new file
provisioning config decrypt secrets.enc.yaml

# Decrypt in-place
provisioning config decrypt secrets.enc.yaml --in-place

# Decrypt to specific location
provisioning config decrypt secrets.enc.yaml --output plaintext.yaml

Edit Encrypted Files

The system provides a secure editing workflow:

# Edit encrypted file (auto decrypt -> edit -> re-encrypt)
provisioning config edit-secure workspace/config/secure.enc.yaml

This will:

1. Decrypt the file temporarily
2. Open in your $EDITOR (vim/nano/etc)
3. Re-encrypt when you save and close
4. Remove temporary decrypted file

Check Encryption Status

# Check if file is encrypted
provisioning config is-encrypted workspace/config/secure.yaml

# Get detailed encryption info
provisioning config encryption-info workspace/config/secure.yaml

KMS Backends

Age (Recommended for Development)

Pros:

• Simple file-based keys
• No external dependencies
• Fast and secure
• Works offline

Setup:

# Initialize
provisioning config init-encryption --kms age

# Set environment variables
export SOPS_AGE_RECIPIENTS="age1..."  # Your public key
export PROVISIONING_KAGE="$HOME/.config/sops/age/keys.txt"

Encrypt/Decrypt:

provisioning config encrypt secrets.yaml --kms age
provisioning config decrypt secrets.enc.yaml

AWS KMS (Production)

Pros:

• Centralized key management
• Audit logging
• IAM integration
• Key rotation

Setup:

1. Create KMS key in AWS Console
2. Configure AWS credentials:

   aws configure
   

3. Update .sops.yaml:

   creation_rules:
     - path_regex: .*\.enc\.yaml$
       kms: "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
   

Encrypt/Decrypt:

provisioning config encrypt secrets.yaml --kms aws-kms
provisioning config decrypt secrets.enc.yaml

HashiCorp Vault (Enterprise)

Pros:

• Dynamic secrets
• Centralized secret management
• Audit logging
• Policy-based access

Setup:

1. Configure Vault address and token:

   export VAULT_ADDR="https://vault.example.com:8200"
   export VAULT_TOKEN="s.xxxxxxxxxxxxxx"
   

2. Update configuration:

   # workspace/config/provisioning.yaml
   kms:
     enabled: true
     mode: "remote"
     vault:
       address: "https://vault.example.com:8200"
       transit_key: "provisioning"
   

Encrypt/Decrypt:

provisioning config encrypt secrets.yaml --kms vault
provisioning config decrypt secrets.enc.yaml

Cosmian KMS (Confidential Computing)

Pros:

• Confidential computing support
• Zero-knowledge architecture
• Post-quantum ready
• Cloud-agnostic

Setup:

1. Deploy Cosmian KMS server
2. Update configuration:

   kms:
     enabled: true
     mode: "remote"
     remote:
       endpoint: "https://kms.example.com:9998"
       auth_method: "certificate"
       client_cert: "/path/to/client.crt"
       client_key: "/path/to/client.key"
   

Encrypt/Decrypt:

provisioning config encrypt secrets.yaml --kms cosmian
provisioning config decrypt secrets.enc.yaml

CLI Commands

Configuration Encryption Commands

Examples

# Encrypt workspace config
provisioning config encrypt workspace/config/secure.yaml --in-place

# Edit encrypted file
provisioning config edit-secure workspace/config/secure.yaml

# Scan for unencrypted sensitive configs
provisioning config scan-sensitive workspace/config --recursive

# Encrypt all sensitive configs in workspace
provisioning config encrypt-all workspace/config --kms age --recursive

# Check encryption status
provisioning config is-encrypted workspace/config/secure.yaml

# Get detailed info
provisioning config encryption-info workspace/config/secure.yaml

# Validate setup
provisioning config validate-encryption

Integration with Config Loader

Automatic Decryption

The config loader automatically detects and decrypts encrypted files:

# Load encrypted config (automatically decrypted in memory)
use lib_provisioning/config/loader.nu

let config = (load-provisioning-config --debug)

Key Features:

• Transparent: No code changes needed
• Memory-Only: Decrypted content never written to disk
• Fallback: If decryption fails, attempts to load as plain file
• Debug Support: Shows decryption status with --debug flag

Manual Loading

use lib_provisioning/config/encryption.nu

# Load encrypted config
let secure_config = (load-encrypted-config "workspace/config/secure.enc.yaml")

# Memory-only decryption (no file created)
let decrypted_content = (decrypt-config-memory "workspace/config/secure.enc.yaml")

Configuration Hierarchy with Encryption

The system supports encrypted files at any level:

1. workspace/{name}/config/provisioning.yaml        ← Can be encrypted
2. workspace/{name}/config/providers/*.toml         ← Can be encrypted
3. workspace/{name}/config/platform/*.toml          ← Can be encrypted
4. ~/.../provisioning/ws_{name}.yaml                ← Can be encrypted
5. Environment variables (PROVISIONING_*)           ← Plain text

Best Practices

1. Encrypt All Sensitive Data

Always encrypt configs containing:

• Passwords
• API keys
• Secret keys
• Private keys
• Tokens
• Credentials

Scan for unencrypted sensitive data:

provisioning config scan-sensitive workspace --recursive

2. Use Appropriate KMS Backend

3. Key Management

Age Keys:

• Store private keys securely: ~/.config/sops/age/keys.txt
• Set file permissions: chmod 600 ~/.config/sops/age/keys.txt
• Backup keys securely (encrypted backup)
• Never commit private keys to git

AWS KMS:

• Use separate keys per environment
• Enable key rotation
• Use IAM policies for access control
• Monitor usage with CloudTrail

Vault:

• Use transit engine for encryption
• Enable audit logging
• Implement least-privilege policies
• Regular policy reviews

4. File Organization

workspace/
└── config/
    ├── provisioning.yaml         # Plain (no secrets)
    ├── secure.yaml                # Encrypted (SOPS auto-detects)
    ├── providers/
    │   ├── aws.toml               # Plain (no secrets)
    │   └── aws-credentials.enc.toml  # Encrypted
    └── platform/
        └── database.enc.yaml      # Encrypted

5. Git Integration

Add to .gitignore:

# Unencrypted sensitive files
**/secrets.yaml
**/credentials.yaml
**/*.dec.yaml
**/*.dec.toml

# Temporary decrypted files
*.tmp.yaml
*.tmp.toml

Commit encrypted files:

# Encrypted files are safe to commit
git add workspace/config/secure.enc.yaml
git commit -m "Add encrypted configuration"

6. Rotation Strategy

Regular Key Rotation:

# Generate new Age key
age-keygen -o ~/.config/sops/age/keys-new.txt

# Update .sops.yaml with new recipient

# Rotate keys for file
provisioning config rotate-keys workspace/config/secure.yaml <new-key-id>

Frequency:

• Development: Annually
• Production: Quarterly
• After team member departure: Immediately

7. Audit and Monitoring

Track encryption status:

# Regular scans
provisioning config scan-sensitive workspace --recursive

# Validate encryption setup
provisioning config validate-encryption

Monitor access (with Vault/AWS KMS):

• Enable audit logging
• Review access patterns
• Alert on anomalies

Troubleshooting

SOPS Not Found

Error:

SOPS binary not found

Solution:

# Install SOPS
brew install sops

# Verify
sops --version

Age Key Not Found

Error:

Age key file not found: ~/.config/sops/age/keys.txt

Solution:

# Generate new key
mkdir -p ~/.config/sops/age
age-keygen -o ~/.config/sops/age/keys.txt

# Set environment variable
export PROVISIONING_KAGE="$HOME/.config/sops/age/keys.txt"

SOPS_AGE_RECIPIENTS Not Set

Error:

no AGE_RECIPIENTS for file.yaml

Solution:

# Extract public key from private key
grep "public key:" ~/.config/sops/age/keys.txt

# Set environment variable
export SOPS_AGE_RECIPIENTS="age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p"

Decryption Failed

Error:

Failed to decrypt configuration file

Solutions:

1. Wrong key:

   # Verify you have the correct private key
   provisioning config validate-encryption
   

2. File corrupted:

   # Check file integrity
   sops --decrypt workspace/config/secure.yaml
   

3. Wrong backend:

   # Check SOPS metadata in file
   head -20 workspace/config/secure.yaml
   

AWS KMS Access Denied

Error:

AccessDeniedException: User is not authorized to perform: kms:Decrypt

Solution:

# Check AWS credentials
aws sts get-caller-identity

# Verify KMS key policy allows your IAM user/role
aws kms describe-key --key-id <key-arn>

Vault Connection Failed

Error:

Vault encryption failed: connection refused

Solution:

# Verify Vault address
echo $VAULT_ADDR

# Check connectivity
curl -k $VAULT_ADDR/v1/sys/health

# Verify token
vault token lookup

Security Considerations

Threat Model

Protected Against:

• ✅ Plaintext secrets in git
• ✅ Accidental secret exposure
• ✅ Unauthorized file access
• ✅ Key compromise (with rotation)

Not Protected Against:

• ❌ Memory dumps during decryption
• ❌ Root/admin access to running process
• ❌ Compromised Age/KMS keys
• ❌ Social engineering

Security Best Practices

1. Principle of Least Privilege: Only grant decryption access to those who need it
2. Key Separation: Use different keys for different environments
3. Regular Audits: Review who has access to keys
4. Secure Key Storage: Never store private keys in git
5. Rotation: Regularly rotate encryption keys
6. Monitoring: Monitor decryption operations (with AWS KMS/Vault)

Additional Resources

• SOPS Documentation: https://github.com/mozilla/sops
• Age Encryption: https://age-encryption.org/
• AWS KMS: https://aws.amazon.com/kms/
• HashiCorp Vault: https://www.vaultproject.io/
• Cosmian KMS: https://www.cosmian.com/

Support

For issues or questions:

• Check troubleshooting section above
• Run: provisioning config validate-encryption
• Review logs with --debug flag

Last Updated: 2025-10-08 Version: 1.0.0

Configuration Encryption Quick Reference

Setup (One-time)

# 1. Initialize encryption
provisioning config init-encryption --kms age

# 2. Set environment variables (add to ~/.zshrc or ~/.bashrc)
export SOPS_AGE_RECIPIENTS="age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p"
export PROVISIONING_KAGE="$HOME/.config/sops/age/keys.txt"

# 3. Validate setup
provisioning config validate-encryption

Common Commands

File Naming Conventions

Automatically encrypted by SOPS:

• workspace/*/config/secure.yaml ← Auto-encrypted
• *.enc.yaml ← Auto-encrypted
• *.enc.yml ← Auto-encrypted
• *.enc.toml ← Auto-encrypted
• workspace/*/config/providers/*credentials*.toml ← Auto-encrypted

Quick Workflow

# Create config with secrets
cat > workspace/config/secure.yaml <<EOF
database:
  password: supersecret
api_key: secret_key_123
EOF

# Encrypt in-place
provisioning config encrypt workspace/config/secure.yaml --in-place

# Verify encrypted
provisioning config is-encrypted workspace/config/secure.yaml

# Edit securely (decrypt -> edit -> re-encrypt)
provisioning config edit-secure workspace/config/secure.yaml

# Configs are auto-decrypted when loaded
provisioning env  # Automatically decrypts secure.yaml

KMS Backends

Security Checklist

• ✅ Encrypt all files with passwords, API keys, secrets
• ✅ Never commit unencrypted secrets to git
• ✅ Set file permissions: chmod 600 ~/.config/sops/age/keys.txt
• ✅ Add plaintext files to .gitignore: *.dec.yaml, secrets.yaml
• ✅ Regular key rotation (quarterly for production)
• ✅ Separate keys per environment (dev/staging/prod)
• ✅ Backup Age keys securely (encrypted backup)

Troubleshooting

Testing

# Run all encryption tests
nu provisioning/core/nulib/lib_provisioning/config/encryption_tests.nu

# Run specific test
nu provisioning/core/nulib/lib_provisioning/config/encryption_tests.nu --test roundtrip

# Test full workflow
nu provisioning/core/nulib/lib_provisioning/config/encryption_tests.nu test-full-encryption-workflow

# Test KMS backend
use lib_provisioning/kms/client.nu
kms-test --backend age

Integration

Configs are automatically decrypted when loaded:

# Nushell code - encryption is transparent
use lib_provisioning/config/loader.nu

# Auto-decrypts encrypted files in memory
let config = (load-provisioning-config)

# Access secrets normally
let db_password = ($config | get database.password)

Emergency Key Recovery

If you lose your Age key:

1. Check backups: ~/.config/sops/age/keys.txt.backup
2. Check other systems: Keys might be on other dev machines
3. Contact team: Team members with access can re-encrypt for you
4. Rotate secrets: If keys are lost, rotate all secrets

Advanced

Multiple Recipients (Team Access)

# .sops.yaml
creation_rules:
  - path_regex: .*\.enc\.yaml$
    age: >-
      age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p,
      age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8q

Key Rotation

# Generate new key
age-keygen -o ~/.config/sops/age/keys-new.txt

# Update .sops.yaml with new recipient

# Rotate keys for file
provisioning config rotate-keys workspace/config/secure.yaml <new-key-id>

Scan and Encrypt All

# Find all unencrypted sensitive configs
provisioning config scan-sensitive workspace --recursive

# Encrypt them all
provisioning config encrypt-all workspace --kms age --recursive

# Verify
provisioning config scan-sensitive workspace --recursive

Documentation

• Full Guide: docs/user/CONFIG_ENCRYPTION_GUIDE.md
• SOPS Docs: https://github.com/mozilla/sops
• Age Docs: https://age-encryption.org/

Last Updated: 2025-10-08

Dynamic Secrets - Quick Reference Guide

Quick Start: Generate temporary credentials instead of using static secrets

Quick Commands

Generate AWS Credentials (1 hour)

secrets generate aws --role deploy --workspace prod --purpose "deployment"

Generate SSH Key (2 hours)

secrets generate ssh --ttl 2 --workspace dev --purpose "server access"

Generate UpCloud Subaccount (2 hours)

secrets generate upcloud --workspace staging --purpose "testing"

List Active Secrets

secrets list

Revoke Secret

secrets revoke <secret-id> --reason "no longer needed"

View Statistics

secrets stats

Secret Types

REST API Endpoints

Base URL: http://localhost:9090/api/v1/secrets

# Generate secret
POST /generate

# Get secret
GET /{id}

# Revoke secret
POST /{id}/revoke

# Renew secret
POST /{id}/renew

# List secrets
GET /list

# List expiring
GET /expiring

# Statistics
GET /stats

AWS STS Example

# Generate
let creds = secrets generate aws `
    --role deploy `
    --region us-west-2 `
    --workspace prod `
    --purpose "Deploy servers"

# Export to environment
export-env {
    AWS_ACCESS_KEY_ID: ($creds.credentials.access_key_id)
    AWS_SECRET_ACCESS_KEY: ($creds.credentials.secret_access_key)
    AWS_SESSION_TOKEN: ($creds.credentials.session_token)
}

# Use credentials
provisioning server create

# Cleanup
secrets revoke ($creds.id) --reason "done"

SSH Key Example

# Generate
let key = secrets generate ssh `
    --ttl 4 `
    --workspace dev `
    --purpose "Debug issue"

# Save key
$key.credentials.private_key | save ~/.ssh/temp_key
chmod 600 ~/.ssh/temp_key

# Use key
ssh -i ~/.ssh/temp_key user@server

# Cleanup
rm ~/.ssh/temp_key
secrets revoke ($key.id) --reason "fixed"

Configuration

File: provisioning/platform/orchestrator/config.defaults.toml

[secrets]
default_ttl_hours = 1
max_ttl_hours = 12
auto_revoke_on_expiry = true
warning_threshold_minutes = 5

aws_account_id = "123456789012"
aws_default_region = "us-east-1"

upcloud_username = "${UPCLOUD_USER}"
upcloud_password = "${UPCLOUD_PASS}"

Troubleshooting

“Provider not found”

→ Check service initialization

“TTL exceeds maximum”

→ Reduce TTL or configure higher max

“Secret not renewable”

→ Generate new secret instead

“Missing required parameter”

→ Check provider requirements (e.g., AWS needs ‘role’)

Security Features

• ✅ No static credentials stored
• ✅ Automatic expiration (1-12 hours)
• ✅ Auto-revocation on expiry
• ✅ Full audit trail
• ✅ Memory-only storage
• ✅ TLS in transit

Support

Orchestrator logs: provisioning/platform/orchestrator/data/orchestrator.log

Debug secrets: secrets list | where is_expired == true

Full documentation: /Users/Akasha/project-provisioning/DYNAMIC_SECRETS_IMPLEMENTATION.md

SSH Temporal Keys - User Guide

Quick Start

Generate and Connect with Temporary Key

The fastest way to use temporal SSH keys:

# Auto-generate, deploy, and connect (key auto-revoked after disconnect)
ssh connect server.example.com

# Connect with custom user and TTL
ssh connect server.example.com --user deploy --ttl 30min

# Keep key active after disconnect
ssh connect server.example.com --keep

Manual Key Management

For more control over the key lifecycle:

# 1. Generate key
ssh generate-key server.example.com --user root --ttl 1hr

# Output:
# ✓ SSH key generated successfully
#   Key ID: abc-123-def-456
#   Type: dynamickeypair
#   User: root
#   Server: server.example.com
#   Expires: 2024-01-01T13:00:00Z
#   Fingerprint: SHA256:...
#
# Private Key (save securely):
# -----BEGIN OPENSSH PRIVATE KEY-----
# ...
# -----END OPENSSH PRIVATE KEY-----

# 2. Deploy key to server
ssh deploy-key abc-123-def-456

# 3. Use the private key to connect
ssh -i /path/to/private/key root@server.example.com

# 4. Revoke when done
ssh revoke-key abc-123-def-456

Key Features

Automatic Expiration

All keys expire automatically after their TTL:

• Default TTL: 1 hour
• Configurable: From 5 minutes to 24 hours
• Background Cleanup: Automatic removal from servers every 5 minutes

Multiple Key Types

Choose the right key type for your use case:

Security Benefits

✅ No static SSH keys to manage ✅ Short-lived credentials (1 hour default) ✅ Automatic cleanup on expiration ✅ Audit trail for all operations ✅ Private keys never stored on disk

Common Usage Patterns

Development Workflow

# Quick SSH for debugging
ssh connect dev-server.local --ttl 30min

# Execute commands
ssh root@dev-server.local "systemctl status nginx"

# Connection closes, key auto-revokes

Production Deployment

# Generate key with longer TTL for deployment
ssh generate-key prod-server.example.com --ttl 2hr

# Deploy to server
ssh deploy-key <key-id>

# Run deployment script
ssh -i /tmp/deploy-key root@prod-server.example.com < deploy.sh

# Manual revoke when done
ssh revoke-key <key-id>

Multi-Server Access

# Generate one key
ssh generate-key server01.example.com --ttl 1hr

# Use the same private key for multiple servers (if you have provisioning access)
# Note: Currently each key is server-specific, multi-server support coming soon

Command Reference

ssh generate-key

Generate a new temporal SSH key.

Syntax:

ssh generate-key <server> [options]

Options:

• --user <name>: SSH user (default: root)
• --ttl <duration>: Key lifetime (default: 1hr)
• --type <ca|otp|dynamic>: Key type (default: dynamic)
• --ip <address>: Allowed IP (OTP mode only)
• --principal <name>: Principal (CA mode only)

Examples:

# Basic usage
ssh generate-key server.example.com

# Custom user and TTL
ssh generate-key server.example.com --user deploy --ttl 30min

# Vault CA mode
ssh generate-key server.example.com --type ca --principal admin

ssh deploy-key

Deploy a generated key to the target server.

Syntax:

ssh deploy-key <key-id>

Example:

ssh deploy-key abc-123-def-456

ssh list-keys

List all active SSH keys.

Syntax:

ssh list-keys [--expired]

Examples:

# List active keys
ssh list-keys

# Show only deployed keys
ssh list-keys | where deployed == true

# Include expired keys
ssh list-keys --expired

ssh get-key

Get detailed information about a specific key.

Syntax:

ssh get-key <key-id>

Example:

ssh get-key abc-123-def-456

ssh revoke-key

Immediately revoke a key (removes from server and tracking).

Syntax:

ssh revoke-key <key-id>

Example:

ssh revoke-key abc-123-def-456

ssh connect

Auto-generate, deploy, connect, and revoke (all-in-one).

Syntax:

ssh connect <server> [options]

Options:

• --user <name>: SSH user (default: root)
• --ttl <duration>: Key lifetime (default: 1hr)
• --type <ca|otp|dynamic>: Key type (default: dynamic)
• --keep: Don’t revoke after disconnect

Examples:

# Quick connection
ssh connect server.example.com

# Custom user
ssh connect server.example.com --user deploy

# Keep key active after disconnect
ssh connect server.example.com --keep

ssh stats

Show SSH key statistics.

Syntax:

ssh stats

Example Output:

SSH Key Statistics:
  Total generated: 42
  Active keys: 10
  Expired keys: 32

Keys by type:
  dynamic: 35
  otp: 5
  certificate: 2

Last cleanup: 2024-01-01T12:00:00Z
  Cleaned keys: 5

ssh cleanup

Manually trigger cleanup of expired keys.

Syntax:

ssh cleanup

ssh test

Run a quick test of the SSH key system.

Syntax:

ssh test <server> [--user <name>]

Example:

ssh test server.example.com --user root

ssh help

Show help information.

Syntax:

ssh help

Duration Formats

The --ttl option accepts various duration formats:

Working with Private Keys

Saving Private Keys

When you generate a key, save the private key immediately:

# Generate and save to file
ssh generate-key server.example.com | get private_key | save -f ~/.ssh/temp_key
chmod 600 ~/.ssh/temp_key

# Use the key
ssh -i ~/.ssh/temp_key root@server.example.com

# Cleanup
rm ~/.ssh/temp_key

Using SSH Agent

Add the temporary key to your SSH agent:

# Generate key and extract private key
ssh generate-key server.example.com | get private_key | save -f /tmp/temp_key
chmod 600 /tmp/temp_key

# Add to agent
ssh-add /tmp/temp_key

# Connect (agent provides the key automatically)
ssh root@server.example.com

# Remove from agent
ssh-add -d /tmp/temp_key
rm /tmp/temp_key

Troubleshooting

Key Deployment Fails

Problem: ssh deploy-key returns error

Solutions:

1. Check SSH connectivity to server:

   ssh root@server.example.com
   

2. Verify provisioning key is configured:

   echo $PROVISIONING_SSH_KEY
   

3. Check server SSH daemon:

   ssh root@server.example.com "systemctl status sshd"
   

Private Key Not Working

Problem: SSH connection fails with “Permission denied (publickey)”

Solutions:

1. Verify key was deployed:

   ssh list-keys | where id == "<key-id>"
   

2. Check key hasn’t expired:

   ssh get-key <key-id> | get expires_at
   

3. Verify private key permissions:

   chmod 600 /path/to/private/key
   

Cleanup Not Running

Problem: Expired keys not being removed

Solutions:

1. Check orchestrator is running:

   curl http://localhost:9090/health
   

2. Trigger manual cleanup:

   ssh cleanup
   

3. Check orchestrator logs:

   tail -f ./data/orchestrator.log | grep SSH
   

Best Practices

Security

1. Short TTLs: Use the shortest TTL that works for your task

   ssh connect server.example.com --ttl 30min
   

2. Immediate Revocation: Revoke keys when you’re done

   ssh revoke-key <key-id>
   

3. Private Key Handling: Never share or commit private keys

   # Save to temp location, delete after use
   ssh generate-key server.example.com | get private_key | save -f /tmp/key
   # ... use key ...
   rm /tmp/key
   

Workflow Integration

1. Automated Deployments: Generate key in CI/CD

   #!/bin/bash
   KEY_ID=$(ssh generate-key prod.example.com --ttl 1hr | get id)
   ssh deploy-key $KEY_ID
   # Run deployment
   ansible-playbook deploy.yml
   ssh revoke-key $KEY_ID
   

2. Interactive Use: Use ssh connect for quick access

   ssh connect dev.example.com
   

3. Monitoring: Check statistics regularly

   ssh stats
   

Advanced Usage

Vault Integration

If your organization uses HashiCorp Vault:

CA Mode (Recommended)

# Generate CA-signed certificate
ssh generate-key server.example.com --type ca --principal admin --ttl 1hr

# Vault signs your public key
# Server must trust Vault CA certificate

Setup (one-time):

# On servers, add to /etc/ssh/sshd_config:
TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem

# Get Vault CA public key:
vault read -field=public_key ssh/config/ca | \
  sudo tee /etc/ssh/trusted-user-ca-keys.pem

# Restart SSH:
sudo systemctl restart sshd

OTP Mode

# Generate one-time password
ssh generate-key server.example.com --type otp --ip 192.168.1.100

# Use the OTP to connect (single use only)

Scripting

Use in scripts for automated operations:

# deploy.nu
def deploy [target: string] {
    let key = (ssh generate-key $target --ttl 1hr)
    ssh deploy-key $key.id

    # Run deployment
    try {
        ssh $"root@($target)" "bash /path/to/deploy.sh"
    } catch {
        print "Deployment failed"
    }

    # Always cleanup
    ssh revoke-key $key.id
}

API Integration

For programmatic access, use the REST API:

# Generate key
curl -X POST http://localhost:9090/api/v1/ssh/generate \
  -H "Content-Type: application/json" \
  -d '{
    "key_type": "dynamickeypair",
    "user": "root",
    "target_server": "server.example.com",
    "ttl_seconds": 3600
  }'

# Deploy key
curl -X POST http://localhost:9090/api/v1/ssh/{key_id}/deploy

# List keys
curl http://localhost:9090/api/v1/ssh/keys

# Get stats
curl http://localhost:9090/api/v1/ssh/stats

FAQ

Q: Can I use the same key for multiple servers? A: Currently, each key is tied to a specific server. Multi-server support is planned.

Q: What happens if the orchestrator crashes? A: Keys in memory are lost, but keys already deployed to servers remain until their expiration time.

Q: Can I extend the TTL of an existing key? A: No, you must generate a new key. This is by design for security.

Q: What’s the maximum TTL? A: Configurable by admin, default maximum is 24 hours.

Q: Are private keys stored anywhere? A: Private keys exist only in memory during generation and are shown once to the user. They are never written to disk by the system.

Q: What happens if cleanup fails? A: The key remains in authorized_keys until the next cleanup run. You can trigger manual cleanup with ssh cleanup.

Q: Can I use this with non-root users? A: Yes, use --user <username> when generating the key.

Q: How do I know when my key will expire? A: Use ssh get-key <key-id> to see the exact expiration timestamp.

Support

For issues or questions:

1. Check orchestrator logs: tail -f ./data/orchestrator.log
2. Run diagnostics: ssh stats
3. Test connectivity: ssh test server.example.com
4. Review documentation: SSH_KEY_MANAGEMENT.md

See Also

• Architecture: SSH_KEY_MANAGEMENT.md
• Implementation: SSH_IMPLEMENTATION_SUMMARY.md
• Configuration: config/ssh-config.toml.example

RustyVault KMS Backend Guide

Version: 1.0.0 Date: 2025-10-08 Status: Production-ready

Overview

RustyVault is a self-hosted, Rust-based secrets management system that provides a Vault-compatible API. The provisioning platform now supports RustyVault as a KMS backend alongside Age, Cosmian, AWS KMS, and HashiCorp Vault.

Why RustyVault?

• Self-hosted: Full control over your key management infrastructure
• Pure Rust: Better performance and memory safety
• Vault-compatible: Drop-in replacement for HashiCorp Vault Transit engine
• OSI-approved License: Apache 2.0 (vs HashiCorp’s BSL)
• Embeddable: Can run as standalone service or embedded library
• No Vendor Lock-in: Open-source alternative to proprietary KMS solutions

Architecture Position

KMS Service Backends:
├── Age (local development, file-based)
├── Cosmian (privacy-preserving, production)
├── AWS KMS (cloud-native AWS)
├── HashiCorp Vault (enterprise, external)
└── RustyVault (self-hosted, embedded) ✨ NEW

Installation

Option 1: Standalone RustyVault Server

# Install RustyVault binary
cargo install rusty_vault

# Start RustyVault server
rustyvault server -config=/path/to/config.hcl