Provisioning Orchestrator

A Rust-based orchestrator service that coordinates infrastructure provisioning workflows with pluggable storage backends and comprehensive migration tools.

Source: provisioning/platform/orchestrator/

Architecture

The orchestrator implements a hybrid multi-storage approach:

• Rust Orchestrator: Handles coordination, queuing, and parallel execution
• Nushell Scripts: Execute the actual provisioning logic
• Pluggable Storage: Multiple storage backends with seamless migration
• REST API: HTTP interface for workflow submission and monitoring

Key Features

• Multi-Storage Backends: Filesystem, SurrealDB Embedded, and SurrealDB Server options
• Task Queue: Priority-based task scheduling with retry logic
• Seamless Migration: Move data between storage backends with zero downtime
• Feature Flags: Compile-time backend selection for minimal dependencies
• Parallel Execution: Multiple tasks can run concurrently
• Status Tracking: Real-time task status and progress monitoring
• Advanced Features: Authentication, audit logging, and metrics (SurrealDB)
• Nushell Integration: Seamless execution of existing provisioning scripts
• RESTful API: HTTP endpoints for workflow management
• Test Environment Service: Automated containerized testing for taskservs, servers, and clusters
• Multi-Node Support: Test complex topologies including Kubernetes and etcd clusters
• Docker Integration: Automated container lifecycle management via Docker API

Quick Start

Build and Run

Default Build (Filesystem Only):

cd provisioning/platform/orchestrator
cargo build --release
cargo run -- --port 8080 --data-dir ./data

With SurrealDB Support:

cargo build --release --features surrealdb

# Run with SurrealDB embedded
cargo run --features surrealdb -- --storage-type surrealdb-embedded --data-dir ./data

# Run with SurrealDB server
cargo run --features surrealdb -- --storage-type surrealdb-server \
  --surrealdb-url ws://localhost:8000 \
  --surrealdb-username admin --surrealdb-password secret

Submit Workflow

curl -X POST http://localhost:8080/workflows/servers/create \
  -H "Content-Type: application/json" \
  -d '{
    "infra": "production",
    "settings": "./settings.yaml",
    "servers": ["web-01", "web-02"],
    "check_mode": false,
    "wait": true
  }'

API Endpoints

Core Endpoints

• GET /health - Service health status
• GET /tasks - List all tasks
• GET /tasks/{id} - Get specific task status

Workflow Endpoints

• POST /workflows/servers/create - Submit server creation workflow
• POST /workflows/taskserv/create - Submit taskserv creation workflow
• POST /workflows/cluster/create - Submit cluster creation workflow

Test Environment Endpoints

• POST /test/environments/create - Create test environment
• GET /test/environments - List all test environments
• GET /test/environments/{id} - Get environment details
• POST /test/environments/{id}/run - Run tests in environment
• DELETE /test/environments/{id} - Cleanup test environment
• GET /test/environments/{id}/logs - Get environment logs

Test Environment Service

The orchestrator includes a comprehensive test environment service for automated containerized testing.

Test Environment Types

1. Single Taskserv

Test individual taskserv in isolated container.

2. Server Simulation

Test complete server configurations with multiple taskservs.

3. Cluster Topology

Test multi-node cluster configurations (Kubernetes, etcd, etc.).

Nushell CLI Integration

# Quick test
provisioning test quick kubernetes

# Single taskserv test
provisioning test env single postgres --auto-start --auto-cleanup

# Server simulation
provisioning test env server web-01 [containerd kubernetes cilium] --auto-start

# Cluster from template
provisioning test topology load kubernetes_3node | test env cluster kubernetes

Topology Templates

Predefined multi-node cluster topologies:

• kubernetes_3node: 3-node HA Kubernetes cluster
• kubernetes_single: All-in-one Kubernetes node
• etcd_cluster: 3-member etcd cluster
• containerd_test: Standalone containerd testing
• postgres_redis: Database stack testing

Storage Backends

Related Documentation

• User Guide: Test Environment Guide
• Architecture: Orchestrator Architecture
• Feature Summary: Orchestrator Features