provisioning/docs/src/operations/orchestrator.md

# Provisioning Orchestrator

A Rust-based orchestrator service that coordinates infrastructure provisioning workflows with pluggable storage backends and comprehensive migration tools.

> **Source**: `provisioning/platform/orchestrator/`

## Architecture

The orchestrator implements a hybrid multi-storage approach:

- **Rust Orchestrator**: Handles coordination, queuing, and parallel execution
- **Nushell Scripts**: Execute the actual provisioning logic
- **Pluggable Storage**: Multiple storage backends with seamless migration
- **REST API**: HTTP interface for workflow submission and monitoring

## Key Features

- **Multi-Storage Backends**: Filesystem, SurrealDB Embedded, and SurrealDB Server options
- **Task Queue**: Priority-based task scheduling with retry logic
- **Seamless Migration**: Move data between storage backends with zero downtime
- **Feature Flags**: Compile-time backend selection for minimal dependencies
- **Parallel Execution**: Multiple tasks can run concurrently
- **Status Tracking**: Real-time task status and progress monitoring
- **Advanced Features**: Authentication, audit logging, and metrics (SurrealDB)
- **Nushell Integration**: Seamless execution of existing provisioning scripts
- **RESTful API**: HTTP endpoints for workflow management
- **Test Environment Service**: Automated containerized testing for taskservs, servers, and clusters
- **Multi-Node Support**: Test complex topologies including Kubernetes and etcd clusters
- **Docker Integration**: Automated container lifecycle management via Docker API

## Quick Start

### Build and Run

**Default Build (Filesystem Only)**:

```text
cd provisioning/platform/orchestrator
cargo build --release
cargo run -- --port 8080 --data-dir ./data
```

**With SurrealDB Support**:

```text
cargo build --release --features surrealdb

# Run with SurrealDB embedded
cargo run --features surrealdb -- --storage-type surrealdb-embedded --data-dir ./data

# Run with SurrealDB server
cargo run --features surrealdb -- --storage-type surrealdb-server 
  --surrealdb-url ws://localhost:8000 
  --surrealdb-username admin --surrealdb-password secret
```

### Submit Workflow

```text
curl -X POST http://localhost:8080/workflows/servers/create 
  -H "Content-Type: application/json" 
  -d '{
    "infra": "production",
    "settings": "./settings.yaml",
    "servers": ["web-01", "web-02"],
    "check_mode": false,
    "wait": true
  }'
```

## API Endpoints

### Core Endpoints

- `GET /health` - Service health status
- `GET /tasks` - List all tasks
- `GET /tasks/{id}` - Get specific task status

### Workflow Endpoints

- `POST /workflows/servers/create` - Submit server creation workflow
- `POST /workflows/taskserv/create` - Submit taskserv creation workflow
- `POST /workflows/cluster/create` - Submit cluster creation workflow

### Test Environment Endpoints

- `POST /test/environments/create` - Create test environment
- `GET /test/environments` - List all test environments
- `GET /test/environments/{id}` - Get environment details
- `POST /test/environments/{id}/run` - Run tests in environment
- `DELETE /test/environments/{id}` - Cleanup test environment
- `GET /test/environments/{id}/logs` - Get environment logs

## Test Environment Service

The orchestrator includes a comprehensive test environment service for automated containerized testing.

### Test Environment Types

#### 1. Single Taskserv

Test individual taskserv in isolated container.

#### 2. Server Simulation

Test complete server configurations with multiple taskservs.

#### 3. Cluster Topology

Test multi-node cluster configurations (Kubernetes, etcd, etc.).

### Nushell CLI Integration

```text
# Quick test
provisioning test quick kubernetes

# Single taskserv test
provisioning test env single postgres --auto-start --auto-cleanup

# Server simulation
provisioning test env server web-01 [containerd kubernetes cilium] --auto-start

# Cluster from template
provisioning test topology load kubernetes_3node | test env cluster kubernetes
```

### Topology Templates

Predefined multi-node cluster topologies:

- **kubernetes_3node**: 3-node HA Kubernetes cluster
- **kubernetes_single**: All-in-one Kubernetes node
- **etcd_cluster**: 3-member etcd cluster
- **containerd_test**: Standalone containerd testing
- **postgres_redis**: Database stack testing

## Storage Backends

| Feature | Filesystem | SurrealDB Embedded | SurrealDB Server |
| --------- | ------------ | ------------------- | ------------------ |
| **Dependencies** | None | Local database | Remote server |
| **Auth/RBAC** | Basic | Advanced | Advanced |
| **Real-time** | No | Yes | Yes |
| **Scalability** | Limited | Medium | High |
| **Complexity** | Low | Medium | High |
| **Best For** | Development | Production | Distributed |

## Related Documentation

- **User Guide**: [Test Environment Guide](../testing/test-environment-guide.md)
- **Architecture**: [Orchestrator Architecture](../architecture/orchestrator-integration-model.md)
- **Feature Summary**: [Orchestrator Features](../architecture/orchestrator-integration-model.md)