# Hybrid Orchestrator Architecture (v3.0.0)\n\n## 🚀 Orchestrator Implementation Completed (2025-09-25)\n\nA production-ready hybrid Rust/Nushell orchestrator has been implemented to solve deep call stack limitations while preserving all Nushell business logic.\n\n## Architecture Overview\n\n- **Rust Orchestrator**: High-performance coordination layer with REST API\n- **Nushell Business Logic**: All existing scripts preserved and enhanced\n- **File-based Persistence**: Reliable task queue using lightweight file storage\n- **Priority Processing**: Intelligent task scheduling with retry logic\n- **Deep Call Stack Solution**: Eliminates template.nu:71 "Type not supported" errors\n\n## Orchestrator Management\n\n```\n# Start orchestrator in background\ncd provisioning/platform/orchestrator\n./scripts/start-orchestrator.nu --background --provisioning-path "/usr/local/bin/provisioning"\n\n# Check orchestrator status\n./scripts/start-orchestrator.nu --check\n\n# Stop orchestrator\n./scripts/start-orchestrator.nu --stop\n\n# View logs\ntail -f ./data/orchestrator.log\n```\n\n## Workflow System\n\nThe orchestrator provides comprehensive workflow management:\n\n### Server Workflows\n\n```\n# Submit server creation workflow\nnu -c "use core/nulib/workflows/server_create.nu *; server_create_workflow 'wuji' '' [] --check"\n\n# Traditional orchestrated server creation\nprovisioning servers create --orchestrated --check\n```\n\n### Taskserv Workflows\n\n```\n# Create taskserv workflow\nnu -c "use core/nulib/workflows/taskserv.nu *; taskserv create 'kubernetes' 'wuji' --check"\n\n# Other taskserv operations\nnu -c "use core/nulib/workflows/taskserv.nu *; taskserv delete 'kubernetes' 'wuji' --check"\nnu -c "use core/nulib/workflows/taskserv.nu *; taskserv generate 'kubernetes' 'wuji'"\nnu -c "use core/nulib/workflows/taskserv.nu *; taskserv check-updates"\n```\n\n### Cluster Workflows\n\n```\n# Create cluster workflow\nnu -c "use core/nulib/workflows/cluster.nu *; cluster create 'buildkit' 'wuji' --check"\n\n# Delete cluster workflow\nnu -c "use core/nulib/workflows/cluster.nu *; cluster delete 'buildkit' 'wuji' --check"\n```\n\n### Workflow Management\n\n```\n# List all workflows\nnu -c "use core/nulib/workflows/management.nu *; workflow list"\n\n# Get workflow statistics\nnu -c "use core/nulib/workflows/management.nu *; workflow stats"\n\n# Monitor workflow in real-time\nnu -c "use core/nulib/workflows/management.nu *; workflow monitor <task_id>"\n\n# Check orchestrator health\nnu -c "use core/nulib/workflows/management.nu *; workflow orchestrator"\n\n# Get specific workflow status\nnu -c "use core/nulib/workflows/management.nu *; workflow status <task_id>"\n```\n\n## REST API Endpoints\n\nThe orchestrator exposes HTTP endpoints for external integration:\n\n- **Health**: `GET http://localhost:9090/v1/health`\n- **List Tasks**: `GET http://localhost:9090/v1/tasks`\n- **Task Status**: `GET http://localhost:9090/v1/tasks/{id}`\n- **Server Workflow**: `POST http://localhost:9090/v1/workflows/servers/create`\n- **Taskserv Workflow**: `POST http://localhost:9090/v1/workflows/taskserv/create`\n- **Cluster Workflow**: `POST http://localhost:9090/v1/workflows/cluster/create`
