# ADR-004: Hybrid Architecture\n\n## Status\n\nAccepted\n\n## Context\n\nProvisioning encountered fundamental limitations with a pure Nushell implementation that required architectural solutions:\n\n1. **Deep Call Stack Limitations**: Nushell's `open` command fails in deep call contexts\n   (`enumerate | each`), causing "Type not supported" errors in template.nu:71\n2. **Performance Bottlenecks**: Complex workflow orchestration hitting Nushell's performance limits\n3. **Concurrency Constraints**: Limited parallel processing capabilities in Nushell for batch operations\n4. **Integration Complexity**: Need for REST API endpoints and external system integration\n5. **State Management**: Complex state tracking and persistence requirements beyond Nushell's capabilities\n6. **Business Logic Preservation**: 65+ existing Nushell files with domain expertise that shouldn't be rewritten\n7. **Developer Productivity**: Nushell excels for configuration management and domain-specific operations\n\nThe system needed an architecture that:\n\n- Solves Nushell's technical limitations without losing business logic\n- Leverages each language's strengths appropriately\n- Maintains existing investment in Nushell domain knowledge\n- Provides performance for coordination-heavy operations\n- Enables modern integration patterns (REST APIs, async workflows)\n- Preserves configuration-driven, Infrastructure as Code principles\n\n## Decision\n\nImplement a **Hybrid Rust/Nushell Architecture** with clear separation of concerns:\n\n### Architecture Layers\n\n#### 1. Coordination Layer (Rust)\n\n- **Orchestrator**: High-performance workflow coordination and task scheduling\n- **REST API Server**: HTTP endpoints for external integration\n- **State Management**: Persistent state tracking with checkpoint recovery\n- **Batch Processing**: Parallel execution of complex workflows\n- **File-based Persistence**: Lightweight task queue using reliable file storage\n- **Error Recovery**: Sophisticated error handling and rollback capabilities\n\n#### 2. Business Logic Layer (Nushell)\n\n- **Provider Implementations**: Cloud provider-specific operations (AWS, UpCloud, local)\n- **Task Services**: Infrastructure service management (Kubernetes, networking, storage)\n- **Configuration Management**: KCL-based configuration processing and validation\n- **Template Processing**: Infrastructure-as-Code template generation\n- **CLI Interface**: User-facing command-line tools and workflows\n- **Domain Operations**: All business-specific logic and operations\n\n### Integration Patterns\n\n#### Rust → Nushell Communication\n\n```\n// Rust orchestrator invokes Nushell scripts via process execution\nlet result = Command::new("nu")\n    .arg("-c")\n    .arg("use core/nulib/workflows/server_create.nu *; server_create_workflow 'name' '' []")\n    .output()?;\n```\n\n#### Nushell → Rust Communication\n\n```\n# Nushell submits workflows to Rust orchestrator via HTTP API\nhttp post "http://localhost:9090/workflows/servers/create" {\n    name: "server-name",\n    provider: "upcloud",\n    config: $server_config\n}\n```\n\n#### Data Exchange Format\n\n- **Structured JSON**: All data exchange via JSON for type safety and interoperability\n- **Configuration TOML**: Configuration data in TOML format for human readability\n- **State Files**: Lightweight file-based state exchange between layers\n\n### Key Architectural Principles\n\n1. **Language Strengths**: Use each language for what it does best\n2. **Business Logic Preservation**: All existing domain knowledge stays in Nushell\n3. **Performance Critical Path**: Coordination and orchestration in Rust\n4. **Clear Boundaries**: Well-defined interfaces between layers\n5. **Configuration Driven**: Both layers respect configuration-driven architecture\n6. **Error Handling**: Coordinated error handling across language boundaries\n7. **State Consistency**: Consistent state management across hybrid system\n\n## Consequences\n\n### Positive\n\n- **Technical Limitations Solved**: Eliminates Nushell deep call stack issues\n- **Performance Optimized**: High-performance coordination while preserving productivity\n- **Business Logic Preserved**: 65+ Nushell files with domain expertise maintained\n- **Modern Integration**: REST APIs and async workflows enabled\n- **Development Efficiency**: Developers can use optimal language for each task\n- **Batch Processing**: Parallel workflow execution with sophisticated state management\n- **Error Recovery**: Advanced error handling and rollback capabilities\n- **Scalability**: Architecture scales to complex multi-provider workflows\n- **Maintainability**: Clear separation of concerns between layers\n\n### Negative\n\n- **Complexity Increase**: Two-language system requires more architectural coordination\n- **Integration Overhead**: Data serialization/deserialization between languages\n- **Development Skills**: Team needs expertise in both Rust and Nushell\n- **Testing Complexity**: Must test integration between language layers\n- **Deployment Complexity**: Two runtime environments must be coordinated\n- **Debugging Challenges**: Debugging across language boundaries more complex\n\n### Neutral\n\n- **Development Patterns**: Different patterns for each layer while maintaining consistency\n- **Documentation Strategy**: Language-specific documentation with integration guides\n- **Tool Chain**: Multiple development tool chains must be maintained\n- **Performance Characteristics**: Different performance characteristics for different operations\n\n## Alternatives Considered\n\n### Alternative 1: Pure Nushell Implementation\n\nContinue with Nushell-only approach and work around limitations.\n**Rejected**: Technical limitations are fundamental and cannot be worked around without compromising functionality. Deep call stack issues are\narchitectural.\n\n### Alternative 2: Complete Rust Rewrite\n\nRewrite entire system in Rust for consistency.\n**Rejected**: Would lose 65+ files of domain expertise and Nushell's productivity advantages for configuration management. Massive development effort.\n\n### Alternative 3: Pure Go Implementation\n\nRewrite system in Go for simplicity and performance.\n**Rejected**: Same issues as Rust rewrite - loses domain expertise and Nushell's configuration strengths. Go doesn't provide significant advantages.\n\n### Alternative 4: Python/Shell Hybrid\n\nUse Python for coordination and shell scripts for operations.\n**Rejected**: Loses type safety and configuration-driven advantages of current system. Python adds dependency complexity.\n\n### Alternative 5: Container-Based Separation\n\nRun Nushell and coordination layer in separate containers.\n**Rejected**: Adds deployment complexity and network communication overhead. Complicates local development significantly.\n\n## Implementation Details\n\n### Orchestrator Components\n\n- **Task Queue**: File-based persistent queue for reliable workflow management\n- **HTTP Server**: REST API for workflow submission and monitoring\n- **State Manager**: Checkpoint-based state tracking with recovery\n- **Process Manager**: Nushell script execution with proper isolation\n- **Error Handler**: Comprehensive error recovery and rollback logic\n\n### Integration Protocols\n\n- **HTTP REST**: Primary API for external integration\n- **JSON Data Exchange**: Structured data format for all communication\n- **File-based State**: Lightweight persistence without database dependencies\n- **Process Execution**: Secure subprocess execution for Nushell operations\n\n### Development Workflow\n\n1. **Rust Development**: Focus on coordination, performance, and integration\n2. **Nushell Development**: Focus on business logic, providers, and task services\n3. **Integration Testing**: Validate communication between layers\n4. **End-to-End Validation**: Complete workflow testing across both layers\n\n### Monitoring and Observability\n\n- **Structured Logging**: JSON logs from both Rust and Nushell components\n- **Metrics Collection**: Performance metrics from coordination layer\n- **Health Checks**: System health monitoring across both layers\n- **Workflow Tracking**: Complete audit trail of workflow execution\n\n## Migration Strategy\n\n### Phase 1: Core Infrastructure (Completed)\n\n- ✅ Rust orchestrator implementation\n- ✅ REST API endpoints\n- ✅ File-based task queue\n- ✅ Basic Nushell integration\n\n### Phase 2: Workflow Integration (Completed)\n\n- ✅ Server creation workflows\n- ✅ Task service workflows\n- ✅ Cluster deployment workflows\n- ✅ State management and recovery\n\n### Phase 3: Advanced Features (Completed)\n\n- ✅ Batch workflow processing\n- ✅ Dependency resolution\n- ✅ Rollback capabilities\n- ✅ Real-time monitoring\n\n## References\n\n- Deep Call Stack Limitations (CLAUDE.md - Architectural Lessons Learned)\n- Configuration-Driven Architecture (ADR-002)\n- Batch Workflow System (CLAUDE.md - v3.1.0)\n- Integration Patterns Documentation\n- Performance Benchmarking Results
