Jesús Pérez 44648e3206

chore: complete nickel migration and consolidate legacy configs

- Remove KCL ecosystem (~220 files deleted)
- Migrate all infrastructure to Nickel schema system
- Consolidate documentation: legacy docs → provisioning/docs/src/
- Add CI/CD workflows (.github/) and Rust build config (.cargo/)
- Update core system for Nickel schema parsing
- Update README.md and CHANGES.md for v5.0.0 release
- Fix pre-commit hooks: end-of-file, trailing-whitespace
- Breaking changes: KCL workspaces require migration
- Migration bridge available in docs/src/development/

2026-01-08 09:55:37 +00:00

3.4 KiB

Raw Blame History

Batch Workflow System (v3.1.0 - TOKEN-OPTIMIZED ARCHITECTURE)

🚀 Batch Workflow System Completed (2025-09-25)

A comprehensive batch workflow system has been implemented using 10 token-optimized agents achieving 85-90% token efficiency over monolithic approaches. The system enables provider-agnostic batch operations with mixed provider support (UpCloud + AWS + local).

Key Achievements

Provider-Agnostic Design: Single workflows supporting multiple cloud providers
Nickel Schema Integration: Type-safe workflow definitions with comprehensive validation
Dependency Resolution: Topological sorting with soft/hard dependency support
State Management: Checkpoint-based recovery with rollback capabilities
Real-time Monitoring: Live workflow progress tracking and health monitoring
Token Optimization: 85-90% efficiency using parallel specialized agents

Batch Workflow Commands

# Submit batch workflow from Nickel definition
nu -c "use core/nulib/workflows/batch.nu *; batch submit workflows/example_batch.ncl"

# Monitor batch workflow progress
nu -c "use core/nulib/workflows/batch.nu *; batch monitor <workflow_id>"

# List batch workflows with filtering
nu -c "use core/nulib/workflows/batch.nu *; batch list --status Running"

# Get detailed batch status
nu -c "use core/nulib/workflows/batch.nu *; batch status <workflow_id>"

# Initiate rollback for failed workflow
nu -c "use core/nulib/workflows/batch.nu *; batch rollback <workflow_id>"

# Show batch workflow statistics
nu -c "use core/nulib/workflows/batch.nu *; batch stats"

Nickel Workflow Schema

Batch workflows are defined using Nickel configuration in schemas/workflows.ncl:

# Example batch workflow with mixed providers
{
  batch_workflow = {
    name = "multi_cloud_deployment",
    version = "1.0.0",
    storage_backend = "surrealdb",  # or "filesystem"
    parallel_limit = 5,
    rollback_enabled = true,

    operations = [
      {
        id = "upcloud_servers",
        type = "server_batch",
        provider = "upcloud",
        dependencies = [],
        server_configs = [
          { name = "web-01", plan = "1xCPU-2 GB", zone = "de-fra1" },
          { name = "web-02", plan = "1xCPU-2 GB", zone = "us-nyc1" }
        ]
      },
      {
        id = "aws_taskservs",
        type = "taskserv_batch",
        provider = "aws",
        dependencies = ["upcloud_servers"],
        taskservs = ["kubernetes", "cilium", "containerd"]
      }
    ]
  }
}

REST API Endpoints (Batch Operations)

Extended orchestrator API for batch workflow management:

Submit Batch: POST http://localhost:9090/v1/workflows/batch/submit
Batch Status: GET http://localhost:9090/v1/workflows/batch/{id}
List Batches: GET http://localhost:9090/v1/workflows/batch
Monitor Progress: GET http://localhost:9090/v1/workflows/batch/{id}/progress
Initiate Rollback: POST http://localhost:9090/v1/workflows/batch/{id}/rollback
Batch Statistics: GET http://localhost:9090/v1/workflows/batch/stats

System Benefits

Provider Agnostic: Mix UpCloud, AWS, and local providers in single workflows
Type Safety: Nickel schema validation prevents runtime errors
Dependency Management: Automatic resolution with failure handling
State Recovery: Checkpoint-based recovery from any failure point
Real-time Monitoring: Live progress tracking with detailed status

3.4 KiB Raw Blame History