# Advanced Workflow Orchestration Guide

Master complex infrastructure workflows with dependency management, conditional execution, and error handling.

## Workflow Components

### Basic Workflow Structure

```nickel
{
  workflows.deploy_production = {
    name = "Production Deployment",
    version = "1.0.0",

    tasks = [
      {
        name = "validate",
        type = "validation",
        config = { ... }
      },
      {
        name = "pre_deploy",
        type = "script",
        depends_on = ["validate"],
        config = { ... }
      }
    ]
  }
}
```

## Advanced Patterns

### 1. Conditional Execution

Execute tasks based on conditions:

```nushell
def deploy-conditional [] {
    # Only deploy if validation passed
    if (provisioning validate --fast) {
        print "✓ Validation passed, proceeding"
        provisioning batch submit workflows/deploy.ncl
    } else {
        print "✗ Validation failed, aborting"
        return 1
    }
}
```

### 2. Parallel Task Execution

Run independent tasks simultaneously:

```nickel
{
  workflows.parallel_deploy = {
    tasks = [
      # These three run in parallel (no dependencies)
      {
        name = "deploy_api",
        type = "kubernetes",
        depends_on = []
      },
      {
        name = "deploy_worker",
        type = "kubernetes",
        depends_on = []
      },
      {
        name = "deploy_cache",
        type = "kubernetes",
        depends_on = []
      },
      # These wait for parallel tasks
      {
        name = "smoke_tests",
        type = "test",
        depends_on = ["deploy_api", "deploy_worker", "deploy_cache"]
      }
    ]
  }
}
```

### 3. Error Handling and Rollback

Handle failures gracefully:

```nushell
def deploy-with-rollback [] {
    let workflow_id = (provisioning batch submit workflows/blue-green.ncl)

    # Wait for completion
    let result = (provisioning batch status --id $workflow_id --wait)

    if $result.status == "failed" {
        print "Deployment failed, rolling back..."
        provisioning rollback --workflow-id $workflow_id --to-previous
        return 1
    } else {
        print "✓ Deployment successful"
        return 0
    }
}
```

### 4. Dynamic Task Generation

Generate tasks based on data:

```nushell
# Deploy microservices from config
def deploy-microservices [config: string] {
    let services = (open $config | get services)

    # Generate deploy task for each service
    $services | each { | service |
        let task = {
            name: $"deploy_($service.name)",
            type: "kubernetes",
            depends_on: ["validate"],
            image: $service.image,
            replicas: $service.replicas
        }

        $task
    } | to json
}

# Usage
let tasks = (deploy-microservices services.json)
provisioning batch submit --tasks $tasks
```

### 5. Canary Deployments

Gradual rollout with validation:

```nickel
{
  workflows.canary_deploy = {
    name = "Canary Deployment",

    tasks = [
      # Stage 1: Deploy to 10% of traffic
      {
        name = "deploy_canary_10",
        type = "kubernetes",
        weight = 0.10,
        duration_minutes = 5
      },
      # Validation: Check error rates
      {
        name = "validate_canary_10",
        type = "test",
        depends_on = ["deploy_canary_10"],
        success_criteria = {
          error_rate_increase = 0.01,  # Max 1% increase
          latency_increase_ms = 50     # Max 50ms increase
        }
      },
      # Stage 2: 50% traffic
      {
        name = "deploy_canary_50",
        type = "kubernetes",
        depends_on = ["validate_canary_10"],
        weight = 0.50,
        duration_minutes = 10
      },
      {
        name = "validate_canary_50",
        type = "test",
        depends_on = ["deploy_canary_50"],
        success_criteria = {
          error_rate_increase = 0.01,
          latency_increase_ms = 50
        }
      },
      # Stage 3: 100% traffic
      {
        name = "deploy_full",
        type = "kubernetes",
        depends_on = ["validate_canary_50"],
        weight = 1.0
      }
    ]
  }
}
```

### 6. Multi-Environment Deployment

Deploy across multiple cloud providers:

```nickel
{
  workflows.multi_cloud_deploy = {
    name = "Multi-Cloud Deployment",

    tasks = [
      # AWS deployment
      {
        name = "deploy_aws",
        type = "terraform",
        provider = "aws",
        region = "us-east-1",
        environment_variables = {
          TF_VAR_instance_count = 3
        }
      },
      # UpCloud deployment (parallel)
      {
        name = "deploy_upcloud",
        type = "terraform",
        provider = "upcloud",
        region = "de-fra",
        environment_variables = {
          TF_VAR_instance_count = 2
        }
      },
      # Health check across all
      {
        name = "health_check",
        type = "test",
        depends_on = ["deploy_aws", "deploy_upcloud"],
        tests = [
          { endpoint = " [https://aws-lb.example.com/health"](https://aws-lb.example.com/health") },
          { endpoint = " [https://upcloud-lb.example.com/health"](https://upcloud-lb.example.com/health") }
        ]
      }
    ]
  }
}
```

### 7. Database Migrations with Rollback

Safe database updates:

```nushell
def migrate-with-verification [migration_file: string] {
    print "Starting database migration..."

    # Backup database before migration
    let backup_id = (provisioning backup create --type full --database prod)
    print $"Backup created: ($backup_id)"

    # Run migration
    let migration_result = (
        provisioning database migrate \
            --database prod \
            --migration-file $migration_file \
            --dry-run false
    )

    if $migration_result.status == "success" {
        print "✓ Migration successful"

        # Verify migration
        let verification = (provisioning database verify --database prod)
        if $verification.status == "healthy" {
            print "✓ Database healthy after migration"
            return 0
        } else {
            print "✗ Database health check failed, rolling back..."
            provisioning backup restore --backup-id $backup_id --target-database prod
            return 1
        }
    } else {
        print $"✗ Migration failed: ($migration_result.error)"
        return 1
    }
}
```

### 8. Cross-Region Failover

Multi-region disaster recovery:

```nickel
{
  workflows.failover_to_secondary = {
    name = "Failover to Secondary Region",

    tasks = [
      # Verify primary is unhealthy
      {
        name = "verify_primary_down",
        type = "health_check",
        target = "primary-region-us-east-1",
        max_retries = 3,
        timeout_seconds = 30
      },
      # Promote secondary to primary
      {
        name = "promote_secondary_db",
        type = "database",
        operation = "promote-replica",
        replica_name = "secondary-db-eu-west-1"
      },
      # Update DNS/load balancer
      {
        name = "update_dns",
        type = "dns",
        depends_on = ["promote_secondary_db"],
        record = "api.example.com",
        target = "secondary-lb-eu-west-1.example.com"
      },
      # Smoke tests in secondary
      {
        name = "smoke_tests_secondary",
        type = "test",
        depends_on = ["update_dns"],
        endpoints = [
          " [https://api.example.com/health",](https://api.example.com/health",)
          " [https://api.example.com/status"](https://api.example.com/status")
        ]
      },
      # Notify team
      {
        name = "notify_team",
        type = "notification",
        depends_on = ["smoke_tests_secondary"],
        channel = "slack",
        message = "Failover to secondary region complete"
      }
    ]
  }
}
```

## Workflow Monitoring

### Real-Time Status

```bash
# Watch workflow execution
provisioning batch status --id wf-12345 --watch

# Output:
# Workflow: Production Deployment (wf-12345)
# Status: Running | Progress: 3/7 tasks
#
# ✓ validate                    (completed in 45s)
# ⏳ deploy_api                 (running, 2m 34s elapsed)
# ⏳ deploy_worker              (running, 2m 12s elapsed)
# ⌛ smoke_tests                (waiting for: deploy_api, deploy_worker)
# ⌛ notify_team                (waiting)
```

### Workflow Metrics

```bash
# Gather workflow statistics
provisioning batch metrics --from 2025-01-01 --to 2025-01-31

# Output:
# Total Workflows: 145
# Success Rate: 98.6% (143/145)
# Average Duration: 12m 45s
#
# Slowest Task: database_migration (avg: 45m)
# Most Failed: smoke_tests (2 failures)
#
# Cost Distribution:
# AWS: $1,234
# UpCloud: $567
# Local: $0 (free tier)
```

## Best Practices

1. **Task Naming**: Use descriptive names (deploy_api_v2, not task1)
2. **Dependencies**: Keep dependencies minimal for parallelization
3. **Timeouts**: Always set realistic timeouts for tasks
4. **Notifications**: Notify teams of failures and completion
5. **Testing**: Include validation tasks after deployments
6. **Idempotency**: Workflows should be safe to retry
7. **Monitoring**: Track workflow metrics for optimization
8. **Documentation**: Document complex workflows in comments

## See Also

- [Batch Workflow System](../infrastructure/batch-workflows.md) - Core concepts
- [Deployment Strategies](../operations/deployment-strategies.md) - Blue-green, canary, rolling updates
- [Orchestrator API](../api-reference/orchestrator-endpoints.md) - Complete API reference
