# Batch Workflows

Batch workflows orchestrate complex multi-step operations across multiple clouds and services with
dependency resolution, parallel execution, and checkpoint recovery.

## Overview

Batch workflows enable:
- Multi-cloud infrastructure orchestration
- Complex deployment pipelines
- Dependency-driven execution
- Parallel task execution
- Checkpoint and recovery
- Rollback on failures

## Workflow Definition

Workflows are defined in Nickel:

```nickel
{
  workflows = [{
    name = "multi-cloud-deployment",
    description = "Deploy application across UpCloud and AWS",
    steps = [
      {
        name = "provision-upcloud",
        type = 'provision,
        provider = 'upcloud,
        resources = {
          servers = [{name = "web-eu", plan = 'medium, zone = "fi-hel1"}]
        }
      },
      {
        name = "provision-aws",
        type = 'provision,
        provider = 'aws,
        resources = {
          servers = [{name = "web-us", plan = 't3.medium, zone = "us-east-1a"}]
        }
      },
      {
        name = "deploy-application",
        type = 'task,
        depends_on = ["provision-upcloud", "provision-aws"],
        tasks = ["install-kubernetes", "deploy-app"]
      },
      {
        name = "configure-dns",
        type = 'configure,
        depends_on = ["deploy-application"],
        config = {
          records = [
            {name = "eu.example.com", target = "web-eu"},
            {name = "us.example.com", target = "web-us"}
          ]
        }
      }
    ],
    rollback_on_failure = true,
    checkpoint_enabled = true
  }]
}
```

## Dependency Resolution

Workflows automatically resolve dependencies:

```plaintext
Execution Graph:

provision-upcloud ──┐
                    ├──> deploy-application ──> configure-dns
provision-aws ──────┘
```

Steps `provision-upcloud` and `provision-aws` run in parallel. `deploy-application` waits for both to complete.

## Step Types

### Provision Steps

Create infrastructure resources:

```nickel
{
  name = "create-servers",
  type = 'provision,
  provider = 'upcloud,
  resources = {
    servers = [...],
    networks = [...],
    storage = [...]
  }
}
```

### Task Steps

Execute task services:

```nickel
{
  name = "install-k8s",
  type = 'task,
  tasks = ["kubernetes", "helm", "monitoring"]
}
```

### Configure Steps

Apply configuration changes:

```nickel
{
  name = "setup-networking",
  type = 'configure,
  config = {
    firewalls = [...],
    routes = [...],
    dns = [...]
  }
}
```

### Validate Steps

Verify conditions before proceeding:

```nickel
{
  name = "health-check",
  type = 'validate,
  checks = [
    {type = 'http, url = " [https://app.example.com",](https://app.example.com",) expected_status = 200},
    {type = 'command, command = "kubectl get nodes", expected_output = "Ready"}
  ]
}
```

## Execution Control

### Parallel Execution

Steps without dependencies run in parallel:

```nickel
steps = [
  {name = "provision-eu", ...},  # Runs in parallel
  {name = "provision-us", ...},  # Runs in parallel
  {name = "provision-asia", ...} # Runs in parallel
]
```

Configure parallelism:

```nickel
{
  max_parallel_tasks = 4,  # Max concurrent steps
  timeout_seconds = 3600   # Step timeout
}
```

### Conditional Execution

Execute steps based on conditions:

```nickel
{
  name = "scale-up",
  type = 'task,
  condition = {
    type = 'expression,
    expression = "cpu_usage > 80"
  }
}
```

### Retry Logic

Automatically retry failed steps:

```nickel
{
  name = "deploy-app",
  type = 'task,
  retry = {
    max_attempts = 3,
    backoff = 'exponential,  # or 'linear, 'constant
    initial_delay_seconds = 10
  }
}
```

## Checkpoint and Recovery

### Checkpointing

Workflows automatically checkpoint state:

```bash
# Enable checkpointing
provisioning workflow run multi-cloud --checkpoint

# Checkpoint saved at each step completion
```

### Recovery

Resume from last successful checkpoint:

```bash
# Workflow failed at step 3
# Resume from checkpoint
provisioning workflow resume multi-cloud --from-checkpoint latest

# Resume from specific checkpoint
provisioning workflow resume multi-cloud --checkpoint-id abc123
```

## Rollback

### Automatic Rollback

Rollback on failure:

```nickel
{
  rollback_on_failure = true,
  rollback_steps = [
    {name = "destroy-resources", type = 'destroy},
    {name = "restore-config", type = 'restore}
  ]
}
```

### Manual Rollback

```bash
# Rollback to previous state
provisioning workflow rollback multi-cloud

# Rollback to specific checkpoint
provisioning workflow rollback multi-cloud --checkpoint-id abc123
```

## Workflow Management

### CLI Commands

```bash
# List workflows
provisioning workflow list

# Show workflow details
provisioning workflow show multi-cloud

# Run workflow
provisioning workflow run multi-cloud

# Check workflow status
provisioning workflow status multi-cloud

# View workflow logs
provisioning workflow logs multi-cloud

# Cancel running workflow
provisioning workflow cancel multi-cloud
```

### Workflow State

Workflows track execution state:

- `pending` - Not yet started
- `running` - Currently executing
- `completed` - Successfully finished
- `failed` - Execution failed
- `rolling_back` - Performing rollback
- `cancelled` - Manually cancelled

## Advanced Features

### Dynamic Workflows

Generate workflows programmatically:

```nickel
let regions = ["fi-hel1", "de-fra1", "uk-lon1"] in
{
  steps = std.array.map (fun region => {
    name = "provision-" ++ region,
    type = 'provision,
    resources = {servers = [{zone = region, ...}]}
  }) regions
}
```

### Workflow Templates

Reusable workflow templates:

```nickel
let DeploymentTemplate = fun app_name regions => {
  name = "deploy-" ++ app_name,
  steps = std.array.map (fun region => {
    name = "deploy-" ++ region,
    type = 'task,
    tasks = ["deploy-app"],
    config = {app_name = app_name, region = region}
  }) regions
}

# Use template
{
  workflows = [
    DeploymentTemplate "frontend" ["eu", "us"],
    DeploymentTemplate "backend" ["eu", "us", "asia"]
  ]
}
```

### Notifications

Send notifications on workflow events:

```nickel
{
  notifications = {
    on_success = {
      type = 'slack,
      webhook_url = std.secret "SLACK_WEBHOOK",
      message = "Deployment completed successfully"
    },
    on_failure = {
      type = 'email,
      to = ["[ops@example.com](mailto:ops@example.com)"],
      subject = "Workflow failed"
    }
  }
}
```

## Best Practices

1. **Define dependencies explicitly** - Clear dependency graph
2. **Enable checkpointing** - Critical for long-running workflows
3. **Implement rollback** - Always have rollback strategy
4. **Use validation steps** - Verify state before proceeding
5. **Configure retries** - Handle transient failures
6. **Monitor execution** - Track workflow progress
7. **Test workflows** - Validate with dry-run mode

## Troubleshooting

### Workflow Stuck

```bash
# Check workflow status
provisioning workflow status <workflow> --verbose

# View logs
provisioning workflow logs <workflow> --tail 100

# Cancel and restart
provisioning workflow cancel <workflow>
provisioning workflow run <workflow>
```

### Step Failures

```bash
# View failed step details
provisioning workflow show <workflow> --step <step-name>

# Retry failed step
provisioning workflow retry <workflow> --step <step-name>

# Skip failed step
provisioning workflow skip <workflow> --step <step-name>
```

## References

- [Task Services](task-services.md) - Available tasks
- [Clusters](clusters.md) - Cluster orchestration
- [Providers](providers.md) - Multi-cloud provisioning
