# Batch Operations Examples

Multi-cloud infrastructure orchestration using Provisioning's batch workflow system.

## Overview

Batch workflows coordinate deployments across multiple providers, environments, and
regions in a single declarative operation.

## Multi-Cloud Deployment

Deploy the same infrastructure across AWS, UpCloud, and Hetzner simultaneously:

```nickel
{
  workspace = "multi-cloud",

  infrastructure = {
    environments = {
      production = {
        providers = [
          { name = "aws", region = "us-east-1" },
          { name = "upcloud", region = "us-sjc" },
          { name = "hetzner", location = "fsn1" }
        ],

        servers = [
          {
            name = "web-01",
            provider = "aws",
            instance_type = "t3.large",
            count = 3
          },
          {
            name = "api-01",
            provider = "upcloud",
            plan = "4xCPU-8GB",
            count = 2
          },
          {
            name = "db-01",
            provider = "hetzner",
            server_type = "cx21",
            count = 1
          }
        ],

        load_balancer = {
          name = "global-lb",
          type = "round-robin",
          backends = ["aws", "upcloud", "hetzner"],
          health_check = {
            interval = 30,
            timeout = 5,
            threshold = 2
          }
        }
      }
    }
  }
}
```

Submit this workflow:

```bash
provisioning batch submit workflows/multi-cloud-deploy.ncl
```

Monitor execution:

```bash
provisioning batch status <workflow-id>
provisioning batch logs <workflow-id> --follow
```

## Blue-Green Deployment

Perform zero-downtime deployment with automatic rollback:

```nickel
{
  batch = {
    name = "blue-green-deployment",
    description = "Zero-downtime app deployment",

    operations = [
      {
        name = "validate-blue",
        action = "verify_infrastructure",
        target = "environment:blue"
      },
      {
        name = "health-check-blue",
        action = "run_health_check",
        target = "environment:blue",
        depends_on = ["validate-blue"]
      },
      {
        name = "deploy-green",
        action = "create_infrastructure",
        target = "environment:green",
        config = "deployment/green-config.ncl",
        depends_on = ["health-check-blue"]
      },
      {
        name = "smoke-tests-green",
        action = "run_tests",
        target = "environment:green",
        test_suite = "smoke-tests",
        depends_on = ["deploy-green"]
      },
      {
        name = "traffic-shift",
        action = "update_load_balancer",
        shift_percentage = 100,
        target = "environment:green",
        depends_on = ["smoke-tests-green"],
        rollback_condition = "error_rate > 0.5%"
      },
      {
        name = "cleanup-blue",
        action = "destroy_infrastructure",
        target = "environment:blue",
        depends_on = ["traffic-shift"],
        post_delay = 300
      }
    ]
  }
}
```

## Disaster Recovery Drill

Automated DR test across regions:

```bash
provisioning batch create-workflow \
  --name "dr-drill-$(date +%s)" \
  --description "Monthly DR test" \
  --operations "
    - name: snapshot-production
      action: create_snapshot
      target: environment:production

    - name: restore-dr
      action: restore_from_snapshot
      target: environment:dr
      depends_on: [snapshot-production]

    - name: verify-dr
      action: validate_infrastructure
      target: environment:dr
      depends_on: [restore-dr]

    - name: failover-test
      action: test_failover
      target: environment:dr
      duration: 1800
      depends_on: [verify-dr]

    - name: cleanup
      action: destroy_test_resources
      target: environment:dr
      depends_on: [failover-test]
  " \
  --max_parallel_tasks 4
```

## Rolling Update with Canary

Gradually roll out new version with traffic analysis:

```nickel
{
  batch = {
    name = "canary-rollout",

    variables = {
      new_version = "v3.2.0",
      canary_percentage = 5,
      analysis_duration = 900
    },

    operations = [
      {
        name = "stage-1-5-percent",
        action = "deploy_version",
        version = "${new_version}",
        replicas = 1,
        traffic_shift = 5
      },
      {
        name = "monitor-stage-1",
        action = "analyze_metrics",
        metric_query = "error_rate{version=\"${new_version}\"}",
        duration = "${analysis_duration}",
        threshold = "< 0.1%",
        depends_on = ["stage-1-5-percent"]
      },
      {
        name = "stage-2-25-percent",
        action = "deploy_version",
        version = "${new_version}",
        replicas = 5,
        traffic_shift = 25,
        depends_on = ["monitor-stage-1"]
      },
      {
        name = "monitor-stage-2",
        action = "analyze_metrics",
        metric_query = "error_rate{version=\"${new_version}\"}",
        duration = "${analysis_duration}",
        threshold = "< 0.1%",
        depends_on = ["stage-2-25-percent"]
      },
      {
        name = "complete-rollout",
        action = "deploy_version",
        version = "${new_version}",
        replicas = 20,
        traffic_shift = 100,
        depends_on = ["monitor-stage-2"]
      }
    ]
  }
}
```

## Parallel Infrastructure Setup

Create multiple independent resources in parallel:

```bash
provisioning batch submit workflows/parallel-setup.ncl \
  --max_parallel_tasks 10
```

This executes up to 10 non-dependent tasks simultaneously, reducing total execution
time from serial O(n) to parallel O(n/10).

## Cross-Region Replication

Set up database replication across regions:

```nickel
{
  batch = {
    name = "cross-region-replication",

    operations = [
      # Primary region setup
      {
        name = "create-primary-db",
        action = "create_database",
        provider = "aws",
        region = "us-east-1",
        engine = "postgres",
        multi_az = true
      },

      # Read replicas in other regions
      {
        name = "create-replica-upcloud",
        action = "create_read_replica",
        provider = "upcloud",
        region = "us-sjc",
        source_db = "primary-db",
        depends_on = ["create-primary-db"]
      },
      {
        name = "create-replica-hetzner",
        action = "create_read_replica",
        provider = "hetzner",
        region = "fsn1",
        source_db = "primary-db",
        depends_on = ["create-primary-db"]
      },

      # Configure replication monitoring
      {
        name = "monitor-replication",
        action = "setup_monitoring",
        targets = ["primary-db", "replica-upcloud", "replica-hetzner"],
        metrics = ["replication_lag", "write_throughput"],
        depends_on = ["create-replica-upcloud", "create-replica-hetzner"]
      }
    ]
  }
}
```

Monitor replication lag:

```bash
provisioning batch logs <workflow-id> --filter "replication_lag"
```

## Related Documentation

- [Batch Workflow System](../infrastructure/batch-workflows.md)
- [Orchestrator Guide](../features/orchestrator.md)
- [Disaster Recovery Guide](../guides/disaster-recovery.md)
