prvng_extensions/taskservs/infrastructure/polkadot/zombienet/README.md

# Polkadot Zombienet Task Service

## Overview

The Polkadot Zombienet task service provides a complete installation and configuration of [Zombienet](https://github.com/paritytech/zombienet), a powerful testing framework for Polkadot and Substrate-based networks. Zombienet enables developers to spawn ephemeral networks with multiple validators, parachains, and test scenarios for comprehensive blockchain testing.

## Features

### Network Orchestration
- **Multi-Node Networks** - Deploy complex networks with multiple validators
- **Parachain Support** - Full parachain deployment and testing capabilities
- **Cross-Chain Messaging (XCM)** - Test cross-chain communication scenarios
- **Network Templates** - Pre-configured network topologies
- **Dynamic Configuration** - Flexible network configuration management

### Provider Support
- **Native Provider** - Local binary execution with process management
- **Kubernetes Provider** - Container-based deployment with k8s orchestration
- **Podman Provider** - Alternative container runtime support
- **Docker Provider** - Standard Docker container deployment

### Testing Framework
- **Test Scripts** - Domain-specific language (DSL) for test scenarios
- **Assertion Engine** - Comprehensive assertion and validation system
- **Network Health Checks** - Automated network status validation
- **Performance Testing** - Load testing and performance analysis
- **Regression Testing** - Automated regression test suites
- **Consensus Testing** - Support for Aura, GRANDPA, ELVES, and hybrid consensus testing

### Management Features
- **Network Lifecycle** - Complete network spawn, test, and teardown
- **Resource Management** - Automatic resource cleanup and management
- **Log Aggregation** - Centralized logging from all network components
- **Monitoring Integration** - Prometheus metrics and Grafana dashboards

## Configuration

### Basic Development Network
```kcl
zombienet: PolkadotZombienet = {
    name: "dev-zombienet"
    version: "1.3.40"
    run_user: {
        name: "zombienet"
        home: "/home/zombienet"
    }
    provider: "native"
    work_path: "/var/lib/zombienet"
    config_path: "/etc/zombienet"
    network_config: "simple-network.toml"
    binaries: {
        polkadot: {
            version: "1.5.0"
            path: "/usr/local/bin/polkadot"
        }
    }
    timeout: 1800
    log_level: "info"
}
```

### Production Testing Environment
```kcl
zombienet: PolkadotZombienet = {
    name: "zombienet-testing"
    version: "1.3.40"
    run_user: {
        name: "zombienet"
        group: "zombienet"
        home: "/opt/zombienet"
    }
    provider: "kubernetes"
    work_path: "/var/lib/zombienet"
    config_path: "/etc/zombienet"
    network_config: "multi-parachain-network.toml"
    binaries: {
        polkadot: {
            version: "1.5.0"
            image: "parity/polkadot:v1.5.0"
        }
        parachain: {
            version: "1.5.0"
            image: "parity/cumulus:v1.5.0"
        }
    }
    kubernetes: {
        namespace: "zombienet-testing"
        node_selector: {
            "node-type": "testing"
        }
        resources: {
            limits: {
                cpu: "2"
                memory: "4Gi"
            }
            requests: {
                cpu: "1"
                memory: "2Gi"
            }
        }
    }
    monitoring: {
        prometheus_enabled: true
        grafana_enabled: true
        metrics_port: 9090
    }
    timeout: 3600
    log_level: "debug"
    cleanup_on_exit: true
}
```

### Multi-Parachain XCM Testing
```kcl
zombienet: PolkadotZombienet = {
    name: "xcm-testing-network"
    # ... base configuration
    network_config: "multi-parachain-network.toml"
    network_template: {
        relay_chain: {
            validators: 4
            chain: "rococo-local"
        }
        parachains: [
            {
                id: 100
                name: "asset-hub"
                validators: 2
                chain: "asset-hub-rococo-local"
            },
            {
                id: 200
                name: "bridge-hub"
                validators: 2
                chain: "bridge-hub-rococo-local"
            },
            {
                id: 1000
                name: "custom-parachain"
                validators: 2
                chain: "custom-parachain-local"
                genesis_wasm: "/opt/zombienet/parachains/custom-parachain.wasm"
                genesis_state: "/opt/zombienet/parachains/custom-parachain-genesis"
            }
        ]
    }
    test_scenarios: [
        "test-basic.zndsl",
        "test-parachain.zndsl",
        "test-xcm.zndsl"
    ]
    xcm_testing: {
        enabled: true
        channels: [
            { from: 100, to: 200 },
            { from: 100, to: 1000 },
            { from: 200, to: 1000 }
        ]
    }
}
```

### Kubernetes Production Setup
```kcl
zombienet: PolkadotZombienet = {
    name: "zombienet-k8s-prod"
    # ... base configuration
    provider: "kubernetes"
    kubernetes: {
        namespace: "polkadot-testing"
        storage_class: "fast-ssd"
        node_selector: {
            "testing-tier": "production"
            "node-type": "compute-optimized"
        }
        tolerations: [
            {
                key: "testing-workload"
                operator: "Equal"
                value: "true"
                effect: "NoSchedule"
            }
        ]
        resources: {
            limits: {
                cpu: "4"
                memory: "8Gi"
                storage: "100Gi"
            }
            requests: {
                cpu: "2"
                memory: "4Gi"
                storage: "50Gi"
            }
        }
        security_context: {
            run_as_user: 1000
            run_as_group: 1000
            fs_group: 1000
        }
    }
    monitoring: {
        prometheus_enabled: true
        grafana_enabled: true
        metrics_port: 9090
        service_monitor: true
    }
    persistence: {
        enabled: true
        storage_class: "fast-ssd"
        size: "100Gi"
    }
}
```

### ELVES Consensus Testing Network
```kcl
zombienet: PolkadotZombienet = {
    name: "elves-consensus-testing"
    # ... base configuration
    network_config: "elves-consensus-network.toml"
    network_template: {
        relay_chain: {
            validators: 4
            chain: "rococo-local"
            consensus: {
                type: "elves"
                elves_config: {
                    epoch_duration: 1200  # shorter for testing
                    validators_per_epoch: 4
                    proposal_timeout: 2000
                    prevote_timeout: 2000
                    precommit_timeout: 2000
                    commit_timeout: 500
                }
            }
        }
        parachains: [
            {
                id: 100
                name: "elves-parachain"
                validators: 2
                chain: "elves-parachain-local"
                consensus: {
                    type: "elves"
                    ethereum_compatibility: true
                }
            }
        ]
    }
    test_scenarios: [
        "test-elves-consensus.zndsl",
        "test-elves-epoch-transitions.zndsl",
        "test-elves-ethereum-compatibility.zndsl"
    ]
    elves_testing: {
        enabled: true
        epoch_monitoring: true
        consensus_transition_tests: true
        ethereum_compatibility_tests: true
    }
}
```

### Hybrid Consensus Testing Network
```kcl
zombienet: PolkadotZombienet = {
    name: "hybrid-consensus-testing"
    # ... base configuration
    network_template: {
        relay_chain: {
            validators: 6
            chain: "rococo-local"
            consensus: {
                type: "hybrid"
                primary_consensus: "aura"
                secondary_consensus: "elves"
                transition_rules: {
                    aura_to_elves: {
                        trigger: "network_partition"
                        min_validators: 2
                    }
                    elves_to_aura: {
                        trigger: "network_recovery"
                        consensus_timeout: 10000
                    }
                }
            }
        }
    }
    test_scenarios: [
        "test-hybrid-consensus.zndsl",
        "test-consensus-transitions.zndsl",
        "test-network-partitions.zndsl",
        "test-recovery-scenarios.zndsl"
    ]
    hybrid_testing: {
        enabled: true
        partition_simulation: true
        recovery_testing: true
        transition_monitoring: true
    }
}
```

## Usage

### Deploy Zombienet
```bash
./core/nulib/provisioning taskserv create polkadot-zombienet --infra <infrastructure-name>
```

### List Available Task Services
```bash
./core/nulib/provisioning taskserv list
```

### SSH to Zombienet Server
```bash
./core/nulib/provisioning server ssh <zombienet-server>
```

### Service Management
```bash
# Check Zombienet status
systemctl status zombienet

# Start/stop Zombienet
systemctl start zombienet
systemctl stop zombienet

# View Zombienet logs
journalctl -u zombienet -f

# Check service management
sudo -u zombienet /opt/zombienet/scripts/zombienet-service.sh status
```

### Network Operations
```bash
# Spawn a network
sudo -u zombienet zombienet spawn network-config.toml

# Run tests against network
sudo -u zombienet zombienet test test-basic.zndsl

# Check network status
sudo -u zombienet zombienet info

# Tear down network
sudo -u zombienet zombienet down
```

### Testing Operations
```bash
# Run basic network tests
cd /var/lib/zombienet
sudo -u zombienet zombienet test simple-network.toml test-basic.zndsl

# Run parachain tests
sudo -u zombienet zombienet test parachain-network.toml test-parachain.zndsl

# Run XCM tests
sudo -u zombienet zombienet test multi-parachain-network.toml test-xcm.zndsl

# Custom test execution
sudo -u zombienet zombienet test custom-network.toml custom-test.zndsl

# ELVES Consensus Testing
# Run ELVES consensus tests
sudo -u zombienet zombienet test elves-consensus-network.toml test-elves-consensus.zndsl

# Test ELVES epoch transitions
sudo -u zombienet zombienet test elves-consensus-network.toml test-elves-epoch-transitions.zndsl

# Test ELVES Ethereum compatibility
sudo -u zombienet zombienet test elves-consensus-network.toml test-elves-ethereum-compatibility.zndsl

# Hybrid Consensus Testing
# Test consensus transitions
sudo -u zombienet zombienet test hybrid-consensus-network.toml test-consensus-transitions.zndsl

# Simulate network partitions
sudo -u zombienet zombienet test hybrid-consensus-network.toml test-network-partitions.zndsl
```

### Kubernetes Operations
```bash
# Check pod status
kubectl -n zombienet-testing get pods

# View pod logs
kubectl -n zombienet-testing logs -f zombienet-validator-0

# Execute commands in pods
kubectl -n zombienet-testing exec -it zombienet-validator-0 -- /bin/bash

# Port forward for debugging
kubectl -n zombienet-testing port-forward zombienet-validator-0 9944:9944
```

### Monitoring and Debugging
```bash
# Check network health
curl http://localhost:9933/health

# Monitor metrics
curl http://localhost:9090/metrics

# View aggregated logs
tail -f /var/lib/zombienet/logs/network.log

# Debug network connectivity
sudo -u zombienet zombienet debug connectivity
```

## Architecture

### System Architecture
```
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Test Runner   │────│  Zombienet Core  │────│   Network       │
│                 │    │                  │    │   Components    │
│ • Test Scripts  │    │ • Orchestration  │    │                 │
│ • Assertions    │────│ • Provider Mgmt  │────│ • Validators    │
│ • Scenarios     │    │ • Network Spawn  │    │ • Parachains    │
│ • Reports       │    │ • Health Checks  │    │ • Collators     │
└─────────────────┘    └──────────────────┘    └─────────────────┘
```

### Network Topology Example
```
┌─────────────────────────────────────────────────────────────┐
│                    Relay Chain                              │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐           │
│  │ Validator 1 │ │ Validator 2 │ │ Validator 3 │  ...      │
│  └─────────────┘ └─────────────┘ └─────────────┘           │
└─────────────────────────────────────────────────────────────┘
                           │
      ┌────────────────────┼────────────────────┐
      │                    │                    │
┌─────▼─────┐       ┌─────▼─────┐       ┌─────▼─────┐
│Parachain A│       │Parachain B│       │Parachain C│
│(ID: 100)  │       │(ID: 200)  │       │(ID: 1000) │
│           │◄─XCM──►│           │◄─XCM──►│           │
│Collator 1 │       │Collator 2 │       │Collator 3 │
│Collator 2 │       │Collator 3 │       │Collator 4 │
└───────────┘       └───────────┘       └───────────┘
```

### Provider Architecture
- **Native Provider** - Direct binary execution on host system
- **Kubernetes Provider** - Pod-based deployment with k8s resources
- **Docker Provider** - Container-based deployment with Docker
- **Podman Provider** - Alternative container runtime

### File Structure
```
/var/lib/zombienet/         # Main working directory
├── networks/               # Network configurations
├── binaries/              # Downloaded binaries
├── logs/                  # Network and test logs
├── temp/                  # Temporary files
└── data/                  # Network data and state

/etc/zombienet/            # Configuration directory
├── networks/              # Network templates
│   ├── simple-network.toml
│   ├── parachain-network.toml
│   └── multi-parachain-network.toml
├── tests/                 # Test scenarios
│   ├── test-basic.zndsl
│   ├── test-parachain.zndsl
│   └── test-xcm.zndsl
└── zombienet.conf         # Main configuration

/opt/zombienet/            # Installation directory
├── bin/                   # Zombienet binary
├── scripts/               # Management scripts
└── templates/             # Configuration templates
```

## Supported Operating Systems

- Ubuntu 20.04+ / Debian 11+
- CentOS 8+ / RHEL 8+ / Fedora 35+

## System Requirements

### Native Provider Requirements
- **RAM**: 8GB (16GB recommended for multi-parachain)
- **Storage**: 100GB SSD (200GB+ for extensive testing)
- **CPU**: 4 cores (8 cores recommended)
- **Network**: Good bandwidth for binary downloads

### Kubernetes Provider Requirements
- **Kubernetes Cluster** - v1.20+ with sufficient resources
- **Storage Classes** - Fast SSD storage classes
- **Network Policies** - Proper network connectivity between pods
- **Resource Limits** - Configured resource quotas

### Development Environment
- **RAM**: 4GB (8GB recommended) 
- **Storage**: 50GB SSD
- **CPU**: 2 cores (4 cores recommended)
- **Network**: Standard internet connection

### CI/CD Environment
- **RAM**: 16GB+ (for parallel test execution)
- **Storage**: 200GB+ SSD
- **CPU**: 8+ cores for faster test execution
- **Network**: High bandwidth for artifact downloads

## Troubleshooting

### Network Spawn Issues
```bash
# Check Zombienet logs
journalctl -u zombienet -n 100

# Verify binary availability
ls -la /var/lib/zombienet/binaries/

# Test binary execution
sudo -u zombienet /var/lib/zombienet/binaries/polkadot --version

# Check network configuration
sudo -u zombienet zombienet validate network-config.toml

# Debug network spawn
sudo -u zombienet zombienet --log-level debug spawn network-config.toml
```

### Kubernetes Issues
```bash
# Check pod status
kubectl -n zombienet-testing get pods -o wide

# Describe problematic pods
kubectl -n zombienet-testing describe pod zombienet-validator-0

# Check events
kubectl -n zombienet-testing get events --sort-by='.lastTimestamp'

# View pod logs
kubectl -n zombienet-testing logs zombienet-validator-0 --previous

# Check storage
kubectl -n zombienet-testing get pvc
```

### Test Execution Issues
```bash
# Validate test syntax
sudo -u zombienet zombienet validate-test test-basic.zndsl

# Run test with debug logging
sudo -u zombienet zombienet --log-level debug test network.toml test.zndsl

# Check test reports
ls -la /var/lib/zombienet/reports/

# View detailed test output
cat /var/lib/zombienet/logs/test-execution.log
```

### Resource Issues
```bash
# Check system resources
htop
df -h /var/lib/zombienet

# Monitor memory usage
free -h
cat /proc/meminfo

# Check disk I/O
iostat -x 1

# Monitor network usage
iftop -i eth0
```

### Binary Issues
```bash
# Check binary permissions
ls -la /var/lib/zombienet/binaries/

# Test binary compatibility
file /var/lib/zombienet/binaries/polkadot
ldd /var/lib/zombienet/binaries/polkadot

# Download latest binaries
sudo -u zombienet zombienet setup

# Verify checksums
sudo -u zombienet zombienet verify-binaries
```

## Security Considerations

### Network Security
- **Isolated Networks** - Use isolated network namespaces
- **Port Management** - Careful management of exposed ports
- **Access Control** - Limit access to testing environments
- **Data Cleanup** - Ensure proper cleanup of test data

### Container Security
- **Image Security** - Use trusted container images
- **Resource Limits** - Set appropriate resource limits
- **Security Contexts** - Configure proper security contexts
- **Network Policies** - Implement network segmentation

### CI/CD Security
- **Secrets Management** - Secure handling of test credentials
- **Build Isolation** - Isolate test environments
- **Artifact Security** - Secure storage of test artifacts
- **Access Logs** - Comprehensive logging of access

## Performance Optimization

### System Performance
- **SSD Storage** - Use high-performance SSD storage
- **Memory Allocation** - Sufficient RAM for all network components
- **CPU Resources** - Adequate CPU for parallel operations
- **Network Optimization** - Optimize network settings for containers

### Test Performance  
- **Parallel Execution** - Run tests in parallel where possible
- **Resource Tuning** - Optimize resource allocation per component
- **Caching** - Cache binaries and frequently used data
- **Cleanup Optimization** - Efficient cleanup between tests

### Kubernetes Optimization
- **Node Selection** - Use appropriate node types for testing
- **Resource Requests** - Set accurate resource requests
- **Storage Classes** - Use high-performance storage classes
- **Network CNI** - Optimize network plugin configuration

## Integration Examples

### CI/CD Pipeline Integration
```yaml
# GitHub Actions example
name: Zombienet Tests
on: [push, pull_request]

jobs:
  zombienet-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      
      - name: Setup Zombienet
        run: |
          curl -L -o zombienet https://github.com/paritytech/zombienet/releases/latest/download/zombienet-linux
          chmod +x zombienet
          
      - name: Run Network Tests
        run: |
          ./zombienet test network-config.toml test-suite.zndsl
          
      - name: Upload Test Reports
        uses: actions/upload-artifact@v2
        with:
          name: test-reports
          path: reports/
```

### Kubernetes Deployment
```yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: zombienet-test
  namespace: testing
spec:
  template:
    spec:
      containers:
      - name: zombienet
        image: paritytech/zombienet:latest
        command: ["zombienet"]
        args: ["test", "network-config.toml", "test-suite.zndsl"]
        resources:
          requests:
            memory: "4Gi"
            cpu: "2"
          limits:
            memory: "8Gi"
            cpu: "4"
        volumeMounts:
        - name: config
          mountPath: /config
        - name: storage
          mountPath: /data
      volumes:
      - name: config
        configMap:
          name: zombienet-config
      - name: storage
        emptyDir:
          sizeLimit: 50Gi
      restartPolicy: Never
```

### Monitoring Integration
```yaml
# Prometheus scrape configuration
- job_name: 'zombienet'
  kubernetes_sd_configs:
  - role: pod
    namespaces:
      names:
      - zombienet-testing
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app]
    action: keep
    regex: zombienet
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: (.+)
    replacement: ${1}:9090
```

## Resources

- **Official Documentation**: [paritytech.github.io/zombienet](https://paritytech.github.io/zombienet/)
- **GitHub Repository**: [paritytech/zombienet](https://github.com/paritytech/zombienet)
- **Polkadot Wiki**: [wiki.polkadot.network/docs/learn-launch](https://wiki.polkadot.network/docs/learn-launch)
- **Substrate Documentation**: [docs.substrate.io](https://docs.substrate.io)
- **Community**: [substrate.stackexchange.com](https://substrate.stackexchange.com)