provisioning/docs/src/testing/test-environment-system.md

# Test Environment Service (v3.4.0)

## 🚀 Test Environment Service Completed (2025-10-06)

A comprehensive containerized test environment service has been integrated into the orchestrator, enabling automated testing of taskservs, complete
servers, and multi-node clusters without manual Docker management.

## Key Features

- **Automated Container Management**: No manual Docker operations required
- **Three Test Environment Types**: Single taskserv, server simulation, multi-node clusters
- **Multi-Node Support**: Test complex topologies (Kubernetes HA, etcd clusters)
- **Network Isolation**: Each test environment gets dedicated Docker networks
- **Resource Management**: Configurable CPU, memory, and disk limits
- **Topology Templates**: Predefined cluster configurations for common scenarios
- **Auto-Cleanup**: Optional automatic cleanup after tests complete
- **CI/CD Integration**: Easy integration into automated pipelines

## Test Environment Types

### 1. Single Taskserv Testing

Test individual taskserv in isolated container:

```bash
# Quick test (create, run, cleanup)
provisioning test quick kubernetes

# With custom resources
provisioning test env single postgres --cpu 2000 --memory 4096 --auto-start --auto-cleanup

# With infrastructure context
provisioning test env single redis --infra my-project
```

### 2. Server Simulation

Test complete server configurations with multiple taskservs:

```bash
# Simulate web server
provisioning test env server web-01 [containerd kubernetes cilium] --auto-start

# Simulate database server
provisioning test env server db-01 [postgres redis] --infra prod-stack --auto-start
```

### 3. Multi-Node Cluster Topology

Test complex cluster configurations before deployment:

```bash
# 3-node Kubernetes HA cluster
provisioning test topology load kubernetes_3node | test env cluster kubernetes --auto-start

# etcd cluster
provisioning test topology load etcd_cluster | test env cluster etcd --auto-start

# Single-node Kubernetes
provisioning test topology load kubernetes_single | test env cluster kubernetes
```

## Test Environment Management

```bash
# List all test environments
provisioning test env list

# Check environment status
provisioning test env status <env-id>

# View environment logs
provisioning test env logs <env-id>

# Run tests in environment
provisioning test env run <env-id>

# Cleanup environment
provisioning test env cleanup <env-id>
```

## Available Topology Templates

Predefined multi-node cluster templates in `provisioning/config/test-topologies.toml`:

| Template | Description | Nodes | Use Case |
| ---------- | ------------- | ------- | ---------- |
| `kubernetes_3node` | K8s HA cluster | 1 CP + 2 workers | Production-like testing |
| `kubernetes_single` | All-in-one K8s | 1 node | Development testing |
| `etcd_cluster` | etcd cluster | 3 members | Distributed consensus |
| `containerd_test` | Standalone containerd | 1 node | Container runtime |
| `postgres_redis` | Database stack | 2 nodes | Database integration |

## REST API Endpoints

The orchestrator exposes test environment endpoints:

- **Create Environment**: `POST http://localhost:9090/v1/test/environments/create`
- **List Environments**: `GET http://localhost:9090/v1/test/environments`
- **Get Environment**: `GET http://localhost:9090/v1/test/environments/{id}`
- **Run Tests**: `POST http://localhost:9090/v1/test/environments/{id}/run`
- **Cleanup**: `DELETE http://localhost:9090/v1/test/environments/{id}`
- **Get Logs**: `GET http://localhost:9090/v1/test/environments/{id}/logs`

## Prerequisites

1. **Docker Running**: Test environments require Docker daemon

   ```bash
   docker ps  # Should work without errors
   ```

1. **Orchestrator Running**: Start the orchestrator to manage test containers

   ```bash
   cd provisioning/platform/orchestrator
   ./scripts/start-orchestrator.nu --background
   ```

## Architecture

```plaintext
User Command (CLI/API)
    ↓
Test Orchestrator (Rust)
    ↓
Container Manager (bollard)
    ↓
Docker API
    ↓
Isolated Test Containers
    • Dedicated networks
    • Resource limits
    • Volume mounts
    • Multi-node support
```

## Configuration

- **Topology Templates**: `provisioning/config/test-topologies.toml`
- **Default Resources**: 1000 millicores CPU, 2048 MB memory
- **Network**: 172.20.0.0/16 (default subnet)
- **Base Image**: ubuntu:22.04 (configurable)

## Use Cases

1. **Taskserv Development**: Test new taskservs before deployment
2. **Integration Testing**: Validate taskserv combinations
3. **Cluster Validation**: Test multi-node configurations
4. **CI/CD Integration**: Automated infrastructure testing
5. **Production Simulation**: Test production-like deployments safely

## CI/CD Integration Example

```yaml
# GitLab CI
test-infrastructure:
  stage: test
  script:
    - ./scripts/start-orchestrator.nu --background
    - provisioning test quick kubernetes
    - provisioning test quick postgres
    - provisioning test quick redis
    - provisioning test topology load kubernetes_3node |
        test env cluster kubernetes --auto-start
  artifacts:
    when: on_failure
    paths:
      - test-logs/
```

## Documentation

Complete documentation available:

- **User Guide**: [Test Environment Guide](../testing/test-environment-guide.md)
- **Detailed Usage**: [Test Environment Usage](../testing/test-environment-usage.md)
- **Orchestrator README**: [Orchestrator](../operations/orchestrator-system.md)

## Command Shortcuts

Test commands are integrated into the CLI with shortcuts:

- `test` or `tst` - Test command prefix
- `test quick <taskserv>` - One-command test
- `test env single/server/cluster` - Create test environments
- `test topology load/list` - Manage topology templates