provisioning/docs/src/testing/test-environment-system.md

Test Environment Service (v3.4.0)

🚀 Test Environment Service Completed (2025-10-06)

A comprehensive containerized test environment service has been integrated into the orchestrator, enabling automated testing of taskservs, complete servers, and multi-node clusters without manual Docker management.

Key Features

• Automated Container Management: No manual Docker operations required
• Three Test Environment Types: Single taskserv, server simulation, multi-node clusters
• Multi-Node Support: Test complex topologies (Kubernetes HA, etcd clusters)
• Network Isolation: Each test environment gets dedicated Docker networks
• Resource Management: Configurable CPU, memory, and disk limits
• Topology Templates: Predefined cluster configurations for common scenarios
• Auto-Cleanup: Optional automatic cleanup after tests complete
• CI/CD Integration: Easy integration into automated pipelines

Test Environment Types

1. Single Taskserv Testing

Test individual taskserv in isolated container:

# Quick test (create, run, cleanup)
provisioning test quick kubernetes

# With custom resources
provisioning test env single postgres --cpu 2000 --memory 4096 --auto-start --auto-cleanup

# With infrastructure context
provisioning test env single redis --infra my-project

2. Server Simulation

Test complete server configurations with multiple taskservs:

# Simulate web server
provisioning test env server web-01 [containerd kubernetes cilium] --auto-start

# Simulate database server
provisioning test env server db-01 [postgres redis] --infra prod-stack --auto-start

3. Multi-Node Cluster Topology

Test complex cluster configurations before deployment:

# 3-node Kubernetes HA cluster
provisioning test topology load kubernetes_3node | test env cluster kubernetes --auto-start

# etcd cluster
provisioning test topology load etcd_cluster | test env cluster etcd --auto-start

# Single-node Kubernetes
provisioning test topology load kubernetes_single | test env cluster kubernetes

Test Environment Management

# List all test environments
provisioning test env list

# Check environment status
provisioning test env status <env-id>

# View environment logs
provisioning test env logs <env-id>

# Run tests in environment
provisioning test env run <env-id>

# Cleanup environment
provisioning test env cleanup <env-id>

Available Topology Templates

Predefined multi-node cluster templates in provisioning/config/test-topologies.toml:

Template	Description	Nodes	Use Case
kubernetes_3node	K8s HA cluster	1 CP + 2 workers	Production-like testing
kubernetes_single	All-in-one K8s	1 node	Development testing
etcd_cluster	etcd cluster	3 members	Distributed consensus
containerd_test	Standalone containerd	1 node	Container runtime
postgres_redis	Database stack	2 nodes	Database integration

REST API Endpoints

The orchestrator exposes test environment endpoints:

• Create Environment: POST http://localhost:9090/v1/test/environments/create
• List Environments: GET http://localhost:9090/v1/test/environments
• Get Environment: GET http://localhost:9090/v1/test/environments/{id}
• Run Tests: POST http://localhost:9090/v1/test/environments/{id}/run
• Cleanup: DELETE http://localhost:9090/v1/test/environments/{id}
• Get Logs: GET http://localhost:9090/v1/test/environments/{id}/logs

Prerequisites

1. Docker Running: Test environments require Docker daemon

   docker ps  # Should work without errors
   

2. Orchestrator Running: Start the orchestrator to manage test containers

   cd provisioning/platform/orchestrator
   ./scripts/start-orchestrator.nu --background
   

Architecture

User Command (CLI/API)
    ↓
Test Orchestrator (Rust)
    ↓
Container Manager (bollard)
    ↓
Docker API
    ↓
Isolated Test Containers
    • Dedicated networks
    • Resource limits
    • Volume mounts
    • Multi-node support

Configuration

• Topology Templates: provisioning/config/test-topologies.toml
• Default Resources: 1000 millicores CPU, 2048 MB memory
• Network: 172.20.0.0/16 (default subnet)
• Base Image: ubuntu:22.04 (configurable)

Use Cases

1. Taskserv Development: Test new taskservs before deployment
2. Integration Testing: Validate taskserv combinations
3. Cluster Validation: Test multi-node configurations
4. CI/CD Integration: Automated infrastructure testing
5. Production Simulation: Test production-like deployments safely

CI/CD Integration Example

# GitLab CI
test-infrastructure:
  stage: test
  script:
    - ./scripts/start-orchestrator.nu --background
    - provisioning test quick kubernetes
    - provisioning test quick postgres
    - provisioning test quick redis
    - provisioning test topology load kubernetes_3node |
        test env cluster kubernetes --auto-start
  artifacts:
    when: on_failure
    paths:
      - test-logs/

Documentation

Complete documentation available:

• User Guide: Test Environment Guide
• Detailed Usage: Test Environment Usage
• Orchestrator README: Orchestrator

Command Shortcuts

Test commands are integrated into the CLI with shortcuts:

• test or tst - Test command prefix
• test quick <taskserv> - One-command test
• test env single/server/cluster - Create test environments
• test topology load/list - Manage topology templates