# Test Environment Guide

**Version**: 1.0.0
**Date**: 2025-10-06
**Status**: Production Ready

---

## Overview

The Test Environment Service provides automated containerized testing for taskservs, servers, and multi-node clusters. Built into the orchestrator, it eliminates manual Docker management and provides realistic test scenarios.

## Architecture

```plaintext
┌─────────────────────────────────────────────────┐
│         Orchestrator (port 8080)                │
│  ┌──────────────────────────────────────────┐  │
│  │  Test Orchestrator                       │  │
│  │  • Container Manager (Docker API)        │  │
│  │  • Network Isolation                     │  │
│  │  • Multi-node Topologies                 │  │
│  │  • Test Execution                        │  │
│  └──────────────────────────────────────────┘  │
└─────────────────────────────────────────────────┘
                      ↓
         ┌────────────────────────┐
         │   Docker Containers    │
         │  • Isolated Networks   │
         │  • Resource Limits     │
         │  • Volume Mounts       │
         └────────────────────────┘
```plaintext

## Test Environment Types

### 1. Single Taskserv Test

Test individual taskserv in isolated container.

```bash
# Basic test
provisioning test env single kubernetes

# With resource limits
provisioning test env single redis --cpu 2000 --memory 4096

# Auto-start and cleanup
provisioning test quick postgres
```plaintext

### 2. Server Simulation

Simulate complete server with multiple taskservs.

```bash
# Server with taskservs
provisioning test env server web-01 [containerd kubernetes cilium]

# With infrastructure context
provisioning test env server db-01 [postgres redis] --infra prod-stack
```plaintext

### 3. Cluster Topology

Multi-node cluster simulation from templates.

```bash
# 3-node Kubernetes cluster
provisioning test topology load kubernetes_3node | test env cluster kubernetes --auto-start

# etcd cluster
provisioning test topology load etcd_cluster | test env cluster etcd
```plaintext

## Quick Start

### Prerequisites

1. **Docker running:**

   ```bash
   docker ps  # Should work without errors
   ```

1. **Orchestrator running:**

   ```bash
   cd provisioning/platform/orchestrator
   ./scripts/start-orchestrator.nu --background
   ```

### Basic Workflow

```bash
# 1. Quick test (fastest)
provisioning test quick kubernetes

# 2. Or step-by-step
# Create environment
provisioning test env single kubernetes --auto-start

# List environments
provisioning test env list

# Check status
provisioning test env status <env-id>

# View logs
provisioning test env logs <env-id>

# Cleanup
provisioning test env cleanup <env-id>
```plaintext

## Topology Templates

### Available Templates

```bash
# List templates
provisioning test topology list
```plaintext

| Template | Description | Nodes |
|----------|-------------|-------|
| `kubernetes_3node` | K8s HA cluster | 1 CP + 2 workers |
| `kubernetes_single` | All-in-one K8s | 1 node |
| `etcd_cluster` | etcd cluster | 3 members |
| `containerd_test` | Standalone containerd | 1 node |
| `postgres_redis` | Database stack | 2 nodes |

### Using Templates

```bash
# Load and use template
provisioning test topology load kubernetes_3node | test env cluster kubernetes

# View template
provisioning test topology load etcd_cluster
```plaintext

### Custom Topology

Create `my-topology.toml`:

```toml
[my_cluster]
name = "My Custom Cluster"
cluster_type = "custom"

[[my_cluster.nodes]]
name = "node-01"
role = "primary"
taskservs = ["postgres", "redis"]
[my_cluster.nodes.resources]
cpu_millicores = 2000
memory_mb = 4096

[[my_cluster.nodes]]
name = "node-02"
role = "replica"
taskservs = ["postgres"]
[my_cluster.nodes.resources]
cpu_millicores = 1000
memory_mb = 2048

[my_cluster.network]
subnet = "172.30.0.0/16"
```plaintext

## Commands Reference

### Environment Management

```bash
# Create from config
provisioning test env create <config>

# Single taskserv
provisioning test env single <taskserv> [--cpu N] [--memory MB]

# Server simulation
provisioning test env server <name> <taskservs> [--infra NAME]

# Cluster topology
provisioning test env cluster <type> <topology>

# List environments
provisioning test env list

# Get details
provisioning test env get <env-id>

# Show status
provisioning test env status <env-id>
```plaintext

### Test Execution

```bash
# Run tests
provisioning test env run <env-id> [--tests [test1, test2]]

# View logs
provisioning test env logs <env-id>

# Cleanup
provisioning test env cleanup <env-id>
```plaintext

### Quick Test

```bash
# One-command test (create, run, cleanup)
provisioning test quick <taskserv> [--infra NAME]
```plaintext

## REST API

### Create Environment

```bash
curl -X POST http://localhost:9090/test/environments/create \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "type": "single_taskserv",
      "taskserv": "kubernetes",
      "base_image": "ubuntu:22.04",
      "environment": {},
      "resources": {
        "cpu_millicores": 2000,
        "memory_mb": 4096
      }
    },
    "infra": "my-project",
    "auto_start": true,
    "auto_cleanup": false
  }'
```plaintext

### List Environments

```bash
curl http://localhost:9090/test/environments
```plaintext

### Run Tests

```bash
curl -X POST http://localhost:9090/test/environments/{id}/run \
  -H "Content-Type: application/json" \
  -d '{
    "tests": [],
    "timeout_seconds": 300
  }'
```plaintext

### Cleanup

```bash
curl -X DELETE http://localhost:9090/test/environments/{id}
```plaintext

## Use Cases

### 1. Taskserv Development

Test taskserv before deployment:

```bash
# Test new taskserv version
provisioning test env single my-taskserv --auto-start

# Check logs
provisioning test env logs <env-id>
```plaintext

### 2. Multi-Taskserv Integration

Test taskserv combinations:

```bash
# Test kubernetes + cilium + containerd
provisioning test env server k8s-test [kubernetes cilium containerd] --auto-start
```plaintext

### 3. Cluster Validation

Test cluster configurations:

```bash
# Test 3-node etcd cluster
provisioning test topology load etcd_cluster | test env cluster etcd --auto-start
```plaintext

### 4. CI/CD Integration

```yaml
# .gitlab-ci.yml
test-taskserv:
  stage: test
  script:
    - provisioning test quick kubernetes
    - provisioning test quick redis
    - provisioning test quick postgres
```plaintext

## Advanced Features

### Resource Limits

```bash
# Custom CPU and memory
provisioning test env single postgres \
  --cpu 4000 \
  --memory 8192
```plaintext

### Network Isolation

Each environment gets isolated network:

- Subnet: 172.20.0.0/16 (default)
- DNS enabled
- Container-to-container communication

### Auto-Cleanup

```bash
# Auto-cleanup after tests
provisioning test env single redis --auto-start --auto-cleanup
```plaintext

### Multiple Environments

Run tests in parallel:

```bash
# Create multiple environments
provisioning test env single kubernetes --auto-start &
provisioning test env single postgres --auto-start &
provisioning test env single redis --auto-start &

wait

# List all
provisioning test env list
```plaintext

## Troubleshooting

### Docker not running

```plaintext
Error: Failed to connect to Docker
```plaintext

**Solution:**

```bash
# Check Docker
docker ps

# Start Docker daemon
sudo systemctl start docker  # Linux
open -a Docker  # macOS
```plaintext

### Orchestrator not running

```plaintext
Error: Connection refused (port 8080)
```plaintext

**Solution:**

```bash
cd provisioning/platform/orchestrator
./scripts/start-orchestrator.nu --background
```plaintext

### Environment creation fails

Check logs:

```bash
provisioning test env logs <env-id>
```plaintext

Check Docker:

```bash
docker ps -a
docker logs <container-id>
```plaintext

### Out of resources

```plaintext
Error: Cannot allocate memory
```plaintext

**Solution:**

```bash
# Cleanup old environments
provisioning test env list | each {|env| provisioning test env cleanup $env.id }

# Or cleanup Docker
docker system prune -af
```plaintext

## Best Practices

### 1. Use Templates

Reuse topology templates instead of recreating:

```bash
provisioning test topology load kubernetes_3node | test env cluster kubernetes
```plaintext

### 2. Auto-Cleanup

Always use auto-cleanup in CI/CD:

```bash
provisioning test quick <taskserv>  # Includes auto-cleanup
```plaintext

### 3. Resource Planning

Adjust resources based on needs:

- Development: 1-2 cores, 2 GB RAM
- Integration: 2-4 cores, 4-8 GB RAM
- Production-like: 4+ cores, 8+ GB RAM

### 4. Parallel Testing

Run independent tests in parallel:

```bash
for taskserv in [kubernetes postgres redis] {
    provisioning test quick $taskserv &
}
wait
```plaintext

## Configuration

### Default Settings

- Base image: `ubuntu:22.04`
- CPU: 1000 millicores (1 core)
- Memory: 2048 MB (2 GB)
- Network: 172.20.0.0/16

### Custom Config

```bash
# Override defaults
provisioning test env single postgres \
  --base-image debian:12 \
  --cpu 2000 \
  --memory 4096
```plaintext

---

## Related Documentation

- [Test Environment API](../api/test-environment-api.md)
- [Topology Templates](../architecture/test-topologies.md)
- [Orchestrator Guide](orchestrator-guide.md)
- [Taskserv Development](taskserv-development.md)

---

## Version History

| Version | Date | Changes |
|---------|------|---------|
| 1.0.0 | 2025-10-06 | Initial test environment service |

---

**Maintained By**: Infrastructure Team
