# Test Environment Guide\n\n**Version**: 1.0.0\n**Date**: 2025-10-06\n**Status**: Production Ready\n\n---\n\n## Overview\n\nThe Test Environment Service provides automated containerized testing for taskservs, servers, and multi-node clusters. Built into the orchestrator, it\neliminates manual Docker management and provides realistic test scenarios.\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────┐\n│         Orchestrator (port 8080)                │\n│  ┌──────────────────────────────────────────┐  │\n│  │  Test Orchestrator                       │  │\n│  │  • Container Manager (Docker API)        │  │\n│  │  • Network Isolation                     │  │\n│  │  • Multi-node Topologies                 │  │\n│  │  • Test Execution                        │  │\n│  └──────────────────────────────────────────┘  │\n└─────────────────────────────────────────────────┘\n                      ↓\n         ┌────────────────────────┐\n         │   Docker Containers    │\n         │  • Isolated Networks   │\n         │  • Resource Limits     │\n         │  • Volume Mounts       │\n         └────────────────────────┘\n```\n\n## Test Environment Types\n\n### 1. Single Taskserv Test\n\nTest individual taskserv in isolated container.\n\n```\n# Basic test\nprovisioning test env single kubernetes\n\n# With resource limits\nprovisioning test env single redis --cpu 2000 --memory 4096\n\n# Auto-start and cleanup\nprovisioning test quick postgres\n```\n\n### 2. Server Simulation\n\nSimulate complete server with multiple taskservs.\n\n```\n# Server with taskservs\nprovisioning test env server web-01 [containerd kubernetes cilium]\n\n# With infrastructure context\nprovisioning test env server db-01 [postgres redis] --infra prod-stack\n```\n\n### 3. Cluster Topology\n\nMulti-node cluster simulation from templates.\n\n```\n# 3-node Kubernetes cluster\nprovisioning test topology load kubernetes_3node | test env cluster kubernetes --auto-start\n\n# etcd cluster\nprovisioning test topology load etcd_cluster | test env cluster etcd\n```\n\n## Quick Start\n\n### Prerequisites\n\n1. **Docker running:**\n\n   ```bash\n   docker ps  # Should work without errors\n   ```\n\n1. **Orchestrator running:**\n\n   ```bash\n   cd provisioning/platform/orchestrator\n   ./scripts/start-orchestrator.nu --background\n   ```\n\n### Basic Workflow\n\n```\n# 1. Quick test (fastest)\nprovisioning test quick kubernetes\n\n# 2. Or step-by-step\n# Create environment\nprovisioning test env single kubernetes --auto-start\n\n# List environments\nprovisioning test env list\n\n# Check status\nprovisioning test env status <env-id>\n\n# View logs\nprovisioning test env logs <env-id>\n\n# Cleanup\nprovisioning test env cleanup <env-id>\n```\n\n## Topology Templates\n\n### Available Templates\n\n```\n# List templates\nprovisioning test topology list\n```\n\n| Template | Description | Nodes |\n| ---------- | ------------- | ------- |\n| `kubernetes_3node` | K8s HA cluster | 1 CP + 2 workers |\n| `kubernetes_single` | All-in-one K8s | 1 node |\n| `etcd_cluster` | etcd cluster | 3 members |\n| `containerd_test` | Standalone containerd | 1 node |\n| `postgres_redis` | Database stack | 2 nodes |\n\n### Using Templates\n\n```\n# Load and use template\nprovisioning test topology load kubernetes_3node | test env cluster kubernetes\n\n# View template\nprovisioning test topology load etcd_cluster\n```\n\n### Custom Topology\n\nCreate `my-topology.toml`:\n\n```\n[my_cluster]\nname = "My Custom Cluster"\ncluster_type = "custom"\n\n[[my_cluster.nodes]]\nname = "node-01"\nrole = "primary"\ntaskservs = ["postgres", "redis"]\n[my_cluster.nodes.resources]\ncpu_millicores = 2000\nmemory_mb = 4096\n\n[[my_cluster.nodes]]\nname = "node-02"\nrole = "replica"\ntaskservs = ["postgres"]\n[my_cluster.nodes.resources]\ncpu_millicores = 1000\nmemory_mb = 2048\n\n[my_cluster.network]\nsubnet = "172.30.0.0/16"\n```\n\n## Commands Reference\n\n### Environment Management\n\n```\n# Create from config\nprovisioning test env create <config>\n\n# Single taskserv\nprovisioning test env single <taskserv> [--cpu N] [--memory MB]\n\n# Server simulation\nprovisioning test env server <name> <taskservs> [--infra NAME]\n\n# Cluster topology\nprovisioning test env cluster <type> <topology>\n\n# List environments\nprovisioning test env list\n\n# Get details\nprovisioning test env get <env-id>\n\n# Show status\nprovisioning test env status <env-id>\n```\n\n### Test Execution\n\n```\n# Run tests\nprovisioning test env run <env-id> [--tests [test1, test2]]\n\n# View logs\nprovisioning test env logs <env-id>\n\n# Cleanup\nprovisioning test env cleanup <env-id>\n```\n\n### Quick Test\n\n```\n# One-command test (create, run, cleanup)\nprovisioning test quick <taskserv> [--infra NAME]\n```\n\n## REST API\n\n### Create Environment\n\n```\ncurl -X POST http://localhost:9090/test/environments/create \\n  -H "Content-Type: application/json" \\n  -d '{\n    "config": {\n      "type": "single_taskserv",\n      "taskserv": "kubernetes",\n      "base_image": "ubuntu:22.04",\n      "environment": {},\n      "resources": {\n        "cpu_millicores": 2000,\n        "memory_mb": 4096\n      }\n    },\n    "infra": "my-project",\n    "auto_start": true,\n    "auto_cleanup": false\n  }'\n```\n\n### List Environments\n\n```\ncurl http://localhost:9090/test/environments\n```\n\n### Run Tests\n\n```\ncurl -X POST http://localhost:9090/test/environments/{id}/run \\n  -H "Content-Type: application/json" \\n  -d '{\n    "tests": [],\n    "timeout_seconds": 300\n  }'\n```\n\n### Cleanup\n\n```\ncurl -X DELETE http://localhost:9090/test/environments/{id}\n```\n\n## Use Cases\n\n### 1. Taskserv Development\n\nTest taskserv before deployment:\n\n```\n# Test new taskserv version\nprovisioning test env single my-taskserv --auto-start\n\n# Check logs\nprovisioning test env logs <env-id>\n```\n\n### 2. Multi-Taskserv Integration\n\nTest taskserv combinations:\n\n```\n# Test kubernetes + cilium + containerd\nprovisioning test env server k8s-test [kubernetes cilium containerd] --auto-start\n```\n\n### 3. Cluster Validation\n\nTest cluster configurations:\n\n```\n# Test 3-node etcd cluster\nprovisioning test topology load etcd_cluster | test env cluster etcd --auto-start\n```\n\n### 4. CI/CD Integration\n\n```\n# .gitlab-ci.yml\ntest-taskserv:\n  stage: test\n  script:\n    - provisioning test quick kubernetes\n    - provisioning test quick redis\n    - provisioning test quick postgres\n```\n\n## Advanced Features\n\n### Resource Limits\n\n```\n# Custom CPU and memory\nprovisioning test env single postgres \\n  --cpu 4000 \\n  --memory 8192\n```\n\n### Network Isolation\n\nEach environment gets isolated network:\n\n- Subnet: 172.20.0.0/16 (default)\n- DNS enabled\n- Container-to-container communication\n\n### Auto-Cleanup\n\n```\n# Auto-cleanup after tests\nprovisioning test env single redis --auto-start --auto-cleanup\n```\n\n### Multiple Environments\n\nRun tests in parallel:\n\n```\n# Create multiple environments\nprovisioning test env single kubernetes --auto-start &\nprovisioning test env single postgres --auto-start &\nprovisioning test env single redis --auto-start &\n\nwait\n\n# List all\nprovisioning test env list\n```\n\n## Troubleshooting\n\n### Docker not running\n\n```\nError: Failed to connect to Docker\n```\n\n**Solution:**\n\n```\n# Check Docker\ndocker ps\n\n# Start Docker daemon\nsudo systemctl start docker  # Linux\nopen -a Docker  # macOS\n```\n\n### Orchestrator not running\n\n```\nError: Connection refused (port 8080)\n```\n\n**Solution:**\n\n```\ncd provisioning/platform/orchestrator\n./scripts/start-orchestrator.nu --background\n```\n\n### Environment creation fails\n\nCheck logs:\n\n```\nprovisioning test env logs <env-id>\n```\n\nCheck Docker:\n\n```\ndocker ps -a\ndocker logs <container-id>\n```\n\n### Out of resources\n\n```\nError: Cannot allocate memory\n```\n\n**Solution:**\n\n```\n# Cleanup old environments\nprovisioning test env list | each {|env| provisioning test env cleanup $env.id }\n\n# Or cleanup Docker\ndocker system prune -af\n```\n\n## Best Practices\n\n### 1. Use Templates\n\nReuse topology templates instead of recreating:\n\n```\nprovisioning test topology load kubernetes_3node | test env cluster kubernetes\n```\n\n### 2. Auto-Cleanup\n\nAlways use auto-cleanup in CI/CD:\n\n```\nprovisioning test quick <taskserv>  # Includes auto-cleanup\n```\n\n### 3. Resource Planning\n\nAdjust resources based on needs:\n\n- Development: 1-2 cores, 2 GB RAM\n- Integration: 2-4 cores, 4-8 GB RAM\n- Production-like: 4+ cores, 8+ GB RAM\n\n### 4. Parallel Testing\n\nRun independent tests in parallel:\n\n```\nfor taskserv in [kubernetes postgres redis] {\n    provisioning test quick $taskserv &\n}\nwait\n```\n\n## Configuration\n\n### Default Settings\n\n- Base image: `ubuntu:22.04`\n- CPU: 1000 millicores (1 core)\n- Memory: 2048 MB (2 GB)\n- Network: 172.20.0.0/16\n\n### Custom Config\n\n```\n# Override defaults\nprovisioning test env single postgres \\n  --base-image debian:12 \\n  --cpu 2000 \\n  --memory 4096\n```\n\n---\n\n## Related Documentation\n\n- [Test Environment API](../api/test-environment-api.md)\n- [Topology Templates](../architecture/test-topologies.md)\n- [Orchestrator Guide](orchestrator-guide.md)\n- [Taskserv Development](taskserv-development.md)\n\n---\n\n## Version History\n\n| Version | Date | Changes |\n| --------- | ------ | --------- |\n| 1.0.0 | 2025-10-06 | Initial test environment service |\n\n---\n\n**Maintained By**: Infrastructure Team
