# Integration Testing Guide\n\n**Version**: 1.0.0\n**Last Updated**: 2025-10-06\n\nThis guide provides comprehensive documentation for the provisioning platform integration testing suite.\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Test Infrastructure](#test-infrastructure)\n3. [Running Tests Locally](#running-tests-locally)\n4. [Running Tests on OrbStack](#running-tests-on-orbstack)\n5. [Writing New Tests](#writing-new-tests)\n6. [Test Organization](#test-organization)\n7. [CI/CD Integration](#cicd-integration)\n8. [Troubleshooting](#troubleshooting)\n\n---\n\n## Overview\n\nThe integration testing suite validates all four execution modes of the provisioning platform:\n\n- **Solo Mode**: Single-user, minimal services (orchestrator, CoreDNS, OCI registry)\n- **Multi-User Mode**: Multi-user support with Gitea, PostgreSQL, RBAC\n- **CI/CD Mode**: Automation mode with API server, service accounts\n- **Enterprise Mode**: Full enterprise features (Harbor, KMS, Prometheus, Grafana, ELK)\n\n### Key Features\n\n- ✅ **Comprehensive Coverage**: Tests for all 4 modes, 15+ services\n- ✅ **OrbStack Integration**: Tests deployable to OrbStack machine "provisioning"\n- ✅ **Parallel Execution**: Run independent tests in parallel for speed\n- ✅ **Automatic Cleanup**: Resources cleaned up automatically after tests\n- ✅ **Multiple Report Formats**: JUnit XML, HTML, JSON\n- ✅ **CI/CD Ready**: GitHub Actions and GitLab CI integration\n\n---\n\n## Test Infrastructure\n\n### Prerequisites\n\n1. **OrbStack Installed**:\n\n   ```bash\n   # Install OrbStack (macOS)\n   brew install --cask orbstack\n   ```\n\n2. **OrbStack Machine Named "provisioning"**:\n\n   ```bash\n   # Create OrbStack machine\n   orb create provisioning\n\n   # Verify machine is running\n   orb status provisioning\n   ```\n\n3. **Nushell 0.107.1+**:\n\n   ```bash\n   # Install Nushell\n   brew install nushell\n   ```\n\n4. **Docker CLI**:\n\n   ```bash\n   # Verify Docker is available\n   docker version\n   ```\n\n### Test Configuration\n\nThe test suite is configured via `provisioning/tests/integration/test_config.yaml`:\n\n```\n# OrbStack connection\norbstack:\n  machine_name: "provisioning"\n  connection:\n    type: "docker"\n    socket: "/var/run/docker.sock"\n\n# Service endpoints\nservices:\n  orchestrator:\n    host: "172.20.0.10"\n    port: 8080\n\n  coredns:\n    host: "172.20.0.2"\n    port: 53\n\n  # ... more services\n```\n\n**Key Settings**:\n\n- `orbstack.machine_name`: Name of OrbStack machine to use\n- `services.*`: IP addresses and ports for deployed services\n- `test_execution.parallel.max_workers`: Number of parallel test workers\n- `test_execution.timeouts.*`: Timeout values for various operations\n\n---\n\n## Running Tests Locally\n\n### Quick Start\n\n1. **Setup Test Environment**:\n\n   ```bash\n   # Setup solo mode environment\n   nu provisioning/tests/integration/setup_test_environment.nu --mode solo\n   ```\n\n1. **Run Tests**:\n\n   ```bash\n   # Run all tests for solo mode\n   nu provisioning/tests/integration/framework/test_runner.nu --mode solo\n\n   # Run specific test file\n   nu provisioning/tests/integration/modes/test_solo_mode.nu\n   ```\n\n2. **Teardown Test Environment**:\n\n   ```bash\n   # Cleanup all resources\n   nu provisioning/tests/integration/teardown_test_environment.nu --force\n   ```\n\n### Test Runner Options\n\n```\nnu provisioning/tests/integration/framework/test_runner.nu \\n  --mode <mode>              # Test specific mode (solo, multiuser, cicd, enterprise)\n  --filter <pattern>         # Filter tests by regex pattern\n  --parallel <n>             # Number of parallel workers (default: 1)\n  --verbose                  # Detailed output\n  --report <path>            # Generate HTML report\n  --skip-setup               # Skip environment setup\n  --skip-teardown            # Skip environment teardown\n```\n\n**Examples**:\n\n```\n# Run all tests for all modes\nnu provisioning/tests/integration/framework/test_runner.nu\n\n# Run only solo mode tests\nnu provisioning/tests/integration/framework/test_runner.nu --mode solo\n\n# Run tests matching pattern\nnu provisioning/tests/integration/framework/test_runner.nu --filter "dns"\n\n# Run tests in parallel with 4 workers\nnu provisioning/tests/integration/framework/test_runner.nu --parallel 4\n\n# Generate HTML report\nnu provisioning/tests/integration/framework/test_runner.nu --report /tmp/test-report.html\n\n# Run tests without cleanup (for debugging)\nnu provisioning/tests/integration/framework/test_runner.nu --skip-teardown\n```\n\n---\n\n## Running Tests on OrbStack\n\n### Setup OrbStack Machine\n\n1. **Create OrbStack Machine**:\n\n   ```bash\n   # Create machine named "provisioning"\n   orb create provisioning --cpu 4 --memory 8192 --disk 100\n\n   # Verify machine is created\n   orb list\n   ```\n\n1. **Configure Machine**:\n\n   ```bash\n   # Start machine\n   orb start provisioning\n\n   # Verify Docker is accessible\n   docker -H /var/run/docker.sock ps\n   ```\n\n### Deploy Platform to OrbStack\n\nThe test setup automatically deploys platform services to OrbStack:\n\n```\n# Deploy solo mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode solo\n\n# Deploy multi-user mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode multiuser\n\n# Deploy CI/CD mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode cicd\n\n# Deploy enterprise mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode enterprise\n```\n\n**Deployed Services**:\n\n| Mode | Services |\n| ------ | ---------- |\n| Solo | Orchestrator, CoreDNS, Zot (OCI registry) |\n| Multi-User | Solo services + Gitea, PostgreSQL |\n| CI/CD | Multi-User services + API server, Prometheus |\n| Enterprise | CI/CD services + Harbor, KMS, Grafana, Elasticsearch |\n\n### Verify Deployment\n\n```\n# Check service health\nnu provisioning/tests/integration/framework/test_helpers.nu check-service-health orchestrator\n\n# View service logs\nnu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs orchestrator\n\n# List running containers\ndocker -H /var/run/docker.sock ps\n```\n\n---\n\n## Writing New Tests\n\n### Test File Structure\n\nAll test files follow this structure:\n\n```\n# Test Description\n# Brief description of what this test validates\n\nuse std log\nuse ../framework/test_helpers.nu *\nuse ../framework/orbstack_helpers.nu *\n\n# Main test suite\nexport def main [] {\n    log info "Running <Test Suite Name>"\n\n    let test_config = (load-test-config)\n\n    mut results = []\n\n    # Run all tests\n    $results = ($results | append (test-case-1 $test_config))\n    $results = ($results | append (test-case-2 $test_config))\n\n    # Report results\n    report-test-results $results\n}\n\n# Individual test case\ndef test-case-1 [test_config: record] {\n    run-test "test-case-1-name" {\n        log info "Testing specific functionality..."\n\n        # Test logic\n        let result = (some-operation)\n\n        # Assertions\n        assert-eq $result.status "success" "Operation should succeed"\n        assert-not-empty $result.data "Result should contain data"\n\n        log info "✓ Test case 1 passed"\n    }\n}\n\n# Report test results\ndef report-test-results [results: list] {\n    # ... reporting logic\n}\n```\n\n### Using Assertion Helpers\n\nThe test framework provides several assertion helpers:\n\n```\n# Equality assertion\nassert-eq $actual $expected "Error message if assertion fails"\n\n# Boolean assertions\nassert-true $condition "Error message"\nassert-false $condition "Error message"\n\n# Collection assertions\nassert-contains $list $item "Error message"\nassert-not-contains $list $item "Error message"\nassert-not-empty $value "Error message"\n\n# HTTP assertions\nassert-http-success $response "Error message"\n```\n\n### Using Test Fixtures\n\nCreate reusable test fixtures:\n\n```\n# Create test workspace\nlet workspace = create-test-workspace "my-test-ws" {\n    provider: "local"\n    environment: "test"\n}\n\n# Create test server\nlet server = create-test-server "test-server" "local" {\n    cores: 4\n    memory: 8192\n}\n\n# Cleanup\ncleanup-test-workspace $workspace\ndelete-test-server $server.id\n```\n\n### Using Retry Logic\n\nFor flaky operations, use retry helpers:\n\n```\n# Retry operation up to 3 times\nlet result = (with-retry --max-attempts 3 --delay 5 {\n    # Operation that might fail\n    http get "http://example.com/api"\n})\n\n# Wait for condition with timeout\nwait-for-condition --timeout 60 --interval 5 {\n    # Condition to check\n    check-service-health "orchestrator"\n} "orchestrator to be healthy"\n```\n\n### Example: Writing a New Service Integration Test\n\n```\n# Test Gitea Integration\n# Validates Gitea workspace git operations and extension publishing\n\nuse std log\nuse ../framework/test_helpers.nu *\n\ndef test-gitea-workspace-operations [test_config: record] {\n    run-test "gitea-workspace-git-operations" {\n        log info "Testing Gitea workspace operations..."\n\n        # Create workspace\n        let workspace = create-test-workspace "gitea-test" {\n            provider: "local"\n        }\n\n        # Initialize git repo\n        cd $workspace.path\n        git init\n\n        # Configure Gitea remote\n        let gitea_url = $"http://($test_config.services.gitea.host):($test_config.services.gitea.port)"\n        git remote add origin $"($gitea_url)/test-user/gitea-test.git"\n\n        # Create test file\n        "test content" | save test.txt\n        git add test.txt\n        git commit -m "Test commit"\n\n        # Push to Gitea\n        git push -u origin main\n\n        # Verify push succeeded\n        let remote_log = (git ls-remote origin)\n        assert-not-empty $remote_log "Remote should have commits"\n\n        log info "✓ Gitea workspace operations work"\n\n        # Cleanup\n        cleanup-test-workspace $workspace\n    }\n}\n```\n\n---\n\n## Test Organization\n\n### Directory Structure\n\n```\nprovisioning/tests/integration/\n├── test_config.yaml                    # Test configuration\n├── setup_test_environment.nu           # Environment setup script\n├── teardown_test_environment.nu        # Cleanup script\n├── framework/                          # Test framework utilities\n│   ├── test_helpers.nu                 # Common test helpers\n│   ├── orbstack_helpers.nu             # OrbStack integration\n│   └── test_runner.nu                  # Test orchestration\n├── modes/                              # Mode-specific tests\n│   ├── test_solo_mode.nu               # Solo mode tests\n│   ├── test_multiuser_mode.nu          # Multi-user mode tests\n│   ├── test_cicd_mode.nu               # CI/CD mode tests\n│   └── test_enterprise_mode.nu         # Enterprise mode tests\n├── services/                           # Service integration tests\n│   ├── test_dns_integration.nu         # CoreDNS tests\n│   ├── test_gitea_integration.nu       # Gitea tests\n│   ├── test_oci_integration.nu         # OCI registry tests\n│   └── test_service_orchestration.nu   # Service manager tests\n├── workflows/                          # Workflow tests\n│   ├── test_extension_loading.nu       # Extension loading tests\n│   └── test_batch_workflows.nu         # Batch workflow tests\n├── e2e/                                # End-to-end tests\n│   ├── test_complete_deployment.nu     # Full deployment workflow\n│   └── test_disaster_recovery.nu       # Backup/restore tests\n├── performance/                        # Performance tests\n│   ├── test_concurrency.nu             # Concurrency tests\n│   └── test_scalability.nu             # Scalability tests\n├── security/                           # Security tests\n│   ├── test_rbac_enforcement.nu        # RBAC tests\n│   └── test_kms_integration.nu         # KMS tests\n└── docs/                               # Documentation\n    ├── TESTING_GUIDE.md                # This guide\n    ├── TEST_COVERAGE.md                # Coverage report\n    └── ORBSTACK_SETUP.md               # OrbStack setup guide\n```\n\n### Test Naming Conventions\n\n- **Test Files**: `test_<feature>_<category>.nu`\n- **Test Functions**: `test-<specific-scenario>`\n- **Test Names**: `<mode>-<category>-<specific-scenario>`\n\n**Examples**:\n\n- File: `test_dns_integration.nu`\n- Function: `test-dns-registration`\n- Test Name: `solo-mode-dns-registration`\n\n---\n\n## CI/CD Integration\n\n### GitHub Actions\n\nCreate `.github/workflows/integration-tests.yml`:\n\n```\nname: Integration Tests\n\non:\n  pull_request:\n  push:\n    branches: [main]\n  schedule:\n    - cron: '0 2 * * *'  # Nightly at 2 AM\n\njobs:\n  integration-tests:\n    runs-on: macos-latest\n\n    strategy:\n      matrix:\n        mode: [solo, multiuser, cicd, enterprise]\n\n    steps:\n      - name: Checkout code\n        uses: actions/checkout@v3\n\n      - name: Install OrbStack\n        run: brew install --cask orbstack\n\n      - name: Create OrbStack machine\n        run: orb create provisioning\n\n      - name: Install Nushell\n        run: brew install nushell\n\n      - name: Setup test environment\n        run: |\n          nu provisioning/tests/integration/setup_test_environment.nu \\n            --mode ${{ matrix.mode }}\n\n      - name: Run integration tests\n        run: |\n          nu provisioning/tests/integration/framework/test_runner.nu \\n            --mode ${{ matrix.mode }} \\n            --report test-report.html\n\n      - name: Upload test results\n        if: always()\n        uses: actions/upload-artifact@v3\n        with:\n          name: test-results-${{ matrix.mode }}\n          path: |\n            /tmp/provisioning-test-reports/\n            test-report.html\n\n      - name: Teardown test environment\n        if: always()\n        run: |\n          nu provisioning/tests/integration/teardown_test_environment.nu --force\n```\n\n### GitLab CI\n\nCreate `.gitlab-ci.yml`:\n\n```\nstages:\n  - test\n\nintegration-tests:\n  stage: test\n  image: ubuntu:22.04\n\n  parallel:\n    matrix:\n      - MODE: [solo, multiuser, cicd, enterprise]\n\n  before_script:\n    # Install dependencies\n    - apt-get update && apt-get install -y docker.io nushell\n\n  script:\n    # Setup test environment\n    - nu provisioning/tests/integration/setup_test_environment.nu --mode $MODE\n\n    # Run tests\n    - nu provisioning/tests/integration/framework/test_runner.nu --mode $MODE --report test-report.html\n\n  after_script:\n    # Cleanup\n    - nu provisioning/tests/integration/teardown_test_environment.nu --force\n\n  artifacts:\n    when: always\n    paths:\n      - /tmp/provisioning-test-reports/\n      - test-report.html\n    reports:\n      junit: /tmp/provisioning-test-reports/junit-results.xml\n```\n\n---\n\n## Troubleshooting\n\n### Common Issues\n\n#### 1. OrbStack Machine Not Found\n\n**Error**: `OrbStack machine 'provisioning' not found`\n\n**Solution**:\n\n```\n# Create OrbStack machine\norb create provisioning\n\n# Verify creation\norb list\n```\n\n#### 2. Docker Connection Failed\n\n**Error**: `Cannot connect to Docker daemon`\n\n**Solution**:\n\n```\n# Verify OrbStack is running\norb status provisioning\n\n# Restart OrbStack\norb restart provisioning\n```\n\n#### 3. Service Health Check Timeout\n\n**Error**: `Timeout waiting for service orchestrator to be healthy`\n\n**Solution**:\n\n```\n# Check service logs\nnu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs orchestrator\n\n# Verify service is running\ndocker -H /var/run/docker.sock ps | grep orchestrator\n\n# Increase timeout in test_config.yaml\n# test_execution.timeouts.test_timeout_seconds: 600\n```\n\n#### 4. Test Environment Cleanup Failed\n\n**Error**: `Failed to remove test workspace`\n\n**Solution**:\n\n```\n# Manual cleanup\nrm -rf /tmp/provisioning-test-workspace*\n\n# Cleanup OrbStack resources\nnu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-cleanup\n```\n\n#### 5. DNS Resolution Failed\n\n**Error**: `DNS record should exist for server`\n\n**Solution**:\n\n```\n# Check CoreDNS logs\nnu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs coredns\n\n# Verify CoreDNS is running\ndocker -H /var/run/docker.sock ps | grep coredns\n\n# Test DNS manually\ndig @172.20.0.2 test-server.local\n```\n\n### Debug Mode\n\nRun tests with verbose logging:\n\n```\n# Enable verbose output\nnu provisioning/tests/integration/framework/test_runner.nu --verbose --mode solo\n\n# Keep environment after tests for debugging\nnu provisioning/tests/integration/framework/test_runner.nu --skip-teardown --mode solo\n\n# Inspect environment manually\ndocker -H /var/run/docker.sock ps\ndocker -H /var/run/docker.sock logs orchestrator\n```\n\n### Viewing Test Logs\n\n```\n# View test execution logs\ncat /tmp/provisioning-test.log\n\n# View service logs\nls /tmp/provisioning-test-reports/logs/\n\n# View HTML report\nopen /tmp/provisioning-test-reports/test-report.html\n```\n\n---\n\n## Performance Benchmarks\n\nExpected test execution times:\n\n| Test Suite | Duration (Solo) | Duration (Enterprise) |\n| ------------ | ----------------- | ------------------------ |\n| Mode Tests | 5-10 min | 15-20 min |\n| Service Tests | 3-5 min | 10-15 min |\n| Workflow Tests | 5-10 min | 15-20 min |\n| E2E Tests | 10-15 min | 30-40 min |\n| **Total** | **25-40 min** | **70-95 min** |\n\n**Parallel Execution** (4 workers):\n\n- Solo mode: ~10-15 min\n- Enterprise mode: ~25-35 min\n\n---\n\n## Best Practices\n\n1. **Idempotent Tests**: Tests should be repeatable without side effects\n2. **Isolated Tests**: Each test should be independent\n3. **Clear Assertions**: Use descriptive error messages\n4. **Cleanup**: Always cleanup resources, even on failure\n5. **Retry Flaky Operations**: Use `with-retry` for network operations\n6. **Meaningful Names**: Use descriptive test names\n7. **Fast Feedback**: Run quick tests first, slow tests later\n8. **Log Important Steps**: Log key operations for debugging\n\n---\n\n## References\n\n- [OrbStack Documentation](https://orbstack.dev/docs)\n- [Nushell Documentation](https://www.nushell.sh)\n- [Provisioning Platform Architecture](/docs/architecture/)\n- [Test Coverage Report](TEST_COVERAGE.md)\n- [OrbStack Setup Guide](ORBSTACK_SETUP.md)\n\n---\n\n**Maintained By**: Platform Team\n**Last Updated**: 2025-10-06