provisioning/tests/integration/docs/TESTING_GUIDE.md

# Integration Testing Guide

**Version**: 1.0.0
**Last Updated**: 2025-10-06

This guide provides comprehensive documentation for the provisioning platform integration testing suite.

## Table of Contents

1. [Overview](#overview)
2. [Test Infrastructure](#test-infrastructure)
3. [Running Tests Locally](#running-tests-locally)
4. [Running Tests on OrbStack](#running-tests-on-orbstack)
5. [Writing New Tests](#writing-new-tests)
6. [Test Organization](#test-organization)
7. [CI/CD Integration](#cicd-integration)
8. [Troubleshooting](#troubleshooting)

---

## Overview

The integration testing suite validates all four execution modes of the provisioning platform:

- **Solo Mode**: Single-user, minimal services (orchestrator, CoreDNS, OCI registry)
- **Multi-User Mode**: Multi-user support with Gitea, PostgreSQL, RBAC
- **CI/CD Mode**: Automation mode with API server, service accounts
- **Enterprise Mode**: Full enterprise features (Harbor, KMS, Prometheus, Grafana, ELK)

### Key Features

- ✅ **Comprehensive Coverage**: Tests for all 4 modes, 15+ services
- ✅ **OrbStack Integration**: Tests deployable to OrbStack machine "provisioning"
- ✅ **Parallel Execution**: Run independent tests in parallel for speed
- ✅ **Automatic Cleanup**: Resources cleaned up automatically after tests
- ✅ **Multiple Report Formats**: JUnit XML, HTML, JSON
- ✅ **CI/CD Ready**: GitHub Actions and GitLab CI integration

---

## Test Infrastructure

### Prerequisites

1. **OrbStack Installed**:
   ```bash
   # Install OrbStack (macOS)
   brew install --cask orbstack
   ```

2. **OrbStack Machine Named "provisioning"**:
   ```bash
   # Create OrbStack machine
   orb create provisioning

   # Verify machine is running
   orb status provisioning
   ```

3. **Nushell 0.107.1+**:
   ```bash
   # Install Nushell
   brew install nushell
   ```

4. **Docker CLI**:
   ```bash
   # Verify Docker is available
   docker version
   ```

### Test Configuration

The test suite is configured via `provisioning/tests/integration/test_config.yaml`:

```yaml
# OrbStack connection
orbstack:
  machine_name: "provisioning"
  connection:
    type: "docker"
    socket: "/var/run/docker.sock"

# Service endpoints
services:
  orchestrator:
    host: "172.20.0.10"
    port: 8080

  coredns:
    host: "172.20.0.2"
    port: 53

  # ... more services
```

**Key Settings**:
- `orbstack.machine_name`: Name of OrbStack machine to use
- `services.*`: IP addresses and ports for deployed services
- `test_execution.parallel.max_workers`: Number of parallel test workers
- `test_execution.timeouts.*`: Timeout values for various operations

---

## Running Tests Locally

### Quick Start

1. **Setup Test Environment**:
   ```bash
   # Setup solo mode environment
   nu provisioning/tests/integration/setup_test_environment.nu --mode solo
   ```

2. **Run Tests**:
   ```bash
   # Run all tests for solo mode
   nu provisioning/tests/integration/framework/test_runner.nu --mode solo

   # Run specific test file
   nu provisioning/tests/integration/modes/test_solo_mode.nu
   ```

3. **Teardown Test Environment**:
   ```bash
   # Cleanup all resources
   nu provisioning/tests/integration/teardown_test_environment.nu --force
   ```

### Test Runner Options

```bash
nu provisioning/tests/integration/framework/test_runner.nu \
  --mode <mode>              # Test specific mode (solo, multiuser, cicd, enterprise)
  --filter <pattern>         # Filter tests by regex pattern
  --parallel <n>             # Number of parallel workers (default: 1)
  --verbose                  # Detailed output
  --report <path>            # Generate HTML report
  --skip-setup               # Skip environment setup
  --skip-teardown            # Skip environment teardown
```

**Examples**:

```bash
# Run all tests for all modes
nu provisioning/tests/integration/framework/test_runner.nu

# Run only solo mode tests
nu provisioning/tests/integration/framework/test_runner.nu --mode solo

# Run tests matching pattern
nu provisioning/tests/integration/framework/test_runner.nu --filter "dns"

# Run tests in parallel with 4 workers
nu provisioning/tests/integration/framework/test_runner.nu --parallel 4

# Generate HTML report
nu provisioning/tests/integration/framework/test_runner.nu --report /tmp/test-report.html

# Run tests without cleanup (for debugging)
nu provisioning/tests/integration/framework/test_runner.nu --skip-teardown
```

---

## Running Tests on OrbStack

### Setup OrbStack Machine

1. **Create OrbStack Machine**:
   ```bash
   # Create machine named "provisioning"
   orb create provisioning --cpu 4 --memory 8192 --disk 100

   # Verify machine is created
   orb list
   ```

2. **Configure Machine**:
   ```bash
   # Start machine
   orb start provisioning

   # Verify Docker is accessible
   docker -H /var/run/docker.sock ps
   ```

### Deploy Platform to OrbStack

The test setup automatically deploys platform services to OrbStack:

```bash
# Deploy solo mode
nu provisioning/tests/integration/setup_test_environment.nu --mode solo

# Deploy multi-user mode
nu provisioning/tests/integration/setup_test_environment.nu --mode multiuser

# Deploy CI/CD mode
nu provisioning/tests/integration/setup_test_environment.nu --mode cicd

# Deploy enterprise mode
nu provisioning/tests/integration/setup_test_environment.nu --mode enterprise
```

**Deployed Services**:

| Mode | Services |
|------|----------|
| Solo | Orchestrator, CoreDNS, Zot (OCI registry) |
| Multi-User | Solo services + Gitea, PostgreSQL |
| CI/CD | Multi-User services + API server, Prometheus |
| Enterprise | CI/CD services + Harbor, KMS, Grafana, Elasticsearch |

### Verify Deployment

```bash
# Check service health
nu provisioning/tests/integration/framework/test_helpers.nu check-service-health orchestrator

# View service logs
nu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs orchestrator

# List running containers
docker -H /var/run/docker.sock ps
```

---

## Writing New Tests

### Test File Structure

All test files follow this structure:

```nushell
# Test Description
# Brief description of what this test validates

use std log
use ../framework/test_helpers.nu *
use ../framework/orbstack_helpers.nu *

# Main test suite
export def main [] {
    log info "Running <Test Suite Name>"

    let test_config = (load-test-config)

    mut results = []

    # Run all tests
    $results = ($results | append (test-case-1 $test_config))
    $results = ($results | append (test-case-2 $test_config))

    # Report results
    report-test-results $results
}

# Individual test case
def test-case-1 [test_config: record] {
    run-test "test-case-1-name" {
        log info "Testing specific functionality..."

        # Test logic
        let result = (some-operation)

        # Assertions
        assert-eq $result.status "success" "Operation should succeed"
        assert-not-empty $result.data "Result should contain data"

        log info "✓ Test case 1 passed"
    }
}

# Report test results
def report-test-results [results: list] {
    # ... reporting logic
}
```

### Using Assertion Helpers

The test framework provides several assertion helpers:

```nushell
# Equality assertion
assert-eq $actual $expected "Error message if assertion fails"

# Boolean assertions
assert-true $condition "Error message"
assert-false $condition "Error message"

# Collection assertions
assert-contains $list $item "Error message"
assert-not-contains $list $item "Error message"
assert-not-empty $value "Error message"

# HTTP assertions
assert-http-success $response "Error message"
```

### Using Test Fixtures

Create reusable test fixtures:

```nushell
# Create test workspace
let workspace = create-test-workspace "my-test-ws" {
    provider: "local"
    environment: "test"
}

# Create test server
let server = create-test-server "test-server" "local" {
    cores: 4
    memory: 8192
}

# Cleanup
cleanup-test-workspace $workspace
delete-test-server $server.id
```

### Using Retry Logic

For flaky operations, use retry helpers:

```nushell
# Retry operation up to 3 times
let result = (with-retry --max-attempts 3 --delay 5 {
    # Operation that might fail
    http get "http://example.com/api"
})

# Wait for condition with timeout
wait-for-condition --timeout 60 --interval 5 {
    # Condition to check
    check-service-health "orchestrator"
} "orchestrator to be healthy"
```

### Example: Writing a New Service Integration Test

```nushell
# Test Gitea Integration
# Validates Gitea workspace git operations and extension publishing

use std log
use ../framework/test_helpers.nu *

def test-gitea-workspace-operations [test_config: record] {
    run-test "gitea-workspace-git-operations" {
        log info "Testing Gitea workspace operations..."

        # Create workspace
        let workspace = create-test-workspace "gitea-test" {
            provider: "local"
        }

        # Initialize git repo
        cd $workspace.path
        git init

        # Configure Gitea remote
        let gitea_url = $"http://($test_config.services.gitea.host):($test_config.services.gitea.port)"
        git remote add origin $"($gitea_url)/test-user/gitea-test.git"

        # Create test file
        "test content" | save test.txt
        git add test.txt
        git commit -m "Test commit"

        # Push to Gitea
        git push -u origin main

        # Verify push succeeded
        let remote_log = (git ls-remote origin)
        assert-not-empty $remote_log "Remote should have commits"

        log info "✓ Gitea workspace operations work"

        # Cleanup
        cleanup-test-workspace $workspace
    }
}
```

---

## Test Organization

### Directory Structure

```
provisioning/tests/integration/
├── test_config.yaml                    # Test configuration
├── setup_test_environment.nu           # Environment setup script
├── teardown_test_environment.nu        # Cleanup script
├── framework/                          # Test framework utilities
│   ├── test_helpers.nu                 # Common test helpers
│   ├── orbstack_helpers.nu             # OrbStack integration
│   └── test_runner.nu                  # Test orchestration
├── modes/                              # Mode-specific tests
│   ├── test_solo_mode.nu               # Solo mode tests
│   ├── test_multiuser_mode.nu          # Multi-user mode tests
│   ├── test_cicd_mode.nu               # CI/CD mode tests
│   └── test_enterprise_mode.nu         # Enterprise mode tests
├── services/                           # Service integration tests
│   ├── test_dns_integration.nu         # CoreDNS tests
│   ├── test_gitea_integration.nu       # Gitea tests
│   ├── test_oci_integration.nu         # OCI registry tests
│   └── test_service_orchestration.nu   # Service manager tests
├── workflows/                          # Workflow tests
│   ├── test_extension_loading.nu       # Extension loading tests
│   └── test_batch_workflows.nu         # Batch workflow tests
├── e2e/                                # End-to-end tests
│   ├── test_complete_deployment.nu     # Full deployment workflow
│   └── test_disaster_recovery.nu       # Backup/restore tests
├── performance/                        # Performance tests
│   ├── test_concurrency.nu             # Concurrency tests
│   └── test_scalability.nu             # Scalability tests
├── security/                           # Security tests
│   ├── test_rbac_enforcement.nu        # RBAC tests
│   └── test_kms_integration.nu         # KMS tests
└── docs/                               # Documentation
    ├── TESTING_GUIDE.md                # This guide
    ├── TEST_COVERAGE.md                # Coverage report
    └── ORBSTACK_SETUP.md               # OrbStack setup guide
```

### Test Naming Conventions

- **Test Files**: `test_<feature>_<category>.nu`
- **Test Functions**: `test-<specific-scenario>`
- **Test Names**: `<mode>-<category>-<specific-scenario>`

**Examples**:
- File: `test_dns_integration.nu`
- Function: `test-dns-registration`
- Test Name: `solo-mode-dns-registration`

---

## CI/CD Integration

### GitHub Actions

Create `.github/workflows/integration-tests.yml`:

```yaml
name: Integration Tests

on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'  # Nightly at 2 AM

jobs:
  integration-tests:
    runs-on: macos-latest

    strategy:
      matrix:
        mode: [solo, multiuser, cicd, enterprise]

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Install OrbStack
        run: brew install --cask orbstack

      - name: Create OrbStack machine
        run: orb create provisioning

      - name: Install Nushell
        run: brew install nushell

      - name: Setup test environment
        run: |
          nu provisioning/tests/integration/setup_test_environment.nu \
            --mode ${{ matrix.mode }}

      - name: Run integration tests
        run: |
          nu provisioning/tests/integration/framework/test_runner.nu \
            --mode ${{ matrix.mode }} \
            --report test-report.html

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: test-results-${{ matrix.mode }}
          path: |
            /tmp/provisioning-test-reports/
            test-report.html

      - name: Teardown test environment
        if: always()
        run: |
          nu provisioning/tests/integration/teardown_test_environment.nu --force
```

### GitLab CI

Create `.gitlab-ci.yml`:

```yaml
stages:
  - test

integration-tests:
  stage: test
  image: ubuntu:22.04

  parallel:
    matrix:
      - MODE: [solo, multiuser, cicd, enterprise]

  before_script:
    # Install dependencies
    - apt-get update && apt-get install -y docker.io nushell

  script:
    # Setup test environment
    - nu provisioning/tests/integration/setup_test_environment.nu --mode $MODE

    # Run tests
    - nu provisioning/tests/integration/framework/test_runner.nu --mode $MODE --report test-report.html

  after_script:
    # Cleanup
    - nu provisioning/tests/integration/teardown_test_environment.nu --force

  artifacts:
    when: always
    paths:
      - /tmp/provisioning-test-reports/
      - test-report.html
    reports:
      junit: /tmp/provisioning-test-reports/junit-results.xml
```

---

## Troubleshooting

### Common Issues

#### 1. OrbStack Machine Not Found

**Error**: `OrbStack machine 'provisioning' not found`

**Solution**:
```bash
# Create OrbStack machine
orb create provisioning

# Verify creation
orb list
```

#### 2. Docker Connection Failed

**Error**: `Cannot connect to Docker daemon`

**Solution**:
```bash
# Verify OrbStack is running
orb status provisioning

# Restart OrbStack
orb restart provisioning
```

#### 3. Service Health Check Timeout

**Error**: `Timeout waiting for service orchestrator to be healthy`

**Solution**:
```bash
# Check service logs
nu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs orchestrator

# Verify service is running
docker -H /var/run/docker.sock ps | grep orchestrator

# Increase timeout in test_config.yaml
# test_execution.timeouts.test_timeout_seconds: 600
```

#### 4. Test Environment Cleanup Failed

**Error**: `Failed to remove test workspace`

**Solution**:
```bash
# Manual cleanup
rm -rf /tmp/provisioning-test-workspace*

# Cleanup OrbStack resources
nu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-cleanup
```

#### 5. DNS Resolution Failed

**Error**: `DNS record should exist for server`

**Solution**:
```bash
# Check CoreDNS logs
nu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs coredns

# Verify CoreDNS is running
docker -H /var/run/docker.sock ps | grep coredns

# Test DNS manually
dig @172.20.0.2 test-server.local
```

### Debug Mode

Run tests with verbose logging:

```bash
# Enable verbose output
nu provisioning/tests/integration/framework/test_runner.nu --verbose --mode solo

# Keep environment after tests for debugging
nu provisioning/tests/integration/framework/test_runner.nu --skip-teardown --mode solo

# Inspect environment manually
docker -H /var/run/docker.sock ps
docker -H /var/run/docker.sock logs orchestrator
```

### Viewing Test Logs

```bash
# View test execution logs
cat /tmp/provisioning-test.log

# View service logs
ls /tmp/provisioning-test-reports/logs/

# View HTML report
open /tmp/provisioning-test-reports/test-report.html
```

---

## Performance Benchmarks

Expected test execution times:

| Test Suite | Duration (Solo) | Duration (Enterprise) |
|------------|-----------------|------------------------|
| Mode Tests | 5-10 min | 15-20 min |
| Service Tests | 3-5 min | 10-15 min |
| Workflow Tests | 5-10 min | 15-20 min |
| E2E Tests | 10-15 min | 30-40 min |
| **Total** | **25-40 min** | **70-95 min** |

**Parallel Execution** (4 workers):
- Solo mode: ~10-15 min
- Enterprise mode: ~25-35 min

---

## Best Practices

1. **Idempotent Tests**: Tests should be repeatable without side effects
2. **Isolated Tests**: Each test should be independent
3. **Clear Assertions**: Use descriptive error messages
4. **Cleanup**: Always cleanup resources, even on failure
5. **Retry Flaky Operations**: Use `with-retry` for network operations
6. **Meaningful Names**: Use descriptive test names
7. **Fast Feedback**: Run quick tests first, slow tests later
8. **Log Important Steps**: Log key operations for debugging

---

## References

- [OrbStack Documentation](https://orbstack.dev/docs)
- [Nushell Documentation](https://www.nushell.sh)
- [Provisioning Platform Architecture](/docs/architecture/)
- [Test Coverage Report](TEST_COVERAGE.md)
- [OrbStack Setup Guide](ORBSTACK_SETUP.md)

---

**Maintained By**: Platform Team
**Last Updated**: 2025-10-06