18 KiB
18 KiB
bash\n # Install OrbStack (macOS)\n brew install --cask orbstack\n \n\n2. OrbStack Machine Named "provisioning":\n\n bash\n # Create OrbStack machine\n orb create provisioning\n\n # Verify machine is running\n orb status provisioning\n \n\n3. Nushell 0.107.1+:\n\n bash\n # Install Nushell\n brew install nushell\n \n\n4. Docker CLI:\n\n bash\n # Verify Docker is available\n docker version\n \n\n### Test Configuration\n\nThe test suite is configured via provisioning/tests/integration/test_config.yaml:\n\n\n# OrbStack connection\norbstack:\n machine_name: "provisioning"\n connection:\n type: "docker"\n socket: "/var/run/docker.sock"\n\n# Service endpoints\nservices:\n orchestrator:\n host: "172.20.0.10"\n port: 8080\n\n coredns:\n host: "172.20.0.2"\n port: 53\n\n # ... more services\n\n\nKey Settings:\n\n- orbstack.machine_name: Name of OrbStack machine to use\n- services.*: IP addresses and ports for deployed services\n- test_execution.parallel.max_workers: Number of parallel test workers\n- test_execution.timeouts.*: Timeout values for various operations\n\n---\n\n## Running Tests Locally\n\n### Quick Start\n\n1. Setup Test Environment:\n\n bash\n # Setup solo mode environment\n nu provisioning/tests/integration/setup_test_environment.nu --mode solo\n \n\n1. Run Tests:\n\n bash\n # Run all tests for solo mode\n nu provisioning/tests/integration/framework/test_runner.nu --mode solo\n\n # Run specific test file\n nu provisioning/tests/integration/modes/test_solo_mode.nu\n \n\n2. Teardown Test Environment:\n\n bash\n # Cleanup all resources\n nu provisioning/tests/integration/teardown_test_environment.nu --force\n \n\n### Test Runner Options\n\n\nnu provisioning/tests/integration/framework/test_runner.nu \n --mode <mode> # Test specific mode (solo, multiuser, cicd, enterprise)\n --filter <pattern> # Filter tests by regex pattern\n --parallel <n> # Number of parallel workers (default: 1)\n --verbose # Detailed output\n --report <path> # Generate HTML report\n --skip-setup # Skip environment setup\n --skip-teardown # Skip environment teardown\n\n\nExamples:\n\n\n# Run all tests for all modes\nnu provisioning/tests/integration/framework/test_runner.nu\n\n# Run only solo mode tests\nnu provisioning/tests/integration/framework/test_runner.nu --mode solo\n\n# Run tests matching pattern\nnu provisioning/tests/integration/framework/test_runner.nu --filter "dns"\n\n# Run tests in parallel with 4 workers\nnu provisioning/tests/integration/framework/test_runner.nu --parallel 4\n\n# Generate HTML report\nnu provisioning/tests/integration/framework/test_runner.nu --report /tmp/test-report.html\n\n# Run tests without cleanup (for debugging)\nnu provisioning/tests/integration/framework/test_runner.nu --skip-teardown\n\n\n---\n\n## Running Tests on OrbStack\n\n### Setup OrbStack Machine\n\n1. Create OrbStack Machine:\n\n bash\n # Create machine named "provisioning"\n orb create provisioning --cpu 4 --memory 8192 --disk 100\n\n # Verify machine is created\n orb list\n \n\n1. Configure Machine:\n\n bash\n # Start machine\n orb start provisioning\n\n # Verify Docker is accessible\n docker -H /var/run/docker.sock ps\n \n\n### Deploy Platform to OrbStack\n\nThe test setup automatically deploys platform services to OrbStack:\n\n\n# Deploy solo mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode solo\n\n# Deploy multi-user mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode multiuser\n\n# Deploy CI/CD mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode cicd\n\n# Deploy enterprise mode\nnu provisioning/tests/integration/setup_test_environment.nu --mode enterprise\n\n\nDeployed Services:\n\n| Mode | Services |\n| ------ | ---------- |\n| Solo | Orchestrator, CoreDNS, Zot (OCI registry) |\n| Multi-User | Solo services + Gitea, PostgreSQL |\n| CI/CD | Multi-User services + API server, Prometheus |\n| Enterprise | CI/CD services + Harbor, KMS, Grafana, Elasticsearch |\n\n### Verify Deployment\n\n\n# Check service health\nnu provisioning/tests/integration/framework/test_helpers.nu check-service-health orchestrator\n\n# View service logs\nnu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs orchestrator\n\n# List running containers\ndocker -H /var/run/docker.sock ps\n\n\n---\n\n## Writing New Tests\n\n### Test File Structure\n\nAll test files follow this structure:\n\n\n# Test Description\n# Brief description of what this test validates\n\nuse std log\nuse ../framework/test_helpers.nu *\nuse ../framework/orbstack_helpers.nu *\n\n# Main test suite\nexport def main [] {\n log info "Running <Test Suite Name>"\n\n let test_config = (load-test-config)\n\n mut results = []\n\n # Run all tests\n $results = ($results | append (test-case-1 $test_config))\n $results = ($results | append (test-case-2 $test_config))\n\n # Report results\n report-test-results $results\n}\n\n# Individual test case\ndef test-case-1 [test_config: record] {\n run-test "test-case-1-name" {\n log info "Testing specific functionality..."\n\n # Test logic\n let result = (some-operation)\n\n # Assertions\n assert-eq $result.status "success" "Operation should succeed"\n assert-not-empty $result.data "Result should contain data"\n\n log info "✓ Test case 1 passed"\n }\n}\n\n# Report test results\ndef report-test-results [results: list] {\n # ... reporting logic\n}\n\n\n### Using Assertion Helpers\n\nThe test framework provides several assertion helpers:\n\n\n# Equality assertion\nassert-eq $actual $expected "Error message if assertion fails"\n\n# Boolean assertions\nassert-true $condition "Error message"\nassert-false $condition "Error message"\n\n# Collection assertions\nassert-contains $list $item "Error message"\nassert-not-contains $list $item "Error message"\nassert-not-empty $value "Error message"\n\n# HTTP assertions\nassert-http-success $response "Error message"\n\n\n### Using Test Fixtures\n\nCreate reusable test fixtures:\n\n\n# Create test workspace\nlet workspace = create-test-workspace "my-test-ws" {\n provider: "local"\n environment: "test"\n}\n\n# Create test server\nlet server = create-test-server "test-server" "local" {\n cores: 4\n memory: 8192\n}\n\n# Cleanup\ncleanup-test-workspace $workspace\ndelete-test-server $server.id\n\n\n### Using Retry Logic\n\nFor flaky operations, use retry helpers:\n\n\n# Retry operation up to 3 times\nlet result = (with-retry --max-attempts 3 --delay 5 {\n # Operation that might fail\n http get "http://example.com/api"\n})\n\n# Wait for condition with timeout\nwait-for-condition --timeout 60 --interval 5 {\n # Condition to check\n check-service-health "orchestrator"\n} "orchestrator to be healthy"\n\n\n### Example: Writing a New Service Integration Test\n\n\n# Test Gitea Integration\n# Validates Gitea workspace git operations and extension publishing\n\nuse std log\nuse ../framework/test_helpers.nu *\n\ndef test-gitea-workspace-operations [test_config: record] {\n run-test "gitea-workspace-git-operations" {\n log info "Testing Gitea workspace operations..."\n\n # Create workspace\n let workspace = create-test-workspace "gitea-test" {\n provider: "local"\n }\n\n # Initialize git repo\n cd $workspace.path\n git init\n\n # Configure Gitea remote\n let gitea_url = $"http://($test_config.services.gitea.host):($test_config.services.gitea.port)"\n git remote add origin $"($gitea_url)/test-user/gitea-test.git"\n\n # Create test file\n "test content" | save test.txt\n git add test.txt\n git commit -m "Test commit"\n\n # Push to Gitea\n git push -u origin main\n\n # Verify push succeeded\n let remote_log = (git ls-remote origin)\n assert-not-empty $remote_log "Remote should have commits"\n\n log info "✓ Gitea workspace operations work"\n\n # Cleanup\n cleanup-test-workspace $workspace\n }\n}\n\n\n---\n\n## Test Organization\n\n### Directory Structure\n\n\nprovisioning/tests/integration/\n├── test_config.yaml # Test configuration\n├── setup_test_environment.nu # Environment setup script\n├── teardown_test_environment.nu # Cleanup script\n├── framework/ # Test framework utilities\n│ ├── test_helpers.nu # Common test helpers\n│ ├── orbstack_helpers.nu # OrbStack integration\n│ └── test_runner.nu # Test orchestration\n├── modes/ # Mode-specific tests\n│ ├── test_solo_mode.nu # Solo mode tests\n│ ├── test_multiuser_mode.nu # Multi-user mode tests\n│ ├── test_cicd_mode.nu # CI/CD mode tests\n│ └── test_enterprise_mode.nu # Enterprise mode tests\n├── services/ # Service integration tests\n│ ├── test_dns_integration.nu # CoreDNS tests\n│ ├── test_gitea_integration.nu # Gitea tests\n│ ├── test_oci_integration.nu # OCI registry tests\n│ └── test_service_orchestration.nu # Service manager tests\n├── workflows/ # Workflow tests\n│ ├── test_extension_loading.nu # Extension loading tests\n│ └── test_batch_workflows.nu # Batch workflow tests\n├── e2e/ # End-to-end tests\n│ ├── test_complete_deployment.nu # Full deployment workflow\n│ └── test_disaster_recovery.nu # Backup/restore tests\n├── performance/ # Performance tests\n│ ├── test_concurrency.nu # Concurrency tests\n│ └── test_scalability.nu # Scalability tests\n├── security/ # Security tests\n│ ├── test_rbac_enforcement.nu # RBAC tests\n│ └── test_kms_integration.nu # KMS tests\n└── docs/ # Documentation\n ├── TESTING_GUIDE.md # This guide\n ├── TEST_COVERAGE.md # Coverage report\n └── ORBSTACK_SETUP.md # OrbStack setup guide\n\n\n### Test Naming Conventions\n\n- Test Files: test_<feature>_<category>.nu\n- Test Functions: test-<specific-scenario>\n- Test Names: <mode>-<category>-<specific-scenario>\n\nExamples:\n\n- File: test_dns_integration.nu\n- Function: test-dns-registration\n- Test Name: solo-mode-dns-registration\n\n---\n\n## CI/CD Integration\n\n### GitHub Actions\n\nCreate .github/workflows/integration-tests.yml:\n\n\nname: Integration Tests\n\non:\n pull_request:\n push:\n branches: [main]\n schedule:\n - cron: '0 2 * * *' # Nightly at 2 AM\n\njobs:\n integration-tests:\n runs-on: macos-latest\n\n strategy:\n matrix:\n mode: [solo, multiuser, cicd, enterprise]\n\n steps:\n - name: Checkout code\n uses: actions/checkout@v3\n\n - name: Install OrbStack\n run: brew install --cask orbstack\n\n - name: Create OrbStack machine\n run: orb create provisioning\n\n - name: Install Nushell\n run: brew install nushell\n\n - name: Setup test environment\n run: |\n nu provisioning/tests/integration/setup_test_environment.nu \n --mode ${{ matrix.mode }}\n\n - name: Run integration tests\n run: |\n nu provisioning/tests/integration/framework/test_runner.nu \n --mode ${{ matrix.mode }} \n --report test-report.html\n\n - name: Upload test results\n if: always()\n uses: actions/upload-artifact@v3\n with:\n name: test-results-${{ matrix.mode }}\n path: |\n /tmp/provisioning-test-reports/\n test-report.html\n\n - name: Teardown test environment\n if: always()\n run: |\n nu provisioning/tests/integration/teardown_test_environment.nu --force\n\n\n### GitLab CI\n\nCreate .gitlab-ci.yml:\n\n\nstages:\n - test\n\nintegration-tests:\n stage: test\n image: ubuntu:22.04\n\n parallel:\n matrix:\n - MODE: [solo, multiuser, cicd, enterprise]\n\n before_script:\n # Install dependencies\n - apt-get update && apt-get install -y docker.io nushell\n\n script:\n # Setup test environment\n - nu provisioning/tests/integration/setup_test_environment.nu --mode $MODE\n\n # Run tests\n - nu provisioning/tests/integration/framework/test_runner.nu --mode $MODE --report test-report.html\n\n after_script:\n # Cleanup\n - nu provisioning/tests/integration/teardown_test_environment.nu --force\n\n artifacts:\n when: always\n paths:\n - /tmp/provisioning-test-reports/\n - test-report.html\n reports:\n junit: /tmp/provisioning-test-reports/junit-results.xml\n\n\n---\n\n## Troubleshooting\n\n### Common Issues\n\n#### 1. OrbStack Machine Not Found\n\nError: OrbStack machine 'provisioning' not found\n\nSolution:\n\n\n# Create OrbStack machine\norb create provisioning\n\n# Verify creation\norb list\n\n\n#### 2. Docker Connection Failed\n\nError: Cannot connect to Docker daemon\n\nSolution:\n\n\n# Verify OrbStack is running\norb status provisioning\n\n# Restart OrbStack\norb restart provisioning\n\n\n#### 3. Service Health Check Timeout\n\nError: Timeout waiting for service orchestrator to be healthy\n\nSolution:\n\n\n# Check service logs\nnu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs orchestrator\n\n# Verify service is running\ndocker -H /var/run/docker.sock ps | grep orchestrator\n\n# Increase timeout in test_config.yaml\n# test_execution.timeouts.test_timeout_seconds: 600\n\n\n#### 4. Test Environment Cleanup Failed\n\nError: Failed to remove test workspace\n\nSolution:\n\n\n# Manual cleanup\nrm -rf /tmp/provisioning-test-workspace*\n\n# Cleanup OrbStack resources\nnu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-cleanup\n\n\n#### 5. DNS Resolution Failed\n\nError: DNS record should exist for server\n\nSolution:\n\n\n# Check CoreDNS logs\nnu provisioning/tests/integration/framework/orbstack_helpers.nu orbstack-logs coredns\n\n# Verify CoreDNS is running\ndocker -H /var/run/docker.sock ps | grep coredns\n\n# Test DNS manually\ndig @172.20.0.2 test-server.local\n\n\n### Debug Mode\n\nRun tests with verbose logging:\n\n\n# Enable verbose output\nnu provisioning/tests/integration/framework/test_runner.nu --verbose --mode solo\n\n# Keep environment after tests for debugging\nnu provisioning/tests/integration/framework/test_runner.nu --skip-teardown --mode solo\n\n# Inspect environment manually\ndocker -H /var/run/docker.sock ps\ndocker -H /var/run/docker.sock logs orchestrator\n\n\n### Viewing Test Logs\n\n\n# View test execution logs\ncat /tmp/provisioning-test.log\n\n# View service logs\nls /tmp/provisioning-test-reports/logs/\n\n# View HTML report\nopen /tmp/provisioning-test-reports/test-report.html\n\n\n---\n\n## Performance Benchmarks\n\nExpected test execution times:\n\n| Test Suite | Duration (Solo) | Duration (Enterprise) |\n| ------------ | ----------------- | ------------------------ |\n| Mode Tests | 5-10 min | 15-20 min |\n| Service Tests | 3-5 min | 10-15 min |\n| Workflow Tests | 5-10 min | 15-20 min |\n| E2E Tests | 10-15 min | 30-40 min |\n| Total | 25-40 min | 70-95 min |\n\nParallel Execution (4 workers):\n\n- Solo mode: ~10-15 min\n- Enterprise mode: ~25-35 min\n\n---\n\n## Best Practices\n\n1. Idempotent Tests: Tests should be repeatable without side effects\n2. Isolated Tests: Each test should be independent\n3. Clear Assertions: Use descriptive error messages\n4. Cleanup: Always cleanup resources, even on failure\n5. Retry Flaky Operations: Use with-retry for network operations\n6. Meaningful Names: Use descriptive test names\n7. Fast Feedback: Run quick tests first, slow tests later\n8. Log Important Steps: Log key operations for debugging\n\n---\n\n## References\n\n- OrbStack Documentation\n- Nushell Documentation\n- Provisioning Platform Architecture\n- Test Coverage Report\n- OrbStack Setup Guide\n\n---\n\nMaintained By: Platform Team\nLast Updated: 2025-10-06