prvng_core/nulib/test/PLUGIN_TEST_README.md

# Plugin Integration Test Suite

Comprehensive test suite for the Provisioning platform's plugin system, covering authentication, KMS, and orchestrator plugins with graceful fallback testing.

## Overview

This test suite validates:
- **Plugin Availability**: Detection of installed Nushell plugins
- **Fallback Behavior**: Graceful degradation to HTTP/SOPS when plugins unavailable
- **Complete Workflows**: End-to-end authentication, encryption, and orchestration
- **Performance**: Benchmarking plugin vs HTTP performance
- **Error Handling**: Network failures, invalid data, concurrent access
- **Cross-Plugin Integration**: Multi-plugin workflows

## Test Files

### Individual Plugin Tests

| File | Purpose | Lines | Tests |
|------|---------|-------|-------|
| `../lib_provisioning/plugins/auth_test.nu` | Authentication plugin | 200 | 9 |
| `../lib_provisioning/plugins/kms_test.nu` | KMS plugin | 250 | 11 |
| `../lib_provisioning/plugins/orchestrator_test.nu` | Orchestrator plugin | 200 | 12 |
| `test_plugin_integration.nu` | Complete integration tests | 400 | 7 workflows |
| `run_plugin_tests.nu` | Test runner and reporter | 300 | - |

**Total**: 1,350 lines, 39+ individual tests

### Configuration

| File | Purpose | Lines |
|------|---------|-------|
| `../../config/plugin-config.toml` | Plugin configuration | 300 |

## Running Tests

### Quick Start

```bash
# Run all tests with report
cd provisioning/core/nulib/test
nu run_plugin_tests.nu

# Run individual test suites
nu ../lib_provisioning/plugins/auth_test.nu
nu ../lib_provisioning/plugins/kms_test.nu
nu ../lib_provisioning/plugins/orchestrator_test.nu
nu test_plugin_integration.nu
```

### Test Options

```bash
# Custom output file
nu run_plugin_tests.nu --output-file my-report.json

# Verbose output
nu run_plugin_tests.nu --verbose

# Skip integration tests (faster)
nu run_plugin_tests.nu --skip-integration
```

### CI/CD Integration

```bash
# GitHub Actions
# See: .github/workflows/plugin-tests.yml

# GitLab CI
# Add to .gitlab-ci.yml:
test:plugins:
  stage: test
  script:
    - nu provisioning/core/nulib/test/run_plugin_tests.nu
  artifacts:
    when: always
    paths:
      - plugin-test-report.json
```

## Test Coverage

### Authentication Plugin Tests (9 tests)

✅ Plugin availability detection
✅ Graceful fallback behavior
✅ Login function signature
✅ Logout function
✅ MFA enrollment signature
✅ MFA verify signature
✅ Configuration integration
✅ Token management
✅ Error handling

### KMS Plugin Tests (11 tests)

✅ Plugin availability detection
✅ Backend detection
✅ KMS status check
✅ Encryption
✅ Decryption
✅ Encryption round-trip
✅ Multiple backends (age, rustyvault, vault)
✅ Configuration integration
✅ Error handling
✅ File encryption
✅ Performance benchmarking

### Orchestrator Plugin Tests (12 tests)

✅ Plugin availability detection
✅ Local vs remote detection
✅ Orchestrator status
✅ Health check
✅ Tasks list
✅ Workflow submission
✅ Workflow status query
✅ Batch operations
✅ Statistics retrieval
✅ KCL validation
✅ Configuration integration
✅ Error handling
✅ Performance benchmarking

### Integration Tests (7 workflows)

✅ Complete authentication workflow (6 steps)
✅ Complete KMS workflow (6 steps)
✅ Complete orchestrator workflow (8 steps)
✅ Performance benchmarks (auth, KMS, orch)
✅ Fallback behavior validation
✅ Cross-plugin integration
✅ Error recovery scenarios

## Test Philosophy

### Graceful Degradation

**All tests pass regardless of plugin availability:**
- ✅ Plugins installed → Use plugins, test performance
- ✅ Plugins missing → Use HTTP/SOPS fallback, warn user
- ✅ Services unavailable → Skip service-dependent tests, report status

### No Hard Dependencies

Tests never fail due to:
- Missing plugins (fallback tested)
- Services not running (gracefully reported)
- Network issues (error handling tested)

### Performance Awareness

Tests measure and report performance:
- **Plugin mode**: <50ms (excellent)
- **HTTP fallback**: <200ms (good)
- **SOPS fallback**: <500ms (acceptable)

## Expected Output

### Successful Run (All Plugins Available)

```
==================================================================
🚀 Running Complete Plugin Integration Test Suite
==================================================================

🧪 Testing authentication workflow...
  Step 1: Verify initial unauthenticated state
    Result: exit_code=1
  Step 2: Attempt login
    ✅ Login succeeded
  Step 3: Verify after login
    ✅ Verification succeeded
  Step 4: Test token refresh
    Token refresh: exit_code=0
  Step 5: Logout
    ✅ Logout succeeded
  Step 6: Verify after logout
    Final verification: exit_code=1
  ✅ Auth workflow tests completed

🧪 Testing KMS workflow...
  Step 1: List KMS backends
    ✅ Backends available: age, rustyvault
  Step 2: Check KMS status
    ✅ KMS status check passed
  Step 3: Encrypt test data
    ✅ Encryption succeeded
  Step 4: Decrypt encrypted data
    ✅ Decryption succeeded
  Step 5: Verify round-trip integrity
    ✅ Round-trip verification passed
  Step 6: Test multiple backends
    ✅ Backend 'age' working
    ✅ Backend 'rustyvault' working
    ⚠️  Backend 'vault' not available
  ✅ KMS workflow tests completed

🧪 Testing orchestrator workflow...
  Step 1: Check orchestrator health
    ✅ Orchestrator is healthy
  Step 2: Get orchestrator status
    ✅ Status retrieved successfully
  Step 3: List all tasks
    ✅ Tasks list retrieved
  Step 4: Submit test workflow
    ✅ Workflow submitted
  Step 5: Check workflow status
    Workflow status check: exit_code=0
  Step 6: Get statistics
    ✅ Statistics retrieved
  Step 7: List batch operations
    ✅ Batch operations listed
  Step 8: Validate KCL content
    ✅ KCL validation passed
  ✅ Orchestrator workflow tests completed

🧪 Running performance benchmarks...
  Benchmarking authentication plugin...
    Auth: avg=12ms, min=10ms, max=15ms
  Benchmarking KMS plugin...
    KMS: avg=25ms, min=20ms, max=30ms
  Benchmarking orchestrator plugin...
    Orch: avg=8ms, min=6ms, max=12ms

  📊 Performance Analysis:
    ✅ Auth performance: Excellent (likely plugin)
    ✅ KMS performance: Excellent (likely plugin)
    ✅ Orch performance: Excellent (likely plugin)
  ✅ Performance benchmarks completed

==================================================================
📊 Generating Test Report...
==================================================================

Plugin Availability:
  • Auth plugin: true
  • KMS plugin: true
  • Orchestrator plugin: true

Service Status:
  • Orchestrator running: true
  • KMS backends available: true

Test Coverage:
  • Authentication workflow: ✅
  • KMS workflow: ✅
  • Orchestrator workflow: ✅
  • Performance benchmarks: ✅
  • Fallback behavior: ✅
  • Cross-plugin integration: ✅
  • Error recovery: ✅

Expected Performance:
  • All plugins available: Excellent performance expected

==================================================================
✅ All plugin integration tests completed successfully!
==================================================================
```

### Fallback Mode (No Plugins)

```
==================================================================
🚀 Running Complete Plugin Integration Test Suite
==================================================================

🔍 Checking Prerequisites

  • Nushell version: 0.107.1
  ✅ Found: ../lib_provisioning/plugins/auth_test.nu
  ✅ Found: ../lib_provisioning/plugins/kms_test.nu
  ✅ Found: ../lib_provisioning/plugins/orchestrator_test.nu
  ✅ Found: ./test_plugin_integration.nu

  Plugin Availability:
    • Auth: false
    • KMS: false
    • Orchestrator: false

  ⚠️  No plugins detected. Tests will use HTTP/SOPS fallback.

🧪 Testing authentication workflow...
  Step 1: Verify initial unauthenticated state
    Result: exit_code=1
  Step 2: Attempt login
    ⚠️  Login failed (expected if control center not running)
  ✅ Auth workflow tests completed

🧪 Testing KMS workflow...
  Step 1: List KMS backends
    ⚠️  No KMS backends available
  ✅ KMS workflow tests completed

🧪 Running performance benchmarks...
  Benchmarking authentication plugin...
    Auth: avg=150ms, min=140ms, max=180ms
  Benchmarking KMS plugin...
    KMS: avg=180ms, min=170ms, max=200ms
  Benchmarking orchestrator plugin...
    Orch: avg=120ms, min=110ms, max=140ms

  📊 Performance Analysis:
    ✅ Auth performance: Good (likely HTTP)
    ✅ KMS performance: Good (likely HTTP/SOPS)
    ✅ Orch performance: Good (likely HTTP)
  ✅ Performance benchmarks completed

==================================================================
📊 Test Report
==================================================================

Summary:
  • Total tests: 4
  • Passed: 4
  • Failed: 0
  • Total duration: 1250ms
  • Average duration: 312ms

Expected Performance:
  • Using HTTP fallback: Good performance expected

==================================================================
✅ All plugin integration tests completed successfully!
==================================================================
```

## Test Report Format

### JSON Report Structure

```json
{
  "timestamp": "2025-10-09 10:30:00",
  "summary": {
    "total": 4,
    "passed": 4,
    "failed": 0,
    "total_duration_ms": 1250,
    "average_duration_ms": 312
  },
  "tests": [
    {
      "name": "Authentication Plugin Tests",
      "file": "../lib_provisioning/plugins/auth_test.nu",
      "success": true,
      "exit_code": 0,
      "duration_ms": 250,
      "stdout": "...",
      "stderr": ""
    }
  ],
  "plugins": {
    "auth": false,
    "kms": false,
    "orchestrator": false
  },
  "environment": {
    "nushell_version": "0.107.1",
    "os": "macos",
    "arch": "aarch64"
  }
}
```

## Troubleshooting

### Tests Fail to Run

**Problem**: `nu: command not found`
**Solution**: Install Nushell 0.107.1+
```bash
brew install nushell  # macOS
cargo install nu      # Any platform
```

### Plugin Tests Show Warnings

**Problem**: All plugin tests show "not available"
**Solution**: This is expected behavior if plugins not installed. Tests will use HTTP fallback.

### Service Connection Errors

**Problem**: "Orchestrator not available" warnings
**Solution**: Start orchestrator service:
```bash
cd provisioning/platform/orchestrator
cargo run --release
```

### KMS Backend Errors

**Problem**: "No KMS backends available"
**Solution**: Configure at least one backend in `provisioning/config/plugin-config.toml`

## Development

### Adding New Tests

1. Create test in appropriate file (`*_test.nu`)
2. Follow naming convention: `test_<feature>_<aspect>`
3. Use `std assert` for assertions
4. Handle both plugin and fallback modes
5. Add to integration test workflow if needed

### Example Test Template

```nushell
export def test_new_feature [] {
    print "  Testing new feature..."

    use plugin_module.nu *

    # Test logic
    let result = (do {
        plugin-function args
    } | complete)

    # Assertions
    if $result.exit_code == 0 {
        print "    ✅ Feature working"
    } else {
        print "    ⚠️  Feature not available"
    }
}
```

## Performance Baselines

### Plugin Mode

| Operation | Target | Excellent | Good | Acceptable |
|-----------|--------|-----------|------|------------|
| Auth verify | <10ms | <20ms | <50ms | <100ms |
| KMS encrypt | <20ms | <40ms | <80ms | <150ms |
| Orch status | <5ms | <10ms | <30ms | <80ms |

### HTTP Fallback Mode

| Operation | Target | Excellent | Good | Acceptable |
|-----------|--------|-----------|------|------------|
| Auth verify | <50ms | <100ms | <200ms | <500ms |
| KMS encrypt | <80ms | <150ms | <300ms | <800ms |
| Orch status | <30ms | <80ms | <150ms | <400ms |

## Continuous Integration

### GitHub Actions

See: `.github/workflows/plugin-tests.yml`

Tests run on:
- Push to main/develop
- Pull requests
- Manual trigger

Platforms:
- Ubuntu latest
- macOS latest

Artifacts:
- Test reports (JSON)
- Benchmark results
- Logs (on failure)

### Badge Status

Add to README:
```markdown
[![Plugin Tests](https://github.com/org/repo/workflows/Plugin%20Integration%20Tests/badge.svg)](https://github.com/org/repo/actions)
```

## Maintenance

### Regular Tasks

- **Weekly**: Review test results for performance regressions
- **Monthly**: Update baselines if performance improves
- **Quarterly**: Add tests for new plugin features
- **Yearly**: Review and update test philosophy

### Test Metrics

Track over time:
- Total test count
- Average execution time
- Plugin availability rate
- Fallback usage rate
- Failure rate

## Contributing

When adding plugin functionality:

1. ✅ Write tests first (TDD)
2. ✅ Test both plugin and fallback modes
3. ✅ Add performance benchmarks
4. ✅ Update this README
5. ✅ Run full test suite before commit

## License

Same as Provisioning Platform (see root LICENSE)

## Support

For issues or questions:
- GitHub Issues: [project-provisioning/issues](https://github.com/org/project-provisioning/issues)
- Documentation: [docs/](../../docs/)
- Plugin Docs: [docs/plugins/](../../docs/plugins/)

---

**Last Updated**: 2025-10-09
**Test Suite Version**: 1.0.0
**Maintained By**: Platform Team