Vapora/docs/adrs/0023-testing-strategy.md

ADR-023: Multi-Layer Testing Strategy

Status: Accepted | Implemented Date: 2024-11-01 Deciders: Quality Assurance Team Technical Story: Building confidence through unit, integration, and real-database tests

---

Decision

Implementar multi-layer testing: unit tests (inline), integration tests (tests/ dir), real DB connections.

---

Rationale

1. Unit Tests: Fast feedback on logic changes
2. Integration Tests: Verify components work together
3. Real DB Tests: Catch database schema/query issues
4. 218+ Tests: Comprehensive coverage across 13 crates

---

Alternatives Considered

❌ Unit Tests Only

• Pros: Fast
• Cons: Miss integration bugs, schema issues

❌ Integration Tests Only

• Pros: Comprehensive
• Cons: Slow, harder to debug

✅ Multi-Layer (CHOSEN)

• All three layers catch different issues

---

Trade-offs

Pros:

• ✅ Fast feedback (unit)
• ✅ Integration validation (integration)
• ✅ Real-world confidence (real DB)
• ✅ 218+ tests total coverage

Cons:

• ⚠️ Slow full test suite (~5 minutes)
• ⚠️ DB tests require test environment
• ⚠️ More test code to maintain

---

Implementation

Unit Tests (Inline):

// crates/vapora-agents/src/learning_profile.rs

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_expertise_score_empty() {
        let profile = TaskTypeLearning {
            agent_id: "test".to_string(),
            task_type: "architecture".to_string(),
            executions_total: 0,
            records: vec![],
            ..Default::default()
        };

        assert_eq!(profile.expertise_score(), 0.0);
    }

    #[test]
    fn test_confidence_weighting() {
        let profile = TaskTypeLearning {
            executions_total: 20,
            ..Default::default()
        };
        assert_eq!(profile.confidence(), 1.0);

        let profile_partial = TaskTypeLearning {
            executions_total: 10,
            ..Default::default()
        };
        assert_eq!(profile_partial.confidence(), 0.5);
    }
}

Integration Tests:

// crates/vapora-backend/tests/integration_tests.rs

#[tokio::test]
async fn test_create_project_full_flow() {
    // Setup: create test database
    let db = setup_test_db().await;
    let app_state = create_test_app_state(db.clone()).await;

    // Execute: create project via HTTP
    let response = app_state
        .handle_request(
            "POST",
            "/api/projects",
            json!({
                "title": "Test Project",
                "description": "A test",
            }),
        )
        .await;

    // Verify: response is 201 Created
    assert_eq!(response.status(), 201);

    // Verify: project in database
    let project = db
        .query("SELECT * FROM projects LIMIT 1")
        .await
        .unwrap()
        .take::<Project>(0)
        .unwrap()
        .unwrap();

    assert_eq!(project.title, "Test Project");
}

Real Database Tests:

// crates/vapora-backend/tests/database_tests.rs

#[tokio::test]
async fn test_multi_tenant_isolation() {
    let db = setup_real_surrealdb().await;

    // Create projects for two tenants
    let project_1 = db
        .create("projects")
        .content(Project {
            tenant_id: "tenant:1".to_string(),
            title: "Project 1".to_string(),
            ..Default::default()
        })
        .await
        .unwrap();

    let project_2 = db
        .create("projects")
        .content(Project {
            tenant_id: "tenant:2".to_string(),
            title: "Project 2".to_string(),
            ..Default::default()
        })
        .await
        .unwrap();

    // Query: tenant 1 should only see their project
    let results = db
        .query("SELECT * FROM projects WHERE tenant_id = 'tenant:1'")
        .await
        .unwrap()
        .take::<Vec<Project>>(0)
        .unwrap();

    assert_eq!(results.len(), 1);
    assert_eq!(results[0].title, "Project 1");
}

Test Utilities:

// crates/vapora-backend/tests/common/mod.rs

pub async fn setup_test_db() -> Surreal<Mem> {
    let db = Surreal::new::<surrealdb::engine::local::Mem>()
        .await
        .unwrap();

    db.use_ns("vapora").use_db("test").await.unwrap();

    // Initialize schema
    init_schema(&db).await.unwrap();

    db
}

pub async fn setup_real_surrealdb() -> Surreal<Ws> {
    // Connect to test SurrealDB instance
    let db = Surreal::new::<Ws>("ws://localhost:8000")
        .await
        .unwrap();

    db.signin(/* test credentials */).await.unwrap();
    db.use_ns("test").use_db("test").await.unwrap();

    db
}

Running Tests:

# Run all tests
cargo test --workspace

# Run unit tests only (fast)
cargo test --workspace --lib

# Run integration tests
cargo test --workspace --test "*"

# Run with output
cargo test --workspace -- --nocapture

# Run specific test
cargo test -p vapora-backend test_multi_tenant_isolation

# Coverage report
cargo tarpaulin --workspace --out Html

Key Files:

• crates/*/src/ (unit tests inline)
• crates/*/tests/ (integration tests)
• crates/*/tests/common/ (test utilities)

---

Verification

# Count tests across workspace
cargo test --workspace -- --list | grep "test " | wc -l

# Run all tests with statistics
cargo test --workspace 2>&1 | grep -E "^test |passed|failed"

# Coverage report
cargo tarpaulin --workspace --out Html
# Output: coverage/index.html

Expected Output:

• 218+ tests total
• All tests passing
• Coverage > 70%
• Unit tests < 5 seconds
• Integration tests < 60 seconds

---

Consequences

Testing Cadence

• Pre-commit: run unit tests
• PR: run all tests
• CI/CD: run all tests + coverage

Test Environment

• Unit tests: in-memory databases
• Integration: SurrealDB in-memory
• Real DB: Docker container (CI/CD only)

Debugging

• Unit test failure: easy to debug (isolated)
• Integration failure: check component interaction
• DB failure: verify schema and queries

---

References

• Rust Testing Documentation
• crates/*/tests/ (integration tests)
• crates/vapora-backend/tests/common/ (test utilities)

---

Related ADRs: ADR-022 (Error Handling), ADR-004 (SurrealDB)