ADR-023: Multi-Layer Testing Strategy

Status: Accepted | Implemented Date: 2024-11-01 Deciders: Quality Assurance Team Technical Story: Building confidence through unit, integration, and real-database tests

Decision

Implementar multi-layer testing: unit tests (inline), integration tests (tests/ dir), real DB connections.

Rationale

Unit Tests: Fast feedback on logic changes
Integration Tests: Verify components work together
Real DB Tests: Catch database schema/query issues
218+ Tests: Comprehensive coverage across 13 crates

Alternatives Considered

❌ Unit Tests Only

Pros: Fast
Cons: Miss integration bugs, schema issues

❌ Integration Tests Only

Pros: Comprehensive
Cons: Slow, harder to debug

✅ Multi-Layer (CHOSEN)

All three layers catch different issues

Trade-offs

Pros:

✅ Fast feedback (unit)
✅ Integration validation (integration)
✅ Real-world confidence (real DB)
✅ 218+ tests total coverage

Cons:

⚠️ Slow full test suite (~5 minutes)
⚠️ DB tests require test environment
⚠️ More test code to maintain

Implementation

Unit Tests (Inline):

#![allow(unused)]
fn main() {
// crates/vapora-agents/src/learning_profile.rs

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_expertise_score_empty() {
        let profile = TaskTypeLearning {
            agent_id: "test".to_string(),
            task_type: "architecture".to_string(),
            executions_total: 0,
            records: vec![],
            ..Default::default()
        };

        assert_eq!(profile.expertise_score(), 0.0);
    }

    #[test]
    fn test_confidence_weighting() {
        let profile = TaskTypeLearning {
            executions_total: 20,
            ..Default::default()
        };
        assert_eq!(profile.confidence(), 1.0);

        let profile_partial = TaskTypeLearning {
            executions_total: 10,
            ..Default::default()
        };
        assert_eq!(profile_partial.confidence(), 0.5);
    }
}
}

Integration Tests:

#![allow(unused)]
fn main() {
// crates/vapora-backend/tests/integration_tests.rs

#[tokio::test]
async fn test_create_project_full_flow() {
    // Setup: create test database
    let db = setup_test_db().await;
    let app_state = create_test_app_state(db.clone()).await;

    // Execute: create project via HTTP
    let response = app_state
        .handle_request(
            "POST",
            "/api/projects",
            json!({
                "title": "Test Project",
                "description": "A test",
            }),
        )
        .await;

    // Verify: response is 201 Created
    assert_eq!(response.status(), 201);

    // Verify: project in database
    let project = db
        .query("SELECT * FROM projects LIMIT 1")
        .await
        .unwrap()
        .take::<Project>(0)
        .unwrap()
        .unwrap();

    assert_eq!(project.title, "Test Project");
}
}

Real Database Tests:

#![allow(unused)]
fn main() {
// crates/vapora-backend/tests/database_tests.rs

#[tokio::test]
async fn test_multi_tenant_isolation() {
    let db = setup_real_surrealdb().await;

    // Create projects for two tenants
    let project_1 = db
        .create("projects")
        .content(Project {
            tenant_id: "tenant:1".to_string(),
            title: "Project 1".to_string(),
            ..Default::default()
        })
        .await
        .unwrap();

    let project_2 = db
        .create("projects")
        .content(Project {
            tenant_id: "tenant:2".to_string(),
            title: "Project 2".to_string(),
            ..Default::default()
        })
        .await
        .unwrap();

    // Query: tenant 1 should only see their project
    let results = db
        .query("SELECT * FROM projects WHERE tenant_id = 'tenant:1'")
        .await
        .unwrap()
        .take::<Vec<Project>>(0)
        .unwrap();

    assert_eq!(results.len(), 1);
    assert_eq!(results[0].title, "Project 1");
}
}

Test Utilities:

#![allow(unused)]
fn main() {
// crates/vapora-backend/tests/common/mod.rs

pub async fn setup_test_db() -> Surreal<Mem> {
    let db = Surreal::new::<surrealdb::engine::local::Mem>()
        .await
        .unwrap();

    db.use_ns("vapora").use_db("test").await.unwrap();

    // Initialize schema
    init_schema(&db).await.unwrap();

    db
}

pub async fn setup_real_surrealdb() -> Surreal<Ws> {
    // Connect to test SurrealDB instance
    let db = Surreal::new::<Ws>("ws://localhost:8000")
        .await
        .unwrap();

    db.signin(/* test credentials */).await.unwrap();
    db.use_ns("test").use_db("test").await.unwrap();

    db
}
}

Running Tests:

# Run all tests
cargo test --workspace

# Run unit tests only (fast)
cargo test --workspace --lib

# Run integration tests
cargo test --workspace --test "*"

# Run with output
cargo test --workspace -- --nocapture

# Run specific test
cargo test -p vapora-backend test_multi_tenant_isolation

# Coverage report
cargo tarpaulin --workspace --out Html

Key Files:

crates/*/src/ (unit tests inline)
crates/*/tests/ (integration tests)
crates/*/tests/common/ (test utilities)

Verification

# Count tests across workspace
cargo test --workspace -- --list | grep "test " | wc -l

# Run all tests with statistics
cargo test --workspace 2>&1 | grep -E "^test |passed|failed"

# Coverage report
cargo tarpaulin --workspace --out Html
# Output: coverage/index.html

Expected Output:

218+ tests total
All tests passing
Coverage > 70%
Unit tests < 5 seconds
Integration tests < 60 seconds

Consequences

Testing Cadence

Pre-commit: run unit tests
PR: run all tests
CI/CD: run all tests + coverage

Test Environment

Unit tests: in-memory databases
Integration: SurrealDB in-memory
Real DB: Docker container (CI/CD only)

Debugging

Unit test failure: easy to debug (isolated)
Integration failure: check component interaction
DB failure: verify schema and queries

References

Rust Testing Documentation
crates/*/tests/ (integration tests)
crates/vapora-backend/tests/common/ (test utilities)

Related ADRs: ADR-022 (Error Handling), ADR-004 (SurrealDB)

Keyboard shortcuts

VAPORA Platform Documentation