# ADR-005: Service Communication Protocol (REST + Async Queue)

**Status**: Accepted | **Date**: 2025-01-16 | **Supersedes**: None

## Context

With 12 microservices, a communication strategy is required balancing reliability, latency, and complexity.

## Decision

Dual communication model:
- **Synchronous**: REST API for request-response (latency < 100ms target)
- **Asynchronous**: Message queues for long-running operations (batch workflows, resource provisioning)

## Rationale

1. **REST for Immediate Operations**:
   - Control flow requires immediate feedback (CLI commands, UI actions)
   - Latency critical for user experience
   - Error handling simpler with synchronous responses

2. **Queues for Long Operations**:
   - Workflow execution may take hours
   - Network failures shouldn't cancel operations
   - Load smoothing across services
   - Better resource utilization

## Implementation

### REST Endpoints

All services expose Actix-web REST APIs:

```rust
// /provisioning/platform/crates/*/src/api/
pub struct ApiServer {
    router: Router,
}

impl ApiServer {
    pub fn new() -> Self {
        Router::new()
            .route("/health", get(health_check))
            .route("/api/v1/resources", get(list_resources))
            .route("/api/v1/resources/:id", get(get_resource))
            .route("/api/v1/resources", post(create_resource))
    }
}
```

### Async Queue Pattern

Using Nushell for queue management and Rust services as workers:

```bash
# Submit workflow to queue
provisioning batch submit workflows/multi-cloud-deploy.ncl \
  --queue async \
  --callback https://control-center/webhooks/workflow-complete

# Queue persists to file-based storage
# Worker (orchestrator) processes asynchronously
# Client polls status or receives webhook notification
```

### Error Handling

**REST failures**:
```rust
match client.get("/vault-service/health").await {
    Ok(response) => { /* continue */ },
    Err(_) => {
        // Fallback: Use cached secrets
        // Retry with exponential backoff
        // Alert monitoring
    }
}
```

**Queue failures**:
```nushell
# Failed messages retry with backoff
orchestrator submit --queue async --max-retries 3 --backoff exponential
# After retries exhausted, move to dead-letter queue for manual review
```

## Request Flow

### Synchronous (REST)

```text
CLI → Control-Center API (100ms)
  → Workspace lookup ✓
  → Return workspace config ✓
Response to user
```

### Asynchronous (Queue)

```text
CLI → Orchestrator (accept immediately)
  → Queue workflow (100ms)
  ✓ Return job_id to user

[Async worker]
Orchestrator processes job
  → Execute tasks
  → Update SurrealDB state
  → Send webhook notification
User polls status or receives notification
```

## Latency Targets

| Operation | Target | SLA |
| ----------- | -------- | ----- |
| Health check | <50ms | 99.95% |
| List workspaces | <200ms | 99.9% |
| Create workspace | <500ms | 99.5% |
| Start workflow | <1s | 99% |
| Task execution | minutes/hours | N/A (monitored) |

## Consequences

- **Positive**:
  - Responsive CLI (immediate feedback)
  - Reliable long operations (queuing)
  - Natural fit for infrastructure workflows
  - Easy horizontal scaling (queue consumers)

- **Negative**:
  - Operational complexity (monitoring queues)
  - Eventual consistency (state updates delayed)
  - Testing asynchronous flows harder
  - Webhook callback management

## Monitoring

```bash
# Queue depth monitoring
provisioning queue status
# Output:
# Queue: async          | Pending: 45    | Failed: 2      | Processed: 1,234
# Queue: priority       | Pending: 0     | Failed: 0      | Processed: 589

# Service latency
curl http://control-center:8080/metrics | grep http_request_duration_seconds
# Output:
# http_request_duration_seconds_bucket{method="GET",path="/api/v1/workspaces",...,le="0.05"} 234
# http_request_duration_seconds_bucket{method="GET",path="/api/v1/workspaces",...,le="0.1"} 456
# http_request_duration_seconds_bucket{method="GET",path="/api/v1/workspaces",...,le="0.5"} 890
```

## Related ADRs

- [ADR-004: Microservice Distribution](./adr-004-microservice-distribution.md) - 12 services communicating
