kogral/docs/architecture/adrs/006-surrealdb-v3-engine-abstraction.md

# ADR-006: SurrealDB 3.0 Engine Abstraction

**Status**: Accepted

**Date**: 2026-02-21

**Deciders**: Architecture Team

**Supersedes**: [ADR-003: Hybrid Storage Strategy](003-hybrid-storage.md) (SurrealDB connection model)

---

## Context

SurrealDB 2.x required a concrete connection type at compile time (`Surreal<Ws>`, `Surreal<Mem>`,
etc.), forcing a single engine choice per binary. This created three problems:

1. **No runtime engine selection**: switching from embedded to remote required recompilation
2. **Tight test coupling**: tests depended on whichever engine was compiled in
3. **No dual-layout support**: using SurrealKV for graph data and RocksDB for hot data required
   separate crates or unsafe casting

SurrealDB 3.0 introduces `surrealdb::engine::any::connect(url)` returning `Surreal<Any>` — a
type-erased connection dispatched at runtime by URL scheme:

| URL scheme | Engine | Characteristics |
|---|---|---|
| `mem://` | In-memory | Ephemeral, test isolation |
| `surrealkv://path` | SurrealKV (B-tree) | Embedded, relational/graph data |
| `rocksdb://path` | RocksDB (LSM) | Embedded, append-heavy hot data |
| `ws://host:port` | WebSocket | Remote, team/shared deployment |

This makes the engine a pure config concern.

---

## Decision

**Use `Surreal<Any>` throughout `SurrealDbStorage`, selecting the engine from `SurrealEngineConfig`
at runtime via URL dispatch.**

### Config Schema

`SurrealEngineConfig` is a serde-tagged enum serialized with `tag = "engine"`:

```rust
#[serde(tag = "engine", rename_all = "snake_case")]
pub enum SurrealEngineConfig {
    Mem,
    SurrealKv { path: String },
    RocksDb   { path: String },
    Ws        { url: String },
}
```

`to_url()` maps variants to the URL scheme `engine::any::connect` expects:

```rust
impl SurrealEngineConfig {
    pub fn to_url(&self) -> String {
        match self {
            Self::Mem               => "mem://".to_string(),
            Self::SurrealKv { path } => format!("surrealkv://{path}"),
            Self::RocksDb   { path } => format!("rocksdb://{path}"),
            Self::Ws        { url }  => url.clone(),
        }
    }
}
```

### Dual Hot/Cold Layout

`SurrealDbStorage` holds two independent `Surreal<Any>` connections:

```rust
pub struct SurrealDbStorage {
    graph_db: Surreal<Any>,   // SurrealKV default — nodes, edges, metadata
    hot_db:   Surreal<Any>,   // RocksDB default  — embeddings, session logs
    namespace: String,
}
```

Default production layout:

```nickel
secondary.surrealdb = {
  graph = { engine = "surreal_kv", path = ".kogral/db/graph" },
  hot   = { engine = "rocks_db",   path = ".kogral/db/hot"   },
  namespace = "kogral",
}
```

Test layout (in-memory, no filesystem side effects):

```nickel
secondary.surrealdb = {
  graph = { engine = "mem" },
  hot   = { engine = "mem" },
  namespace = "test",
}
```

Remote team deployment (single SurrealDB instance, two databases):

```nickel
secondary.surrealdb = {
  graph = { engine = "ws", url = "ws://kb.company.com:8000" },
  hot   = { engine = "ws", url = "ws://kb.company.com:8000" },
  namespace = "engineering",
}
```

### Storage Factory

`storage::factory::build()` performs the runtime dispatch, keeping SurrealDB-specific logic
behind the `surrealdb-backend` feature gate:

```rust
#[allow(clippy::unused_async)]
pub async fn build(config: &StorageConfig, base_path: PathBuf) -> Result<Box<dyn Storage>> {
    #[cfg(feature = "surrealdb-backend")]
    if config.secondary.enabled && config.secondary.storage_type == SecondaryStorageType::Surrealdb {
        let db = SurrealDbStorage::from_config(&config.secondary.surrealdb).await?;
        return Ok(Box::new(db));
    }
    Ok(match config.primary {
        StorageType::Memory     => Box::new(MemoryStorage::new()),
        StorageType::Filesystem => Box::new(FilesystemStorage::new(base_path)),
    })
}
```

### Type Erasure Composition

`impl Storage for Box<dyn Storage>` enables `EventingStorage<Box<dyn Storage>>` without
knowing the concrete backend at compile time:

```rust
#[async_trait]
impl Storage for Box<dyn Storage> {
    async fn save_graph(&mut self, graph: &Graph) -> Result<()> {
        (**self).save_graph(graph).await
    }
    // ... full delegation for all methods
}
```

### CRUD via `serde_json::Value`

SurrealDB 3.0 removed the `IntoSurrealValue`/`SurrealValue` traits. All CRUD goes through
`serde_json::Value` as the intermediary:

```rust
// save_node
let row = serde_json::to_value(node)
    .map_err(|e| KbError::Storage(e.to_string()))?;
let _: Option<serde_json::Value> = self.graph_db
    .upsert(("nodes", format!("{graph_name}__{}", node.id)))
    .content(row)
    .await
    .map_err(|e| KbError::Storage(e.to_string()))?;
```

`.bind()` parameters require `'static` values — `&str` arguments must be `.to_string()`:

```rust
let nodes: Vec<Node> = self.graph_db
    .query("SELECT * FROM nodes WHERE project = $g")
    .bind(("g", graph_name.to_string()))   // .to_string() required
    .await...
```

### Error Handling

`KbError::Database(#[from] surrealdb::Error)` was removed. All SurrealDB errors convert via
`map_err(|e| KbError::Storage(e.to_string()))`, avoiding `#[from]` coupling to a feature-gated
type that would break compilation on default features.

---

## Consequences

### Positive

- Engine selection is a config value, not a compile-time decision
- Tests run fully in-memory (`engine = "mem"`) with zero filesystem side effects
- Dual layout (SurrealKV + RocksDB) is the embedded production default
- Remote deployment (WebSocket) requires only a config change
- `Storage` trait consumers never see SurrealDB types — `Box<dyn Storage>` is the boundary

### Negative

- `Surreal<Any>` has slightly higher dispatch overhead than concrete types (negligible vs. I/O)
- `serde_json::Value` intermediary adds one extra allocation per CRUD call
- `engine::any` requires all four engine feature flags compiled in when `surrealdb-backend` is
  enabled (larger binary)

---

## Feature Matrix

```toml
[features]
default           = ["filesystem"]
filesystem        = []
surrealdb-backend = ["dep:surrealdb"]
nats-events       = ["dep:platform-nats", "dep:bytes"]
orchestration     = ["nats-events", "dep:stratum-orchestrator"]
full              = ["surrealdb-backend", "nats-events", "orchestration"]
```

SurrealDB dependency activates all four engines:

```toml
surrealdb = { workspace = true, optional = true,
              features = ["kv-surrealkv", "kv-rocksdb", "protocol-ws", "rustls"] }
```

---

## References

- [SurrealDB `engine::any` docs](https://surrealdb.com/docs/sdk/rust/setup)
- [SurrealDbStorage](../../../crates/kogral-core/src/storage/surrealdb.rs)
- [Storage Factory](../../../crates/kogral-core/src/storage/factory.rs)
- [Config Schema](../../../crates/kogral-core/src/config/schema.rs)
- [Nickel Defaults](../../../schemas/kogral/defaults.ncl)

---

## Revision History

| Date | Author | Change |
|---|---|---|
| 2026-02-21 | Architecture Team | Initial decision — SurrealDB 3.0 + engine::any |

---

**Previous ADR**: [ADR-005: MCP Protocol](005-mcp-protocol.md)

**Next ADR**: [ADR-007: NATS Event Publishing](007-nats-event-publishing.md)