# SurrealDB Migration Guide for syntaxis ## Executive Summary **Current System**: SQLite with sqlx (sync schema in code) **Proposed System**: SurrealDB (multi-model NoSQL + graph database) **Migration Complexity**: **MEDIUM** - Doable but requires careful abstraction **Estimated Effort**: 40-60 hours (depends on feature set) **Risk Level**: **MEDIUM** - Good separation of concerns exists --- ## Current Architecture Analysis ### Database Layer (syntaxis-core/src/persistence.rs) **Current Design:** ```rust pub struct PersistenceLayer { pool: SqlitePool, // ← Tightly coupled to SQLite } impl PersistenceLayer { pub async fn new(db_path: &str) -> Result { ... } async fn run_migrations(pool: &SqlitePool) -> Result<()> { ... } // Query methods (200+ lines of direct SQLx queries) pub async fn create_project(&self, ...) -> Result { ... } pub async fn get_project(&self, id: &str) -> Result { ... } // ... 50+ more query methods } ``` **Coupling Points:** 1. ✗ `SqlitePool` directly in struct (SQLx-specific) 2. ✗ Raw SQL strings in `run_migrations()` (SQLite dialect) 3. ✗ `sqlx::query!` macros throughout (compile-time SQLite checking) 4. ✗ Hard-coded migration logic (SQLite PRAGMA, CREATE TABLE IF NOT EXISTS) **Good News:** - ✅ Single responsibility (only persistence) - ✅ Result return types (error abstraction) - ✅ No leakage into business logic layers - ✅ 1500 line file (manageable, not monolithic) --- ## Migration Strategy ### Phase 1: Create Database Trait (Low Risk) **Goal**: Abstract database operations behind a trait ```rust // syntaxis-core/src/persistence/db_trait.rs #[async_trait] pub trait Database: Send + Sync + Clone { // Project operations async fn create_project(&self, req: &CreateProjectRequest) -> Result; async fn get_project(&self, id: &str) -> Result; async fn list_projects(&self) -> Result>; async fn update_project(&self, id: &str, req: &UpdateProjectRequest) -> Result; async fn delete_project(&self, id: &str) -> Result<()>; // Checklist operations async fn create_checklist_item(&self, item: &ChecklistItem) -> Result; async fn list_checklist_items(&self, project_id: &str) -> Result>; async fn update_checklist_item(&self, id: &str, completed: bool) -> Result; // Phase operations async fn record_phase_transition(&self, transition: &PhaseTransition) -> Result<()>; async fn get_phase_history(&self, project_id: &str) -> Result>; // ... more operations } ``` **Benefits:** - Allows SQLite and SurrealDB implementations to coexist - Can test with mock implementations - No immediate breaking changes - Incremental migration possible ### Phase 2: Implement SQLite Adapter ```rust // syntaxis-core/src/persistence/sqlite_impl.rs pub struct SqliteDatabase { pool: SqlitePool, } #[async_trait] impl Database for SqliteDatabase { async fn create_project(&self, req: &CreateProjectRequest) -> Result { // Current implementation moved here } // ... all other methods } ``` **Outcome**: Existing code works unchanged, but now isolated in adapter ### Phase 3: Implement SurrealDB Adapter ```rust // syntaxis-core/src/persistence/surrealdb_impl.rs pub struct SurrealDatabase { db: Surreal, // SurrealDB connection } #[async_trait] impl Database for SurrealDatabase { async fn create_project(&self, req: &CreateProjectRequest) -> Result { // SurrealDB implementation let project = Project::new(req); // SurrealDB: Graph-native queries self.db.create::("projects", Some(&project.id)) .content(&project) .await?; Ok(project) } async fn list_projects(&self) -> Result> { // SurrealDB: Natural graph queries let projects: Vec = self.db .query("SELECT * FROM projects WHERE archived = false") .await? .take(0)?; Ok(projects) } async fn record_phase_transition(&self, transition: &PhaseTransition) -> Result<()> { // SurrealDB: Graph edge creation self.db.query(" LET $project = type::thing('projects', $project_id); LET $from = type::thing('phases', $from_phase); LET $to = type::thing('phases', $to_phase); RELATE $project->transitioned_to->$to SET reason = $reason, timestamp = $timestamp, from_phase = $from_phase ") .bind(("project_id", &transition.project_id)) .bind(("from_phase", &transition.from_phase)) .bind(("to_phase", &transition.to_phase)) .bind(("reason", &transition.reason)) .bind(("timestamp", &transition.timestamp)) .await?; Ok(()) } } ``` **SurrealDB Advantages:** - **Graph Queries**: Phase transitions become edges (natural relationships) - **Relations**: Checklist items → Projects → Phases (built-in) - **Multi-model**: Document + Graph + SQL-like syntax - **Type Safety**: Serde integration for documents - **Async Native**: Built for tokio runtime ### Phase 4: Configuration-Driven Selection ```toml # configs/default.toml [database] engine = "sqlite" # or "surrealdb" sqlite_path = "workspace.db" [database.surrealdb] url = "ws://localhost:8000" # WebSocket for local, TCP for remote namespace = "syntaxis" database = "projects" username = "root" password = "root" ``` ```rust // syntaxis-core/src/persistence/mod.rs pub enum DatabaseEngine { Sqlite(SqliteDatabase), SurrealDB(SurrealDatabase), } impl DatabaseEngine { pub async fn from_config(config: &DatabaseConfig) -> Result { match config.engine.as_str() { "sqlite" => { let db = SqliteDatabase::new(&config.sqlite_path).await?; Ok(DatabaseEngine::Sqlite(db)) } "surrealdb" => { let db = SurrealDatabase::new(&config.surrealdb_url).await?; Ok(DatabaseEngine::SurrealDB(db)) } _ => Err(LifecycleError::Config(format!("Unknown database engine: {}", config.engine))), } } } // All callers use trait, not specific implementation pub async fn use_database(engine: DatabaseEngine) -> Result<()> { let projects = engine.list_projects().await?; // Works with both! Ok(()) } ``` --- ## SurrealDB Benefits for syntaxis ### 1. **Graph-Native Phase Transitions** **Current (SQLite):** ```sql -- Store transitions in table, query with JOINs SELECT pt.*, p.name FROM phase_transitions pt JOIN projects p ON pt.project_id = p.id WHERE pt.project_id = ? ORDER BY timestamp DESC ``` **SurrealDB (Graph):** ```surql -- Query graph edges directly SELECT * FROM project:12345 -> transitioned_to -> * ``` ### 2. **Dynamic Relationships (Checklist Dependencies)** **Current (SQLite):** ```sql -- Store JSON array of task_deps, parse manually SELECT task_deps FROM checklist_items WHERE id = ? -- Then manually fetch related tasks ``` **SurrealDB (Relations):** ```surql -- Query relationships directly SELECT * FROM task:123 -> depends_on -> * SELECT * -> depends_on -> * FROM task:123 -- Recursive ``` ### 3. **Flexible Schema for Tool Configurations** **Current (SQLite):** ```sql -- config_json stored as string, must parse CREATE TABLE tool_configurations ( id TEXT PRIMARY KEY, config_json TEXT, -- {"rust_version": "1.75", "lint_level": "pedantic"} ... ) ``` **SurrealDB (Document):** ```surql -- Native document storage CREATE toolconfig:123 CONTENT { tool_name: 'cargo-clippy', enabled: true, rust_version: '1.75', lint_level: 'pedantic', lint_rules: ['all', 'pedantic'], updated_at: }; ``` ### 4. **Audit Trail (Built-in Time-Travel)** **Current (SQLite):** ```sql -- Activity log in separate table CREATE TABLE activity_logs ( id TEXT, action TEXT, timestamp TEXT, ... ) ``` **SurrealDB (Versioning):** ```surql -- SurrealDB versions automatically SELECT * FROM projects VERSION AT ; SELECT * FROM projects DIFF FROM ; ``` ### 5. **Real-time Notifications (WebSocket)** **Current (SQLite):** - Poll database at intervals - Manual change tracking **SurrealDB:** - Built-in WebSocket subscriptions - Push notifications on changes - Perfect for TUI/Dashboard live updates ```rust // Subscribe to project changes let mut stream = db.query("LIVE SELECT * FROM projects") .await? .take(0)?; while let Some(change) = stream.next().await { println!("Project changed: {:?}", change); } ``` --- ## Implementation Roadmap ### Step 1: Add SurrealDB Dependency (1 hour) ```toml # Cargo.toml [workspace.dependencies] sqlx = { version = "0.8", features = ["runtime-tokio-native-tls", "sqlite", "macros"] } surrealdb = { version = "2.0", features = ["ws"] } # Add this ``` ### Step 2: Create Trait Layer (4-6 hours) - Define `Database` trait in `syntaxis-core/src/persistence/mod.rs` - Move 50+ methods signatures to trait - Create error wrapper for SurrealDB errors ### Step 3: Refactor SQLite to Adapter (6-8 hours) - Move SQLite implementation to `sqlite_impl.rs` - Implement `Database` trait for `SqliteDatabase` - Test all existing functionality ### Step 4: Implement SurrealDB Adapter (12-16 hours) - Create schema (tables/types in SurrealDB) - Implement `Database` trait for `SurrealDatabase` - Handle type conversions (Serde) ### Step 5: Update Binaries (4-6 hours) - `syntaxis-api`: Accept database config, use trait - `syntaxis-cli`: Load database from config - `syntaxis-tui`: Switch databases at runtime - Tests: Update to use trait ### Step 6: Testing & Validation (8-10 hours) - Unit tests for both adapters - Integration tests - Performance benchmarks (SQLite vs SurrealDB) - E2E tests with TUI/API --- ## Database Schema Mapping ### SQLite Tables → SurrealDB Collections ``` SQLite SurrealDB projects → projects:{id} checklist_items → checklist_items:{id} phase_transitions → phase_transitions:{id} security_assessments → assessments:{id} tool_configurations → tool_configs:{id} activity_logs → activities:{id} backup_history → backups:{id} phase_history → phase_history:{id} Foreign Keys → Graph Relations (RELATE) Indices → SurrealDB DEFINE INDEX ``` ### Example Schema Definition ```surql -- Create collections with schema DEFINE TABLE projects SCHEMAFULL AS SELECT id, name, version, description, project_type, current_phase, created_at, updated_at FROM projects; DEFINE FIELD projects.id AS string ASSERT $before IS NONE; DEFINE FIELD projects.name AS string; DEFINE FIELD projects.version AS string; DEFINE FIELD projects.current_phase AS enum; -- Create graph relation for phase transitions DEFINE TABLE transitioned_to SCHEMALESS AS SELECT * FROM transitioned_to; -- Create indices DEFINE INDEX idx_projects_created ON TABLE projects COLUMNS created_at DESC; DEFINE INDEX idx_checklist_project ON TABLE checklist_items COLUMNS project_id; ``` --- ## Configuration Example ```toml # .env.development - Use SQLite [database] engine = "sqlite" sqlite_path = "data/workspace.db" # .env.production - Use SurrealDB [database] engine = "surrealdb" [database.surrealdb] url = "ws://surrealdb-server:8000" namespace = "syntaxis" database = "projects" username = "${SURREALDB_USER}" # From env var password = "${SURREALDB_PASS}" ``` --- ## Risk Assessment & Mitigation | Risk | Impact | Probability | Mitigation | |------|--------|-------------|-----------| | **Breaking Changes** | High | Medium | Trait pattern allows gradual migration, keep SQLite as fallback | | **Data Migration** | High | High | Create migration script: SQLite → SurrealDB before switching | | **SurrealDB Performance** | Medium | Low | Benchmark both; can tune indexes and caching | | **Test Coverage** | Medium | Medium | Mock implementation of Database trait for tests | | **Operator Complexity** | Medium | High | Keep SQLite as simple default, SurrealDB for advanced features | --- ## Recommendation **✅ YES, migrate to SurrealDB** because: 1. **Clean separation exists** - Single persistence module, no leakage 2. **Natural fit for domain** - Graph relationships (phases, tasks, dependencies) are core 3. **Future features easier** - Real-time subscriptions, versioning, complex queries 4. **Abstraction pattern** - Trait-based approach enables both systems to coexist 5. **Not urgent** - Can be done incrementally without disrupting current work **Best Approach:** 1. Keep SQLite as default (current behavior) 2. Implement SurrealDB as optional feature 3. Configuration-driven selection (TOML) 4. Tests work with both backends 5. Gradual migration path for users **Timeline:** Phase 1-3 in 2-3 sprints, full deployment by Q2 2025 --- ## Code References - **Current Persistence Layer**: `core/crates/syntaxis-core/src/persistence.rs` (1492 lines) - **Database Dependencies**: `Cargo.toml` lines 72-74 - **Config Loading**: `syntaxis-core/src/config.rs` - **Error Handling**: `syntaxis-core/src/error.rs` --- ## Further Reading - [SurrealDB Docs](https://surrealdb.com/docs) - [SurrealDB Rust Client](https://github.com/surrealdb/surrealdb.rs) - [Graph Database Patterns](https://surrealdb.com/docs/surrealql/statements/relate) - [Trait-based Abstraction in Rust](https://doc.rust-lang.org/book/ch17-02-using-trait-objects.html)