2026-01-14 03:25:20 +00:00
|
|
|
# Storage Backends Guide\n\nThis document provides comprehensive guidance on the orchestrator's storage backend options, configuration, and migration between them.\n\n## Overview\n\nThe orchestrator supports three storage backends through a pluggable architecture:\n\n1. **Filesystem** - JSON file-based storage (default)\n2. **SurrealDB Embedded** - Local database with RocksDB engine\n3. **SurrealDB Server** - Remote SurrealDB server connection\n\nAll backends implement the same `TaskStorage` trait, ensuring consistent behavior and seamless migration.\n\n## Backend Comparison\n\n| Feature | Filesystem | SurrealDB Embedded | SurrealDB Server |\n| --------- | ------------ | ------------------- | ------------------ |\n| **Setup Complexity** | Minimal | Low | Medium |\n| **External Dependencies** | None | None | SurrealDB Server |\n| **Storage Format** | JSON Files | RocksDB | Remote DB |\n| **ACID Transactions** | No | Yes | Yes |\n| **Authentication/RBAC** | Basic | Advanced | Advanced |\n| **Real-time Subscriptions** | No | Yes | Yes |\n| **Audit Logging** | Manual | Automatic | Automatic |\n| **Metrics Collection** | Basic | Advanced | Advanced |\n| **Task Dependencies** | Simple | Graph-based | Graph-based |\n| **Horizontal Scaling** | No | No | Yes |\n| **Backup/Recovery** | File Copy | Database Backup | Server Backup |\n| **Performance** | Good | Excellent | Variable |\n| **Memory Usage** | Low | Medium | Low |\n| **Disk Usage** | Medium | Optimized | Minimal |\n\n## 1. Filesystem Backend\n\n### Overview\n\nThe default storage backend using JSON files for task persistence. Ideal for development and simple deployments.\n\n### Configuration\n\n```{$detected_lang}\n# Default configuration\n./orchestrator --storage-type filesystem --data-dir ./data\n\n# Custom data directory\n./orchestrator --storage-type filesystem --data-dir /var/lib/orchestrator\n```\n\n### File Structure\n\n```{$detected_lang}\ndata/\n└── queue.rkvs/\n ├── tasks/\n │ ├── uuid1.json # Individual task records\n │ ├── uuid2.json\n │ └── ...\n └── queue/\n ├── uuid1.json # Queue entries with priority\n ├── uuid2.json\n └── ...\n```\n\n### Features\n\n- ✅ **Simple Setup**: No external dependencies\n- ✅ **Transparency**: Human-readable JSON files\n- ✅ **Backup**: Standard file system tools\n- ✅ **Debugging**: Direct file inspection\n- ❌ **ACID**: No transaction guarantees\n- ❌ **Concurrency**: Basic file locking\n- ❌ **Advanced Features**: Limited auth/audit\n\n### Best Use Cases\n\n- Development environments\n- Single-instance deployments\n- Simple task orchestration\n- Environments with strict dependency requirements\n\n## 2. SurrealDB Embedded\n\n### Overview\n\nLocal SurrealDB database using RocksDB storage engine. Provides advanced database features without external dependencies.\n\n### Configuration\n\n```{$detected_lang}\n# Build with SurrealDB support\ncargo build --features surrealdb\n\n# Run with embedded SurrealDB\n./orchestrator --storage-type surrealdb-embedded --data-dir ./data\n```\n\n### Database Schema\n\n- **tasks**: Main task records with full metadata\n- **task_queue**: Priority queue with scheduling info\n- **users**: Authentication and RBAC\n- **audit_log**: Complete operation history\n- **metrics**: Performance and usage statistics\n- **task_events**: Real-time event stream\n\n### Features\n\n- ✅ **ACID Transactions**: Reliable data consistency\n- ✅ **Advanced Queries**: SQL-like syntax with graph support\n- ✅ **Real-time Events**: Live query subscriptions\n- ✅ **Built-in Auth**: User management and RBAC\n- ✅ **Audit Logging**: Automatic operation tracking\n- ✅ **No External Deps**: Self-contained database\n- ❌ **Horizontal Scaling**: Single-node only\n\n### Configuration Options\n\n```{$detected_lang}\n# Custom database location\n./orchestrator --storage-type surrealdb-embedded \\n --data-dir /var/lib/orchestrator/db\n\n# With specific namespace/database\n./orchestrator --storage-type surrealdb-embe
|
