468 lines
12 KiB
Markdown
468 lines
12 KiB
Markdown
|
# SurrealDB Implementation Checklist
|
||
|
|
|
||
|
|
Complete checklist for implementing SurrealDB support with configuration-based selection and migration tool.
|
||
|
|
|
||
|
|
## ✅ Phase 1: Design & Planning (Week 1)
|
||
|
|
|
||
|
|
### Architecture
|
||
|
|
- [ ] Design Database trait with all required methods
|
||
|
|
- [ ] Plan SQLite adapter implementation
|
||
|
|
- [ ] Plan SurrealDB adapter implementation
|
||
|
|
- [ ] Plan configuration schema (TOML structure)
|
||
|
|
- [ ] Design migration tool architecture
|
||
|
|
|
||
|
|
### Configuration System
|
||
|
|
- [ ] Define DatabaseConfig struct with engine selection
|
||
|
|
- [ ] Create config examples (sqlite.toml, surrealdb.toml)
|
||
|
|
- [ ] Plan environment variable overrides
|
||
|
|
- [ ] Design config validation logic
|
||
|
|
|
||
|
|
### Testing Strategy
|
||
|
|
- [ ] Plan mock Database implementation
|
||
|
|
- [ ] Plan unit tests for each backend
|
||
|
|
- [ ] Plan integration tests
|
||
|
|
- [ ] Plan E2E tests
|
||
|
|
- [ ] Plan performance benchmarks
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## ✅ Phase 2: Trait Layer & SQLite Refactoring (Weeks 1-2)
|
||
|
|
|
||
|
|
### Core Trait Definition (syntaxis-core/src/persistence/mod.rs)
|
||
|
|
- [ ] Define `#[async_trait] pub trait Database`
|
||
|
|
- [ ] Extract all 50+ method signatures
|
||
|
|
- [ ] Document trait with rustdoc
|
||
|
|
- [ ] Add error handling trait bounds
|
||
|
|
- [ ] Create tests for trait interface
|
||
|
|
|
||
|
|
### Configuration Loading
|
||
|
|
- [ ] Create DatabaseConfig struct (serde)
|
||
|
|
- [ ] Implement config loading from TOML
|
||
|
|
- [ ] Add environment variable override support
|
||
|
|
- [ ] Add config validation
|
||
|
|
- [ ] Create example configs
|
||
|
|
|
||
|
|
### SQLite Adapter
|
||
|
|
- [ ] Move current persistence.rs code to sqlite_impl.rs
|
||
|
|
- [ ] Implement Database trait for SqliteDatabase
|
||
|
|
- [ ] Test all existing functionality
|
||
|
|
- [ ] Verify backward compatibility
|
||
|
|
|
||
|
|
### File Structure
|
||
|
|
- [ ] Create syntaxis-core/src/persistence/
|
||
|
|
- [ ] mod.rs (trait definition)
|
||
|
|
- [ ] sqlite_impl.rs (SQLite implementation)
|
||
|
|
- [ ] error.rs (unified error handling)
|
||
|
|
- [ ] config.rs (configuration management)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## ✅ Phase 3: SurrealDB Implementation (Weeks 2-3)
|
||
|
|
|
||
|
|
### SurrealDB Adapter (syntaxis-core/src/persistence/surrealdb_impl.rs)
|
||
|
|
- [ ] Create SurrealDatabase struct
|
||
|
|
- [ ] Initialize SurrealDB connection
|
||
|
|
- [ ] Implement schema initialization
|
||
|
|
- [ ] Implement Database trait for SurrealDatabase
|
||
|
|
- [ ] Project operations (CRUD)
|
||
|
|
- [ ] Checklist operations
|
||
|
|
- [ ] Phase transition operations
|
||
|
|
- [ ] Security assessment operations
|
||
|
|
- [ ] Tool configuration operations
|
||
|
|
- [ ] Activity logging operations
|
||
|
|
- [ ] Backup history operations
|
||
|
|
|
||
|
|
### Schema Definition
|
||
|
|
- [ ] Create SurrealQL schema definitions
|
||
|
|
- [ ] Define collections (projects, checklist_items, etc.)
|
||
|
|
- [ ] Define relationships (RELATE for phase transitions)
|
||
|
|
- [ ] Create indices for performance
|
||
|
|
- [ ] Document schema structure
|
||
|
|
|
||
|
|
### Query Optimization
|
||
|
|
- [ ] Test graph queries (phase transitions)
|
||
|
|
- [ ] Test document queries (tool configs)
|
||
|
|
- [ ] Test filtering and sorting
|
||
|
|
- [ ] Optimize query performance
|
||
|
|
- [ ] Create performance benchmarks
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## ✅ Phase 4: Migration Tool (Week 3)
|
||
|
|
|
||
|
|
### Data Migration Strategy
|
||
|
|
- [ ] Design migration architecture
|
||
|
|
- [ ] Create SQLite → SurrealDB migration tool
|
||
|
|
- [ ] Read all data from SQLite
|
||
|
|
- [ ] Transform to SurrealDB format
|
||
|
|
- [ ] Validate data integrity
|
||
|
|
- [ ] Handle relationships (foreign keys → RELATE)
|
||
|
|
|
||
|
|
### CLI for Migration
|
||
|
|
- [ ] Create `workspace migrate` command
|
||
|
|
- [ ] Add `--from sqlite --to surrealdb` flags
|
||
|
|
- [ ] Add `--dry-run` option
|
||
|
|
- [ ] Add `--backup` option (backup before migration)
|
||
|
|
- [ ] Add validation and error reporting
|
||
|
|
|
||
|
|
### Rollback Support
|
||
|
|
- [ ] Design rollback strategy
|
||
|
|
- [ ] Implement backup mechanism
|
||
|
|
- [ ] Create `workspace migrate --rollback` command
|
||
|
|
- [ ] Document recovery procedures
|
||
|
|
|
||
|
|
### Testing
|
||
|
|
- [ ] Create test data
|
||
|
|
- [ ] Test migration with various data sizes
|
||
|
|
- [ ] Test rollback functionality
|
||
|
|
- [ ] Verify data integrity post-migration
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## ✅ Phase 5: Runtime Selection (Weeks 3-4)
|
||
|
|
|
||
|
|
### Backend Selection Enum
|
||
|
|
- [ ] Create DatabaseBackend enum
|
||
|
|
- [ ] DatabaseBackend::Sqlite(SqliteDatabase)
|
||
|
|
- [ ] DatabaseBackend::SurrealDB(SurrealDatabase)
|
||
|
|
- [ ] Implement Database trait for enum
|
||
|
|
- [ ] Add runtime selection logic
|
||
|
|
|
||
|
|
### Configuration-Driven Initialization
|
||
|
|
- [ ] Load config file
|
||
|
|
- [ ] Parse engine selection
|
||
|
|
- [ ] Create appropriate backend
|
||
|
|
- [ ] Handle initialization errors gracefully
|
||
|
|
- [ ] Log selected backend
|
||
|
|
|
||
|
|
### Documentation
|
||
|
|
- [ ] Create user guide for config selection
|
||
|
|
- [ ] Document each configuration option
|
||
|
|
- [ ] Provide example configurations
|
||
|
|
- [ ] Document environment variables
|
||
|
|
- [ ] Document migration procedures
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## ✅ Phase 6: Integration & Testing (Week 4)
|
||
|
|
|
||
|
|
### Binary Updates
|
||
|
|
- [ ] **syntaxis-cli** - Accept database config
|
||
|
|
- [ ] **syntaxis-tui** - Handle database initialization
|
||
|
|
- [ ] **syntaxis-api** - Use DatabaseBackend
|
||
|
|
- [ ] **syntaxis-dashboard** - Connect to backend
|
||
|
|
|
||
|
|
### Unit Tests
|
||
|
|
- [ ] SQLite adapter tests (170+ tests)
|
||
|
|
- [ ] SurrealDB adapter tests (170+ tests)
|
||
|
|
- [ ] Configuration loading tests
|
||
|
|
- [ ] Migration tool tests
|
||
|
|
- [ ] Backend selection tests
|
||
|
|
|
||
|
|
### Integration Tests
|
||
|
|
- [ ] Create project → store → retrieve (SQLite)
|
||
|
|
- [ ] Create project → store → retrieve (SurrealDB)
|
||
|
|
- [ ] Phase transitions (graph queries)
|
||
|
|
- [ ] Complex filtering
|
||
|
|
- [ ] Concurrent operations
|
||
|
|
|
||
|
|
### E2E Tests
|
||
|
|
- [ ] CLI workflow (SQLite)
|
||
|
|
- [ ] CLI workflow (SurrealDB)
|
||
|
|
- [ ] TUI operations (SQLite)
|
||
|
|
- [ ] TUI operations (SurrealDB)
|
||
|
|
- [ ] API endpoints (SQLite)
|
||
|
|
- [ ] API endpoints (SurrealDB)
|
||
|
|
|
||
|
|
### Performance Benchmarks
|
||
|
|
- [ ] Read performance (SQLite vs SurrealDB)
|
||
|
|
- [ ] Write performance (SQLite vs SurrealDB)
|
||
|
|
- [ ] Complex query performance
|
||
|
|
- [ ] Memory usage
|
||
|
|
- [ ] Connection pooling
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## ✅ Phase 7: Documentation & Deployment (Week 5)
|
||
|
|
|
||
|
|
### User Documentation
|
||
|
|
- [ ] Configuration guide (how to choose backend)
|
||
|
|
- [ ] Migration guide (SQLite → SurrealDB)
|
||
|
|
- [ ] Troubleshooting guide
|
||
|
|
- [ ] Performance tuning guide
|
||
|
|
- [ ] Backup/restore procedures
|
||
|
|
|
||
|
|
### Developer Documentation
|
||
|
|
- [ ] Architecture documentation
|
||
|
|
- [ ] Database trait reference
|
||
|
|
- [ ] SQLite implementation guide
|
||
|
|
- [ ] SurrealDB implementation guide
|
||
|
|
- [ ] Adding new backends (extension guide)
|
||
|
|
|
||
|
|
### Deployment Documentation
|
||
|
|
- [ ] Docker setup (SurrealDB server)
|
||
|
|
- [ ] Production deployment guide
|
||
|
|
- [ ] Kubernetes deployment guide
|
||
|
|
- [ ] Monitoring and observability
|
||
|
|
- [ ] Disaster recovery procedures
|
||
|
|
|
||
|
|
### Release Artifacts
|
||
|
|
- [ ] Update CHANGELOG.md
|
||
|
|
- [ ] Create migration guide
|
||
|
|
- [ ] Create release notes
|
||
|
|
- [ ] Update README with backend options
|
||
|
|
- [ ] Create compatibility matrix
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Configuration Examples
|
||
|
|
|
||
|
|
### SQLite (Current/Default)
|
||
|
|
|
||
|
|
```toml
|
||
|
|
# configs/database-sqlite.toml
|
||
|
|
[database]
|
||
|
|
engine = "sqlite"
|
||
|
|
sqlite_path = "~/.local/share/core/workspace.db"
|
||
|
|
max_connections = 5
|
||
|
|
timeout_secs = 30
|
||
|
|
```
|
||
|
|
|
||
|
|
### SurrealDB (Cloud-Ready)
|
||
|
|
|
||
|
|
```toml
|
||
|
|
# configs/database-surrealdb.toml
|
||
|
|
[database]
|
||
|
|
engine = "surrealdb"
|
||
|
|
|
||
|
|
[database.surrealdb]
|
||
|
|
url = "ws://surrealdb-server:8000"
|
||
|
|
namespace = "syntaxis"
|
||
|
|
database = "projects"
|
||
|
|
username = "${SURREALDB_USER}" # From env var
|
||
|
|
password = "${SURREALDB_PASSWORD}"
|
||
|
|
max_connections = 10
|
||
|
|
timeout_secs = 30
|
||
|
|
```
|
||
|
|
|
||
|
|
### PostgreSQL (Future)
|
||
|
|
|
||
|
|
```toml
|
||
|
|
# configs/database-postgres.toml (for future implementation)
|
||
|
|
[database]
|
||
|
|
engine = "postgres"
|
||
|
|
|
||
|
|
[database.postgres]
|
||
|
|
host = "${POSTGRES_HOST}"
|
||
|
|
port = 5432
|
||
|
|
username = "${POSTGRES_USER}"
|
||
|
|
password = "${POSTGRES_PASSWORD}"
|
||
|
|
database = "syntaxis"
|
||
|
|
max_connections = 20
|
||
|
|
timeout_secs = 30
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Migration Tool CLI
|
||
|
|
|
||
|
|
### Commands
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Show current database
|
||
|
|
workspace database status
|
||
|
|
|
||
|
|
# List available backends
|
||
|
|
workspace database list-backends
|
||
|
|
|
||
|
|
# Migrate from SQLite to SurrealDB
|
||
|
|
workspace database migrate \
|
||
|
|
--from sqlite \
|
||
|
|
--to surrealdb \
|
||
|
|
--target-url ws://surrealdb-server:8000 \
|
||
|
|
--backup
|
||
|
|
|
||
|
|
# Dry run (show what would happen)
|
||
|
|
workspace database migrate \
|
||
|
|
--from sqlite \
|
||
|
|
--to surrealdb \
|
||
|
|
--dry-run
|
||
|
|
|
||
|
|
# Rollback migration
|
||
|
|
workspace database migrate --rollback
|
||
|
|
|
||
|
|
# Validate data integrity
|
||
|
|
workspace database validate
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Testing Checklist
|
||
|
|
|
||
|
|
### Unit Tests (Minimum Coverage: 80%)
|
||
|
|
- [ ] Database trait contract tests
|
||
|
|
- [ ] SQLite implementation tests (all 50+ methods)
|
||
|
|
- [ ] SurrealDB implementation tests (all 50+ methods)
|
||
|
|
- [ ] Configuration loading tests
|
||
|
|
- [ ] Error handling tests
|
||
|
|
- [ ] Backend selection tests
|
||
|
|
|
||
|
|
### Integration Tests
|
||
|
|
- [ ] Full syntaxis (SQLite)
|
||
|
|
- [ ] Full syntaxis (SurrealDB)
|
||
|
|
- [ ] Data consistency across backends
|
||
|
|
- [ ] Transaction handling
|
||
|
|
- [ ] Concurrent operations
|
||
|
|
- [ ] Error recovery
|
||
|
|
|
||
|
|
### Migration Tests
|
||
|
|
- [ ] SQLite → SurrealDB migration (small dataset)
|
||
|
|
- [ ] SQLite → SurrealDB migration (large dataset)
|
||
|
|
- [ ] Data integrity post-migration
|
||
|
|
- [ ] Rollback functionality
|
||
|
|
- [ ] Performance during migration
|
||
|
|
- [ ] Concurrent operations during migration
|
||
|
|
|
||
|
|
### Performance Tests
|
||
|
|
- [ ] Read throughput (1000 operations)
|
||
|
|
- [ ] Write throughput (1000 operations)
|
||
|
|
- [ ] Complex queries (10 scenarios)
|
||
|
|
- [ ] Memory usage over time
|
||
|
|
- [ ] Connection pool efficiency
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Documentation Deliverables
|
||
|
|
|
||
|
|
### User-Facing
|
||
|
|
- [ ] Configuration Quick Start Guide
|
||
|
|
- [ ] How to Choose Your Backend (decision matrix)
|
||
|
|
- [ ] SQLite Configuration Guide
|
||
|
|
- [ ] SurrealDB Configuration Guide
|
||
|
|
- [ ] Migration Guide (SQLite → SurrealDB)
|
||
|
|
- [ ] Troubleshooting Guide
|
||
|
|
- [ ] Performance Tuning Guide
|
||
|
|
- [ ] Backup & Restore Guide
|
||
|
|
|
||
|
|
### Developer-Facing
|
||
|
|
- [ ] Architecture Overview
|
||
|
|
- [ ] Database Trait Reference
|
||
|
|
- [ ] SQLite Implementation Details
|
||
|
|
- [ ] SurrealDB Implementation Details
|
||
|
|
- [ ] Adding New Backends (extension guide)
|
||
|
|
- [ ] Testing Strategy
|
||
|
|
- [ ] Performance Benchmarks
|
||
|
|
|
||
|
|
### Operations-Facing
|
||
|
|
- [ ] Production Deployment Guide
|
||
|
|
- [ ] Docker Compose Setup
|
||
|
|
- [ ] Kubernetes Deployment
|
||
|
|
- [ ] Monitoring & Observability
|
||
|
|
- [ ] Disaster Recovery
|
||
|
|
- [ ] Capacity Planning
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Code Quality Checklist
|
||
|
|
|
||
|
|
Before release:
|
||
|
|
|
||
|
|
### Formatting & Linting
|
||
|
|
- [ ] `cargo fmt --all` ✅
|
||
|
|
- [ ] `cargo clippy --all-targets` ✅
|
||
|
|
- [ ] No warnings
|
||
|
|
- [ ] Documentation complete
|
||
|
|
|
||
|
|
### Testing
|
||
|
|
- [ ] All unit tests pass (>80% coverage)
|
||
|
|
- [ ] All integration tests pass
|
||
|
|
- [ ] All E2E tests pass
|
||
|
|
- [ ] Performance benchmarks documented
|
||
|
|
- [ ] No flaky tests
|
||
|
|
|
||
|
|
### Security
|
||
|
|
- [ ] No `unsafe` code
|
||
|
|
- [ ] No `unwrap()` in production
|
||
|
|
- [ ] SQL injection protected (parameterized queries)
|
||
|
|
- [ ] Authentication validated
|
||
|
|
- [ ] No hardcoded credentials
|
||
|
|
- [ ] Secrets from environment variables
|
||
|
|
|
||
|
|
### Performance
|
||
|
|
- [ ] Benchmarks < 10ms per query (simple)
|
||
|
|
- [ ] Benchmarks < 100ms per query (complex)
|
||
|
|
- [ ] Memory usage reasonable
|
||
|
|
- [ ] Connection pooling working
|
||
|
|
- [ ] No N+1 queries
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Success Criteria
|
||
|
|
|
||
|
|
✅ **Phase Complete When:**
|
||
|
|
|
||
|
|
1. **Configuration System**
|
||
|
|
- [ ] Both backends configurable via TOML
|
||
|
|
- [ ] Environment variables supported
|
||
|
|
- [ ] Runtime selection working
|
||
|
|
|
||
|
|
2. **Both Backends Working**
|
||
|
|
- [ ] SQLite implementation passes all tests
|
||
|
|
- [ ] SurrealDB implementation passes all tests
|
||
|
|
- [ ] Data integrity verified across both
|
||
|
|
|
||
|
|
3. **Migration Tool**
|
||
|
|
- [ ] SQLite → SurrealDB migration working
|
||
|
|
- [ ] Data integrity verified post-migration
|
||
|
|
- [ ] Rollback working
|
||
|
|
- [ ] Backup/restore working
|
||
|
|
|
||
|
|
4. **Documentation**
|
||
|
|
- [ ] User guide complete
|
||
|
|
- [ ] Developer guide complete
|
||
|
|
- [ ] API documentation complete
|
||
|
|
- [ ] Configuration examples provided
|
||
|
|
|
||
|
|
5. **Quality**
|
||
|
|
- [ ] All tests passing (632+ tests)
|
||
|
|
- [ ] >80% code coverage
|
||
|
|
- [ ] Performance benchmarks documented
|
||
|
|
- [ ] Zero breaking changes (backward compatible)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Sign-off
|
||
|
|
|
||
|
|
- [ ] **Development Team** - Implementation complete
|
||
|
|
- [ ] **QA Team** - Testing complete, issues resolved
|
||
|
|
- [ ] **Security Team** - Security review passed
|
||
|
|
- [ ] **Operations Team** - Deployment ready
|
||
|
|
- [ ] **Product Team** - Feature approved for release
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Timeline Estimate
|
||
|
|
|
||
|
|
| Phase | Duration | Start | End |
|
||
|
|
|-------|----------|-------|-----|
|
||
|
|
| 1 | 1 week | Week 1 | Week 1 |
|
||
|
|
| 2 | 2 weeks | Week 1 | Week 2 |
|
||
|
|
| 3 | 2 weeks | Week 2 | Week 3 |
|
||
|
|
| 4 | 1 week | Week 3 | Week 3 |
|
||
|
|
| 5 | 1 week | Week 3 | Week 4 |
|
||
|
|
| 6 | 1 week | Week 4 | Week 4 |
|
||
|
|
| 7 | 1 week | Week 5 | Week 5 |
|
||
|
|
| **Total** | **5-6 weeks** | | |
|
||
|
|
|
||
|
|
**With 1 full-time developer + 0.5 QA + 0.5 DevOps**
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Notes
|
||
|
|
|
||
|
|
- All phases can overlap (agile approach)
|
||
|
|
- Testing should be continuous, not end-phase
|
||
|
|
- Documentation should be written as you code
|
||
|
|
- Regular demos to stakeholders every Friday
|
||
|
|
- Regular sync-ups (3x per week recommended)
|