# SurrealDB Implementation Checklist

Complete checklist for implementing SurrealDB support with configuration-based selection and migration tool.

## ✅ Phase 1: Design & Planning (Week 1)

### Architecture
- [ ] Design Database trait with all required methods
- [ ] Plan SQLite adapter implementation
- [ ] Plan SurrealDB adapter implementation
- [ ] Plan configuration schema (TOML structure)
- [ ] Design migration tool architecture

### Configuration System
- [ ] Define DatabaseConfig struct with engine selection
- [ ] Create config examples (sqlite.toml, surrealdb.toml)
- [ ] Plan environment variable overrides
- [ ] Design config validation logic

### Testing Strategy
- [ ] Plan mock Database implementation
- [ ] Plan unit tests for each backend
- [ ] Plan integration tests
- [ ] Plan E2E tests
- [ ] Plan performance benchmarks

---

## ✅ Phase 2: Trait Layer & SQLite Refactoring (Weeks 1-2)

### Core Trait Definition (syntaxis-core/src/persistence/mod.rs)
- [ ] Define `#[async_trait] pub trait Database`
- [ ] Extract all 50+ method signatures
- [ ] Document trait with rustdoc
- [ ] Add error handling trait bounds
- [ ] Create tests for trait interface

### Configuration Loading
- [ ] Create DatabaseConfig struct (serde)
- [ ] Implement config loading from TOML
- [ ] Add environment variable override support
- [ ] Add config validation
- [ ] Create example configs

### SQLite Adapter
- [ ] Move current persistence.rs code to sqlite_impl.rs
- [ ] Implement Database trait for SqliteDatabase
- [ ] Test all existing functionality
- [ ] Verify backward compatibility

### File Structure
- [ ] Create syntaxis-core/src/persistence/
  - [ ] mod.rs (trait definition)
  - [ ] sqlite_impl.rs (SQLite implementation)
  - [ ] error.rs (unified error handling)
  - [ ] config.rs (configuration management)

---

## ✅ Phase 3: SurrealDB Implementation (Weeks 2-3)

### SurrealDB Adapter (syntaxis-core/src/persistence/surrealdb_impl.rs)
- [ ] Create SurrealDatabase struct
- [ ] Initialize SurrealDB connection
- [ ] Implement schema initialization
- [ ] Implement Database trait for SurrealDatabase
  - [ ] Project operations (CRUD)
  - [ ] Checklist operations
  - [ ] Phase transition operations
  - [ ] Security assessment operations
  - [ ] Tool configuration operations
  - [ ] Activity logging operations
  - [ ] Backup history operations

### Schema Definition
- [ ] Create SurrealQL schema definitions
- [ ] Define collections (projects, checklist_items, etc.)
- [ ] Define relationships (RELATE for phase transitions)
- [ ] Create indices for performance
- [ ] Document schema structure

### Query Optimization
- [ ] Test graph queries (phase transitions)
- [ ] Test document queries (tool configs)
- [ ] Test filtering and sorting
- [ ] Optimize query performance
- [ ] Create performance benchmarks

---

## ✅ Phase 4: Migration Tool (Week 3)

### Data Migration Strategy
- [ ] Design migration architecture
- [ ] Create SQLite → SurrealDB migration tool
  - [ ] Read all data from SQLite
  - [ ] Transform to SurrealDB format
  - [ ] Validate data integrity
  - [ ] Handle relationships (foreign keys → RELATE)

### CLI for Migration
- [ ] Create `workspace migrate` command
- [ ] Add `--from sqlite --to surrealdb` flags
- [ ] Add `--dry-run` option
- [ ] Add `--backup` option (backup before migration)
- [ ] Add validation and error reporting

### Rollback Support
- [ ] Design rollback strategy
- [ ] Implement backup mechanism
- [ ] Create `workspace migrate --rollback` command
- [ ] Document recovery procedures

### Testing
- [ ] Create test data
- [ ] Test migration with various data sizes
- [ ] Test rollback functionality
- [ ] Verify data integrity post-migration

---

## ✅ Phase 5: Runtime Selection (Weeks 3-4)

### Backend Selection Enum
- [ ] Create DatabaseBackend enum
  - [ ] DatabaseBackend::Sqlite(SqliteDatabase)
  - [ ] DatabaseBackend::SurrealDB(SurrealDatabase)
- [ ] Implement Database trait for enum
- [ ] Add runtime selection logic

### Configuration-Driven Initialization
- [ ] Load config file
- [ ] Parse engine selection
- [ ] Create appropriate backend
- [ ] Handle initialization errors gracefully
- [ ] Log selected backend

### Documentation
- [ ] Create user guide for config selection
- [ ] Document each configuration option
- [ ] Provide example configurations
- [ ] Document environment variables
- [ ] Document migration procedures

---

## ✅ Phase 6: Integration & Testing (Week 4)

### Binary Updates
- [ ] **syntaxis-cli** - Accept database config
- [ ] **syntaxis-tui** - Handle database initialization
- [ ] **syntaxis-api** - Use DatabaseBackend
- [ ] **syntaxis-dashboard** - Connect to backend

### Unit Tests
- [ ] SQLite adapter tests (170+ tests)
- [ ] SurrealDB adapter tests (170+ tests)
- [ ] Configuration loading tests
- [ ] Migration tool tests
- [ ] Backend selection tests

### Integration Tests
- [ ] Create project → store → retrieve (SQLite)
- [ ] Create project → store → retrieve (SurrealDB)
- [ ] Phase transitions (graph queries)
- [ ] Complex filtering
- [ ] Concurrent operations

### E2E Tests
- [ ] CLI workflow (SQLite)
- [ ] CLI workflow (SurrealDB)
- [ ] TUI operations (SQLite)
- [ ] TUI operations (SurrealDB)
- [ ] API endpoints (SQLite)
- [ ] API endpoints (SurrealDB)

### Performance Benchmarks
- [ ] Read performance (SQLite vs SurrealDB)
- [ ] Write performance (SQLite vs SurrealDB)
- [ ] Complex query performance
- [ ] Memory usage
- [ ] Connection pooling

---

## ✅ Phase 7: Documentation & Deployment (Week 5)

### User Documentation
- [ ] Configuration guide (how to choose backend)
- [ ] Migration guide (SQLite → SurrealDB)
- [ ] Troubleshooting guide
- [ ] Performance tuning guide
- [ ] Backup/restore procedures

### Developer Documentation
- [ ] Architecture documentation
- [ ] Database trait reference
- [ ] SQLite implementation guide
- [ ] SurrealDB implementation guide
- [ ] Adding new backends (extension guide)

### Deployment Documentation
- [ ] Docker setup (SurrealDB server)
- [ ] Production deployment guide
- [ ] Kubernetes deployment guide
- [ ] Monitoring and observability
- [ ] Disaster recovery procedures

### Release Artifacts
- [ ] Update CHANGELOG.md
- [ ] Create migration guide
- [ ] Create release notes
- [ ] Update README with backend options
- [ ] Create compatibility matrix

---

## Configuration Examples

### SQLite (Current/Default)

```toml
# configs/database-sqlite.toml
[database]
engine = "sqlite"
sqlite_path = "~/.local/share/core/workspace.db"
max_connections = 5
timeout_secs = 30
```

### SurrealDB (Cloud-Ready)

```toml
# configs/database-surrealdb.toml
[database]
engine = "surrealdb"

[database.surrealdb]
url = "ws://surrealdb-server:8000"
namespace = "syntaxis"
database = "projects"
username = "${SURREALDB_USER}"    # From env var
password = "${SURREALDB_PASSWORD}"
max_connections = 10
timeout_secs = 30
```

### PostgreSQL (Future)

```toml
# configs/database-postgres.toml (for future implementation)
[database]
engine = "postgres"

[database.postgres]
host = "${POSTGRES_HOST}"
port = 5432
username = "${POSTGRES_USER}"
password = "${POSTGRES_PASSWORD}"
database = "syntaxis"
max_connections = 20
timeout_secs = 30
```

---

## Migration Tool CLI

### Commands

```bash
# Show current database
workspace database status

# List available backends
workspace database list-backends

# Migrate from SQLite to SurrealDB
workspace database migrate \
  --from sqlite \
  --to surrealdb \
  --target-url ws://surrealdb-server:8000 \
  --backup

# Dry run (show what would happen)
workspace database migrate \
  --from sqlite \
  --to surrealdb \
  --dry-run

# Rollback migration
workspace database migrate --rollback

# Validate data integrity
workspace database validate
```

---

## Testing Checklist

### Unit Tests (Minimum Coverage: 80%)
- [ ] Database trait contract tests
- [ ] SQLite implementation tests (all 50+ methods)
- [ ] SurrealDB implementation tests (all 50+ methods)
- [ ] Configuration loading tests
- [ ] Error handling tests
- [ ] Backend selection tests

### Integration Tests
- [ ] Full syntaxis (SQLite)
- [ ] Full syntaxis (SurrealDB)
- [ ] Data consistency across backends
- [ ] Transaction handling
- [ ] Concurrent operations
- [ ] Error recovery

### Migration Tests
- [ ] SQLite → SurrealDB migration (small dataset)
- [ ] SQLite → SurrealDB migration (large dataset)
- [ ] Data integrity post-migration
- [ ] Rollback functionality
- [ ] Performance during migration
- [ ] Concurrent operations during migration

### Performance Tests
- [ ] Read throughput (1000 operations)
- [ ] Write throughput (1000 operations)
- [ ] Complex queries (10 scenarios)
- [ ] Memory usage over time
- [ ] Connection pool efficiency

---

## Documentation Deliverables

### User-Facing
- [ ] Configuration Quick Start Guide
- [ ] How to Choose Your Backend (decision matrix)
- [ ] SQLite Configuration Guide
- [ ] SurrealDB Configuration Guide
- [ ] Migration Guide (SQLite → SurrealDB)
- [ ] Troubleshooting Guide
- [ ] Performance Tuning Guide
- [ ] Backup & Restore Guide

### Developer-Facing
- [ ] Architecture Overview
- [ ] Database Trait Reference
- [ ] SQLite Implementation Details
- [ ] SurrealDB Implementation Details
- [ ] Adding New Backends (extension guide)
- [ ] Testing Strategy
- [ ] Performance Benchmarks

### Operations-Facing
- [ ] Production Deployment Guide
- [ ] Docker Compose Setup
- [ ] Kubernetes Deployment
- [ ] Monitoring & Observability
- [ ] Disaster Recovery
- [ ] Capacity Planning

---

## Code Quality Checklist

Before release:

### Formatting & Linting
- [ ] `cargo fmt --all` ✅
- [ ] `cargo clippy --all-targets` ✅
- [ ] No warnings
- [ ] Documentation complete

### Testing
- [ ] All unit tests pass (>80% coverage)
- [ ] All integration tests pass
- [ ] All E2E tests pass
- [ ] Performance benchmarks documented
- [ ] No flaky tests

### Security
- [ ] No `unsafe` code
- [ ] No `unwrap()` in production
- [ ] SQL injection protected (parameterized queries)
- [ ] Authentication validated
- [ ] No hardcoded credentials
- [ ] Secrets from environment variables

### Performance
- [ ] Benchmarks < 10ms per query (simple)
- [ ] Benchmarks < 100ms per query (complex)
- [ ] Memory usage reasonable
- [ ] Connection pooling working
- [ ] No N+1 queries

---

## Success Criteria

✅ **Phase Complete When:**

1. **Configuration System**
   - [ ] Both backends configurable via TOML
   - [ ] Environment variables supported
   - [ ] Runtime selection working

2. **Both Backends Working**
   - [ ] SQLite implementation passes all tests
   - [ ] SurrealDB implementation passes all tests
   - [ ] Data integrity verified across both

3. **Migration Tool**
   - [ ] SQLite → SurrealDB migration working
   - [ ] Data integrity verified post-migration
   - [ ] Rollback working
   - [ ] Backup/restore working

4. **Documentation**
   - [ ] User guide complete
   - [ ] Developer guide complete
   - [ ] API documentation complete
   - [ ] Configuration examples provided

5. **Quality**
   - [ ] All tests passing (632+ tests)
   - [ ] >80% code coverage
   - [ ] Performance benchmarks documented
   - [ ] Zero breaking changes (backward compatible)

---

## Sign-off

- [ ] **Development Team** - Implementation complete
- [ ] **QA Team** - Testing complete, issues resolved
- [ ] **Security Team** - Security review passed
- [ ] **Operations Team** - Deployment ready
- [ ] **Product Team** - Feature approved for release

---

## Timeline Estimate

| Phase | Duration | Start | End |
|-------|----------|-------|-----|
| 1 | 1 week | Week 1 | Week 1 |
| 2 | 2 weeks | Week 1 | Week 2 |
| 3 | 2 weeks | Week 2 | Week 3 |
| 4 | 1 week | Week 3 | Week 3 |
| 5 | 1 week | Week 3 | Week 4 |
| 6 | 1 week | Week 4 | Week 4 |
| 7 | 1 week | Week 5 | Week 5 |
| **Total** | **5-6 weeks** | | |

**With 1 full-time developer + 0.5 QA + 0.5 DevOps**

---

## Notes

- All phases can overlap (agile approach)
- Testing should be continuous, not end-phase
- Documentation should be written as you code
- Regular demos to stakeholders every Friday
- Regular sync-ups (3x per week recommended)