syntaxis/docs/surrealdb-setup-guide.md

# SurrealDB 2.3 Setup & Deployment Guide

**Version**: 2.3
**Status**: Production Ready
**Last Updated**: November 15, 2025

---

## Overview

This guide covers deploying and configuring SurrealDB 2.3 for the syntaxis project in three deployment contexts:

1. **Local Development** - Server mode for development/testing
2. **Docker Container** - Production containerized deployment
3. **Kubernetes** - Enterprise orchestration with HA/scaling

---

## Part 1: Local Development Setup

### 1.1 Install SurrealDB

#### macOS (via Homebrew)
```bash
brew install surrealdb
# Verify installation
surreal version
# Output: surrealdb version X.X.X
```

#### Linux (via Cargo)
```bash
cargo install surreal
# Or download binary from https://github.com/surrealdb/surrealdb/releases
```

#### Windows (via Chocolatey)
```powershell
choco install surrealdb
```

### 1.2 Start SurrealDB Server (Local Development)

#### Basic In-Memory Server
```bash
# Start SurrealDB listening on localhost:8000
surreal start --bind 127.0.0.1:8000 memory

# Surreal outputs:
# 2025-11-15T10:00:00.000Z INFO  surreal::cli: Starting SurrealDB server...
# 2025-11-15T10:00:00.123Z INFO  surreal::net: Database server now listening on 0.0.0.0:8000
```

#### File-Based RocksDB Server
```bash
# Persistent storage in local directory
surreal start --bind 127.0.0.1:8000 file:///tmp/surrealdb.db

# Or with expanded home path
surreal start --bind 127.0.0.1:8000 file://~/.local/share/surrealdb/workspace.db
```

#### With Authentication
```bash
# Start with credentials (recommended for production-like testing)
surreal start \
  --bind 127.0.0.1:8000 \
  --username admin \
  --password secure_password_123 \
  file:///tmp/surrealdb.db
```

### 1.3 Configure Application to Connect

**Configuration File**: `configs/database.toml`
```toml
# Database Configuration - syntaxis
engine = "surrealdb"  # or "sqlite" for testing

# SQLite Configuration (Default for development)
[sqlite]
path = "~/.local/share/core/workspace.db"
max_connections = 5
timeout_secs = 30
wal_mode = true
pragma_synchronous = "NORMAL"
pragma_cache_size = 2000

# SurrealDB Configuration (Optional - requires server running)
[surrealdb]
# Connection modes (choose one)
# In-Memory (embedded, no server needed):
url = "mem://"

# File-based (embedded RocksDB, no server needed):
# url = "file:///tmp/surrealdb.db"

# Remote Server (requires surreal start):
url = "ws://localhost:8000"

# Optional: ws:// for WebSocket or http:// for HTTP
# url = "http://localhost:8000"

# Namespace and Database selection
namespace = "syntaxis"
database = "projects"

# Authentication (if server started with --username/--password)
# username = "admin"
# password = "secure_password_123"

# Connection pooling
max_connections = 10
timeout_secs = 30

# TLS Configuration (for production)
# tls_enabled = true
# tls_ca_cert = "/path/to/ca.pem"
# tls_client_cert = "/path/to/client.pem"
# tls_client_key = "/path/to/client.key"
```

### 1.4 Enable SurrealDB in Application

**In Rust Code** (syntaxis-core):
```rust
use workspace_core::persistence::{Database, SurrealDatabase};

#[tokio::main]
async fn main() -> Result<()> {
    // Embedded mode (no server required)
    let db = SurrealDatabase::new_memory().await?;
    // or
    let db = SurrealDatabase::new_file("/tmp/surrealdb.db").await?;

    // Server mode (requires `surreal start` running)
    // let db = SurrealDatabase::new_server(
    //     "ws://localhost:8000",
    //     "syntaxis",
    //     "projects",
    //     None,  // username
    //     None,  // password
    // ).await?;

    Ok(())
}
```

### 1.5 Verify Connection

```bash
# Test if server is running
curl http://localhost:8000/health
# Expected: {"status":"ok"}

# Check with surreal CLI
surreal sql --endpoint ws://localhost:8000
# Then at prompt:
# > SELECT 1;
```

---

## Part 2: Docker Container Deployment

### 2.1 Create Dockerfile

**File**: `Dockerfile.surrealdb`
```dockerfile
# Multi-stage build for minimal image size
FROM surrealdb/surrealdb:latest AS surrealdb

# Production runtime image
FROM debian:bookworm-slim

# Install minimal dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Copy SurrealDB binary from builder
COPY --from=surrealdb /usr/local/bin/surreal /usr/local/bin/surreal

# Create data directory
RUN mkdir -p /data && chmod 755 /data

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8000/health || exit 1

# Expose port
EXPOSE 8000

# Labels for container metadata
LABEL maintainer="syntaxis"
LABEL version="2.3"
LABEL description="SurrealDB 2.3 for syntaxis"

# Run SurrealDB
ENTRYPOINT ["surreal", "start"]
CMD ["--bind", "0.0.0.0:8000", "file:///data/surrealdb.db"]

# Volume for data persistence
VOLUME ["/data"]
```

### 2.2 Docker Compose Configuration

**File**: `docker-compose.surrealdb.yml`
```yaml
version: "3.9"

services:
  surrealdb:
    # Option 1: Use official SurrealDB image
    image: surrealdb/surrealdb:latest

    # Option 2: Build from local Dockerfile
    # build:
    #   context: .
    #   dockerfile: Dockerfile.surrealdb

    container_name: workspace-surrealdb

    # Port mapping
    ports:
      - "8000:8000"

    # Volume for persistent data
    volumes:
      - surrealdb-data:/data

    # Environment variables
    environment:
      # SURREAL_LOG: debug  # Enable debug logging
      TZ: UTC

    # Command override
    command:
      - start
      - --bind
      - "0.0.0.0:8000"
      - --username
      - "admin"
      - --password
      - "workspace_password_123"
      - file:///data/surrealdb.db

    # Resource limits
    deploy:
      resources:
        limits:
          cpus: "2"
          memory: 2G
        reservations:
          cpus: "1"
          memory: 1G

    # Restart policy
    restart: unless-stopped

    # Health check
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 5s

    # Network
    networks:
      - workspace-net

  # Optional: Redis cache layer (for performance)
  redis:
    image: redis:7-alpine
    container_name: workspace-redis
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    restart: unless-stopped
    networks:
      - workspace-net

volumes:
  surrealdb-data:
    driver: local
  redis-data:
    driver: local

networks:
  workspace-net:
    driver: bridge
```

### 2.3 Docker Compose Usage

```bash
# Start SurrealDB
docker-compose -f docker-compose.surrealdb.yml up -d

# View logs
docker-compose -f docker-compose.surrealdb.yml logs -f surrealdb

# Connect to database
docker-compose -f docker-compose.surrealdb.yml exec surrealdb \
  surreal sql --endpoint ws://localhost:8000 \
  --username admin \
  --password workspace_password_123

# Stop services
docker-compose -f docker-compose.surrealdb.yml down

# Remove volumes (caution: deletes data)
docker-compose -f docker-compose.surrealdb.yml down -v
```

### 2.4 Multi-Environment Compose Override

**File**: `docker-compose.override.yml` (local development)
```yaml
version: "3.9"

services:
  surrealdb:
    # Expose debug port
    ports:
      - "8000:8000"
      - "8001:8001"  # Debug port

    # Use memory backend for faster development
    command:
      - start
      - --bind
      - "0.0.0.0:8000"
      - memory

    # Less resource restriction for dev
    deploy:
      resources:
        limits:
          cpus: "4"
          memory: 4G
```

**File**: `docker-compose.prod.yml` (production)
```yaml
version: "3.9"

services:
  surrealdb:
    image: surrealdb/surrealdb:2.3.0  # Pin version for prod

    # Stricter resource limits
    deploy:
      resources:
        limits:
          cpus: "1"
          memory: 1.5G
        reservations:
          cpus: "0.5"
          memory: 1G

    # Stricter restart policy
    restart: on-failure:5
```

---

## Part 3: Kubernetes Deployment

### 3.1 Kubernetes ConfigMap

**File**: `k8s/surrealdb-config.yaml`
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: surrealdb-config
  namespace: workspace
spec:
  database.toml: |
    engine = "surrealdb"

    [surrealdb]
    url = "ws://surrealdb:8000"
    namespace = "syntaxis"
    database = "projects"
    max_connections = 20
    timeout_secs = 60

  init-schema.sql: |
    -- SurrealDB schema initialization
    DEFINE TABLE projects SCHEMAFULL;
    DEFINE TABLE checklist_items SCHEMAFULL;
    DEFINE TABLE phase_transitions SCHEMAFULL;
    DEFINE TABLE security_assessments SCHEMAFULL;
    DEFINE TABLE phase_history SCHEMAFULL;
    DEFINE TABLE tool_configurations SCHEMAFULL;
    DEFINE TABLE tool_dependencies SCHEMAFULL;
    DEFINE TABLE security_assessment_details SCHEMAFULL;
    DEFINE TABLE team_members SCHEMAFULL;
```

### 3.2 Kubernetes Secret

**File**: `k8s/surrealdb-secret.yaml`
```yaml
apiVersion: v1
kind: Secret
metadata:
  name: surrealdb-credentials
  namespace: workspace
type: Opaque
stringData:
  username: "admin"
  password: "secure_kubernetes_password_here_at_least_32_chars_minimum"
  # Note: In production, use external secret management (Vault, AWS Secrets Manager, etc)
```

### 3.3 SurrealDB StatefulSet

**File**: `k8s/surrealdb-statefulset.yaml`
```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: surrealdb
  namespace: workspace
  labels:
    app: surrealdb
    version: "2.3"
spec:
  serviceName: surrealdb
  replicas: 1  # For HA, scale to 3 with clustering (future)
  selector:
    matchLabels:
      app: surrealdb
  template:
    metadata:
      labels:
        app: surrealdb
        version: "2.3"
    spec:
      # Node affinity for stability
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: node-type
                    operator: In
                    values: ["database"]

      # Security context
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
        fsGroup: 999

      # Init containers for schema setup
      initContainers:
        - name: init-schema
          image: surrealdb/surrealdb:2.3
          command:
            - /bin/sh
            - -c
            - |
              # Wait for SurrealDB to be ready
              surreal sql \
                --endpoint ws://localhost:8000 \
                --username admin \
                --password $SURREALDB_PASSWORD \
                < /config/init-schema.sql
          env:
            - name: SURREALDB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: surrealdb-credentials
                  key: password
          volumeMounts:
            - name: config
              mountPath: /config

      containers:
        - name: surrealdb
          image: surrealdb/surrealdb:2.3
          imagePullPolicy: IfNotPresent

          # Resource requests and limits
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "2000m"
              memory: "2Gi"

          # Port exposure
          ports:
            - name: http
              containerPort: 8000
              protocol: TCP

          # Environment variables
          env:
            - name: SURREALDB_USER
              valueFrom:
                secretKeyRef:
                  name: surrealdb-credentials
                  key: username
            - name: SURREALDB_PASS
              valueFrom:
                secretKeyRef:
                  name: surrealdb-credentials
                  key: password

          # Startup probe (for slow startup)
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 30

          # Liveness probe
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 10
            periodSeconds: 30
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 3

          # Readiness probe
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 3

          # Volume mounts
          volumeMounts:
            - name: config
              mountPath: /etc/surrealdb
            - name: data
              mountPath: /data

          # Security context for container
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL

          # Startup command
          args:
            - start
            - --bind
            - 0.0.0.0:8000
            - --username
            - $(SURREALDB_USER)
            - --password
            - $(SURREALDB_PASS)
            - file:///data/surrealdb.db

      # Volumes
      volumes:
        - name: config
          configMap:
            name: surrealdb-config

  # Persistent volume claim
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: standard  # Change to appropriate storage class
        resources:
          requests:
            storage: 10Gi  # Adjust based on data needs
```

### 3.4 Kubernetes Service

**File**: `k8s/surrealdb-service.yaml`
```yaml
---
# Headless service for StatefulSet
apiVersion: v1
kind: Service
metadata:
  name: surrealdb
  namespace: workspace
  labels:
    app: surrealdb
spec:
  clusterIP: None  # Headless service
  selector:
    app: surrealdb
  ports:
    - name: http
      port: 8000
      targetPort: 8000
      protocol: TCP

---
# Load balancer service for external access (optional)
apiVersion: v1
kind: Service
metadata:
  name: surrealdb-lb
  namespace: workspace
  labels:
    app: surrealdb
spec:
  type: LoadBalancer
  selector:
    app: surrealdb
  ports:
    - name: http
      port: 8000
      targetPort: 8000
      protocol: TCP

---
# ClusterIP for internal access
apiVersion: v1
kind: Service
metadata:
  name: surrealdb-internal
  namespace: workspace
  labels:
    app: surrealdb
spec:
  type: ClusterIP
  selector:
    app: surrealdb
  ports:
    - name: http
      port: 8000
      targetPort: 8000
      protocol: TCP
```

### 3.5 Kubernetes Deployment Script

**File**: `k8s/deploy.sh`
```bash
#!/bin/bash
set -e

NAMESPACE="workspace"
CONTEXT="kubernetes-cluster"  # Change to your context

echo "📦 Deploying SurrealDB to Kubernetes..."

# Switch context
kubectl config use-context $CONTEXT
echo "✓ Using context: $CONTEXT"

# Create namespace if needed
kubectl create namespace $NAMESPACE --dry-run=client -o yaml | kubectl apply -f -
echo "✓ Namespace: $NAMESPACE"

# Apply resources
echo "📝 Applying ConfigMap..."
kubectl apply -f k8s/surrealdb-config.yaml

echo "🔐 Applying Secrets..."
kubectl apply -f k8s/surrealdb-secret.yaml

echo "📊 Applying StatefulSet..."
kubectl apply -f k8s/surrealdb-statefulset.yaml

echo "🌐 Applying Services..."
kubectl apply -f k8s/surrealdb-service.yaml

# Wait for rollout
echo "⏳ Waiting for SurrealDB to be ready..."
kubectl rollout status statefulset/surrealdb -n $NAMESPACE --timeout=5m

# Verify
echo "✓ Checking pod status..."
kubectl get pods -n $NAMESPACE -l app=surrealdb

echo "✓ Checking services..."
kubectl get svc -n $NAMESPACE -l app=surrealdb

echo ""
echo "🎉 SurrealDB deployed successfully!"
echo ""
echo "Access information:"
echo "  Internal: ws://surrealdb:8000"
echo "  External: kubectl port-forward -n $NAMESPACE svc/surrealdb-lb 8000:8000"
echo ""
echo "To connect:"
echo "  kubectl -n $NAMESPACE exec -it surrealdb-0 -- surreal sql"
```

### 3.6 Kubernetes Values for Helm (Optional)

**File**: `k8s/helm-values.yaml` (if using Helm)
```yaml
# SurrealDB Helm Chart Values
# Install: helm install surrealdb surrealdb/surrealdb -f helm-values.yaml

replicaCount: 1

image:
  repository: surrealdb/surrealdb
  pullPolicy: IfNotPresent
  tag: "2.3"

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  create: true
  annotations: {}
  name: "surrealdb"

podAnnotations: {}

podSecurityContext:
  runAsNonRoot: true
  runAsUser: 999
  fsGroup: 999

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

service:
  type: ClusterIP
  port: 8000

resources:
  limits:
    cpu: 2000m
    memory: 2Gi
  requests:
    cpu: 500m
    memory: 512Mi

persistence:
  enabled: true
  storageClassName: standard
  size: 10Gi
  path: /data

auth:
  username: admin
  password: "secure_password_here"

config:
  namespace: "syntaxis"
  database: "projects"
  logLevel: info
```

---

## Part 4: Production Considerations

### 4.1 High Availability Setup

```yaml
# For HA, scale replicas to 3 and configure clustering
# k8s/surrealdb-statefulset.yaml (updated for HA)
spec:
  replicas: 3  # Scale to 3 nodes

  # In container args, add clustering:
  args:
    - start
    - --bind
    - 0.0.0.0:8000
    - --auth
    - enable
    - --log
    - debug
    - --clock
    - "2ms"  # Clock precision for clustering
    # Enable clustering for multi-node setup
    # Note: Requires SurrealDB enterprise or cloud version
```

### 4.2 Backup and Recovery

```bash
# Backup SurrealDB data
kubectl -n workspace exec surrealdb-0 -- \
  tar czf /tmp/backup.tar.gz /data/

# Extract backup
kubectl -n workspace cp surrealdb-0:/tmp/backup.tar.gz ./backup.tar.gz

# Restore from backup
kubectl -n workspace cp backup.tar.gz surrealdb-0:/tmp/
kubectl -n workspace exec surrealdb-0 -- \
  tar xzf /tmp/backup.tar.gz -C /
```

### 4.3 Monitoring and Logging

```yaml
# Prometheus ServiceMonitor (if using Prometheus)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: surrealdb
  namespace: workspace
spec:
  selector:
    matchLabels:
      app: surrealdb
  endpoints:
    - port: http
      interval: 30s
      path: /metrics
```

### 4.4 Network Security

```yaml
# NetworkPolicy to restrict access
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: surrealdb-netpol
  namespace: workspace
spec:
  podSelector:
    matchLabels:
      app: surrealdb
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: workspace-app
      ports:
        - protocol: TCP
          port: 8000
  egress:
    - to:
        - podSelector: {}
      ports:
        - protocol: TCP
          port: 53  # DNS
```

---

## Part 5: Troubleshooting

### Issue: Connection Refused
**Error**: `connection refused` when connecting to localhost:8000

**Solution**:
```bash
# Check if SurrealDB is running
ps aux | grep surreal

# If not running, start it
surreal start --bind 127.0.0.1:8000 memory

# Test connection
curl http://localhost:8000/health
```

### Issue: Permission Denied in Docker
**Error**: `permission denied` when accessing `/data`

**Solution**:
```bash
# Fix file permissions
docker-compose exec surrealdb chmod 755 /data
docker-compose exec surrealdb chown -R 999:999 /data
```

### Issue: Kubernetes Pod CrashLoopBackOff
**Error**: Pod keeps restarting

**Solution**:
```bash
# Check logs
kubectl logs -n workspace surrealdb-0 --previous

# Check events
kubectl describe pod -n workspace surrealdb-0

# Common causes:
# 1. Credentials incorrect in secret
# 2. PVC not available
# 3. Insufficient resources
```

### Issue: Slow Queries
**Solution**:
```bash
# Enable query logging
surreal start --log debug --bind 127.0.0.1:8000 file:///tmp/surrealdb.db

# Check query performance in logs
# Optimize indexes if needed
```

---

## Summary

### Quick Start

**Local Development**:
```bash
# Terminal 1: Start SurrealDB
surreal start --bind 127.0.0.1:8000 memory

# Terminal 2: Run your application
cargo run -p syntaxis-cli
```

**Docker Development**:
```bash
docker-compose -f docker-compose.surrealdb.yml up -d
# Update configs/database.toml to use:
# url = "ws://localhost:8000"
```

**Kubernetes Production**:
```bash
cd k8s
./deploy.sh
# Access: kubectl port-forward -n workspace svc/surrealdb 8000:8000
```

---

## References

- [SurrealDB Official Docs](https://surrealdb.com/docs)
- [SurrealDB Docker Hub](https://hub.docker.com/r/surrealdb/surrealdb)
- [Kubernetes Documentation](https://kubernetes.io/docs/)
- [syntaxis PERSISTENCE_QUICK_START.md](PERSISTENCE_QUICK_START.md)
- [SURREALDB_2_3_MIGRATION.md](SURREALDB_2_3_MIGRATION.md)

---

**Last Updated**: November 15, 2025
**Version**: SurrealDB 2.3
**Status**: Production Ready