syntaxis/docs/provision/deployment-guide.md

# 🚀 Deployment Guide: syntaxis with provctl-bridge

**Version**: 1.0
**Date**: 2025-11-19
**Status**: ✅ Ready for Deployment

---

## Table of Contents

1. [Quick Start](#quick-start)
2. [Prerequisites](#prerequisites)
3. [Local Development Deployment](#local-development-deployment)
4. [Kubernetes Deployment](#kubernetes-deployment)
5. [Docker Compose Deployment](#docker-compose-deployment)
6. [Health Check Verification](#health-check-verification)
7. [Service Communication](#service-communication)
8. [Troubleshooting](#troubleshooting)
9. [Scaling & Optimization](#scaling--optimization)

---

## Quick Start

### 1. Generate Deployment Files

```bash
# Generate Kubernetes manifests
cargo run --example generate-k8s > k8s.yaml

# Generate Docker Compose
cargo run --example generate-docker-compose > docker-compose.yml

# Validate Kubernetes syntax
kubectl apply --dry-run=client -f k8s.yaml
```

### 2. Deploy Services

**Kubernetes**:
```bash
kubectl apply -f k8s.yaml
kubectl get all -n syntaxis
```

**Docker Compose**:
```bash
docker-compose -f docker-compose.yml up -d
docker-compose ps
```

### 3. Verify Services

```bash
# Check service status
kubectl get services -n syntaxis

# Check pod status
kubectl get pods -n syntaxis

# View logs
kubectl logs -n syntaxis deployment/syntaxis-api
```

---

## Prerequisites

### System Requirements
- **RAM**: Minimum 2GB (1GB for services + 1GB overhead)
- **Disk**: Minimum 2GB (1GB estimated for all services)
- **CPU**: Minimum 2 cores recommended

### Software Requirements

**For Kubernetes Deployment**:
- Kubernetes cluster 1.24+
- kubectl CLI configured
- Access to cluster with deployment permissions

**For Docker Compose Deployment**:
- Docker 20.10+
- Docker Compose 2.0+
- Docker daemon running

**For Local Development**:
- Rust toolchain
- Cargo package manager
- Git

### Catalog Access
- Access to syntaxis service catalog at `/Users/Akasha/Development/syntaxis/configs/services-catalog.toml`
- Read permissions on catalog file

---

## Local Development Deployment

### Setup Development Environment

```bash
# 1. Clone/navigate to project
cd /Users/Akasha/Development/syntaxis

# 2. Verify catalog exists
ls -la configs/services-catalog.toml

# 3. Build all services locally
cargo build --release

# 4. Install binaries locally
cargo install --path core/crates/syntaxis-cli
cargo install --path core/crates/syntaxis-tui
cargo install --path core/crates/syntaxis-api
```

### Run Services Individually

**syntaxis-cli** (required for others):
```bash
syntaxis-cli --help
syntaxis-cli project list
syntaxis-cli init my-project
```

**syntaxis-tui**:
```bash
syntaxis-tui
# Navigate with h/j/k/l, enter to select, q to quit
```

**syntaxis-api** (REST server):
```bash
# Start on port 3000
syntaxis-api --bind 127.0.0.1:3000

# In another terminal, test health check
curl http://127.0.0.1:3000/health
```

**SurrealDB** (database):
```bash
# Start in-memory mode (for development)
surreal start --bind 127.0.0.1:8000 memory

# For persistent storage
surreal start --bind 127.0.0.1:8000 file:///tmp/syntaxis.db
```

**NATS** (messaging):
```bash
# Start NATS server
nats-server --port 4222

# Test connectivity
nats connect
```

### Verify Local Setup

```bash
# Check all services are running
ps aux | grep -E "(syntaxis|surreal|nats-server)"

# Test API connectivity
curl -s http://127.0.0.1:3000/health | jq .

# Test database connectivity
curl -s http://127.0.0.1:8000

# Test messaging
nats sub test.subject &
nats pub test.subject "Hello"
```

---

## Kubernetes Deployment

### Prerequisites

```bash
# Verify kubectl is configured
kubectl cluster-info

# Verify cluster is accessible
kubectl get nodes

# Check available resources
kubectl top nodes
kubectl describe node $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
```

### Generate & Deploy Manifests

**Using provctl-bridge**:

```bash
# Programmatically generate manifests
cat > generate_k8s.rs << 'EOF'
use provctl_bridge::ServiceCatalog;
use std::fs;

fn main() {
    let catalog = ServiceCatalog::from_file(
        "/Users/Akasha/Development/syntaxis/configs/services-catalog.toml"
    ).expect("Failed to load catalog");

    let k8s = catalog.generate_kubernetes("production")
        .expect("Failed to generate Kubernetes manifests");

    fs::write("k8s.yaml", k8s).expect("Failed to write manifests");
    println!("Generated k8s.yaml");
}
EOF

rustc generate_k8s.rs -L dependency=/path/to/provctl-bridge/target/release
./generate_k8s
```

**Manual deployment**:

```bash
# 1. Create namespace
kubectl create namespace syntaxis

# 2. Apply manifests (generated earlier)
kubectl apply -f k8s.yaml

# 3. Wait for deployments
kubectl wait --for=condition=available \
  --timeout=300s \
  deployment/syntaxis-api -n syntaxis

# 4. Verify all services are running
kubectl get all -n syntaxis
```

### Monitor Deployments

```bash
# Watch deployment progress
kubectl rollout status deployment/syntaxis-api -n syntaxis
kubectl rollout status deployment/surrealdb -n syntaxis
kubectl rollout status deployment/nats -n syntaxis

# Check pod status
kubectl get pods -n syntaxis -w

# View pod events
kubectl describe pod <pod-name> -n syntaxis

# Check resource usage
kubectl top pods -n syntaxis
kubectl top nodes
```

### Access Services

```bash
# Port forward to access API
kubectl port-forward svc/syntaxis-api 3000:3000 -n syntaxis &

# Test API health
curl http://localhost:3000/health

# Access service logs
kubectl logs deployment/syntaxis-api -n syntaxis
kubectl logs deployment/surrealdb -n syntaxis
kubectl logs deployment/nats -n syntaxis

# Stream logs in real-time
kubectl logs -f deployment/syntaxis-api -n syntaxis
```

### Resource Management

```bash
# Check actual resource usage
kubectl describe nodes
kubectl top pods -n syntaxis

# Scale services
kubectl scale deployment syntaxis-api --replicas=3 -n syntaxis

# View resource limits
kubectl describe deployment syntaxis-api -n syntaxis
kubectl describe limits -n syntaxis
```

---

## Docker Compose Deployment

### Setup Docker Environment

```bash
# Verify Docker is running
docker ps

# Check Docker version
docker version

# Check Docker Compose version
docker-compose version
```

### Generate & Deploy Compose File

**Generate from catalog**:

```bash
# Using provctl-bridge
cat > generate_compose.rs << 'EOF'
use provctl_bridge::ServiceCatalog;
use std::fs;

fn main() {
    let catalog = ServiceCatalog::from_file(
        "/Users/Akasha/Development/syntaxis/configs/services-catalog.toml"
    ).expect("Failed to load catalog");

    let compose = catalog.generate_docker_compose("production")
        .expect("Failed to generate Docker Compose");

    fs::write("docker-compose.yml", compose)
        .expect("Failed to write compose file");
    println!("Generated docker-compose.yml");
}
EOF
```

### Start Services

```bash
# 1. Create docker-compose.yml from catalog
# (See above generation method)

# 2. Start services
docker-compose up -d

# 3. Check service status
docker-compose ps

# 4. View logs
docker-compose logs -f

# 5. Check individual service logs
docker-compose logs syntaxis-api
docker-compose logs surrealdb
docker-compose logs nats
```

### Manage Services

```bash
# Stop all services
docker-compose down

# Stop specific service
docker-compose stop syntaxis-api

# Restart service
docker-compose restart syntaxis-api

# View resource usage
docker stats

# Execute command in running service
docker-compose exec syntaxis-api /bin/sh

# Scale services
docker-compose up -d --scale syntaxis-api=3
```

### Access Services

```bash
# Test API (usually port 3000)
curl http://localhost:3000/health

# Test database (usually port 8000)
curl http://localhost:8000

# Test messaging (port 4222)
nats connect localhost:4222
```

---

## Health Check Verification

### Configured Health Checks

Based on service catalog:

| Service | Endpoint | Method | Expected Status |
|---------|----------|--------|-----------------|
| syntaxis-api | `/health` | GET | 200 |
| surrealdb | `/` | GET | 200 |
| nats | `http://127.0.0.1:8222/healthz` | GET | 200 |

### Manual Health Verification

```bash
# API health
curl -v http://localhost:3000/health

# Database health
curl -v http://localhost:8000

# Messaging health
curl -v http://127.0.0.1:8222/healthz

# All at once
for service in "http://localhost:3000/health" \
               "http://localhost:8000" \
               "http://127.0.0.1:8222/healthz"; do
  echo "Testing: $service"
  curl -s "$service" && echo "✅ UP" || echo "❌ DOWN"
done
```

### Automated Health Monitoring

**Using Kubernetes**:

```yaml
# Automatically added to generated manifests
livenessProbe:
  httpGet:
    path: /health
    port: 3000
  initialDelaySeconds: 10
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /health
    port: 3000
  initialDelaySeconds: 5
  periodSeconds: 5
```

**Using Docker Compose**:

```yaml
# Add to docker-compose.yml
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 40s
```

---

## Service Communication

### Dependency Chain

```
syntaxis-cli (base)
    ↓
    ├─ syntaxis-tui (requires CLI)
    └─ syntaxis-api (requires CLI)
         ↓
         └─ syntaxis-dashboard (requires API)
```

### Service Networking

**API to Database Communication**:
```bash
# Verify API can reach database
curl http://localhost:8000

# Check connection from API logs
kubectl logs deployment/syntaxis-api -n syntaxis | grep -i database
```

**API to Messaging Communication**:
```bash
# Verify API can reach NATS
nats connect localhost:4222

# Check connection from API logs
kubectl logs deployment/syntaxis-api -n syntaxis | grep -i nats
```

**Inter-Service Communication**:
```bash
# Test from within API pod
kubectl exec deployment/syntaxis-api -n syntaxis -- \
  curl http://surrealdb:8000

# Test from within NATS pod
kubectl exec deployment/nats -n syntaxis -- \
  nats sub test.subject
```

### Service Discovery

**Kubernetes DNS**:
```bash
# Services are automatically discovered via DNS
# Within cluster, use service name:
# surrealdb.syntaxis.svc.cluster.local
# nats.syntaxis.svc.cluster.local
# syntaxis-api.syntaxis.svc.cluster.local

# Test DNS resolution
kubectl run -it --rm debug --image=busybox --restart=Never -- \
  nslookup syntaxis-api.syntaxis.svc.cluster.local
```

**Docker Compose**:
```bash
# Services can communicate using service names
# From within container:
curl http://surrealdb:8000

# Test container communication
docker-compose exec syntaxis-api \
  curl http://surrealdb:8000
```

---

## Troubleshooting

### Service Won't Start

```bash
# Check pod events
kubectl describe pod <pod-name> -n syntaxis

# Check logs for errors
kubectl logs <pod-name> -n syntaxis --previous

# Check resource limits
kubectl get resourcequota -n syntaxis
kubectl describe nodes | grep -A 5 "Allocated resources"

# Verify image availability
kubectl get events -n syntaxis
```

### Connection Issues

```bash
# Test pod-to-pod communication
kubectl exec deployment/syntaxis-api -n syntaxis -- \
  curl http://surrealdb:8000

# Check service endpoints
kubectl get endpoints -n syntaxis

# Verify network policy (if any)
kubectl get networkpolicies -n syntaxis
```

### Resource Issues

```bash
# Check memory usage
kubectl top pods -n syntaxis

# Check disk usage
kubectl exec deployment/surrealdb -n syntaxis -- \
  du -sh /data

# Increase pod resource limits
kubectl set resources deployment syntaxis-api \
  --limits=memory=512Mi,cpu=500m \
  -n syntaxis
```

### Port Conflicts

```bash
# Check running ports
netstat -tuln | grep -E "(3000|8000|4222)"

# On macOS
lsof -i :3000
lsof -i :8000
lsof -i :4222

# Kill conflicting process
kill -9 <pid>

# Or use different port
docker-compose -p myapp up -d
```

---

## Scaling & Optimization

### Scale Services

**Kubernetes**:
```bash
# Scale API service to 3 replicas
kubectl scale deployment syntaxis-api --replicas=3 -n syntaxis

# Scale all services
for deployment in syntaxis-api surrealdb nats; do
  kubectl scale deployment $deployment --replicas=2 -n syntaxis
done

# Auto-scaling (requires metrics-server)
kubectl autoscale deployment syntaxis-api \
  --min=1 --max=5 \
  --cpu-percent=80 \
  -n syntaxis
```

**Docker Compose**:
```bash
# Scale services
docker-compose up -d --scale syntaxis-api=3

# Verify scaling
docker-compose ps
```

### Resource Optimization

**Reduce Memory Usage**:
```bash
# Current requirements (from catalog)
# - syntaxis-api: 256Mi
# - surrealdb: 256Mi
# - nats: 256Mi
# - Total: 768Mi

# Can be reduced for development
kubectl set resources deployment syntaxis-api \
  --requests=memory=128Mi,cpu=50m \
  --limits=memory=256Mi,cpu=100m \
  -n syntaxis
```

**Enable Compression**:
```bash
# In API configuration
syntaxis-api --bind 127.0.0.1:3000 --gzip
```

**Database Optimization**:
```bash
# SurrealDB file mode (instead of memory)
surreal start --bind 127.0.0.1:8000 \
  file:///data/syntaxis.db

# Enable write-ahead logging
surreal start --bind 127.0.0.1:8000 \
  --log info \
  file:///data/syntaxis.db
```

---

## Production Checklist

- [ ] All services are running and healthy
- [ ] Health checks are passing
- [ ] Service-to-service communication works
- [ ] Logs are accessible and monitored
- [ ] Resource limits are appropriate
- [ ] Backups are configured (if needed)
- [ ] Security policies are in place
- [ ] RBAC is configured (Kubernetes)
- [ ] Network policies are defined (Kubernetes)
- [ ] Monitoring and alerting are set up

---

## Next Steps

1. **Deploy to Kubernetes** - Use generated manifests
2. **Configure Monitoring** - Set up Prometheus + Grafana
3. **Enable Logging** - Configure ELK or similar
4. **Setup Backups** - For persistent data (SurrealDB)
5. **Configure CI/CD** - For automated deployments
6. **Security Hardening** - RBAC, network policies, secrets

---

## Additional Resources

- **Kubernetes Docs**: https://kubernetes.io/docs/
- **Docker Compose Docs**: https://docs.docker.com/compose/
- **syntaxis Docs**: See project README
- **SurrealDB Docs**: https://surrealdb.com/docs
- **NATS Docs**: https://docs.nats.io/

---

**Status**: 🎉 Ready for Production Deployment

**Last Updated**: 2025-11-19
**Maintained By**: syntaxis Team