# Platform Operations Cheatsheet

**Quick reference for daily operations, deployments, and troubleshooting**

---

## Mode Selection (One Command)

```bash
# Development/Testing
export VAULT_MODE=solo REGISTRY_MODE=solo RAG_MODE=solo AI_SERVICE_MODE=solo DAEMON_MODE=solo

# Team Environment
export VAULT_MODE=multiuser REGISTRY_MODE=multiuser RAG_MODE=multiuser AI_SERVICE_MODE=multiuser DAEMON_MODE=multiuser

# CI/CD Pipelines
export VAULT_MODE=cicd REGISTRY_MODE=cicd RAG_MODE=cicd AI_SERVICE_MODE=cicd DAEMON_MODE=cicd

# Production HA
export VAULT_MODE=enterprise REGISTRY_MODE=enterprise RAG_MODE=enterprise AI_SERVICE_MODE=enterprise DAEMON_MODE=enterprise
```

---

## Service Ports & Endpoints

| Service | Port | Endpoint | Health Check |
|---------|------|----------|--------------|
| **Vault** | 8200 | `http://localhost:8200` | `curl http://localhost:8200/health` |
| **Registry** | 8081 | `http://localhost:8081` | `curl http://localhost:8081/health` |
| **RAG** | 8083 | `http://localhost:8083` | `curl http://localhost:8083/health` |
| **AI Service** | 8082 | `http://localhost:8082` | `curl http://localhost:8082/health` |
| **Orchestrator** | 9090 | `http://localhost:9090` | `curl http://localhost:9090/health` |
| **Control Center** | 8080 | `http://localhost:8080` | `curl http://localhost:8080/health` |
| **MCP Server** | 8084 | `http://localhost:8084` | `curl http://localhost:8084/health` |
| **Installer** | 8085 | `http://localhost:8085` | `curl http://localhost:8085/health` |

---

## Service Startup (Order Matters)

```bash
# Build everything first
cargo build --release

# Then start in dependency order:
# 1. Infrastructure
cargo run --release -p vault-service &
sleep 2

# 2. Configuration & Extensions
cargo run --release -p extension-registry &
sleep 2

# 3. AI/RAG Layer
cargo run --release -p provisioning-rag &
cargo run --release -p ai-service &
sleep 2

# 4. Orchestration
cargo run --release -p orchestrator &
cargo run --release -p control-center &
cargo run --release -p mcp-server &
sleep 2

# 5. Background Operations
cargo run --release -p provisioning-daemon &

# 6. Optional: Installer
cargo run --release -p installer &
```

---

## Quick Checks (All Services)

```bash
# Check all services running
pgrep -a cargo | grep "release -p"

# All health endpoints (fast)
for port in 8200 8081 8083 8082 9090 8080 8084 8085; do
  echo "Port $port: $(curl -s http://localhost:$port/health | jq -r .status 2>/dev/null || echo 'DOWN')"
done

# Check all listening ports
ss -tlnp | grep -E "8200|8081|8083|8082|9090|8080|8084|8085"

# Show PIDs of all services
ps aux | grep "cargo run --release" | grep -v grep
```

---

## Configuration Management

### View Config Files

```bash
# List all available schemas
ls -la provisioning/schemas/platform/schemas/

# View specific service schema
cat provisioning/schemas/platform/schemas/vault-service.ncl

# Check schema syntax
nickel typecheck provisioning/schemas/platform/schemas/vault-service.ncl
```

### Apply Config Changes

```bash
# 1. Update schema or defaults
vim provisioning/schemas/platform/schemas/vault-service.ncl
# Or update defaults:
vim provisioning/schemas/platform/defaults/vault-service-defaults.ncl

# 2. Validate
nickel typecheck provisioning/schemas/platform/schemas/vault-service.ncl

# 3. Re-generate runtime configs (local, private)
./provisioning/.typedialog/platform/scripts/generate-configs.nu vault-service multiuser

# 4. Restart service (graceful)
pkill -SIGTERM vault-service
sleep 2
export VAULT_MODE=multiuser
cargo run --release -p vault-service &

# 5. Verify loaded
curl http://localhost:8200/api/config | jq .
```

---

## Service Control

### Stop Services

```bash
# Stop all gracefully
pkill -SIGTERM -f "cargo run --release"

# Wait for shutdown
sleep 5

# Verify all stopped
pgrep -f "cargo run --release" || echo "All stopped"

# Force kill if needed
pkill -9 -f "cargo run --release"
```

### Restart Services

```bash
# Single service
pkill -SIGTERM vault-service && sleep 2 && cargo run --release -p vault-service &

# All services
pkill -SIGTERM -f "cargo run --release"
sleep 5
cargo build --release
# Then restart using startup commands above
```

### Check Logs

```bash
# Follow service logs (if using journalctl)
journalctl -fu provisioning-vault
journalctl -fu provisioning-orchestrator

# Or tail application logs
tail -f /var/log/provisioning/*.log

# Filter errors
grep -i error /var/log/provisioning/*.log
```

---

## Database Management

### SurrealDB (Multiuser/Enterprise)

```bash
# Check SurrealDB status
curl -s http://surrealdb:8000/health | jq .

# Connect to SurrealDB
surreal sql --endpoint http://surrealdb:8000 --username root --password root

# Run query
surreal sql --endpoint http://surrealdb:8000 --username root --password root \
  --query "SELECT * FROM services"

# Backup database
surreal export --endpoint http://surrealdb:8000 \
  --username root --password root > backup.sql

# Restore database
surreal import --endpoint http://surrealdb:8000 \
  --username root --password root < backup.sql
```

### Etcd (Enterprise HA)

```bash
# Check Etcd cluster health
etcdctl --endpoints=http://etcd:2379 endpoint health

# List members
etcdctl --endpoints=http://etcd:2379 member list

# Get key from Etcd
etcdctl --endpoints=http://etcd:2379 get /provisioning/config

# Set key in Etcd
etcdctl --endpoints=http://etcd:2379 put /provisioning/config "value"

# Backup Etcd
etcdctl --endpoints=http://etcd:2379 snapshot save backup.db

# Restore Etcd from snapshot
etcdctl --endpoints=http://etcd:2379 snapshot restore backup.db
```

---

## Environment Variable Overrides

### Override Individual Settings

```bash
# Vault overrides
export VAULT_SERVER_URL=http://vault-custom:8200
export VAULT_STORAGE_BACKEND=etcd
export VAULT_TLS_VERIFY=true

# Registry overrides
export REGISTRY_SERVER_PORT=9081
export REGISTRY_SERVER_WORKERS=8
export REGISTRY_GITEA_URL=http://gitea:3000
export REGISTRY_OCI_REGISTRY=registry.local:5000

# RAG overrides
export RAG_ENABLED=true
export RAG_EMBEDDINGS_PROVIDER=openai
export RAG_EMBEDDINGS_API_KEY=sk-xxx
export RAG_LLM_PROVIDER=anthropic

# AI Service overrides
export AI_SERVICE_SERVER_PORT=9082
export AI_SERVICE_RAG_ENABLED=true
export AI_SERVICE_MCP_ENABLED=false
export AI_SERVICE_DAG_MAX_CONCURRENT_TASKS=50

# Daemon overrides
export DAEMON_POLL_INTERVAL=30
export DAEMON_MAX_WORKERS=8
export DAEMON_LOGGING_LEVEL=info
```

---

## Health & Status Checks

### Quick Status (30 seconds)

```bash
# Test all services with visual status
curl -s http://localhost:8200/health && echo "✓ Vault" || echo "✗ Vault"
curl -s http://localhost:8081/health && echo "✓ Registry" || echo "✗ Registry"
curl -s http://localhost:8083/health && echo "✓ RAG" || echo "✗ RAG"
curl -s http://localhost:8082/health && echo "✓ AI Service" || echo "✗ AI Service"
curl -s http://localhost:9090/health && echo "✓ Orchestrator" || echo "✗ Orchestrator"
curl -s http://localhost:8080/health && echo "✓ Control Center" || echo "✗ Control Center"
```

### Detailed Status

```bash
# Orchestrator cluster status
curl -s http://localhost:9090/api/v1/cluster/status | jq .

# Service integration check
curl -s http://localhost:9090/api/v1/services | jq .

# Queue status
curl -s http://localhost:9090/api/v1/queue/status | jq .

# Worker status
curl -s http://localhost:9090/api/v1/workers | jq .

# Recent tasks (last 10)
curl -s http://localhost:9090/api/v1/tasks?limit=10 | jq .
```

---

## Performance & Monitoring

### System Resources

```bash
# Memory usage
free -h

# Disk usage
df -h /var/lib/provisioning

# CPU load
top -bn1 | head -5

# Network connections count
ss -s

# Count established connections
netstat -an | grep ESTABLISHED | wc -l

# Watch resources in real-time
watch -n 1 'free -h && echo "---" && df -h'
```

### Service Performance

```bash
# Monitor service memory usage
ps aux | grep "cargo run" | awk '{print $2, $6}' | while read pid mem; do
  echo "$pid: $(bc <<< "$mem / 1024")MB"
done

# Monitor request latency (Orchestrator)
curl -s http://localhost:9090/api/v1/metrics/latency | jq .

# Monitor error rate
curl -s http://localhost:9090/api/v1/metrics/errors | jq .
```

---

## Troubleshooting Quick Fixes

### Service Won't Start

```bash
# Check port in use
lsof -i :8200
ss -tlnp | grep 8200

# Kill process using port
pkill -9 -f "vault-service"

# Start with verbose logging
RUST_LOG=debug cargo run -p vault-service 2>&1 | head -50

# Verify schema exists
nickel typecheck provisioning/schemas/platform/schemas/vault-service.ncl

# Check mode defaults
ls -la provisioning/schemas/platform/defaults/deployment/$VAULT_MODE-defaults.ncl
```

### High Memory Usage

```bash
# Identify top memory consumers
ps aux --sort=-%mem | head -10

# Reduce worker count for affected service
export VAULT_SERVER_WORKERS=2
pkill -SIGTERM vault-service
sleep 2
cargo run --release -p vault-service &

# Run memory analysis (if valgrind available)
valgrind --leak-check=full target/release/vault-service
```

### Database Connection Error

```bash
# Test database connectivity
curl http://surrealdb:8000/health
etcdctl --endpoints=http://etcd:2379 endpoint health

# Update connection string
export SURREALDB_URL=ws://surrealdb:8000
export ETCD_ENDPOINTS=http://etcd:2379

# Restart service with new config
pkill vault-service
sleep 2
cargo run --release -p vault-service &

# Check logs for connection errors
grep -i "connection" /var/log/provisioning/*.log
```

### Services Not Communicating

```bash
# Test inter-service connectivity
curl http://localhost:8200/health
curl http://localhost:8081/health
curl -H "X-Service: vault" http://localhost:9090/api/v1/health

# Check DNS resolution (if using hostnames)
nslookup vault.internal
dig vault.internal

# Add to /etc/hosts if DNS fails
echo "127.0.0.1 vault.internal" >> /etc/hosts
```

---

## Emergency Procedures

### Full Service Recovery

```bash
# 1. Stop everything
pkill -9 -f "cargo run"

# 2. Backup current data
tar -czf /backup/provisioning-$(date +%s).tar.gz /var/lib/provisioning/

# 3. Clean slate (solo mode only)
rm -rf /tmp/provisioning-solo

# 4. Restart services
export VAULT_MODE=solo
cargo build --release
cargo run --release -p vault-service &
sleep 2
cargo run --release -p extension-registry &

# 5. Verify recovery
curl http://localhost:8200/health
curl http://localhost:8081/health
```

### Rollback to Previous Configuration

```bash
# 1. Stop affected service
pkill -SIGTERM vault-service

# 2. Restore previous schema from version control
git checkout HEAD~1 -- provisioning/schemas/platform/schemas/vault-service.ncl
git checkout HEAD~1 -- provisioning/schemas/platform/defaults/vault-service-defaults.ncl

# 3. Re-generate runtime config
./provisioning/.typedialog/platform/scripts/generate-configs.nu vault-service solo

# 4. Restart with restored config
export VAULT_MODE=solo
sleep 2
cargo run --release -p vault-service &

# 5. Verify restored state
curl http://localhost:8200/health
curl http://localhost:8200/api/config | jq .
```

### Data Recovery

```bash
# Restore SurrealDB from backup
surreal import --endpoint http://surrealdb:8000 \
  --username root --password root < /backup/surreal-20260105.sql

# Restore Etcd from snapshot
etcdctl --endpoints=http://etcd:2379 snapshot restore /backup/etcd-20260105.db

# Restore filesystem data (solo mode)
cp -r /backup/vault-data/* /tmp/provisioning-solo/vault/
chmod -R 755 /tmp/provisioning-solo/vault/
```

---

## File Locations

```bash
# Configuration files (PUBLIC - version controlled)
provisioning/schemas/platform/                   # Nickel schemas & defaults
provisioning/.typedialog/platform/               # Forms & generation scripts

# Configuration files (PRIVATE - gitignored)
provisioning/config/runtime/                     # Actual deployment configs

# Build artifacts
target/release/vault-service
target/release/extension-registry
target/release/provisioning-rag
target/release/ai-service
target/release/orchestrator
target/release/control-center
target/release/provisioning-daemon

# Logs (if configured)
/var/log/provisioning/
/tmp/provisioning-solo/logs/

# Data directories
/var/lib/provisioning/      # Production data
/tmp/provisioning-solo/     # Solo mode data
/mnt/provisioning-data/     # Shared storage (multiuser)

# Backups
/mnt/provisioning-backups/  # Automated backups
/backup/                    # Manual backups
```

---

## Mode Quick Reference Matrix

| Aspect | Solo | Multiuser | CICD | Enterprise |
|--------|------|-----------|------|------------|
| **Workers** | 2-4 | 4-6 | 8-12 | 16-32 |
| **Storage** | Filesystem | SurrealDB | Memory | Etcd+Replicas |
| **Startup** | 2-5 min | 3-8 min | 1-2 min | 5-15 min |
| **Data** | Ephemeral | Persistent | None | Replicated |
| **TLS** | No | Optional | No | Yes |
| **HA** | No | No | No | Yes |
| **Machines** | 1 | 2-4 | 1 | 3+ |
| **Logging** | Debug | Info | Warn | Info+Audit |

---

## Common Command Patterns

### Deploy Mode Change

```bash
# Migrate solo to multiuser
pkill -SIGTERM -f "cargo run"
sleep 5
tar -czf backup-solo.tar.gz /var/lib/provisioning/
export VAULT_MODE=multiuser REGISTRY_MODE=multiuser
cargo run --release -p vault-service &
sleep 2
cargo run --release -p extension-registry &
```

### Restart Single Service Without Downtime

```bash
# For load-balanced deployments:
# 1. Remove from load balancer
# 2. Graceful shutdown
pkill -SIGTERM vault-service
# 3. Wait for connections to drain
sleep 10
# 4. Restart service
cargo run --release -p vault-service &
# 5. Health check
curl http://localhost:8200/health
# 6. Return to load balancer
```

### Scale Workers for Load

```bash
# Increase workers when under load
export VAULT_SERVER_WORKERS=16
pkill -SIGTERM vault-service
sleep 2
cargo run --release -p vault-service &

# Alternative: Edit schema/defaults
vim provisioning/schemas/platform/schemas/vault-service.ncl
# Or: vim provisioning/schemas/platform/defaults/vault-service-defaults.ncl
# Change: server.workers = 16, then re-generate and restart
./provisioning/.typedialog/platform/scripts/generate-configs.nu vault-service enterprise
pkill -SIGTERM vault-service
sleep 2
cargo run --release -p vault-service &
```

---

## Diagnostic Bundle

```bash
# Generate complete diagnostics for support
echo "=== Processes ===" && pgrep -a cargo
echo "=== Listening Ports ===" && ss -tlnp
echo "=== System Resources ===" && free -h && df -h
echo "=== Schema Info ===" && nickel typecheck provisioning/schemas/platform/schemas/vault-service.ncl
echo "=== Active Env Vars ===" && env | grep -E "VAULT_|REGISTRY_|RAG_|AI_SERVICE_"
echo "=== Service Health ===" && for port in 8200 8081 8083 8082 9090 8080; do
  curl -s http://localhost:$port/health || echo "Port $port DOWN"
done

# Package diagnostics for support ticket
tar -czf diagnostics-$(date +%Y%m%d-%H%M%S).tar.gz \
  /var/log/provisioning/ \
  provisioning/schemas/platform/ \
  provisioning/.typedialog/platform/ \
  <(ps aux) \
  <(env | grep -E "VAULT_|REGISTRY_|RAG_")
```

---

## Essential References

- **Full Deployment Guide**: `provisioning/docs/src/operations/deployment-guide.md`
- **Service Management**: `provisioning/docs/src/operations/service-management-guide.md`
- **Config Guide**: `provisioning/docs/src/development/typedialog-platform-config-guide.md`
- **Troubleshooting**: `provisioning/docs/src/operations/troubleshooting-guide.md`
- **Platform Status**: Check `.coder/2026-01-05-phase13-19-completion.md` for latest platform info

---

**Last Updated**: 2026-01-05
**Version**: 1.0.0
**Status**: Production Ready ✅
