# Platform Operations Cheatsheet\n\n**Quick reference for daily operations, deployments, and troubleshooting**\n\n---\n\n## Mode Selection (One Command)\n\n```\n# Development/Testing\nexport VAULT_MODE=solo REGISTRY_MODE=solo RAG_MODE=solo AI_SERVICE_MODE=solo DAEMON_MODE=solo\n\n# Team Environment\nexport VAULT_MODE=multiuser REGISTRY_MODE=multiuser RAG_MODE=multiuser AI_SERVICE_MODE=multiuser DAEMON_MODE=multiuser\n\n# CI/CD Pipelines\nexport VAULT_MODE=cicd REGISTRY_MODE=cicd RAG_MODE=cicd AI_SERVICE_MODE=cicd DAEMON_MODE=cicd\n\n# Production HA\nexport VAULT_MODE=enterprise REGISTRY_MODE=enterprise RAG_MODE=enterprise AI_SERVICE_MODE=enterprise DAEMON_MODE=enterprise\n```\n\n---\n\n## Service Ports & Endpoints\n\n| Service | Port | Endpoint | Health Check |\n| --------- | ------ | ---------- | -------------- |\n| **Vault** | 8200 | `http://localhost:8200` | `curl http://localhost:8200/health` |\n| **Registry** | 8081 | `http://localhost:8081` | `curl http://localhost:8081/health` |\n| **RAG** | 8083 | `http://localhost:8083` | `curl http://localhost:8083/health` |\n| **AI Service** | 8082 | `http://localhost:8082` | `curl http://localhost:8082/health` |\n| **Orchestrator** | 9090 | `http://localhost:9090` | `curl http://localhost:9090/health` |\n| **Control Center** | 8080 | `http://localhost:8080` | `curl http://localhost:8080/health` |\n| **MCP Server** | 8084 | `http://localhost:8084` | `curl http://localhost:8084/health` |\n| **Installer** | 8085 | `http://localhost:8085` | `curl http://localhost:8085/health` |\n\n---\n\n## Service Startup (Order Matters)\n\n```\n# Build everything first\ncargo build --release\n\n# Then start in dependency order:\n# 1. Infrastructure\ncargo run --release -p vault-service &\nsleep 2\n\n# 2. Configuration & Extensions\ncargo run --release -p extension-registry &\nsleep 2\n\n# 3. AI/RAG Layer\ncargo run --release -p provisioning-rag &\ncargo run --release -p ai-service &\nsleep 2\n\n# 4. Orchestration\ncargo run --release -p orchestrator &\ncargo run --release -p control-center &\ncargo run --release -p mcp-server &\nsleep 2\n\n# 5. Background Operations\ncargo run --release -p provisioning-daemon &\n\n# 6. Optional: Installer\ncargo run --release -p installer &\n```\n\n---\n\n## Quick Checks (All Services)\n\n```\n# Check all services running\npgrep -a cargo | grep "release -p"\n\n# All health endpoints (fast)\nfor port in 8200 8081 8083 8082 9090 8080 8084 8085; do\n  echo "Port $port: $(curl -s http://localhost:$port/health | jq -r .status 2>/dev/null || echo 'DOWN')"\ndone\n\n# Check all listening ports\nss -tlnp | grep -E "8200|8081|8083|8082|9090|8080|8084|8085"\n\n# Show PIDs of all services\nps aux | grep "cargo run --release" | grep -v grep\n```\n\n---\n\n## Configuration Management\n\n### View Config Files\n\n```\n# List all available schemas\nls -la provisioning/schemas/platform/schemas/\n\n# View specific service schema\ncat provisioning/schemas/platform/schemas/vault-service.ncl\n\n# Check schema syntax\nnickel typecheck provisioning/schemas/platform/schemas/vault-service.ncl\n```\n\n### Apply Config Changes\n\n```\n# 1. Update schema or defaults\nvim provisioning/schemas/platform/schemas/vault-service.ncl\n# Or update defaults:\nvim provisioning/schemas/platform/defaults/vault-service-defaults.ncl\n\n# 2. Validate\nnickel typecheck provisioning/schemas/platform/schemas/vault-service.ncl\n\n# 3. Re-generate runtime configs (local, private)\n./provisioning/.typedialog/platform/scripts/generate-configs.nu vault-service multiuser\n\n# 4. Restart service (graceful)\npkill -SIGTERM vault-service\nsleep 2\nexport VAULT_MODE=multiuser\ncargo run --release -p vault-service &\n\n# 5. Verify loaded\ncurl http://localhost:8200/api/config | jq .\n```\n\n---\n\n## Service Control\n\n### Stop Services\n\n```\n# Stop all gracefully\npkill -SIGTERM -f "cargo run --release"\n\n# Wait for shutdown\nsleep 5\n\n# Verify all stopped\npgrep -f "cargo run --release" || echo "All stopped"\n\n# Force kill if needed\npkill -9 -f "cargo run --release"\n```\n\n### Restart Services\n\n```\n# Single service\npkill -SIGTERM vault-service && sleep 2 && cargo run --release -p vault-service &\n\n# All services\npkill -SIGTERM -f "cargo run --release"\nsleep 5\ncargo build --release\n# Then restart using startup commands above\n```\n\n### Check Logs\n\n```\n# Follow service logs (if using journalctl)\njournalctl -fu provisioning-vault\njournalctl -fu provisioning-orchestrator\n\n# Or tail application logs\ntail -f /var/log/provisioning/*.log\n\n# Filter errors\ngrep -i error /var/log/provisioning/*.log\n```\n\n---\n\n## Database Management\n\n### SurrealDB (Multiuser/Enterprise)\n\n```\n# Check SurrealDB status\ncurl -s http://surrealdb:8000/health | jq .\n\n# Connect to SurrealDB\nsurreal sql --endpoint http://surrealdb:8000 --username root --password root\n\n# Run query\nsurreal sql --endpoint http://surrealdb:8000 --username root --password root \\n  --query "SELECT * FROM services"\n\n# Backup database\nsurreal export --endpoint http://surrealdb:8000 \\n  --username root --password root > backup.sql\n\n# Restore database\nsurreal import --endpoint http://surrealdb:8000 \\n  --username root --password root < backup.sql\n```\n\n### Etcd (Enterprise HA)\n\n```\n# Check Etcd cluster health\netcdctl --endpoints=http://etcd:2379 endpoint health\n\n# List members\netcdctl --endpoints=http://etcd:2379 member list\n\n# Get key from Etcd\netcdctl --endpoints=http://etcd:2379 get /provisioning/config\n\n# Set key in Etcd\netcdctl --endpoints=http://etcd:2379 put /provisioning/config "value"\n\n# Backup Etcd\netcdctl --endpoints=http://etcd:2379 snapshot save backup.db\n\n# Restore Etcd from snapshot\netcdctl --endpoints=http://etcd:2379 snapshot restore backup.db\n```\n\n---\n\n## Environment Variable Overrides\n\n### Override Individual Settings\n\n```\n# Vault overrides\nexport VAULT_SERVER_URL=http://vault-custom:8200\nexport VAULT_STORAGE_BACKEND=etcd\nexport VAULT_TLS_VERIFY=true\n\n# Registry overrides\nexport REGISTRY_SERVER_PORT=9081\nexport REGISTRY_SERVER_WORKERS=8\nexport REGISTRY_GITEA_URL=http://gitea:3000\nexport REGISTRY_OCI_REGISTRY=registry.local:5000\n\n# RAG overrides\nexport RAG_ENABLED=true\nexport RAG_EMBEDDINGS_PROVIDER=openai\nexport RAG_EMBEDDINGS_API_KEY=sk-xxx\nexport RAG_LLM_PROVIDER=anthropic\n\n# AI Service overrides\nexport AI_SERVICE_SERVER_PORT=9082\nexport AI_SERVICE_RAG_ENABLED=true\nexport AI_SERVICE_MCP_ENABLED=false\nexport AI_SERVICE_DAG_MAX_CONCURRENT_TASKS=50\n\n# Daemon overrides\nexport DAEMON_POLL_INTERVAL=30\nexport DAEMON_MAX_WORKERS=8\nexport DAEMON_LOGGING_LEVEL=info\n```\n\n---\n\n## Health & Status Checks\n\n### Quick Status (30 seconds)\n\n```\n# Test all services with visual status\ncurl -s http://localhost:8200/health && echo "✓ Vault" || echo "✗ Vault"\ncurl -s http://localhost:8081/health && echo "✓ Registry" || echo "✗ Registry"\ncurl -s http://localhost:8083/health && echo "✓ RAG" || echo "✗ RAG"\ncurl -s http://localhost:8082/health && echo "✓ AI Service" || echo "✗ AI Service"\ncurl -s http://localhost:9090/health && echo "✓ Orchestrator" || echo "✗ Orchestrator"\ncurl -s http://localhost:8080/health && echo "✓ Control Center" || echo "✗ Control Center"\n```\n\n### Detailed Status\n\n```\n# Orchestrator cluster status\ncurl -s http://localhost:9090/api/v1/cluster/status | jq .\n\n# Service integration check\ncurl -s http://localhost:9090/api/v1/services | jq .\n\n# Queue status\ncurl -s http://localhost:9090/api/v1/queue/status | jq .\n\n# Worker status\ncurl -s http://localhost:9090/api/v1/workers | jq .\n\n# Recent tasks (last 10)\ncurl -s http://localhost:9090/api/v1/tasks?limit=10 | jq .\n```\n\n---\n\n## Performance & Monitoring\n\n### System Resources\n\n```\n# Memory usage\nfree -h\n\n# Disk usage\ndf -h /var/lib/provisioning\n\n# CPU load\ntop -bn1 | head -5\n\n# Network connections count\nss -s\n\n# Count established connections\nnetstat -an | grep ESTABLISHED | wc -l\n\n# Watch resources in real-time\nwatch -n 1 'free -h && echo "---" && df -h'\n```\n\n### Service Performance\n\n```\n# Monitor service memory usage\nps aux | grep "cargo run" | awk '{print $2, $6}' | while read pid mem; do\n  echo "$pid: $(bc <<< "$mem / 1024")MB"\ndone\n\n# Monitor request latency (Orchestrator)\ncurl -s http://localhost:9090/api/v1/metrics/latency | jq .\n\n# Monitor error rate\ncurl -s http://localhost:9090/api/v1/metrics/errors | jq .\n```\n\n---\n\n## Troubleshooting Quick Fixes\n\n### Service Won't Start\n\n```\n# Check port in use\nlsof -i :8200\nss -tlnp | grep 8200\n\n# Kill process using port\npkill -9 -f "vault-service"\n\n# Start with verbose logging\nRUST_LOG=debug cargo run -p vault-service 2>&1 | head -50\n\n# Verify schema exists\nnickel typecheck provisioning/schemas/platform/schemas/vault-service.ncl\n\n# Check mode defaults\nls -la provisioning/schemas/platform/defaults/deployment/$VAULT_MODE-defaults.ncl\n```\n\n### High Memory Usage\n\n```\n# Identify top memory consumers\nps aux --sort=-%mem | head -10\n\n# Reduce worker count for affected service\nexport VAULT_SERVER_WORKERS=2\npkill -SIGTERM vault-service\nsleep 2\ncargo run --release -p vault-service &\n\n# Run memory analysis (if valgrind available)\nvalgrind --leak-check=full target/release/vault-service\n```\n\n### Database Connection Error\n\n```\n# Test database connectivity\ncurl http://surrealdb:8000/health\netcdctl --endpoints=http://etcd:2379 endpoint health\n\n# Update connection string\nexport SURREALDB_URL=ws://surrealdb:8000\nexport ETCD_ENDPOINTS=http://etcd:2379\n\n# Restart service with new config\npkill vault-service\nsleep 2\ncargo run --release -p vault-service &\n\n# Check logs for connection errors\ngrep -i "connection" /var/log/provisioning/*.log\n```\n\n### Services Not Communicating\n\n```\n# Test inter-service connectivity\ncurl http://localhost:8200/health\ncurl http://localhost:8081/health\ncurl -H "X-Service: vault" http://localhost:9090/api/v1/health\n\n# Check DNS resolution (if using hostnames)\nnslookup vault.internal\ndig vault.internal\n\n# Add to /etc/hosts if DNS fails\necho "127.0.0.1 vault.internal" >> /etc/hosts\n```\n\n---\n\n## Emergency Procedures\n\n### Full Service Recovery\n\n```\n# 1. Stop everything\npkill -9 -f "cargo run"\n\n# 2. Backup current data\ntar -czf /backup/provisioning-$(date +%s).tar.gz /var/lib/provisioning/\n\n# 3. Clean slate (solo mode only)\nrm -rf /tmp/provisioning-solo\n\n# 4. Restart services\nexport VAULT_MODE=solo\ncargo build --release\ncargo run --release -p vault-service &\nsleep 2\ncargo run --release -p extension-registry &\n\n# 5. Verify recovery\ncurl http://localhost:8200/health\ncurl http://localhost:8081/health\n```\n\n### Rollback to Previous Configuration\n\n```\n# 1. Stop affected service\npkill -SIGTERM vault-service\n\n# 2. Restore previous schema from version control\ngit checkout HEAD~1 -- provisioning/schemas/platform/schemas/vault-service.ncl\ngit checkout HEAD~1 -- provisioning/schemas/platform/defaults/vault-service-defaults.ncl\n\n# 3. Re-generate runtime config\n./provisioning/.typedialog/platform/scripts/generate-configs.nu vault-service solo\n\n# 4. Restart with restored config\nexport VAULT_MODE=solo\nsleep 2\ncargo run --release -p vault-service &\n\n# 5. Verify restored state\ncurl http://localhost:8200/health\ncurl http://localhost:8200/api/config | jq .\n```\n\n### Data Recovery\n\n```\n# Restore SurrealDB from backup\nsurreal import --endpoint http://surrealdb:8000 \\n  --username root --password root < /backup/surreal-20260105.sql\n\n# Restore Etcd from snapshot\netcdctl --endpoints=http://etcd:2379 snapshot restore /backup/etcd-20260105.db\n\n# Restore filesystem data (solo mode)\ncp -r /backup/vault-data/* /tmp/provisioning-solo/vault/\nchmod -R 755 /tmp/provisioning-solo/vault/\n```\n\n---\n\n## File Locations\n\n```\n# Configuration files (PUBLIC - version controlled)\nprovisioning/schemas/platform/                   # Nickel schemas & defaults\nprovisioning/.typedialog/platform/               # Forms & generation scripts\n\n# Configuration files (PRIVATE - gitignored)\nprovisioning/config/runtime/                     # Actual deployment configs\n\n# Build artifacts\ntarget/release/vault-service\ntarget/release/extension-registry\ntarget/release/provisioning-rag\ntarget/release/ai-service\ntarget/release/orchestrator\ntarget/release/control-center\ntarget/release/provisioning-daemon\n\n# Logs (if configured)\n/var/log/provisioning/\n/tmp/provisioning-solo/logs/\n\n# Data directories\n/var/lib/provisioning/      # Production data\n/tmp/provisioning-solo/     # Solo mode data\n/mnt/provisioning-data/     # Shared storage (multiuser)\n\n# Backups\n/mnt/provisioning-backups/  # Automated backups\n/backup/                    # Manual backups\n```\n\n---\n\n## Mode Quick Reference Matrix\n\n| Aspect | Solo | Multiuser | CICD | Enterprise |\n| -------- | ------ | ----------- | ------ | ------------ |\n| **Workers** | 2-4 | 4-6 | 8-12 | 16-32 |\n| **Storage** | Filesystem | SurrealDB | Memory | Etcd+Replicas |\n| **Startup** | 2-5 min | 3-8 min | 1-2 min | 5-15 min |\n| **Data** | Ephemeral | Persistent | None | Replicated |\n| **TLS** | No | Optional | No | Yes |\n| **HA** | No | No | No | Yes |\n| **Machines** | 1 | 2-4 | 1 | 3+ |\n| **Logging** | Debug | Info | Warn | Info+Audit |\n\n---\n\n## Common Command Patterns\n\n### Deploy Mode Change\n\n```\n# Migrate solo to multiuser\npkill -SIGTERM -f "cargo run"\nsleep 5\ntar -czf backup-solo.tar.gz /var/lib/provisioning/\nexport VAULT_MODE=multiuser REGISTRY_MODE=multiuser\ncargo run --release -p vault-service &\nsleep 2\ncargo run --release -p extension-registry &\n```\n\n### Restart Single Service Without Downtime\n\n```\n# For load-balanced deployments:\n# 1. Remove from load balancer\n# 2. Graceful shutdown\npkill -SIGTERM vault-service\n# 3. Wait for connections to drain\nsleep 10\n# 4. Restart service\ncargo run --release -p vault-service &\n# 5. Health check\ncurl http://localhost:8200/health\n# 6. Return to load balancer\n```\n\n### Scale Workers for Load\n\n```\n# Increase workers when under load\nexport VAULT_SERVER_WORKERS=16\npkill -SIGTERM vault-service\nsleep 2\ncargo run --release -p vault-service &\n\n# Alternative: Edit schema/defaults\nvim provisioning/schemas/platform/schemas/vault-service.ncl\n# Or: vim provisioning/schemas/platform/defaults/vault-service-defaults.ncl\n# Change: server.workers = 16, then re-generate and restart\n./provisioning/.typedialog/platform/scripts/generate-configs.nu vault-service enterprise\npkill -SIGTERM vault-service\nsleep 2\ncargo run --release -p vault-service &\n```\n\n---\n\n## Diagnostic Bundle\n\n```\n# Generate complete diagnostics for support\necho "=== Processes ===" && pgrep -a cargo\necho "=== Listening Ports ===" && ss -tlnp\necho "=== System Resources ===" && free -h && df -h\necho "=== Schema Info ===" && nickel typecheck provisioning/schemas/platform/schemas/vault-service.ncl\necho "=== Active Env Vars ===" && env | grep -E "VAULT_|REGISTRY_|RAG_|AI_SERVICE_"\necho "=== Service Health ===" && for port in 8200 8081 8083 8082 9090 8080; do\n  curl -s http://localhost:$port/health || echo "Port $port DOWN"\ndone\n\n# Package diagnostics for support ticket\ntar -czf diagnostics-$(date +%Y%m%d-%H%M%S).tar.gz \\n  /var/log/provisioning/ \\n  provisioning/schemas/platform/ \\n  provisioning/.typedialog/platform/ \\n  <(ps aux) \\n  <(env | grep -E "VAULT_|REGISTRY_|RAG_")\n```\n\n---\n\n## Essential References\n\n- **Full Deployment Guide**: `provisioning/docs/src/operations/deployment-guide.md`\n- **Service Management**: `provisioning/docs/src/operations/service-management-guide.md`\n- **Config Guide**: `provisioning/docs/src/development/typedialog-platform-config-guide.md`\n- **Troubleshooting**: `provisioning/docs/src/operations/troubleshooting-guide.md`\n- **Platform Status**: Check `.coder/2026-01-05-phase13-19-completion.md` for latest platform info\n\n---\n\n**Last Updated**: 2026-01-05\n**Version**: 1.0.0\n**Status**: Production Ready ✅
