Vapora/docs/setup/deployment.md

VAPORA v1.0 Deployment Guide

Complete guide for deploying VAPORA v1.0 to Kubernetes (self-hosted).

Version: 0.1.0 Status: Production Ready Last Updated: 2025-11-10

---

Table of Contents

1. Overview
2. Prerequisites
3. Architecture
4. Deployment Methods
5. Building Docker Images
6. Kubernetes Deployment
7. Provisioning Deployment
8. Configuration
9. Monitoring & Health Checks
10. Scaling
11. Troubleshooting
12. Rollback
13. Security

---

Overview

VAPORA v1.0 is a cloud-native multi-agent software development platform that runs on Kubernetes. It consists of:

• 6 Rust services: Backend API, Frontend UI, Agents, MCP Server, LLM Router (embedded), Shared library
• 2 Infrastructure services: SurrealDB (database), NATS JetStream (messaging)
• Multi-IA routing: Claude, OpenAI, Gemini, Ollama support
• 12 specialized agents: Architect, Developer, Reviewer, Tester, Documenter, etc.

All services are containerized and deployed as Kubernetes workloads.

---

Prerequisites

Required Tools

• Kubernetes 1.25+ (K3s, RKE2, or managed Kubernetes)
• kubectl (configured and connected to cluster)
• Docker or Podman (for building images)
• Nushell (for deployment scripts)

Optional Tools

• Provisioning CLI (for advanced deployment)
• Helm (if using Helm charts)
• cert-manager (for automatic TLS certificates)
• Prometheus/Grafana (for monitoring)

Cluster Requirements

• Minimum: 4 CPU, 8GB RAM, 50GB storage
• Recommended: 8 CPU, 16GB RAM, 100GB storage
• Production: 16+ CPU, 32GB+ RAM, 200GB+ storage

Storage

• Storage Class: Required for SurrealDB PersistentVolumeClaim
• Options: local-path, nfs-client, rook-ceph, or cloud provider storage
• Minimum: 20Gi for database

Ingress

• nginx-ingress controller installed
• Domain name pointing to cluster ingress IP
• TLS certificate (optional, recommended for production)

---

Architecture

┌─────────────────────────────────────────────────────┐
│                  Internet / Users                   │
└───────────────────────┬─────────────────────────────┘
                        │
┌───────────────────────▼─────────────────────────────┐
│  Ingress (nginx)                                    │
│  - vapora.example.com                               │
│  - TLS termination                                  │
└────┬────────┬─────────┬─────────┬──────────────────┘
     │        │         │         │
     │        │         │         │
┌────▼────┐ ┌▼─────┐ ┌▼─────┐  ┌▼──────────┐
│Frontend │ │Backend│ │ MCP  │  │           │
│(Leptos) │ │(Axum) │ │Server│  │           │
│ 2 pods  │ │2 pods │ │1 pod │  │           │
└─────────┘ └───┬───┘ └──────┘  │           │
                │                 │           │
         ┌──────┴──────┬──────────┤           │
         │             │          │           │
    ┌────▼────┐   ┌───▼─────┐  ┌▼───────┐   │
    │SurrealDB│   │  NATS   │  │ Agents │   │
    │StatefulS│   │JetStream│  │ 3 pods │   │
    │  1 pod  │   │  1 pod  │  └────────┘   │
    └─────────┘   └─────────┘                │
         │                                   │
    ┌────▼────────────────────────────────┐  │
    │  Persistent Volume (20Gi)           │  │
    │  - SurrealDB data                   │  │
    └─────────────────────────────────────┘  │
                                              │
┌─────────────────────────────────────────────▼──┐
│  External LLM APIs                            │
│  - Anthropic Claude API                       │
│  - OpenAI API                                 │
│  - Google Gemini API                          │
│  - (Optional) Ollama local                    │
└───────────────────────────────────────────────┘

---

Deployment Methods

VAPORA supports two deployment methods:

Method 1: Vanilla Kubernetes (Recommended for Getting Started)

Pros:

• Simple, well-documented
• Standard K8s manifests
• Easy to understand and modify
• No additional tools required

Cons:

• Manual cluster management
• Manual service ordering
• No built-in rollback

Use when: Learning, testing, or simple deployments

Method 2: Provisioning (Recommended for Production)

Pros:

• Automated cluster creation
• Declarative workflows
• Built-in rollback
• Service mesh integration
• Secret management

Cons:

• Requires Provisioning CLI
• More complex configuration
• Steeper learning curve

Use when: Production deployments, complex environments

---

Building Docker Images

Option 1: Using Nushell Script (Recommended)

# Build all images (local registry)
nu scripts/build-docker.nu

# Build and push to Docker Hub
nu scripts/build-docker.nu --registry docker.io --push

# Build with specific tag
nu scripts/build-docker.nu --tag v0.1.0

# Build without cache
nu scripts/build-docker.nu --no-cache

Option 2: Manual Docker Build

# From project root

# Backend
docker build -f crates/vapora-backend/Dockerfile -t vapora/backend:latest .

# Frontend
docker build -f crates/vapora-frontend/Dockerfile -t vapora/frontend:latest .

# Agents
docker build -f crates/vapora-agents/Dockerfile -t vapora/agents:latest .

# MCP Server
docker build -f crates/vapora-mcp-server/Dockerfile -t vapora/mcp-server:latest .

Image Sizes (Approximate)

• vapora/backend: ~50MB (Alpine + Rust binary)
• vapora/frontend: ~30MB (nginx + WASM)
• vapora/agents: ~50MB (Alpine + Rust binary)
• vapora/mcp-server: ~45MB (Alpine + Rust binary)

---

Kubernetes Deployment

Step 1: Configure Secrets

Edit kubernetes/03-secrets.yaml:

stringData:
  # Generate strong JWT secret
  jwt-secret: "$(openssl rand -base64 32)"

  # Add your LLM API keys
  anthropic-api-key: "sk-ant-xxxxx"
  openai-api-key: "sk-xxxxx"
  gemini-api-key: "xxxxx"  # Optional

  # Database credentials
  surrealdb-user: "root"
  surrealdb-pass: "$(openssl rand -base64 32)"

IMPORTANT: Never commit real secrets to version control!

Step 2: Configure Ingress

Edit kubernetes/08-ingress.yaml:

spec:
  rules:
  - host: vapora.yourdomain.com  # Change this!

Step 3: Deploy Using Script (Recommended)

# Dry run to validate
nu scripts/deploy-k8s.nu --dry-run

# Deploy to default namespace (vapora)
nu scripts/deploy-k8s.nu

# Deploy to custom namespace
nu scripts/deploy-k8s.nu --namespace my-vapora

# Skip secrets (if already created)
nu scripts/deploy-k8s.nu --skip-secrets

Step 4: Manual Deploy (Alternative)

# Apply manifests in order
kubectl apply -f kubernetes/00-namespace.yaml
kubectl apply -f kubernetes/01-surrealdb.yaml
kubectl apply -f kubernetes/02-nats.yaml
kubectl apply -f kubernetes/03-secrets.yaml
kubectl apply -f kubernetes/04-backend.yaml
kubectl apply -f kubernetes/05-frontend.yaml
kubectl apply -f kubernetes/06-agents.yaml
kubectl apply -f kubernetes/07-mcp-server.yaml
kubectl apply -f kubernetes/08-ingress.yaml

# Wait for rollout
kubectl rollout status deployment/vapora-backend -n vapora
kubectl rollout status deployment/vapora-frontend -n vapora

Step 5: Verify Deployment

# Check all pods are running
kubectl get pods -n vapora

# Expected output:
# NAME                                READY   STATUS    RESTARTS
# surrealdb-0                         1/1     Running   0
# nats-xxx                            1/1     Running   0
# vapora-backend-xxx                  1/1     Running   0
# vapora-backend-yyy                  1/1     Running   0
# vapora-frontend-xxx                 1/1     Running   0
# vapora-frontend-yyy                 1/1     Running   0
# vapora-agents-xxx                   1/1     Running   0
# vapora-agents-yyy                   1/1     Running   0
# vapora-agents-zzz                   1/1     Running   0
# vapora-mcp-server-xxx               1/1     Running   0

# Check services
kubectl get svc -n vapora

# Check ingress
kubectl get ingress -n vapora

Step 6: Access VAPORA

# Get ingress IP/hostname
kubectl get ingress vapora -n vapora

# Configure DNS
# Point vapora.yourdomain.com to ingress IP

# Access UI
open https://vapora.yourdomain.com

---

Provisioning Deployment

Step 1: Validate Configuration

# Validate Provisioning workspace
nu scripts/validate-provisioning.nu

Step 2: Create Cluster

cd provisioning/vapora-wrksp

# Validate configuration
provisioning validate --all

# Create cluster
provisioning cluster create --config workspace.toml

Step 3: Deploy Services

# Deploy infrastructure (database, messaging)
provisioning workflow run workflows/deploy-infra.yaml

# Deploy services (backend, frontend, agents)
provisioning workflow run workflows/deploy-services.yaml

# Or deploy full stack at once
provisioning workflow run workflows/deploy-full-stack.yaml

Step 4: Health Check

provisioning workflow run workflows/health-check.yaml

See provisioning-integration/README.md for details.

---

Configuration

Environment Variables

Backend (vapora-backend)

RUST_LOG=info,vapora=debug
SURREALDB_URL=http://surrealdb:8000
SURREALDB_USER=root
SURREALDB_PASS=<secret>
NATS_URL=nats://nats:4222
JWT_SECRET=<secret>
BIND_ADDR=0.0.0.0:8080

Agents (vapora-agents)

RUST_LOG=info,vapora_agents=debug
NATS_URL=nats://nats:4222
BIND_ADDR=0.0.0.0:9000
ANTHROPIC_API_KEY=<secret>
OPENAI_API_KEY=<secret>
GEMINI_API_KEY=<secret>
VAPORA_AGENT_CONFIG=/etc/vapora/agents.toml  # Optional

MCP Server (vapora-mcp-server)

RUST_LOG=info,vapora_mcp_server=debug
# Port configured via --port flag

ConfigMaps

Create custom configuration:

kubectl create configmap agent-config -n vapora \
  --from-file=agents.toml

Mount in deployment:

volumeMounts:
- name: config
  mountPath: /etc/vapora
volumes:
- name: config
  configMap:
    name: agent-config

---

Monitoring & Health Checks

Health Endpoints

All services expose health check endpoints:

• Backend: GET /health
• Frontend: GET /health.html
• Agents: GET /health, GET /ready
• MCP Server: GET /health
• SurrealDB: GET /health
• NATS: GET /healthz (port 8222)

Manual Health Checks

# Backend health
kubectl exec -n vapora deploy/vapora-backend -- \
  curl -s http://localhost:8080/health

# Database health
kubectl exec -n vapora deploy/vapora-backend -- \
  curl -s http://surrealdb:8000/health

# NATS health
kubectl exec -n vapora deploy/vapora-backend -- \
  curl -s http://nats:8222/healthz

Kubernetes Probes

All deployments have:

• Liveness Probe: Restarts unhealthy pods
• Readiness Probe: Removes pod from service until ready

Logs

# View backend logs
kubectl logs -n vapora -l app=vapora-backend -f

# View agent logs
kubectl logs -n vapora -l app=vapora-agents -f

# View all logs
kubectl logs -n vapora -l app --all-containers=true -f

Metrics (Optional)

Deploy Prometheus + Grafana:

# Install Prometheus Operator
helm install prometheus prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

# Access Grafana
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

VAPORA services expose metrics on /metrics endpoint (future enhancement).

---

Scaling

Manual Scaling

# Scale backend
kubectl scale deployment vapora-backend -n vapora --replicas=4

# Scale frontend
kubectl scale deployment vapora-frontend -n vapora --replicas=3

# Scale agents (for higher workload)
kubectl scale deployment vapora-agents -n vapora --replicas=10

Horizontal Pod Autoscaler (HPA)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vapora-backend-hpa
  namespace: vapora
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vapora-backend
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Apply:

kubectl apply -f hpa.yaml

Resource Limits

Adjust in deployment YAML:

resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: 1000m
    memory: 1Gi

---

Troubleshooting

Pods Not Starting

# Check pod status
kubectl get pods -n vapora

# Describe pod for events
kubectl describe pod -n vapora <pod-name>

# Check logs
kubectl logs -n vapora <pod-name>

# Check previous logs (if crashed)
kubectl logs -n vapora <pod-name> --previous

Database Connection Issues

# Check SurrealDB is running
kubectl get pod -n vapora -l app=surrealdb

# Test connection from backend
kubectl exec -n vapora deploy/vapora-backend -- \
  curl -v http://surrealdb:8000/health

# Check SurrealDB logs
kubectl logs -n vapora surrealdb-0

NATS Connection Issues

# Check NATS is running
kubectl get pod -n vapora -l app=nats

# Test connection
kubectl exec -n vapora deploy/vapora-backend -- \
  curl http://nats:8222/varz

# Check NATS logs
kubectl logs -n vapora -l app=nats

Image Pull Errors

# Check image pull secrets
kubectl get secrets -n vapora

# Create Docker registry secret
kubectl create secret docker-registry regcred \
  -n vapora \
  --docker-server=<registry> \
  --docker-username=<username> \
  --docker-password=<password>

# Add to deployment
spec:
  imagePullSecrets:
  - name: regcred

Ingress Not Working

# Check ingress controller is installed
kubectl get pods -n ingress-nginx

# Check ingress resource
kubectl describe ingress vapora -n vapora

# Check ingress logs
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx

---

Rollback

Kubernetes Rollback

# View rollout history
kubectl rollout history deployment/vapora-backend -n vapora

# Rollback to previous version
kubectl rollout undo deployment/vapora-backend -n vapora

# Rollback to specific revision
kubectl rollout undo deployment/vapora-backend -n vapora --to-revision=2

Provisioning Rollback

cd provisioning/vapora-wrksp

# List versions
provisioning version list

# Rollback to previous version
provisioning rollback --to-version <version-id>

---

Security

Secrets Management

• Kubernetes Secrets: Encrypted at rest (if configured in K8s)
• External Secrets Operator: Sync from Vault, AWS Secrets Manager, etc.
• RustyVault: Integrated with Provisioning

Network Policies

Apply network policies to restrict pod-to-pod communication:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vapora-backend
  namespace: vapora
spec:
  podSelector:
    matchLabels:
      app: vapora-backend
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: vapora-frontend
    ports:
    - protocol: TCP
      port: 8080

TLS Certificates

Use cert-manager for automatic TLS:

# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml

# Create ClusterIssuer
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@yourdomain.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx
EOF

Update ingress:

metadata:
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - vapora.yourdomain.com
    secretName: vapora-tls

---

Backup & Restore

SurrealDB Backup

# Create backup
kubectl exec -n vapora surrealdb-0 -- \
  surreal export --conn http://localhost:8000 \
  --user root --pass <password> \
  --ns vapora --db main backup.surql

# Copy backup locally
kubectl cp vapora/surrealdb-0:/backup.surql ./backup-$(date +%Y%m%d).surql

SurrealDB Restore

# Copy backup to pod
kubectl cp ./backup.surql vapora/surrealdb-0:/restore.surql

# Restore
kubectl exec -n vapora surrealdb-0 -- \
  surreal import --conn http://localhost:8000 \
  --user root --pass <password> \
  --ns vapora --db main /restore.surql

PVC Backup

# Snapshot PVC (if supported by storage class)
kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: surrealdb-snapshot
  namespace: vapora
spec:
  source:
    persistentVolumeClaimName: data-surrealdb-0
EOF

---

Uninstall

Delete All Resources

# Delete namespace (deletes all resources)
kubectl delete namespace vapora

# Or delete manifests individually
kubectl delete -f kubernetes/

Delete PVCs

# List PVCs
kubectl get pvc -n vapora

# Delete PVC (data will be lost!)
kubectl delete pvc data-surrealdb-0 -n vapora

---

Next Steps

After successful deployment:

1. Configure DNS: Point domain to ingress IP
2. Set up TLS: Configure cert-manager for HTTPS
3. Enable monitoring: Deploy Prometheus/Grafana
4. Configure backups: Schedule SurrealDB backups
5. Set up CI/CD: Automate deployments
6. Configure HPA: Enable autoscaling
7. Test disaster recovery: Practice rollback procedures

---

Support

• Deployment Issues: Check kubernetes/README.md
• Provisioning Issues: Check provisioning-integration/README.md
• Scripts Help: Run nu scripts/<script-name>.nu --help
• Kubernetes Docs: https://kubernetes.io/docs/

---

VAPORA v1.0 - Cloud-Native Multi-Agent Platform Status: Production Ready ✅