Vapora/provisioning/vapora-wrksp/README.md

# VAPORA Workspace for Provisioning
## Multi-Agent Multi-IA Platform Deployment

**Version**: 0.2.0
**Status**: Specification + Structure (Ready for Implementation)

This workspace contains all configuration for deploying VAPORA using Provisioning.

---

## 📁 Structure

```
vapora-wrksp/
├── workspace.toml           # Workspace configuration
├── kcl/                     # Infrastructure-as-Code (KCL schemas)
│   ├── cluster.k           # K8s cluster definition
│   ├── services.k          # Microservices
│   ├── storage.k           # Databases + storage
│   ├── agents.k            # Agent pools
│   └── multi-ia.k          # LLM Router configuration
├── taskservs/              # Taskserv definitions (TOML)
│   ├── vapora-backend.toml
│   ├── vapora-frontend.toml
│   ├── vapora-agents.toml
│   ├── vapora-mcp-gateway.toml
│   └── vapora-llm-router.toml
├── workflows/              # Batch operations
│   ├── deploy-full-stack.yaml
│   ├── scale-agents.yaml
│   ├── upgrade-vapora.yaml
│   └── disaster-recovery.yaml
└── README.md              # This file
```

---

## 🚀 Quick Start

### 1. Prerequisites

```bash
# Install Provisioning
./vapora-install.sh

# Verify installation
provisioning --version

# Ensure kubectl access
kubectl cluster-info
```

### 2. Deploy Cluster

```bash
cd provisioning/vapora-wrksp

# Create K8s cluster from KCL
provisioning cluster create --config kcl/cluster.k

# This will:
# - Deploy K3s or RKE2 (auto-detect)
# - Install Istio service mesh
# - Install Rook Ceph for storage
# - Set up monitoring (Prometheus + Grafana)
```

### 3. Deploy Full Stack

```bash
# Deploy all VAPORA services
provisioning workflow run workflows/deploy-full-stack.yaml

# This will:
# - Deploy SurrealDB (3 replicas)
# - Deploy NATS JetStream (3 replicas)
# - Deploy Redis
# - Deploy Axum backend (3 replicas)
# - Deploy Leptos frontend (2 replicas)
# - Deploy Agent Runtime (3 replicas)
# - Deploy MCP Gateway (2 replicas)
# - Deploy LLM Router (2 replicas)
# - Initialize database
# - Configure ingress
```

### 4. Verify Deployment

```bash
# Check all services
provisioning health-check --services all

# Get status
provisioning taskserv list

# View logs
provisioning logs tail -f vapora-backend

# Access frontend
open https://vapora.example.com
```

### 5. Scale Agents

```bash
# Scale developer agents to 10
provisioning taskserv scale vapora-agents --agent developer --replicas 10

# View autoscaling status
provisioning taskserv autoscale status vapora-agents
```

---

## 📋 Configuration

### workspace.toml

Master configuration for entire deployment:
- Cluster settings
- Agent pool sizes
- LLM providers
- Monitoring & scaling
- Backup policies

Edit this file to customize:

```toml
[agents]
developer = { min = 5, max = 20 }   # Scale developers to 20 max

[llm_router]
warn_threshold_daily = 5000  # Alert if daily LLM cost > $50
```

### Taskservs

Each taskserv (in `taskservs/`) defines:
- Source repository
- Build configuration
- Deployment resources
- Health checks
- Scaling policies
- Environment variables
- Secrets

Customize per-service as needed.

---

## 🔧 Common Operations

### Deploy New Version

```bash
# Update image version in workspace.toml or taskserv
# Then upgrade service:
provisioning taskserv upgrade vapora-backend --image vapora/backend:0.3.0

# This uses rolling update strategy (no downtime)
```

### Rollback Service

```bash
provisioning taskserv rollback vapora-backend --to-version 0.2.0
```

### View Agent Logs

```bash
provisioning logs tail -f vapora-agents --agent developer-001
```

### Access Database

```bash
# Port-forward to SurrealDB
kubectl port-forward -n vapora-system svc/surrealdb 8000:8000

# Then access at http://localhost:8000
```

### Monitor Costs

```bash
# View LLM usage & costs
provisioning metrics llm-router --daily

# Output shows:
# - Cost per provider
# - Tokens used per agent
# - Cost per task type
```

---

## 🎯 Workflows

Pre-built batch operations in `workflows/`:

### deploy-full-stack.yaml
Complete deployment from scratch (1 hour)

### scale-agents.yaml
Dynamic agent scaling based on queue depth

### upgrade-vapora.yaml
Rolling upgrade of all VAPORA services

### disaster-recovery.yaml
Restore from backups and verify

---

## 🔒 Security

VAPORA deployment includes:
- **mTLS**: Mutual TLS between all services (Istio)
- **Network Policies**: Namespace isolation
- **RBAC**: Kubernetes role-based access
- **Secrets**: RustyVault integration
- **Audit Logs**: Full audit trail via Cedar

---

## 📊 Monitoring

Post-deployment, access:

- **Grafana Dashboard**: `https://vapora.example.com/grafana`
  - Agent health & queue depth
  - API latency & error rates
  - LLM costs & usage

- **Prometheus**: `https://vapora.example.com:9090`
  - Raw metrics scraping

- **Logs**: `provisioning logs tail vapora-backend`

---

## 🆘 Troubleshooting

### Pods not starting?
```bash
kubectl describe pod -n vapora-system <pod-name>
provisioning logs tail vapora-backend
```

### Service unavailable?
```bash
# Check health
provisioning health-check vapora-backend

# Check ingress
kubectl get ingress -n vapora-system

# Check Istio VirtualService
kubectl get vs -n vapora-system
```

### Database connection issues?
```bash
# Check SurrealDB
provisioning logs tail vapora-system surrealdb

# Port-forward and test
kubectl port-forward svc/surrealdb 8000:8000
curl http://localhost:8000/health
```

---

## 📚 References

- **Workspace Configuration**: `workspace.toml`
- **Full Architecture**: `../../guides/core/VAPORA-ARCHITECTURE-V2.md`
- **Provisioning Integration**: `../../guides/integration/PROVISIONING-INTEGRATION.md`
- **KCL Schemas**: Read `.k` files in `kcl/` directory
- **Taskserv Format**: Read `.toml` files in `taskservs/` directory

---

## 🚀 Next Steps

1. ✅ Review `workspace.toml` and customize
2. ✅ Run `provisioning cluster create`
3. ✅ Run `provisioning workflow run deploy-full-stack.yaml`
4. ✅ Access VAPORA at `https://vapora.example.com`
5. ✅ Scale agents as needed for your workload

---

**Version**: 0.2.0
**Status**: Ready for Implementation
**Maintained**: By VAPORA Team