prvng_platform/docs/deployment/guide.md

# Provisioning Platform - Deployment Guide

**Last Updated**: 2025-10-07
**Platform**: macOS (Apple Silicon / Intel) + OrbStack/Docker

---

## ✅ Fixed: Docker Builds

Docker builds have been **fixed** to properly handle the Rust workspace structure. Both deployment methods (Native and Docker) are now fully supported.

**Note**: Docker builds use Rust nightly to support edition2024 (required by async-graphql 7.x from surrealdb). RocksDB has been replaced with SurrealDB in-memory backend (kv-mem) to simplify Docker builds (no libclang requirement).

---

## 📦 Quick Start

### Prerequisites

**For Native Deployment:**

- Rust 1.75+: `brew install rust`
- Nushell 0.107+: `brew install nushell`

**For Docker Deployment:**

- OrbStack (recommended): <https://orbstack.dev>
- Or Docker Desktop: `brew install --cask docker`

---

## 🚀 Deployment Methods

### Method 1: Native Execution (Recommended for Development)

**Fastest startup, easiest debugging, direct access to logs**

```bash
cd provisioning/platform/scripts

# 1. Build all services
nu run-native.nu build

# 2. Start all services in background
nu run-native.nu start-all --background

# 3. Check status
nu run-native.nu status

# 4. View logs
nu run-native.nu logs orchestrator --follow

# 5. Stop all
nu run-native.nu stop-all
```

**Services will run on:**

- Orchestrator: <http://localhost:8080>
- Control Center: <http://localhost:8081>

**Data stored in:**

- `~/.provisioning-platform/data/`
- `~/.provisioning-platform/logs/`

---

### Method 2: Docker Execution (Recommended for Production-like Testing)

**Isolated environments, easy cleanup, supports all deployment modes**

```bash
cd provisioning/platform/scripts

# 1. Build Docker images (Solo mode)
nu run-docker.nu build solo

# 2. Start services in background
nu run-docker.nu start solo --detach

# 3. Check status
nu run-docker.nu status

# 4. View logs
nu run-docker.nu logs orchestrator --follow

# 5. Stop all
nu run-docker.nu stop
```

**Deployment Modes:**

- `solo` - 2 CPU / 4GB RAM (dev/test)
- `multiuser` - 4 CPU / 8GB RAM (team)
- `cicd` - 8 CPU / 16GB RAM (automation)
- `enterprise` - 16 CPU / 32GB RAM (production + KMS)

---

## 📋 Complete Command Reference

### Native Execution (`run-native.nu`)

| Command | Description |
|---------|-------------|
| `build` | Build all services |
| `start <service>` | Start orchestrator or control_center |
| `start-all` | Start all services |
| `stop <service>` | Stop a specific service |
| `stop-all` | Stop all services |
| `status` | Show service status |
| `logs <service>` | Show logs (add `--follow`) |
| `health` | Check service health |

**Examples:**

```bash
nu run-native.nu build
nu run-native.nu start orchestrator --background
nu run-native.nu start control_center --background
nu run-native.nu logs orchestrator --follow
nu run-native.nu health
nu run-native.nu stop-all
```

---

### Docker Execution (`run-docker.nu`)

| Command | Description |
|---------|-------------|
| `build [mode]` | Build Docker images |
| `start [mode]` | Start services (add `--detach`) |
| `stop` | Stop all services (add `--volumes` to delete data) |
| `restart [mode]` | Restart services |
| `status` | Show container status |
| `logs <service>` | Show logs (add `--follow`) |
| `exec <service> <cmd>` | Execute command in container |
| `stats` | Show resource usage |
| `health` | Check service health |
| `config [mode]` | Show docker-compose config |
| `clean` | Remove containers (add `--all` for images too) |

**Examples:**

```bash
# Solo mode (fastest)
nu run-docker.nu build solo
nu run-docker.nu start solo --detach

# Enterprise mode (with KMS)
nu run-docker.nu build enterprise
nu run-docker.nu start enterprise --detach

# Operations
nu run-docker.nu status
nu run-docker.nu logs control-center --follow
nu run-docker.nu exec orchestrator bash
nu run-docker.nu stats
nu run-docker.nu stop
```

---

## 🗄️ Database Information

### Control-Center Database

**Type**: SurrealDB with in-memory backend (kv-mem)
**Location**: In-memory (data persisted during container/process lifetime)
**Production Alternative**: SurrealDB with remote WebSocket connection for persistent storage

**No separate database server required** - SurrealDB in-memory backend is embedded in the control-center process.

### Orchestrator Storage

**Type**: Filesystem queue (default)
**Location**:

- Native: `~/.provisioning-platform/data/orchestrator/queue.rkvs`
- Docker: `/data/queue.rkvs` (inside container)

**Production Option**: Switch to SurrealDB via config for distributed deployments.

---

## ⚙️ Configuration Loading

Services load configuration in this order (priority: low → high):

1. **System Defaults** - `provisioning/config/config.defaults.toml`
2. **Service Defaults** - `provisioning/platform/{service}/config.defaults.toml`
3. **Workspace Config** - `workspace/{name}/config/provisioning.yaml`
4. **User Config** - `~/Library/Application Support/provisioning/user_config.yaml`
5. **Environment Variables** - `CONTROL_CENTER_*`, `ORCHESTRATOR_*`
6. **Runtime Overrides** - `--config` flag

**See full documentation**: `docs/architecture/DATABASE_AND_CONFIG_ARCHITECTURE.md`

---

## 🐛 Troubleshooting

### Native Deployment Issues

**Build fails:**

```bash
# Clean and rebuild
cd provisioning/platform
cargo clean
cargo build --release
```

**Port already in use:**

```bash
# Check what's using the port
lsof -i :8080
lsof -i :8081

# Kill the process or use different ports via environment variables
export ORCHESTRATOR_SERVER_PORT=8090
export CONTROL_CENTER_SERVER_PORT=8091
```

**Service won't start:**

```bash
# Check logs for errors
nu run-native.nu logs orchestrator

# Run in foreground to see output
nu run-native.nu start orchestrator
```

---

### Docker Deployment Issues

**Build fails with workspace errors:**

- **Fixed!** Dockerfiles now properly handle workspace structure
- If still failing: `nu run-docker.nu build solo --no-cache`

**Containers won't start:**

```bash
# Check container logs
nu run-docker.nu logs orchestrator

# Check Docker daemon
docker ps
docker info

# Restart Docker/OrbStack
```

**Port conflicts:**

```bash
# Check what's using ports
lsof -i :8080
lsof -i :8081

# Stop conflicting services or modify docker-compose.yaml ports
```

**Out of resources:**

```bash
# Check current usage
nu run-docker.nu stats

# Clean up unused containers/images
docker system prune -a

# Or use the script
nu run-docker.nu clean --all
```

---

## 🔐 KMS Integration (Enterprise Mode)

Enterprise mode includes Cosmian KMS for production-grade secret management.

**Start with KMS:**

```bash
nu run-docker.nu build enterprise
nu run-docker.nu start enterprise --detach
```

**Access KMS:**

- KMS API: <http://localhost:9998>
- KMS Health: <http://localhost:9998/health>

**KMS Features:**

- SSL certificate lifecycle management
- SSH private key rotation
- Cloud credential auto-refresh
- Audit trails
- Automatic key rotation

**See full KMS documentation**: `provisioning/platform/control-center/src/kms/README.md`

---

## 📊 Monitoring

### Health Checks

**Native:**

```bash
nu run-native.nu health
```

**Docker:**

```bash
nu run-docker.nu health
```

**Manual:**

```bash
curl http://localhost:8080/health  # Orchestrator
curl http://localhost:8081/health  # Control Center
curl http://localhost:9998/health  # KMS (enterprise only)
```

### Resource Usage

**Docker:**

```bash
nu run-docker.nu stats
```

**Native:**

```bash
ps aux | grep -E "provisioning-orchestrator|control-center"
top -pid <pid>
```

---

## 🧪 Testing Both Methods

### Test Native Deployment

```bash
cd provisioning/platform/scripts

# 1. Build
nu run-native.nu build

# 2. Start services
nu run-native.nu start-all --background

# 3. Verify
nu run-native.nu status
nu run-native.nu health

# 4. Test API
curl http://localhost:8080/health
curl http://localhost:8081/health

# 5. Clean up
nu run-native.nu stop-all
```

### Test Docker Deployment

```bash
cd provisioning/platform/scripts

# 1. Build
nu run-docker.nu build solo

# 2. Start services
nu run-docker.nu start solo --detach

# 3. Verify
nu run-docker.nu status
nu run-docker.nu health

# 4. Test API
curl http://localhost:8080/health
curl http://localhost:8081/health

# 5. Clean up
nu run-docker.nu stop --volumes
```

---

## 🎯 Best Practices

### Development Workflow

1. **Use Native for Active Development**
   - Faster iteration (no Docker rebuild)
   - Direct log access
   - Easy debugging with IDE

2. **Use Docker for Integration Testing**
   - Test deployment configurations
   - Verify Docker builds
   - Simulate production environment

### Production Deployment

1. **Use Docker/Kubernetes**
   - Isolated environments
   - Easy scaling
   - Standard deployment

2. **Use Enterprise Mode**
   - KMS for secret management
   - Full monitoring stack
   - High availability

---

## 📚 Related Documentation

- **Database Architecture**: `docs/architecture/DATABASE_AND_CONFIG_ARCHITECTURE.md`
- **KMS Integration**: `provisioning/platform/control-center/src/kms/README.md`
- **Configuration System**: `.claude/features/configuration-system.md`
- **Workspace Switching**: `.claude/features/workspace-switching.md`
- **Orchestrator Architecture**: `.claude/features/orchestrator-architecture.md`

---

## ✅ Summary

### Native Execution

- ✅ **Fixed**: Workspace builds work correctly
- ✅ **Fast**: No container overhead
- ✅ **Simple**: Direct binary execution
- ✅ **Best for**: Development, debugging

### Docker Execution

- ✅ **Fixed**: Dockerfiles now workspace-aware
- ✅ **Isolated**: Clean environments
- ✅ **Flexible**: Multiple deployment modes
- ✅ **Best for**: Testing, production-like deployments

**Both methods fully supported and tested!**

---

**Quick Links:**

- Native Script: `provisioning/platform/scripts/run-native.nu`
- Docker Script: `provisioning/platform/scripts/run-docker.nu`
- Docker Files: `provisioning/platform/docker-compose.yaml` + mode-specific overrides