prvng_platform/docs/deployment/deployment-guide.md

Provisioning Platform Deployment Guide

Version: 3.0.0 Date: 2025-10-06 Deployment Modes: Solo, Multi-User, CI/CD, Enterprise

---

Table of Contents

1. Overview
2. Prerequisites
3. Deployment Modes
4. Quick Start
5. Configuration
6. Deployment Methods
7. Post-Deployment
8. Troubleshooting

---

Overview

The Provisioning Platform is a comprehensive infrastructure automation system that can be deployed in four modes:

• Solo: Single-user local development (minimal services)
• Multi-User: Team collaboration with source control
• CI/CD: Automated deployment pipelines
• Enterprise: Full production with monitoring, KMS, and audit logging

Architecture Components

Component	Solo	Multi-User	CI/CD	Enterprise
Orchestrator	✓	✓	✓	✓
Control Center	✓	✓	✓	✓
CoreDNS	✓	✓	✓	✓
OCI Registry (Zot)	✓	✓	✓	-
Extension Registry	✓	✓	✓	✓
Gitea	-	✓	✓	✓
PostgreSQL	-	✓	✓	✓
API Server	-	-	✓	✓
Harbor	-	-	-	✓
Cosmian KMS	-	-	-	✓
Prometheus	-	-	-	✓
Grafana	-	-	-	✓
Loki + Promtail	-	-	-	✓
Elasticsearch + Kibana	-	-	-	✓
Nginx Reverse Proxy	-	-	-	✓

---

Prerequisites

Required Software

1. Docker (version 20.10+)

   docker --version
   # Docker version 20.10.0 or higher
   

2. Docker Compose (version 2.0+)

   docker-compose --version
   # Docker Compose version 2.0.0 or higher
   

3. Nushell (version 0.107.1+ for automation scripts)

   nu --version
   # 0.107.1 or higher
   

System Requirements

Solo Mode

• CPU: 2 cores
• Memory: 4GB RAM
• Disk: 20GB free space
• Network: Internet connection for pulling images

Multi-User Mode

• CPU: 4 cores
• Memory: 8GB RAM
• Disk: 50GB free space
• Network: Internet connection + internal network

CI/CD Mode

• CPU: 8 cores
• Memory: 16GB RAM
• Disk: 100GB free space
• Network: Internet + dedicated CI/CD network

Enterprise Mode

• CPU: 16 cores
• Memory: 32GB RAM
• Disk: 500GB free space (SSD recommended)
• Network: High-bandwidth, low-latency network

Optional Tools

• OpenSSL (for generating secrets)
• kubectl (for Kubernetes deployment)
• Helm (for Kubernetes package management)

---

Deployment Modes

Solo Mode

Use Case: Local development, testing, personal use

Features:

• Minimal resource usage
• No authentication required
• SQLite databases
• Local file storage

Limitations:

• Single user only
• No version control integration
• No audit logging

Multi-User Mode

Use Case: Small team collaboration

Features:

• Multi-user authentication
• Gitea for source control
• PostgreSQL shared database
• User management

Limitations:

• No automated pipelines
• No advanced monitoring

CI/CD Mode

Use Case: Automated deployment pipelines

Features:

• All Multi-User features
• Provisioning API Server
• Webhook support
• Jenkins/GitLab Runner integration

Limitations:

• Basic monitoring only

Enterprise Mode

Use Case: Production deployments, compliance requirements

Features:

• All CI/CD features
• Harbor registry (enterprise OCI)
• Cosmian KMS (secret management)
• Full monitoring stack (Prometheus, Grafana)
• Log aggregation (Loki, Elasticsearch)
• Audit logging
• TLS/SSL encryption
• Nginx reverse proxy

---

Quick Start

1. Clone Repository

cd /opt
git clone https://github.com/your-org/project-provisioning.git
cd project-provisioning/provisioning/platform
```plaintext

### 2. Generate Secrets

```bash
# Generate .env file with random secrets
./scripts/generate-secrets.nu

# Or copy and edit manually
cp .env.example .env
nano .env
```plaintext

### 3. Choose Deployment Mode and Deploy

#### Solo Mode

```bash
./scripts/deploy-platform.nu --mode solo
```plaintext

#### Multi-User Mode

```bash
# Generate secrets first
./scripts/generate-secrets.nu

# Deploy
./scripts/deploy-platform.nu --mode multi-user
```plaintext

#### CI/CD Mode

```bash
./scripts/deploy-platform.nu --mode cicd --build
```plaintext

#### Enterprise Mode

```bash
# Full production deployment
./scripts/deploy-platform.nu --mode enterprise --build --wait 600
```plaintext

### 4. Verify Deployment

```bash
# Check all services
./scripts/health-check.nu

# View logs
docker-compose logs -f
```plaintext

### 5. Access Services

- **Orchestrator**: <http://localhost:9090>
- **Control Center**: <http://localhost:8081>
- **OCI Registry**: <http://localhost:5000>
- **Gitea** (Multi-User+): <http://localhost:3000>
- **Grafana** (Enterprise): <http://localhost:3001>

---

## Configuration

### Environment Variables

The `.env` file controls all deployment settings. Key variables:

#### Platform Configuration

```bash
PROVISIONING_MODE=solo           # solo, multi-user, cicd, enterprise
PLATFORM_ENVIRONMENT=development # development, staging, production
```plaintext

#### Service Ports

```bash
ORCHESTRATOR_PORT=8080
CONTROL_CENTER_PORT=8081
GITEA_HTTP_PORT=3000
OCI_REGISTRY_PORT=5000
```plaintext

#### Security Settings

```bash
# Generate with: openssl rand -base64 32
CONTROL_CENTER_JWT_SECRET=<random-secret>
API_SERVER_JWT_SECRET=<random-secret>
POSTGRES_PASSWORD=<random-password>
```plaintext

#### Resource Limits

```bash
ORCHESTRATOR_CPU_LIMIT=2000m
ORCHESTRATOR_MEMORY_LIMIT=2048M
```plaintext

### Configuration Files

#### Docker Compose

- **Main**: `docker-compose.yaml` (base services)
- **Solo**: `infrastructure/docker/docker-compose.solo.yaml`
- **Multi-User**: `infrastructure/docker/docker-compose.multi-user.yaml`
- **CI/CD**: `infrastructure/docker/docker-compose.cicd.yaml`
- **Enterprise**: `infrastructure/docker/docker-compose.enterprise.yaml`

#### Service Configurations

- **Orchestrator**: `orchestrator/config.defaults.toml`
- **Control Center**: `control-center/config.defaults.toml`
- **CoreDNS**: `config/coredns/Corefile`
- **OCI Registry**: `infrastructure/oci-registry/config.json`
- **Nginx**: `infrastructure/nginx/nginx.conf`
- **Prometheus**: `infrastructure/monitoring/prometheus/prometheus.yml`

---

## Deployment Methods

### Method 1: Docker Compose (Recommended)

#### Deploy

```bash
# Solo mode
docker-compose -f docker-compose.yaml \
               -f infrastructure/docker/docker-compose.solo.yaml \
               up -d

# Multi-user mode
docker-compose -f docker-compose.yaml \
               -f infrastructure/docker/docker-compose.multi-user.yaml \
               up -d

# CI/CD mode
docker-compose -f docker-compose.yaml \
               -f infrastructure/docker/docker-compose.multi-user.yaml \
               -f infrastructure/docker/docker-compose.cicd.yaml \
               up -d

# Enterprise mode
docker-compose -f docker-compose.yaml \
               -f infrastructure/docker/docker-compose.multi-user.yaml \
               -f infrastructure/docker/docker-compose.cicd.yaml \
               -f infrastructure/docker/docker-compose.enterprise.yaml \
               up -d
```plaintext

#### Manage Services

```bash
# View logs
docker-compose logs -f [service-name]

# Restart service
docker-compose restart orchestrator

# Stop all services
docker-compose down

# Stop and remove volumes (WARNING: data loss)
docker-compose down --volumes
```plaintext

### Method 2: Systemd (Linux Production)

#### Install Services

```bash
cd systemd
sudo ./install-services.sh
```plaintext

#### Manage via systemd

```bash
# Start platform
sudo systemctl start provisioning-platform

# Enable auto-start on boot
sudo systemctl enable provisioning-platform

# Check status
sudo systemctl status provisioning-platform

# View logs
sudo journalctl -u provisioning-platform -f

# Restart
sudo systemctl restart provisioning-platform

# Stop
sudo systemctl stop provisioning-platform
```plaintext

### Method 3: Kubernetes

See [KUBERNETES_DEPLOYMENT.md](./KUBERNETES_DEPLOYMENT.md) for detailed instructions.

#### Quick Deploy

```bash
# Create namespace
kubectl apply -f k8s/base/namespace.yaml

# Deploy services
kubectl apply -f k8s/deployments/
kubectl apply -f k8s/services/
kubectl apply -f k8s/ingress/

# Check status
kubectl get pods -n provisioning
```plaintext

### Method 4: Automation Script (Nushell)

```bash
# Deploy with options
./scripts/deploy-platform.nu --mode enterprise \
                              --build \
                              --wait 300

# Health check
./scripts/health-check.nu

# Dry run (show what would be deployed)
./scripts/deploy-platform.nu --mode enterprise --dry-run
```plaintext

---

## Post-Deployment

### 1. Verify Services

```bash
# Quick health check
./scripts/health-check.nu

# Detailed Docker status
docker-compose ps

# Check individual service
curl http://localhost:9090/health
```plaintext

### 2. Initial Configuration

#### Create Admin User (Multi-User+)

Access Gitea at <http://localhost:3000> and complete setup wizard.

#### Configure DNS (Optional)

Add to `/etc/hosts` or configure local DNS:

```plaintext
127.0.0.1  provisioning.local
127.0.0.1  gitea.provisioning.local
127.0.0.1  grafana.provisioning.local
```plaintext

#### Configure Monitoring (Enterprise)

1. Access Grafana: <http://localhost:3001>
2. Login with credentials from `.env`:
   - Username: `admin`
   - Password: `${GRAFANA_ADMIN_PASSWORD}`
3. Dashboards are auto-provisioned from `infrastructure/monitoring/grafana/dashboards/`

### 3. Load Extensions

```bash
# List available extensions
curl http://localhost:8082/api/v1/extensions

# Upload extension (example)
curl -X POST http://localhost:8082/api/v1/extensions/upload \
     -F "file=@my-extension.tar.gz"
```plaintext

### 4. Test Workflows

```bash
# Create test server (via orchestrator API)
curl -X POST http://localhost:9090/workflows/servers/create \
     -H "Content-Type: application/json" \
     -d '{"name": "test-server", "plan": "1xCPU-2GB"}'

# Check workflow status
curl http://localhost:9090/tasks/<task-id>
```plaintext

---

## Troubleshooting

### Common Issues

#### Services Not Starting

**Symptom**: `docker-compose up` fails or services crash

**Solutions**:

1. Check Docker daemon:

   ```bash
   systemctl status docker

1. Check logs:

   docker-compose logs orchestrator
   

2. Check resource limits:

   docker stats
   

3. Increase Docker resources in Docker Desktop settings

Port Conflicts

Symptom: Error: port is already allocated

Solutions:

1. Find conflicting process:

   lsof -i :9090
   

2. Change port in .env:

   ORCHESTRATOR_PORT=9080
   

3. Restart deployment:

   docker-compose down
   docker-compose up -d
   

Health Checks Failing

Symptom: Health check script reports unhealthy services

Solutions:

1. Check service logs:

   docker-compose logs -f <service>
   

2. Verify network connectivity:

   docker network inspect provisioning-net
   

3. Check firewall rules:

   sudo ufw status
   

4. Wait longer for services to start:

   ./scripts/deploy-platform.nu --wait 600
   

Database Connection Errors

Symptom: PostgreSQL connection refused

Solutions:

1. Check PostgreSQL health:

   docker exec provisioning-postgres pg_isready
   

2. Verify credentials in .env:

   grep POSTGRES_ .env
   

3. Check PostgreSQL logs:

   docker-compose logs postgres
   

4. Recreate database:

   docker-compose down
   docker volume rm provisioning_postgres-data
   docker-compose up -d
   

Out of Disk Space

Symptom: No space left on device

Solutions:

1. Clean Docker volumes:

   docker volume prune
   

2. Clean Docker images:

   docker image prune -a
   

3. Check volume sizes:

   docker system df -v
   

Getting Help

• Logs: Always check logs first: docker-compose logs -f
• Health: Run health check: ./scripts/health-check.nu --json
• Documentation: See docs/ directory
• Issues: File bug reports at GitHub repository

---

Security Best Practices

1. Secret Management

• Never commit .env files to version control
• Use ./scripts/generate-secrets.nu to generate strong secrets
• Rotate secrets regularly
• Use KMS in enterprise mode

2. Network Security

• Use TLS/SSL in production (enterprise mode)

• Configure firewall rules:

  sudo ufw allow 80/tcp
  sudo ufw allow 443/tcp
  sudo ufw enable
  

• Use private networks for backend services

3. Access Control

• Enable authentication in multi-user mode
• Use strong passwords (16+ characters)
• Configure API keys for CI/CD access
• Enable audit logging in enterprise mode

4. Regular Updates

# Pull latest images
docker-compose pull

# Rebuild with updates
./scripts/deploy-platform.nu --pull --build
```plaintext

---

## Backup and Recovery

### Backup

```bash
# Backup volumes
docker run --rm -v provisioning_orchestrator-data:/data \
           -v $(pwd)/backup:/backup \
           alpine tar czf /backup/orchestrator-data.tar.gz -C /data .

# Backup PostgreSQL
docker exec provisioning-postgres pg_dumpall -U provisioning > backup/postgres-backup.sql
```plaintext

### Restore

```bash
# Restore volume
docker run --rm -v provisioning_orchestrator-data:/data \
           -v $(pwd)/backup:/backup \
           alpine tar xzf /backup/orchestrator-data.tar.gz -C /data

# Restore PostgreSQL
docker exec -i provisioning-postgres psql -U provisioning < backup/postgres-backup.sql
```plaintext

---

## Maintenance

### Updates

```bash
# Pull latest images
docker-compose pull

# Recreate containers
docker-compose up -d --force-recreate

# Remove old images
docker image prune
```plaintext

### Monitoring

- **Prometheus**: <http://localhost:9090>
- **Grafana**: <http://localhost:3001>
- **Logs**: `docker-compose logs -f`

### Health Checks

```bash
# Automated health check
./scripts/health-check.nu

# Manual checks
curl http://localhost:9090/health
curl http://localhost:8081/health
```plaintext

---

## Next Steps

- [Production Deployment Guide](./PRODUCTION_DEPLOYMENT.md)
- [Kubernetes Deployment Guide](./KUBERNETES_DEPLOYMENT.md)
- [Docker Compose Reference](./DOCKER_COMPOSE_REFERENCE.md)
- [Monitoring Setup](./MONITORING_SETUP.md)
- [Security Hardening](./SECURITY_HARDENING.md)

---

**Documentation Version**: 1.0.0
**Last Updated**: 2025-10-06
**Maintained By**: Platform Team