secretumvault/docs/operations/deployment.md

# SecretumVault Deployment Guide

This guide covers deployment of SecretumVault using Docker, Docker Compose, Kubernetes, and Helm.

## Table of Contents

1. [Local Development with Docker Compose](#local-development-with-docker-compose -f deploy/docker/docker-compose.yml)
2. [Kubernetes Deployment](#kubernetes-deployment)
3. [Helm Installation](#helm-installation)
4. [Configuration](#configuration)
5. [Initializing and Unsealing](#initializing-and-unsealing)
6. [Accessing the API](#accessing-the-api)
7. [TLS Configuration](#tls-configuration)
8. [Monitoring with Prometheus](#monitoring-with-prometheus)
9. [Troubleshooting](#troubleshooting)

## Local Development with Docker Compose

### Prerequisites

- Docker 20.10+
- Docker Compose 2.0+

### Quick Start

```bash
# Build the vault image
docker build -t secretumvault:latest -f deploy/docker/Dockerfile .

# Start all services
docker-compose -f deploy/docker/docker-compose.yml up -d

# Verify services are running
docker-compose -f deploy/docker/docker-compose.yml ps

# View logs
docker-compose -f deploy/docker/docker-compose.yml logs -f vault
```

### Services Included

The docker-compose -f deploy/docker/docker-compose.yml.yml includes:

- **vault**: SecretumVault server (port 8200 API, 9090 metrics)
- **etcd**: Distributed key-value store for secrets (port 2379)
- **surrealdb**: Alternative database backend (port 8000)
- **postgres**: PostgreSQL for dynamic secrets (port 5432)
- **prometheus**: Metrics scraping and storage (port 9090)
- **grafana**: Metrics visualization (port 3000)

### Configuration

Configuration is mounted from `docker/config/svault.toml`. Modify this file to:

- Change storage backend: `backend = "etcd"` or `"surrealdb"` or `"postgresql"`
- Change crypto backend: `crypto_backend = "openssl"` or `"aws-lc"`
- Enable/disable engines in the `[engines]` section
- Adjust logging level: `level = "info"`

### Health Check

```bash
# Check vault health
curl http://localhost:8200/v1/sys/health

# Check etcd health
docker-compose -f deploy/docker/docker-compose.yml exec etcd etcdctl --endpoints=http://localhost:2379 endpoint health
```

### Cleanup

```bash
# Stop all services
docker-compose -f deploy/docker/docker-compose.yml down

# Remove volumes (WARNING: deletes all data)
docker-compose -f deploy/docker/docker-compose.yml down -v
```

## Kubernetes Deployment

### Prerequisites

- Kubernetes 1.20+
- kubectl configured with cluster access
- StorageClass available for persistent volumes
- 2+ CPU and 2Gi RAM available cluster-wide

### Quick Start

```bash
# Deploy to 'secretumvault' namespace
kubectl apply -f deploy/k8s/01-namespace.yaml
kubectl apply -f deploy/k8s/02-configmap.yaml
kubectl apply -f deploy/k8s/03-deployment.yaml
kubectl apply -f deploy/k8s/04-service.yaml
kubectl apply -f deploy/k8s/05-etcd.yaml

# Optional: Additional storage backends
kubectl apply -f deploy/k8s/06-surrealdb.yaml
kubectl apply -f deploy/k8s/07-postgresql.yaml

# Verify deployment
kubectl -n secretumvault get pods -w
kubectl -n secretumvault get svc
```

### Accessing Vault

**From within cluster:**
```bash
# Using ClusterIP service
curl http://vault:8200/v1/sys/health

# Using headless service (direct pod access)
curl http://vault-headless:8200/v1/sys/health
```

**Port-forward from outside cluster:**
```bash
kubectl -n secretumvault port-forward svc/vault 8200:8200
curl http://localhost:8200/v1/sys/health
```

### Configuring Secrets

To pass database password or other secrets:

```bash
# Create secret for PostgreSQL
kubectl -n secretumvault create secret generic vault-postgresql-secret \
  --from-literal=password='your-secure-password'

# Create secret for SurrealDB
kubectl -n secretumvault create secret generic vault-surrealdb-secret \
  --from-literal=password='your-secure-password'

# Create secret for etcd (if authentication enabled)
kubectl -n secretumvault create secret generic vault-etcd-secret \
  --from-literal=password='your-secure-password'
```

### Scaling etcd

etcd is deployed as a StatefulSet with 3 replicas for high availability:

```bash
# View etcd pods
kubectl -n secretumvault get statefulset vault-etcd

# Scale if needed
kubectl -n secretumvault scale statefulset vault-etcd --replicas=5
```

### Cleanup

```bash
# Delete all vault resources
kubectl delete namespace secretumvault

# Or delete individually
kubectl delete -f k8s/
```

## Helm Installation

### Prerequisites

- Helm 3.0+
- kubectl configured with cluster access

### Installation

```bash
# Add repository (if using remote repo)
# helm repo add secretumvault https://charts.secretumvault.io
# helm repo update

# Install from local chart
helm install vault deploy/helm/ \
  --namespace secretumvault \
  --create-namespace

# Or with custom values
helm install vault deploy/helm/ \
  --namespace secretumvault \
  --create-namespace \
  --values helm/custom-values.yaml
```

### Upgrade

```bash
# List releases
helm list -n secretumvault

# Upgrade deployment
helm upgrade vault helm/ -n secretumvault

# Rollback to previous release
helm rollback vault -n secretumvault
```

### Customization

Customize deployment via values overrides:

```bash
# Enable SurrealDB backend
helm install vault deploy/helm/ -n secretumvault --create-namespace \
  --set vault.config.storageBackend=surrealdb \
  --set surrealdb.enabled=true

# Enable PostgreSQL for dynamic secrets
helm install vault deploy/helm/ -n secretumvault --create-namespace \
  --set postgresql.enabled=true \
  --set vault.config.engines.database=true

# Enable monitoring
helm install vault deploy/helm/ -n secretumvault --create-namespace \
  --set monitoring.prometheus.enabled=true \
  --set monitoring.grafana.enabled=true

# Change vault replicas
helm install vault deploy/helm/ -n secretumvault --create-namespace \
  --set vault.replicas=3
```

### Uninstall

```bash
# Remove Helm release
helm uninstall vault -n secretumvault

# Remove namespace
kubectl delete namespace secretumvault
```

## Configuration

### Configuration File Location

- **Docker**: `/etc/secretumvault/svault.toml` (mounted from `docker/config/`)
- **Kubernetes**: ConfigMap `vault-config` (from `k8s/02-configmap.yaml`)
- **Helm**: Templated from `helm/templates/configmap.yaml` (values in `helm/values.yaml`)

### Common Configuration Changes

**Switch Storage Backend:**

```toml
[storage]
backend = "surrealdb"  # or "etcd", "postgresql", "filesystem"

[storage.surrealdb]
url = "ws://vault-surrealdb:8000"
password = "${SURREAL_PASSWORD}"
```

**Change Crypto Backend:**

```toml
[vault]
crypto_backend = "aws-lc"  # or "openssl", "rustcrypto"
```

**Mount Additional Engines:**

```toml
[engines.kv]
path = "secret/"
versioned = true

[engines.transit]
path = "transit/"

[engines.pki]
path = "pki/"

[engines.database]
path = "database/"
```

**Adjust Logging:**

```toml
[logging]
level = "debug"
format = "json"
ansi = true
```

**Telemetry and Metrics:**

```toml
[telemetry]
prometheus_port = 9090
enable_trace = false
```

## Initializing and Unsealing

### Initialize Vault

```bash
# HTTP request to initialize
curl -X POST http://localhost:8200/v1/sys/init \
  -H "Content-Type: application/json" \
  -d '{
    "shares": 3,
    "threshold": 2
  }'

# Response contains unseal keys and root token
# Save these securely in a password manager (e.g., Bitwarden, 1Password)
```

### Unseal Vault

To unseal after restart, provide threshold number of unseal keys:

```bash
# Unseal with first key
curl -X POST http://localhost:8200/v1/sys/unseal \
  -H "Content-Type: application/json" \
  -d '{"key": "unseal-key-1"}'

# Unseal with second key
curl -X POST http://localhost:8200/v1/sys/unseal \
  -H "Content-Type: application/json" \
  -d '{"key": "unseal-key-2"}'

# Check seal status
curl http://localhost:8200/v1/sys/seal-status
```

### Check Status

```bash
# Health endpoint
curl http://localhost:8200/v1/sys/health

# Seal status
curl http://localhost:8200/v1/sys/seal-status
```

## Accessing the API

### Authentication

SecretumVault uses token-based authentication. Use the root token obtained during initialization:

```bash
export VAULT_TOKEN="root-token-from-initialization"
export VAULT_ADDR="http://localhost:8200"
```

### Example API Calls

**Create a secret:**
```bash
curl -X POST "$VAULT_ADDR/v1/secret/data/myapp" \
  -H "X-Vault-Token: $VAULT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "data": {
      "username": "admin",
      "password": "secret123"
    }
  }'
```

**Read a secret:**
```bash
curl -X GET "$VAULT_ADDR/v1/secret/data/myapp" \
  -H "X-Vault-Token: $VAULT_TOKEN"
```

**Delete a secret:**
```bash
curl -X DELETE "$VAULT_ADDR/v1/secret/data/myapp" \
  -H "X-Vault-Token: $VAULT_TOKEN"
```

**List all secrets:**
```bash
curl -X LIST "$VAULT_ADDR/v1/secret/metadata" \
  -H "X-Vault-Token: $VAULT_TOKEN"
```

**Encrypt with Transit engine:**
```bash
curl -X POST "$VAULT_ADDR/v1/transit/encrypt/my-key" \
  -H "X-Vault-Token: $VAULT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"plaintext": "dGhlIHF1aWNrIGJyb3duIGZveA=="}'
```

## TLS Configuration

### Self-Signed Certificate (Development)

```bash
# Generate self-signed cert
openssl req -x509 -newkey rsa:4096 -keyout tls.key -out tls.crt \
  -days 365 -nodes -subj "/CN=localhost"

# In Docker Compose, mount cert and key:
# volumes:
#   - ./tls.crt:/etc/secretumvault/tls.crt:ro
#   - ./tls.key:/etc/secretumvault/tls.key:ro

# Update svault.toml:
# [server]
# tls_cert = "/etc/secretumvault/tls.crt"
# tls_key = "/etc/secretumvault/tls.key"
```

### Kubernetes with cert-manager

```bash
# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

# Create ClusterIssuer
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx
EOF

# Update Helm values
# vault:
#   tls:
#     enabled: true
#     certManager:
#       enabled: true
#       issuer: letsencrypt-prod
```

### mTLS (Mutual TLS)

For client certificate authentication:

```bash
# Generate client certificate
openssl req -x509 -newkey rsa:2048 -keyout client.key -out client.crt \
  -days 365 -nodes -subj "/CN=vault-client"

# Update svault.toml
# [server]
# tls_client_ca = "/etc/secretumvault/client-ca.crt"

# Clients must provide certificate
curl --cert client.crt --key client.key \
  https://localhost:8200/v1/sys/health
```

## Monitoring with Prometheus

### Prometheus Configuration

Prometheus is configured to scrape vault metrics every 10 seconds:

```yaml
scrape_configs:
  - job_name: 'vault'
    static_configs:
      - targets: ['vault:9090']  # Docker Compose
      # Or for Kubernetes:
      # - targets: ['vault.secretumvault.svc.cluster.local:9090']
    scrape_interval: 10s
```

### Accessing Prometheus

**Docker Compose:**
```bash
# Metrics endpoint
curl http://localhost:9090/metrics

# Prometheus UI
open http://localhost:9090
```

**Kubernetes:**
```bash
# Port-forward
kubectl -n secretumvault port-forward svc/prometheus 9090:9090

# Metrics from vault
curl http://vault.secretumvault.svc.cluster.local:9090/metrics
```

### Available Metrics

- `vault_secrets_stored_total` - Total secrets stored
- `vault_secrets_read_total` - Total secrets read
- `vault_secrets_deleted_total` - Total secrets deleted
- `vault_operations_encrypt` - Encryption operations
- `vault_operations_decrypt` - Decryption operations
- `vault_operations_sign` - Signing operations
- `vault_operations_verify` - Verification operations
- `vault_policies_evaluated` - Cedar policies evaluated
- `vault_tokens_created` - Tokens created
- `vault_tokens_revoked` - Tokens revoked
- `vault_leases_issued` - Dynamic secret leases issued
- `vault_leases_revoked` - Dynamic secret leases revoked
- `vault_errors_total` - Total errors encountered

### Grafana Integration

If monitoring is enabled via Helm:

```bash
# Port-forward to Grafana
kubectl -n secretumvault port-forward svc/grafana 3000:3000

# Default login
# Username: admin
# Password: (from values.yaml monitoring.grafana.adminPassword)

# Add Prometheus data source
# http://prometheus:9090
```

## Troubleshooting

### Vault Pod Not Starting

```bash
# Check pod status
kubectl -n secretumvault describe pod <pod-name>

# Check logs
kubectl -n secretumvault logs <pod-name>

# Check events
kubectl -n secretumvault get events --sort-by='.lastTimestamp'
```

### etcd Connection Issues

```bash
# Check etcd service
kubectl -n secretumvault get svc vault-etcd-client

# Check etcd pods
kubectl -n secretumvault get statefulset vault-etcd

# Test connectivity from vault pod
kubectl -n secretumvault exec <vault-pod> -- \
  curl http://vault-etcd-client:2379/health
```

### Storage Backend Connection Error

```bash
# Verify ConfigMap
kubectl -n secretumvault get cm vault-config -o yaml

# Check if endpoints match service names
# For etcd: vault-etcd-client:2379
# For SurrealDB: vault-surrealdb-client:8000
# For PostgreSQL: vault-postgresql:5432
```

### High Memory Usage

```bash
# Check resource usage
kubectl -n secretumvault top pods

# If memory limit exceeded, increase in Helm values:
# vault:
#   resources:
#     limits:
#       memory: "1Gi"
```

### Metrics Not Appearing

```bash
# Check Prometheus targets
curl http://localhost:9090/api/v1/targets

# Check vault metrics endpoint directly
curl http://localhost:9090/metrics

# Verify prometheus port in config
# telemetry.prometheus_port = 9090
```

### Volume Mounting Issues

```bash
# Check PVC status
kubectl -n secretumvault get pvc

# Check StorageClass available
kubectl get storageclass

# For development without persistent storage:
# Update etcd StatefulSet to use emptyDir:
# volumes:
# - name: data
#   emptyDir: {}
```

### Vault Initialization Failed

If vault initialization fails, ensure:

1. Vault is unsealed (check `/v1/sys/seal-status`)
2. Storage backend is accessible
3. Master key can be encrypted/decrypted
4. Sufficient resources available

```bash
# Restart vault to retry
kubectl -n secretumvault delete pod <vault-pod>
```

## Production Checklist

- [ ] Enable TLS with valid certificates (not self-signed)
- [ ] Configure mTLS for client authentication
- [ ] Set strong unseal key threshold (2-3 of 5+)
- [ ] Store unseal keys securely in external vault (not in version control)
- [ ] Enable audit logging for compliance
- [ ] Configure Cedar policies for fine-grained access control
- [ ] Set up monitoring and alerting
- [ ] Configure high availability (3+ replicas for vault)
- [ ] Configure persistent storage backend (etcd or PostgreSQL)
- [ ] Set resource requests and limits appropriately
- [ ] Configure network policies to restrict traffic
- [ ] Enable pod security policies
- [ ] Set up automated backups
- [ ] Test disaster recovery procedures
- [ ] Enable secret rotation policies
- [ ] Configure lease expiration and revocation

## Additional Resources

- Architecture: `docs/secretumvault-complete-architecture.md`
- Configuration Guide: `docs/CONFIGURATION.md`
- API Reference: `docs/API.md`
- Security Guidelines: `docs/SECURITY.md`