secretumvault/docs/operations/deployment.md

SecretumVault Deployment Guide

This guide covers deployment of SecretumVault using Docker, Docker Compose, Kubernetes, and Helm.

Table of Contents

1. [Local Development with Docker Compose](#local-development-with-docker-compose -f deploy/docker/docker-compose.yml)
2. Kubernetes Deployment
3. Helm Installation
4. Configuration
5. Initializing and Unsealing
6. Accessing the API
7. TLS Configuration
8. Monitoring with Prometheus
9. Troubleshooting

Local Development with Docker Compose

Prerequisites

• Docker 20.10+
• Docker Compose 2.0+

Quick Start

# Build the vault image
docker build -t secretumvault:latest -f deploy/docker/Dockerfile .

# Start all services
docker-compose -f deploy/docker/docker-compose.yml up -d

# Verify services are running
docker-compose -f deploy/docker/docker-compose.yml ps

# View logs
docker-compose -f deploy/docker/docker-compose.yml logs -f vault

Services Included

The docker-compose -f deploy/docker/docker-compose.yml.yml includes:

• vault: SecretumVault server (port 8200 API, 9090 metrics)
• etcd: Distributed key-value store for secrets (port 2379)
• surrealdb: Alternative database backend (port 8000)
• postgres: PostgreSQL for dynamic secrets (port 5432)
• prometheus: Metrics scraping and storage (port 9090)
• grafana: Metrics visualization (port 3000)

Configuration

Configuration is mounted from docker/config/svault.toml. Modify this file to:

• Change storage backend: backend = "etcd" or "surrealdb" or "postgresql"
• Change crypto backend: crypto_backend = "openssl" or "aws-lc"
• Enable/disable engines in the [engines] section
• Adjust logging level: level = "info"

Health Check

# Check vault health
curl http://localhost:8200/v1/sys/health

# Check etcd health
docker-compose -f deploy/docker/docker-compose.yml exec etcd etcdctl --endpoints=http://localhost:2379 endpoint health

Cleanup

# Stop all services
docker-compose -f deploy/docker/docker-compose.yml down

# Remove volumes (WARNING: deletes all data)
docker-compose -f deploy/docker/docker-compose.yml down -v

Kubernetes Deployment

Prerequisites

• Kubernetes 1.20+
• kubectl configured with cluster access
• StorageClass available for persistent volumes
• 2+ CPU and 2Gi RAM available cluster-wide

Quick Start

# Deploy to 'secretumvault' namespace
kubectl apply -f deploy/k8s/01-namespace.yaml
kubectl apply -f deploy/k8s/02-configmap.yaml
kubectl apply -f deploy/k8s/03-deployment.yaml
kubectl apply -f deploy/k8s/04-service.yaml
kubectl apply -f deploy/k8s/05-etcd.yaml

# Optional: Additional storage backends
kubectl apply -f deploy/k8s/06-surrealdb.yaml
kubectl apply -f deploy/k8s/07-postgresql.yaml

# Verify deployment
kubectl -n secretumvault get pods -w
kubectl -n secretumvault get svc

Accessing Vault

From within cluster:

# Using ClusterIP service
curl http://vault:8200/v1/sys/health

# Using headless service (direct pod access)
curl http://vault-headless:8200/v1/sys/health

Port-forward from outside cluster:

kubectl -n secretumvault port-forward svc/vault 8200:8200
curl http://localhost:8200/v1/sys/health

Configuring Secrets

To pass database password or other secrets:

# Create secret for PostgreSQL
kubectl -n secretumvault create secret generic vault-postgresql-secret \
  --from-literal=password='your-secure-password'

# Create secret for SurrealDB
kubectl -n secretumvault create secret generic vault-surrealdb-secret \
  --from-literal=password='your-secure-password'

# Create secret for etcd (if authentication enabled)
kubectl -n secretumvault create secret generic vault-etcd-secret \
  --from-literal=password='your-secure-password'

Scaling etcd

etcd is deployed as a StatefulSet with 3 replicas for high availability:

# View etcd pods
kubectl -n secretumvault get statefulset vault-etcd

# Scale if needed
kubectl -n secretumvault scale statefulset vault-etcd --replicas=5

Cleanup

# Delete all vault resources
kubectl delete namespace secretumvault

# Or delete individually
kubectl delete -f k8s/

Helm Installation

Prerequisites

• Helm 3.0+
• kubectl configured with cluster access

Installation

# Add repository (if using remote repo)
# helm repo add secretumvault https://charts.secretumvault.io
# helm repo update

# Install from local chart
helm install vault deploy/helm/ \
  --namespace secretumvault \
  --create-namespace

# Or with custom values
helm install vault deploy/helm/ \
  --namespace secretumvault \
  --create-namespace \
  --values helm/custom-values.yaml

Upgrade

# List releases
helm list -n secretumvault

# Upgrade deployment
helm upgrade vault helm/ -n secretumvault

# Rollback to previous release
helm rollback vault -n secretumvault

Customization

Customize deployment via values overrides:

# Enable SurrealDB backend
helm install vault deploy/helm/ -n secretumvault --create-namespace \
  --set vault.config.storageBackend=surrealdb \
  --set surrealdb.enabled=true

# Enable PostgreSQL for dynamic secrets
helm install vault deploy/helm/ -n secretumvault --create-namespace \
  --set postgresql.enabled=true \
  --set vault.config.engines.database=true

# Enable monitoring
helm install vault deploy/helm/ -n secretumvault --create-namespace \
  --set monitoring.prometheus.enabled=true \
  --set monitoring.grafana.enabled=true

# Change vault replicas
helm install vault deploy/helm/ -n secretumvault --create-namespace \
  --set vault.replicas=3

Uninstall

# Remove Helm release
helm uninstall vault -n secretumvault

# Remove namespace
kubectl delete namespace secretumvault

Configuration

Configuration File Location

• Docker: /etc/secretumvault/svault.toml (mounted from docker/config/)
• Kubernetes: ConfigMap vault-config (from k8s/02-configmap.yaml)
• Helm: Templated from helm/templates/configmap.yaml (values in helm/values.yaml)

Common Configuration Changes

Switch Storage Backend:

[storage]
backend = "surrealdb"  # or "etcd", "postgresql", "filesystem"

[storage.surrealdb]
url = "ws://vault-surrealdb:8000"
password = "${SURREAL_PASSWORD}"

Change Crypto Backend:

[vault]
crypto_backend = "aws-lc"  # or "openssl", "rustcrypto"

Mount Additional Engines:

[engines.kv]
path = "secret/"
versioned = true

[engines.transit]
path = "transit/"

[engines.pki]
path = "pki/"

[engines.database]
path = "database/"

Adjust Logging:

[logging]
level = "debug"
format = "json"
ansi = true

Telemetry and Metrics:

[telemetry]
prometheus_port = 9090
enable_trace = false

Initializing and Unsealing

Initialize Vault

# HTTP request to initialize
curl -X POST http://localhost:8200/v1/sys/init \
  -H "Content-Type: application/json" \
  -d '{
    "shares": 3,
    "threshold": 2
  }'

# Response contains unseal keys and root token
# Save these securely in a password manager (e.g., Bitwarden, 1Password)

Unseal Vault

To unseal after restart, provide threshold number of unseal keys:

# Unseal with first key
curl -X POST http://localhost:8200/v1/sys/unseal \
  -H "Content-Type: application/json" \
  -d '{"key": "unseal-key-1"}'

# Unseal with second key
curl -X POST http://localhost:8200/v1/sys/unseal \
  -H "Content-Type: application/json" \
  -d '{"key": "unseal-key-2"}'

# Check seal status
curl http://localhost:8200/v1/sys/seal-status

Check Status

# Health endpoint
curl http://localhost:8200/v1/sys/health

# Seal status
curl http://localhost:8200/v1/sys/seal-status

Accessing the API

Authentication

SecretumVault uses token-based authentication. Use the root token obtained during initialization:

export VAULT_TOKEN="root-token-from-initialization"
export VAULT_ADDR="http://localhost:8200"

Example API Calls

Create a secret:

curl -X POST "$VAULT_ADDR/v1/secret/data/myapp" \
  -H "X-Vault-Token: $VAULT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "data": {
      "username": "admin",
      "password": "secret123"
    }
  }'

Read a secret:

curl -X GET "$VAULT_ADDR/v1/secret/data/myapp" \
  -H "X-Vault-Token: $VAULT_TOKEN"

Delete a secret:

curl -X DELETE "$VAULT_ADDR/v1/secret/data/myapp" \
  -H "X-Vault-Token: $VAULT_TOKEN"

List all secrets:

curl -X LIST "$VAULT_ADDR/v1/secret/metadata" \
  -H "X-Vault-Token: $VAULT_TOKEN"

Encrypt with Transit engine:

curl -X POST "$VAULT_ADDR/v1/transit/encrypt/my-key" \
  -H "X-Vault-Token: $VAULT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"plaintext": "dGhlIHF1aWNrIGJyb3duIGZveA=="}'

TLS Configuration

Self-Signed Certificate (Development)

# Generate self-signed cert
openssl req -x509 -newkey rsa:4096 -keyout tls.key -out tls.crt \
  -days 365 -nodes -subj "/CN=localhost"

# In Docker Compose, mount cert and key:
# volumes:
#   - ./tls.crt:/etc/secretumvault/tls.crt:ro
#   - ./tls.key:/etc/secretumvault/tls.key:ro

# Update svault.toml:
# [server]
# tls_cert = "/etc/secretumvault/tls.crt"
# tls_key = "/etc/secretumvault/tls.key"

Kubernetes with cert-manager

# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

# Create ClusterIssuer
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx
EOF

# Update Helm values
# vault:
#   tls:
#     enabled: true
#     certManager:
#       enabled: true
#       issuer: letsencrypt-prod

mTLS (Mutual TLS)

For client certificate authentication:

# Generate client certificate
openssl req -x509 -newkey rsa:2048 -keyout client.key -out client.crt \
  -days 365 -nodes -subj "/CN=vault-client"

# Update svault.toml
# [server]
# tls_client_ca = "/etc/secretumvault/client-ca.crt"

# Clients must provide certificate
curl --cert client.crt --key client.key \
  https://localhost:8200/v1/sys/health

Monitoring with Prometheus

Prometheus Configuration

Prometheus is configured to scrape vault metrics every 10 seconds:

scrape_configs:
  - job_name: 'vault'
    static_configs:
      - targets: ['vault:9090']  # Docker Compose
      # Or for Kubernetes:
      # - targets: ['vault.secretumvault.svc.cluster.local:9090']
    scrape_interval: 10s

Accessing Prometheus

Docker Compose:

# Metrics endpoint
curl http://localhost:9090/metrics

# Prometheus UI
open http://localhost:9090

Kubernetes:

# Port-forward
kubectl -n secretumvault port-forward svc/prometheus 9090:9090

# Metrics from vault
curl http://vault.secretumvault.svc.cluster.local:9090/metrics

Available Metrics

• vault_secrets_stored_total - Total secrets stored
• vault_secrets_read_total - Total secrets read
• vault_secrets_deleted_total - Total secrets deleted
• vault_operations_encrypt - Encryption operations
• vault_operations_decrypt - Decryption operations
• vault_operations_sign - Signing operations
• vault_operations_verify - Verification operations
• vault_policies_evaluated - Cedar policies evaluated
• vault_tokens_created - Tokens created
• vault_tokens_revoked - Tokens revoked
• vault_leases_issued - Dynamic secret leases issued
• vault_leases_revoked - Dynamic secret leases revoked
• vault_errors_total - Total errors encountered

Grafana Integration

If monitoring is enabled via Helm:

# Port-forward to Grafana
kubectl -n secretumvault port-forward svc/grafana 3000:3000

# Default login
# Username: admin
# Password: (from values.yaml monitoring.grafana.adminPassword)

# Add Prometheus data source
# http://prometheus:9090

Troubleshooting

Vault Pod Not Starting

# Check pod status
kubectl -n secretumvault describe pod <pod-name>

# Check logs
kubectl -n secretumvault logs <pod-name>

# Check events
kubectl -n secretumvault get events --sort-by='.lastTimestamp'

etcd Connection Issues

# Check etcd service
kubectl -n secretumvault get svc vault-etcd-client

# Check etcd pods
kubectl -n secretumvault get statefulset vault-etcd

# Test connectivity from vault pod
kubectl -n secretumvault exec <vault-pod> -- \
  curl http://vault-etcd-client:2379/health

Storage Backend Connection Error

# Verify ConfigMap
kubectl -n secretumvault get cm vault-config -o yaml

# Check if endpoints match service names
# For etcd: vault-etcd-client:2379
# For SurrealDB: vault-surrealdb-client:8000
# For PostgreSQL: vault-postgresql:5432

High Memory Usage

# Check resource usage
kubectl -n secretumvault top pods

# If memory limit exceeded, increase in Helm values:
# vault:
#   resources:
#     limits:
#       memory: "1Gi"

Metrics Not Appearing

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets

# Check vault metrics endpoint directly
curl http://localhost:9090/metrics

# Verify prometheus port in config
# telemetry.prometheus_port = 9090

Volume Mounting Issues

# Check PVC status
kubectl -n secretumvault get pvc

# Check StorageClass available
kubectl get storageclass

# For development without persistent storage:
# Update etcd StatefulSet to use emptyDir:
# volumes:
# - name: data
#   emptyDir: {}

Vault Initialization Failed

If vault initialization fails, ensure:

1. Vault is unsealed (check /v1/sys/seal-status)
2. Storage backend is accessible
3. Master key can be encrypted/decrypted
4. Sufficient resources available

# Restart vault to retry
kubectl -n secretumvault delete pod <vault-pod>

Production Checklist

• Enable TLS with valid certificates (not self-signed)
• Configure mTLS for client authentication
• Set strong unseal key threshold (2-3 of 5+)
• Store unseal keys securely in external vault (not in version control)
• Enable audit logging for compliance
• Configure Cedar policies for fine-grained access control
• Set up monitoring and alerting
• Configure high availability (3+ replicas for vault)
• Configure persistent storage backend (etcd or PostgreSQL)
• Set resource requests and limits appropriately
• Configure network policies to restrict traffic
• Enable pod security policies
• Set up automated backups
• Test disaster recovery procedures
• Enable secret rotation policies
• Configure lease expiration and revocation

Additional Resources

• Architecture: docs/secretumvault-complete-architecture.md
• Configuration Guide: docs/CONFIGURATION.md
• API Reference: docs/API.md
• Security Guidelines: docs/SECURITY.md