secretumvault/DEPLOYMENT.md

SecretumVault Deployment Guide

This guide covers deployment of SecretumVault using Docker, Docker Compose, Kubernetes, and Helm.

Table of Contents

1. Local Development with Docker Compose
2. Kubernetes Deployment
3. Helm Installation
4. Configuration
5. Initializing and Unsealing
6. Accessing the API
7. TLS Configuration
8. Monitoring with Prometheus
9. Troubleshooting

Local Development with Docker Compose

Prerequisites

• Docker 20.10+
• Docker Compose 2.0+

Quick Start

# Build the vault image
docker build -t secretumvault:latest .

# Start all services
docker-compose up -d

# Verify services are running
docker-compose ps

# View logs
docker-compose logs -f vault

Services Included

The docker-compose.yml includes:

• vault: SecretumVault server (port 8200 API, 9090 metrics)
• etcd: Distributed key-value store for secrets (port 2379)
• surrealdb: Alternative database backend (port 8000)
• postgres: PostgreSQL for dynamic secrets (port 5432)
• prometheus: Metrics scraping and storage (port 9090)
• grafana: Metrics visualization (port 3000)

Configuration

Configuration is mounted from docker/config/svault.toml. Modify this file to:

• Change storage backend: backend = "etcd" or "surrealdb" or "postgresql"
• Change crypto backend: crypto_backend = "openssl" or "aws-lc"
• Enable/disable engines in the [engines] section
• Adjust logging level: level = "info"

Health Check

# Check vault health
curl http://localhost:8200/v1/sys/health

# Check etcd health
docker-compose exec etcd etcdctl --endpoints=http://localhost:2379 endpoint health

Cleanup

# Stop all services
docker-compose down

# Remove volumes (WARNING: deletes all data)
docker-compose down -v

Kubernetes Deployment

Prerequisites

• Kubernetes 1.20+
• kubectl configured with cluster access
• StorageClass available for persistent volumes
• 2+ CPU and 2Gi RAM available cluster-wide

Quick Start

# Deploy to 'secretumvault' namespace
kubectl apply -f k8s/01-namespace.yaml
kubectl apply -f k8s/02-configmap.yaml
kubectl apply -f k8s/03-deployment.yaml
kubectl apply -f k8s/04-service.yaml
kubectl apply -f k8s/05-etcd.yaml

# Optional: Additional storage backends
kubectl apply -f k8s/06-surrealdb.yaml
kubectl apply -f k8s/07-postgresql.yaml

# Verify deployment
kubectl -n secretumvault get pods -w
kubectl -n secretumvault get svc

Accessing Vault

From within cluster:

# Using ClusterIP service
curl http://vault:8200/v1/sys/health

# Using headless service (direct pod access)
curl http://vault-headless:8200/v1/sys/health

Port-forward from outside cluster:

kubectl -n secretumvault port-forward svc/vault 8200:8200
curl http://localhost:8200/v1/sys/health

Configuring Secrets

To pass database password or other secrets:

# Create secret for PostgreSQL
kubectl -n secretumvault create secret generic vault-postgresql-secret \
  --from-literal=password='your-secure-password'

# Create secret for SurrealDB
kubectl -n secretumvault create secret generic vault-surrealdb-secret \
  --from-literal=password='your-secure-password'

# Create secret for etcd (if authentication enabled)
kubectl -n secretumvault create secret generic vault-etcd-secret \
  --from-literal=password='your-secure-password'

Scaling etcd

etcd is deployed as a StatefulSet with 3 replicas for high availability:

# View etcd pods
kubectl -n secretumvault get statefulset vault-etcd

# Scale if needed
kubectl -n secretumvault scale statefulset vault-etcd --replicas=5

Cleanup

# Delete all vault resources
kubectl delete namespace secretumvault

# Or delete individually
kubectl delete -f k8s/

Helm Installation

Prerequisites

• Helm 3.0+
• kubectl configured with cluster access

Installation

# Add repository (if using remote repo)
# helm repo add secretumvault https://charts.secretumvault.io
# helm repo update

# Install from local chart
helm install vault helm/ \
  --namespace secretumvault \
  --create-namespace

# Or with custom values
helm install vault helm/ \
  --namespace secretumvault \
  --create-namespace \
  --values helm/custom-values.yaml

Upgrade

# List releases
helm list -n secretumvault

# Upgrade deployment
helm upgrade vault helm/ -n secretumvault

# Rollback to previous release
helm rollback vault -n secretumvault

Customization

Customize deployment via values overrides:

# Enable SurrealDB backend
helm install vault helm/ -n secretumvault --create-namespace \
  --set vault.config.storageBackend=surrealdb \
  --set surrealdb.enabled=true

# Enable PostgreSQL for dynamic secrets
helm install vault helm/ -n secretumvault --create-namespace \
  --set postgresql.enabled=true \
  --set vault.config.engines.database=true

# Enable monitoring
helm install vault helm/ -n secretumvault --create-namespace \
  --set monitoring.prometheus.enabled=true \
  --set monitoring.grafana.enabled=true

# Change vault replicas
helm install vault helm/ -n secretumvault --create-namespace \
  --set vault.replicas=3

Uninstall

# Remove Helm release
helm uninstall vault -n secretumvault

# Remove namespace
kubectl delete namespace secretumvault

Configuration

Configuration File Location

• Docker: /etc/secretumvault/svault.toml (mounted from docker/config/)
• Kubernetes: ConfigMap vault-config (from k8s/02-configmap.yaml)
• Helm: Templated from helm/templates/configmap.yaml (values in helm/values.yaml)

Common Configuration Changes

Switch Storage Backend:

[storage]
backend = "surrealdb"  # or "etcd", "postgresql", "filesystem"

[storage.surrealdb]
url = "ws://vault-surrealdb:8000"
password = "${SURREAL_PASSWORD}"

Change Crypto Backend:

[vault]
crypto_backend = "aws-lc"  # or "openssl", "rustcrypto"

Mount Additional Engines:

[engines.kv]
path = "secret/"
versioned = true

[engines.transit]
path = "transit/"

[engines.pki]
path = "pki/"

[engines.database]
path = "database/"

Adjust Logging:

[logging]
level = "debug"
format = "json"
ansi = true

Telemetry and Metrics:

[telemetry]
prometheus_port = 9090
enable_trace = false

Initializing and Unsealing

Initialize Vault

# HTTP request to initialize
curl -X POST http://localhost:8200/v1/sys/init \
  -H "Content-Type: application/json" \
  -d '{
    "shares": 3,
    "threshold": 2
  }'

# Response contains unseal keys and root token
# Save these securely in a password manager (e.g., Bitwarden, 1Password)

Unseal Vault

To unseal after restart, provide threshold number of unseal keys:

# Unseal with first key
curl -X POST http://localhost:8200/v1/sys/unseal \
  -H "Content-Type: application/json" \
  -d '{"key": "unseal-key-1"}'

# Unseal with second key
curl -X POST http://localhost:8200/v1/sys/unseal \
  -H "Content-Type: application/json" \
  -d '{"key": "unseal-key-2"}'

# Check seal status
curl http://localhost:8200/v1/sys/seal-status

Check Status

# Health endpoint
curl http://localhost:8200/v1/sys/health

# Seal status
curl http://localhost:8200/v1/sys/seal-status

Accessing the API

Authentication

SecretumVault uses token-based authentication. Use the root token obtained during initialization:

export VAULT_TOKEN="root-token-from-initialization"
export VAULT_ADDR="http://localhost:8200"

Example API Calls

Create a secret:

curl -X POST "$VAULT_ADDR/v1/secret/data/myapp" \
  -H "X-Vault-Token: $VAULT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "data": {
      "username": "admin",
      "password": "secret123"
    }
  }'

Read a secret:

curl -X GET "$VAULT_ADDR/v1/secret/data/myapp" \
  -H "X-Vault-Token: $VAULT_TOKEN"

Delete a secret:

curl -X DELETE "$VAULT_ADDR/v1/secret/data/myapp" \
  -H "X-Vault-Token: $VAULT_TOKEN"

List all secrets:

curl -X LIST "$VAULT_ADDR/v1/secret/metadata" \
  -H "X-Vault-Token: $VAULT_TOKEN"

Encrypt with Transit engine:

curl -X POST "$VAULT_ADDR/v1/transit/encrypt/my-key" \
  -H "X-Vault-Token: $VAULT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"plaintext": "dGhlIHF1aWNrIGJyb3duIGZveA=="}'

TLS Configuration

Self-Signed Certificate (Development)

# Generate self-signed cert
openssl req -x509 -newkey rsa:4096 -keyout tls.key -out tls.crt \
  -days 365 -nodes -subj "/CN=localhost"

# In Docker Compose, mount cert and key:
# volumes:
#   - ./tls.crt:/etc/secretumvault/tls.crt:ro
#   - ./tls.key:/etc/secretumvault/tls.key:ro

# Update svault.toml:
# [server]
# tls_cert = "/etc/secretumvault/tls.crt"
# tls_key = "/etc/secretumvault/tls.key"

Kubernetes with cert-manager

# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

# Create ClusterIssuer
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx
EOF

# Update Helm values
# vault:
#   tls:
#     enabled: true
#     certManager:
#       enabled: true
#       issuer: letsencrypt-prod

mTLS (Mutual TLS)

For client certificate authentication:

# Generate client certificate
openssl req -x509 -newkey rsa:2048 -keyout client.key -out client.crt \
  -days 365 -nodes -subj "/CN=vault-client"

# Update svault.toml
# [server]
# tls_client_ca = "/etc/secretumvault/client-ca.crt"

# Clients must provide certificate
curl --cert client.crt --key client.key \
  https://localhost:8200/v1/sys/health

Monitoring with Prometheus

Prometheus Configuration

Prometheus is configured to scrape vault metrics every 10 seconds:

scrape_configs:
  - job_name: 'vault'
    static_configs:
      - targets: ['vault:9090']  # Docker Compose
      # Or for Kubernetes:
      # - targets: ['vault.secretumvault.svc.cluster.local:9090']
    scrape_interval: 10s

Accessing Prometheus

Docker Compose:

# Metrics endpoint
curl http://localhost:9090/metrics

# Prometheus UI
open http://localhost:9090

Kubernetes:

# Port-forward
kubectl -n secretumvault port-forward svc/prometheus 9090:9090

# Metrics from vault
curl http://vault.secretumvault.svc.cluster.local:9090/metrics

Available Metrics

• vault_secrets_stored_total - Total secrets stored
• vault_secrets_read_total - Total secrets read
• vault_secrets_deleted_total - Total secrets deleted
• vault_operations_encrypt - Encryption operations
• vault_operations_decrypt - Decryption operations
• vault_operations_sign - Signing operations
• vault_operations_verify - Verification operations
• vault_policies_evaluated - Cedar policies evaluated
• vault_tokens_created - Tokens created
• vault_tokens_revoked - Tokens revoked
• vault_leases_issued - Dynamic secret leases issued
• vault_leases_revoked - Dynamic secret leases revoked
• vault_errors_total - Total errors encountered

Grafana Integration

If monitoring is enabled via Helm:

# Port-forward to Grafana
kubectl -n secretumvault port-forward svc/grafana 3000:3000

# Default login
# Username: admin
# Password: (from values.yaml monitoring.grafana.adminPassword)

# Add Prometheus data source
# http://prometheus:9090

Troubleshooting

Vault Pod Not Starting

# Check pod status
kubectl -n secretumvault describe pod <pod-name>

# Check logs
kubectl -n secretumvault logs <pod-name>

# Check events
kubectl -n secretumvault get events --sort-by='.lastTimestamp'

etcd Connection Issues

# Check etcd service
kubectl -n secretumvault get svc vault-etcd-client

# Check etcd pods
kubectl -n secretumvault get statefulset vault-etcd

# Test connectivity from vault pod
kubectl -n secretumvault exec <vault-pod> -- \
  curl http://vault-etcd-client:2379/health

Storage Backend Connection Error

# Verify ConfigMap
kubectl -n secretumvault get cm vault-config -o yaml

# Check if endpoints match service names
# For etcd: vault-etcd-client:2379
# For SurrealDB: vault-surrealdb-client:8000
# For PostgreSQL: vault-postgresql:5432

High Memory Usage

# Check resource usage
kubectl -n secretumvault top pods

# If memory limit exceeded, increase in Helm values:
# vault:
#   resources:
#     limits:
#       memory: "1Gi"

Metrics Not Appearing

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets

# Check vault metrics endpoint directly
curl http://localhost:9090/metrics

# Verify prometheus port in config
# telemetry.prometheus_port = 9090

Volume Mounting Issues

# Check PVC status
kubectl -n secretumvault get pvc

# Check StorageClass available
kubectl get storageclass

# For development without persistent storage:
# Update etcd StatefulSet to use emptyDir:
# volumes:
# - name: data
#   emptyDir: {}

Vault Initialization Failed

If vault initialization fails, ensure:

1. Vault is unsealed (check /v1/sys/seal-status)
2. Storage backend is accessible
3. Master key can be encrypted/decrypted
4. Sufficient resources available

# Restart vault to retry
kubectl -n secretumvault delete pod <vault-pod>

Production Checklist

• Enable TLS with valid certificates (not self-signed)
• Configure mTLS for client authentication
• Set strong unseal key threshold (2-3 of 5+)
• Store unseal keys securely in external vault (not in version control)
• Enable audit logging for compliance
• Configure Cedar policies for fine-grained access control
• Set up monitoring and alerting
• Configure high availability (3+ replicas for vault)
• Configure persistent storage backend (etcd or PostgreSQL)
• Set resource requests and limits appropriately
• Configure network policies to restrict traffic
• Enable pod security policies
• Set up automated backups
• Test disaster recovery procedures
• Enable secret rotation policies
• Configure lease expiration and revocation

Additional Resources

• Architecture: docs/secretumvault-complete-architecture.md
• Configuration Guide: docs/CONFIGURATION.md
• API Reference: docs/API.md
• Security Guidelines: docs/SECURITY.md