# SecretumVault KMS Backend Guide\n\nSecretumVault is an enterprise-grade, post-quantum ready secrets management system integrated as the fourth KMS backend in the provisioning platform,\nalongside Age (dev), Cosmian (prod), and RustyVault (self-hosted).\n\n## Overview\n\n### What is SecretumVault\n\nSecretumVault provides:\n\n- **Post-Quantum Cryptography**: Ready for quantum-resistant algorithms\n- **Enterprise Features**: Policy-as-code (Cedar), audit logging, compliance tracking\n- **Multiple Storage Backends**: Filesystem (dev), SurrealDB (staging), etcd (prod), PostgreSQL\n- **Transit Engine**: Encryption-as-a-service for data protection\n- **KV Engine**: Versioned secret storage with rotation policies\n- **High Availability**: Seamless transition from embedded to distributed modes\n\n### When to Use SecretumVault\n\n| Scenario | Backend | Reason |\n| ---------- | --------- | -------- |\n| Local development | Age | Simple, no dependencies |\n| Testing/Staging | SecretumVault | Enterprise features, production-like |\n| Production | Cosmian or SecretumVault | Enterprise security, compliance |\n| Self-Hosted Enterprise | SecretumVault + etcd | Full control, HA support |\n\n## Deployment Modes\n\n### Development Mode (Embedded)\n\n**Storage**: Filesystem (`~/.config/provisioning/secretumvault/data`)\n**Performance**: <3 ms encryption/decryption\n**Setup**: No separate service required\n**Best For**: Local development and testing\n\n```\nexport PROVISIONING_ENV=dev\nexport KMS_DEV_BACKEND=secretumvault\nprovisioning kms encrypt config.yaml\n```\n\n### Staging Mode (Service + SurrealDB)\n\n**Storage**: SurrealDB (document database)\n**Performance**: <10 ms operations\n**Setup**: Start SecretumVault service separately\n**Best For**: Team testing, staging environments\n\n```\n# Start SecretumVault service\nsecretumvault server --storage-backend surrealdb\n\n# Configure provisioning\nexport PROVISIONING_ENV=staging\nexport SECRETUMVAULT_URL=http://localhost:8200\nexport SECRETUMVAULT_TOKEN=your-auth-token\n\nprovisioning kms encrypt config.yaml\n```\n\n### Production Mode (Service + etcd)\n\n**Storage**: etcd cluster (3+ nodes)\n**Performance**: <10 ms operations (ninety-ninth percentile)\n**Setup**: etcd cluster + SecretumVault service\n**Best For**: Production deployments with HA requirements\n\n```\n# Setup etcd cluster (3 nodes minimum)\netcd --name etcd1 --data-dir etcd1-data \\n     --advertise-client-urls http://localhost:2379 \\n     --listen-client-urls http://localhost:2379\n\n# Start SecretumVault with etcd\nsecretumvault server \\n  --storage-backend etcd \\n  --etcd-endpoints http://etcd1:2379,http://etcd2:2379,http://etcd3:2379\n\n# Configure provisioning\nexport PROVISIONING_ENV=prod\nexport SECRETUMVAULT_URL=https://your-secretumvault:8200\nexport SECRETUMVAULT_TOKEN=your-auth-token\nexport SECRETUMVAULT_STORAGE=etcd\n\nprovisioning kms encrypt config.yaml\n```\n\n## Configuration\n\n### Environment Variables\n\n| Variable | Purpose | Default | Example |\n| ---------- | --------- | --------- | --------- |\n| `PROVISIONING_ENV` | Deployment environment | `dev` | `staging`, `prod` |\n| `KMS_DEV_BACKEND` | Development KMS backend | `age` | `secretumvault` |\n| `KMS_STAGING_BACKEND` | Staging KMS backend | `secretumvault` | `cosmian` |\n| `KMS_PROD_BACKEND` | Production KMS backend | `cosmian` | `secretumvault` |\n| `SECRETUMVAULT_URL` | Server URL | `http://localhost:8200` | `https://kms.example.com` |\n| `SECRETUMVAULT_TOKEN` | Authentication token | (none) | (Bearer token) |\n| `SECRETUMVAULT_STORAGE` | Storage backend | `filesystem` | `surrealdb`, `etcd` |\n| `SECRETUMVAULT_TLS_VERIFY` | Verify TLS certificates | `false` | `true` |\n\n### Configuration Files\n\n**System Defaults**: `provisioning/config/secretumvault.toml`\n**KMS Config**: `provisioning/config/kms.toml`\n\nEdit these files to customize:\n\n- Engine mount points\n- Key names\n- Storage backend settings\n- Performance tuning\n- Audit logging\n- Key rotation policies\n\n## Operations\n\n### Encrypt Data\n\n```\n# Encrypt a file\nprovisioning kms encrypt config.yaml\n# Output: config.yaml.enc\n\n# Encrypt with specific key\nprovisioning kms encrypt --key-id my-key config.yaml\n\n# Encrypt and sign\nprovisioning kms encrypt --sign config.yaml\n```\n\n### Decrypt Data\n\n```\n# Decrypt a file\nprovisioning kms decrypt config.yaml.enc\n# Output: config.yaml\n\n# Decrypt with specific key\nprovisioning kms decrypt --key-id my-key config.yaml.enc\n\n# Verify and decrypt\nprovisioning kms decrypt --verify config.yaml.enc\n```\n\n### Generate Data Keys\n\n```\n# Generate AES-256 data key\nprovisioning kms generate-key --spec AES256\n\n# Generate AES-128 data key\nprovisioning kms generate-key --spec AES128\n\n# Generate RSA-4096 key\nprovisioning kms generate-key --spec RSA4096\n```\n\n### Health and Status\n\n```\n# Check KMS health\nprovisioning kms health\n\n# Get KMS version\nprovisioning kms version\n\n# Detailed KMS status\nprovisioning kms status\n```\n\n### Key Rotation\n\n```\n# Rotate encryption key\nprovisioning kms rotate-key provisioning-master\n\n# Check rotation policy\nprovisioning kms rotation-policy provisioning-master\n\n# Update rotation interval\nprovisioning kms update-rotation 90  # Rotate every 90 days\n```\n\n## Storage Backends\n\n### Filesystem (Development)\n\nLocal file-based storage with no external dependencies.\n\n**Pros**:\n\n- Zero external dependencies\n- Fast (local disk access)\n- Easy to inspect/backup\n\n**Cons**:\n\n- Single-node only\n- No HA\n- Manual backup required\n\n**Configuration**:\n\n```\n[secretumvault.storage.filesystem]\ndata_dir = "~/.config/provisioning/secretumvault/data"\npermissions = "0700"\n```\n\n### SurrealDB (Staging)\n\nEmbedded or standalone document database.\n\n**Pros**:\n\n- Embedded or distributed\n- Flexible schema\n- Real-time syncing\n\n**Cons**:\n\n- More complex than filesystem\n- New technology (less tested than etcd)\n\n**Configuration**:\n\n```\n[secretumvault.storage.surrealdb]\nconnection_url = "ws://localhost:8000"\nnamespace = "provisioning"\ndatabase = "secrets"\nusername = "${SECRETUMVAULT_SURREALDB_USER:-admin}"\npassword = "${SECRETUMVAULT_SURREALDB_PASS:-password}"\n```\n\n### etcd (Production)\n\nDistributed key-value store for high availability.\n\n**Pros**:\n\n- Proven in production\n- HA and disaster recovery\n- Consistent consensus protocol\n- Multi-site replication\n\n**Cons**:\n\n- Operational complexity\n- Requires 3+ nodes\n- More infrastructure\n\n**Configuration**:\n\n```\n[secretumvault.storage.etcd]\nendpoints = ["http://etcd1:2379", "http://etcd2:2379", "http://etcd3:2379"]\ntls_enabled = true\ntls_cert_file = "/path/to/client.crt"\ntls_key_file = "/path/to/client.key"\n```\n\n### PostgreSQL (Enterprise)\n\nRelational database backend.\n\n**Pros**:\n\n- Mature and reliable\n- Advanced querying\n- Full ACID transactions\n\n**Cons**:\n\n- Schema requirements\n- External database dependency\n- More operational overhead\n\n**Configuration**:\n\n```\n[secretumvault.storage.postgresql]\nconnection_url = "postgresql://user:pass@localhost:5432/secretumvault"\nmax_connections = 10\nssl_mode = "require"\n```\n\n## Troubleshooting\n\n### Connection Errors\n\n**Error**: "Failed to connect to SecretumVault service"\n\n**Solutions**:\n\n1. Verify SecretumVault is running:\n\n   ```bash\n   curl http://localhost:8200/v1/sys/health\n   ```\n\n2. Check server URL configuration:\n\n   ```bash\n   provisioning config show secretumvault.server_url\n   ```\n\n3. Verify network connectivity:\n\n   ```bash\n   nc -zv localhost 8200\n   ```\n\n### Authentication Failures\n\n**Error**: "Authentication failed: X-Vault-Token missing or invalid"\n\n**Solutions**:\n\n1. Set authentication token:\n\n   ```bash\n   export SECRETUMVAULT_TOKEN=your-token\n   ```\n\n2. Verify token is still valid:\n\n   ```bash\n   provisioning secrets verify-token\n   ```\n\n3. Get new token from SecretumVault:\n\n   ```bash\n   secretumvault auth login\n   ```\n\n### Storage Backend Errors\n\n#### Filesystem Backend\n\n**Error**: "Permission denied: ~/.config/provisioning/secretumvault/data"\n\n**Solution**: Check directory permissions:\n\n```\nls -la ~/.config/provisioning/secretumvault/\n# Should be: drwx------ (0700)\nchmod 700 ~/.config/provisioning/secretumvault/data\n```\n\n#### SurrealDB Backend\n\n**Error**: "Failed to connect to SurrealDB at ws://localhost:8000"\n\n**Solution**: Start SurrealDB first:\n\n```\nsurreal start --bind 0.0.0.0:8000 file://secretum.db\n```\n\n#### etcd Backend\n\n**Error**: "etcd cluster unhealthy"\n\n**Solution**: Check etcd cluster status:\n\n```\netcdctl member list\netcdctl endpoint health\n\n# Verify all nodes are reachable\ncurl http://etcd1:2379/health\ncurl http://etcd2:2379/health\ncurl http://etcd3:2379/health\n```\n\n### Performance Issues\n\n**Slow encryption/decryption**:\n\n1. Check network latency (for service mode):\n\n   ```bash\n   ping -c 3 secretumvault-server\n   ```\n\n2. Monitor SecretumVault performance:\n\n   ```bash\n   provisioning kms metrics\n   ```\n\n3. Check storage backend performance:\n   - Filesystem: Check disk I/O\n   - SurrealDB: Monitor database load\n   - etcd: Check cluster consensus state\n\n**High memory usage**:\n\n1. Check cache settings:\n\n   ```bash\n   provisioning config show secretumvault.performance.cache_ttl\n   ```\n\n2. Reduce cache TTL:\n\n   ```bash\n   provisioning config set secretumvault.performance.cache_ttl 60\n   ```\n\n3. Monitor active connections:\n\n   ```bash\n   provisioning kms status\n   ```\n\n### Debugging\n\n**Enable debug logging**:\n\n```\nexport RUST_LOG=debug\nprovisioning kms encrypt config.yaml\n```\n\n**Check configuration**:\n\n```\nprovisioning config show secretumvault\nprovisioning config validate\n```\n\n**Test connectivity**:\n\n```\nprovisioning kms health --verbose\n```\n\n**View audit logs**:\n\n```\ntail -f ~/.config/provisioning/logs/secretumvault-audit.log\n```\n\n## Security Best Practices\n\n### Token Management\n\n- Never commit tokens to version control\n- Use environment variables or `.env` files (gitignored)\n- Rotate tokens regularly\n- Use different tokens per environment\n\n### TLS/SSL\n\n- Enable TLS verification in production:\n\n  ```bash\n  export SECRETUMVAULT_TLS_VERIFY=true\n  ```\n\n- Use proper certificates (not self-signed in production)\n- Pin certificates to prevent MITM attacks\n\n### Access Control\n\n- Restrict who can access SecretumVault admin UI\n- Use strong authentication (MFA preferred)\n- Audit all secrets access\n- Implement least-privilege principle\n\n### Key Rotation\n\n- Rotate keys regularly (every 90 days recommended)\n- Keep old versions for decryption\n- Test rotation procedures in staging first\n- Monitor rotation status\n\n### Backup and Recovery\n\n- Backup SecretumVault data regularly\n- Test restore procedures\n- Store backups securely\n- Keep backup keys separate from encrypted data\n\n## Migration Guide\n\n### From Age to SecretumVault\n\n```\n# Export all secrets encrypted with Age\nprovisioning secrets export --backend age --output secrets.json\n\n# Import into SecretumVault\nprovisioning secrets import --backend secretumvault secrets.json\n\n# Re-encrypt all configurations\nfind workspace/infra -name "*.enc" -exec provisioning kms reencrypt {} \;\n```\n\n### From RustyVault to SecretumVault\n\n```\n# Both use Vault-compatible APIs, so migration is simpler:\n# 1. Ensure SecretumVault keys are available\n# 2. Update KMS_PROD_BACKEND=secretumvault\n# 3. Test with staging first\n# 4. Monitor during transition\n```\n\n### From Cosmian to SecretumVault\n\n```\n# For production migration:\n# 1. Set up SecretumVault with etcd backend\n# 2. Verify high availability is working\n# 3. Run parallel encryption with both systems\n# 4. Validate all decryptions work\n# 5. Update KMS_PROD_BACKEND=secretumvault\n# 6. Monitor closely for 24 hours\n# 7. Keep Cosmian as fallback for 7 days\n```\n\n## Performance Tuning\n\n### Development (Filesystem)\n\n```\n[secretumvault.performance]\nmax_connections = 5\nconnection_timeout = 5\nrequest_timeout = 30\ncache_ttl = 60\n```\n\n### Staging (SurrealDB)\n\n```\n[secretumvault.performance]\nmax_connections = 20\nconnection_timeout = 5\nrequest_timeout = 30\ncache_ttl = 300\n```\n\n### Production (etcd)\n\n```\n[secretumvault.performance]\nmax_connections = 50\nconnection_timeout = 10\nrequest_timeout = 30\ncache_ttl = 600\n```\n\n## Compliance and Audit\n\n### Audit Logging\n\nAll operations are logged:\n\n```\n# View recent audit events\nprovisioning kms audit --limit 100\n\n# Export audit logs\nprovisioning kms audit export --output audit.json\n\n# Audit specific operations\nprovisioning kms audit --action encrypt --from 24h\n```\n\n### Compliance Reports\n\n```\n# Generate compliance report\nprovisioning compliance report --backend secretumvault\n\n# GDPR data export\nprovisioning compliance gdpr-export user@example.com\n\n# SOC2 audit trail\nprovisioning compliance soc2-export --output soc2-audit.json\n```\n\n## Advanced Topics\n\n### Cedar Authorization Policies\n\nEnable fine-grained access control:\n\n```\n# Enable Cedar integration\nprovisioning config set secretumvault.authorization.cedar_enabled true\n\n# Define access policies\nprovisioning policy define-kms-access user@example.com admin\nprovisioning policy define-kms-access deployer@example.com deploy-only\n```\n\n### Key Encryption Keys (KEK)\n\nConfigure master key settings:\n\n```\n# Set KEK rotation interval\nprovisioning config set secretumvault.rotation.rotation_interval_days 90\n\n# Enable automatic rotation\nprovisioning config set secretumvault.rotation.auto_rotate true\n\n# Retain old versions for decryption\nprovisioning config set secretumvault.rotation.retain_old_versions true\n```\n\n### Multi-Region Setup\n\nFor production deployments across regions:\n\n```\n# Region 1\nexport SECRETUMVAULT_URL=https://kms-us-east.example.com\nexport SECRETUMVAULT_STORAGE=etcd\n\n# Region 2 (for failover)\nexport SECRETUMVAULT_URL_FALLBACK=https://kms-us-west.example.com\n```\n\n## Support and Resources\n\n- **Documentation**: `docs/user/SECRETUMVAULT_KMS_GUIDE.md` (this file)\n- **Configuration Template**: `provisioning/config/secretumvault.toml`\n- **KMS Configuration**: `provisioning/config/kms.toml`\n- **Issues**: Report issues with `provisioning kms debug`\n- **Logs**: Check `~/.config/provisioning/logs/secretumvault-*.log`\n\n## See Also\n\n- [Age KMS Guide](guides/AGE_KMS_GUIDE.md) - Simple local encryption\n- [Cosmian KMS Guide](guides/COSMIAN_KMS_GUIDE.md) - Enterprise confidential computing\n- [RustyVault Guide](RUSTYVAULT_KMS_GUIDE.md) - Self-hosted Vault\n- [KMS Overview](KMS_OVERVIEW.md) - KMS backend comparison
