# SecretumVault KMS Backend Guide

SecretumVault is an enterprise-grade, post-quantum ready secrets management system integrated as the fourth KMS backend in the provisioning platform, alongside Age (dev), Cosmian (prod), and RustyVault (self-hosted).

## Overview

### What is SecretumVault

SecretumVault provides:

- **Post-Quantum Cryptography**: Ready for quantum-resistant algorithms
- **Enterprise Features**: Policy-as-code (Cedar), audit logging, compliance tracking
- **Multiple Storage Backends**: Filesystem (dev), SurrealDB (staging), etcd (prod), PostgreSQL
- **Transit Engine**: Encryption-as-a-service for data protection
- **KV Engine**: Versioned secret storage with rotation policies
- **High Availability**: Seamless transition from embedded to distributed modes

### When to Use SecretumVault

| Scenario | Backend | Reason |
|----------|---------|--------|
| Local development | Age | Simple, no dependencies |
| Testing/Staging | SecretumVault | Enterprise features, production-like |
| Production | Cosmian or SecretumVault | Enterprise security, compliance |
| Self-Hosted Enterprise | SecretumVault + etcd | Full control, HA support |

## Deployment Modes

### Development Mode (Embedded)

**Storage**: Filesystem (`~/.config/provisioning/secretumvault/data`)
**Performance**: <3 ms encryption/decryption
**Setup**: No separate service required
**Best For**: Local development and testing

```bash
export PROVISIONING_ENV=dev
export KMS_DEV_BACKEND=secretumvault
provisioning kms encrypt config.yaml
```

### Staging Mode (Service + SurrealDB)

**Storage**: SurrealDB (document database)
**Performance**: <10 ms operations
**Setup**: Start SecretumVault service separately
**Best For**: Team testing, staging environments

```bash
# Start SecretumVault service
secretumvault server --storage-backend surrealdb

# Configure provisioning
export PROVISIONING_ENV=staging
export SECRETUMVAULT_URL=http://localhost:8200
export SECRETUMVAULT_TOKEN=your-auth-token

provisioning kms encrypt config.yaml
```

### Production Mode (Service + etcd)

**Storage**: etcd cluster (3+ nodes)
**Performance**: <10 ms operations (ninety-ninth percentile)
**Setup**: etcd cluster + SecretumVault service
**Best For**: Production deployments with HA requirements

```bash
# Setup etcd cluster (3 nodes minimum)
etcd --name etcd1 --data-dir etcd1-data \
     --advertise-client-urls http://localhost:2379 \
     --listen-client-urls http://localhost:2379

# Start SecretumVault with etcd
secretumvault server \
  --storage-backend etcd \
  --etcd-endpoints http://etcd1:2379,http://etcd2:2379,http://etcd3:2379

# Configure provisioning
export PROVISIONING_ENV=prod
export SECRETUMVAULT_URL=https://your-secretumvault:8200
export SECRETUMVAULT_TOKEN=your-auth-token
export SECRETUMVAULT_STORAGE=etcd

provisioning kms encrypt config.yaml
```

## Configuration

### Environment Variables

| Variable | Purpose | Default | Example |
|----------|---------|---------|---------|
| `PROVISIONING_ENV` | Deployment environment | `dev` | `staging`, `prod` |
| `KMS_DEV_BACKEND` | Development KMS backend | `age` | `secretumvault` |
| `KMS_STAGING_BACKEND` | Staging KMS backend | `secretumvault` | `cosmian` |
| `KMS_PROD_BACKEND` | Production KMS backend | `cosmian` | `secretumvault` |
| `SECRETUMVAULT_URL` | Server URL | `http://localhost:8200` | `https://kms.example.com` |
| `SECRETUMVAULT_TOKEN` | Authentication token | (none) | (Bearer token) |
| `SECRETUMVAULT_STORAGE` | Storage backend | `filesystem` | `surrealdb`, `etcd` |
| `SECRETUMVAULT_TLS_VERIFY` | Verify TLS certificates | `false` | `true` |

### Configuration Files

**System Defaults**: `provisioning/config/secretumvault.toml`
**KMS Config**: `provisioning/config/kms.toml`

Edit these files to customize:

- Engine mount points
- Key names
- Storage backend settings
- Performance tuning
- Audit logging
- Key rotation policies

## Operations

### Encrypt Data

```bash
# Encrypt a file
provisioning kms encrypt config.yaml
# Output: config.yaml.enc

# Encrypt with specific key
provisioning kms encrypt --key-id my-key config.yaml

# Encrypt and sign
provisioning kms encrypt --sign config.yaml
```

### Decrypt Data

```bash
# Decrypt a file
provisioning kms decrypt config.yaml.enc
# Output: config.yaml

# Decrypt with specific key
provisioning kms decrypt --key-id my-key config.yaml.enc

# Verify and decrypt
provisioning kms decrypt --verify config.yaml.enc
```

### Generate Data Keys

```bash
# Generate AES-256 data key
provisioning kms generate-key --spec AES256

# Generate AES-128 data key
provisioning kms generate-key --spec AES128

# Generate RSA-4096 key
provisioning kms generate-key --spec RSA4096
```

### Health and Status

```bash
# Check KMS health
provisioning kms health

# Get KMS version
provisioning kms version

# Detailed KMS status
provisioning kms status
```

### Key Rotation

```bash
# Rotate encryption key
provisioning kms rotate-key provisioning-master

# Check rotation policy
provisioning kms rotation-policy provisioning-master

# Update rotation interval
provisioning kms update-rotation 90  # Rotate every 90 days
```

## Storage Backends

### Filesystem (Development)

Local file-based storage with no external dependencies.

**Pros**:

- Zero external dependencies
- Fast (local disk access)
- Easy to inspect/backup

**Cons**:

- Single-node only
- No HA
- Manual backup required

**Configuration**:

```toml
[secretumvault.storage.filesystem]
data_dir = "~/.config/provisioning/secretumvault/data"
permissions = "0700"
```

### SurrealDB (Staging)

Embedded or standalone document database.

**Pros**:

- Embedded or distributed
- Flexible schema
- Real-time syncing

**Cons**:

- More complex than filesystem
- New technology (less tested than etcd)

**Configuration**:

```toml
[secretumvault.storage.surrealdb]
connection_url = "ws://localhost:8000"
namespace = "provisioning"
database = "secrets"
username = "${SECRETUMVAULT_SURREALDB_USER:-admin}"
password = "${SECRETUMVAULT_SURREALDB_PASS:-password}"
```

### etcd (Production)

Distributed key-value store for high availability.

**Pros**:

- Proven in production
- HA and disaster recovery
- Consistent consensus protocol
- Multi-site replication

**Cons**:

- Operational complexity
- Requires 3+ nodes
- More infrastructure

**Configuration**:

```toml
[secretumvault.storage.etcd]
endpoints = ["http://etcd1:2379", "http://etcd2:2379", "http://etcd3:2379"]
tls_enabled = true
tls_cert_file = "/path/to/client.crt"
tls_key_file = "/path/to/client.key"
```

### PostgreSQL (Enterprise)

Relational database backend.

**Pros**:

- Mature and reliable
- Advanced querying
- Full ACID transactions

**Cons**:

- Schema requirements
- External database dependency
- More operational overhead

**Configuration**:

```toml
[secretumvault.storage.postgresql]
connection_url = "postgresql://user:pass@localhost:5432/secretumvault"
max_connections = 10
ssl_mode = "require"
```

## Troubleshooting

### Connection Errors

**Error**: "Failed to connect to SecretumVault service"

**Solutions**:

1. Verify SecretumVault is running:

   ```bash
   curl http://localhost:8200/v1/sys/health
   ```

2. Check server URL configuration:

   ```bash
   provisioning config show secretumvault.server_url
   ```

3. Verify network connectivity:

   ```bash
   nc -zv localhost 8200
   ```

### Authentication Failures

**Error**: "Authentication failed: X-Vault-Token missing or invalid"

**Solutions**:

1. Set authentication token:

   ```bash
   export SECRETUMVAULT_TOKEN=your-token
   ```

2. Verify token is still valid:

   ```bash
   provisioning secrets verify-token
   ```

3. Get new token from SecretumVault:

   ```bash
   secretumvault auth login
   ```

### Storage Backend Errors

#### Filesystem Backend

**Error**: "Permission denied: ~/.config/provisioning/secretumvault/data"

**Solution**: Check directory permissions:

```bash
ls -la ~/.config/provisioning/secretumvault/
# Should be: drwx------ (0700)
chmod 700 ~/.config/provisioning/secretumvault/data
```

#### SurrealDB Backend

**Error**: "Failed to connect to SurrealDB at ws://localhost:8000"

**Solution**: Start SurrealDB first:

```bash
surreal start --bind 0.0.0.0:8000 file://secretum.db
```

#### etcd Backend

**Error**: "etcd cluster unhealthy"

**Solution**: Check etcd cluster status:

```bash
etcdctl member list
etcdctl endpoint health

# Verify all nodes are reachable
curl http://etcd1:2379/health
curl http://etcd2:2379/health
curl http://etcd3:2379/health
```

### Performance Issues

**Slow encryption/decryption**:

1. Check network latency (for service mode):

   ```bash
   ping -c 3 secretumvault-server
   ```

2. Monitor SecretumVault performance:

   ```bash
   provisioning kms metrics
   ```

3. Check storage backend performance:
   - Filesystem: Check disk I/O
   - SurrealDB: Monitor database load
   - etcd: Check cluster consensus state

**High memory usage**:

1. Check cache settings:

   ```bash
   provisioning config show secretumvault.performance.cache_ttl
   ```

2. Reduce cache TTL:

   ```bash
   provisioning config set secretumvault.performance.cache_ttl 60
   ```

3. Monitor active connections:

   ```bash
   provisioning kms status
   ```

### Debugging

**Enable debug logging**:

```bash
export RUST_LOG=debug
provisioning kms encrypt config.yaml
```

**Check configuration**:

```bash
provisioning config show secretumvault
provisioning config validate
```

**Test connectivity**:

```bash
provisioning kms health --verbose
```

**View audit logs**:

```bash
tail -f ~/.config/provisioning/logs/secretumvault-audit.log
```

## Security Best Practices

### Token Management

- Never commit tokens to version control
- Use environment variables or `.env` files (gitignored)
- Rotate tokens regularly
- Use different tokens per environment

### TLS/SSL

- Enable TLS verification in production:

  ```bash
  export SECRETUMVAULT_TLS_VERIFY=true
  ```

- Use proper certificates (not self-signed in production)
- Pin certificates to prevent MITM attacks

### Access Control

- Restrict who can access SecretumVault admin UI
- Use strong authentication (MFA preferred)
- Audit all secrets access
- Implement least-privilege principle

### Key Rotation

- Rotate keys regularly (every 90 days recommended)
- Keep old versions for decryption
- Test rotation procedures in staging first
- Monitor rotation status

### Backup and Recovery

- Backup SecretumVault data regularly
- Test restore procedures
- Store backups securely
- Keep backup keys separate from encrypted data

## Migration Guide

### From Age to SecretumVault

```bash
# Export all secrets encrypted with Age
provisioning secrets export --backend age --output secrets.json

# Import into SecretumVault
provisioning secrets import --backend secretumvault secrets.json

# Re-encrypt all configurations
find workspace/infra -name "*.enc" -exec provisioning kms reencrypt {} \;
```

### From RustyVault to SecretumVault

```bash
# Both use Vault-compatible APIs, so migration is simpler:
# 1. Ensure SecretumVault keys are available
# 2. Update KMS_PROD_BACKEND=secretumvault
# 3. Test with staging first
# 4. Monitor during transition
```

### From Cosmian to SecretumVault

```bash
# For production migration:
# 1. Set up SecretumVault with etcd backend
# 2. Verify high availability is working
# 3. Run parallel encryption with both systems
# 4. Validate all decryptions work
# 5. Update KMS_PROD_BACKEND=secretumvault
# 6. Monitor closely for 24 hours
# 7. Keep Cosmian as fallback for 7 days
```

## Performance Tuning

### Development (Filesystem)

```toml
[secretumvault.performance]
max_connections = 5
connection_timeout = 5
request_timeout = 30
cache_ttl = 60
```

### Staging (SurrealDB)

```toml
[secretumvault.performance]
max_connections = 20
connection_timeout = 5
request_timeout = 30
cache_ttl = 300
```

### Production (etcd)

```toml
[secretumvault.performance]
max_connections = 50
connection_timeout = 10
request_timeout = 30
cache_ttl = 600
```

## Compliance and Audit

### Audit Logging

All operations are logged:

```bash
# View recent audit events
provisioning kms audit --limit 100

# Export audit logs
provisioning kms audit export --output audit.json

# Audit specific operations
provisioning kms audit --action encrypt --from 24h
```

### Compliance Reports

```bash
# Generate compliance report
provisioning compliance report --backend secretumvault

# GDPR data export
provisioning compliance gdpr-export user@example.com

# SOC2 audit trail
provisioning compliance soc2-export --output soc2-audit.json
```

## Advanced Topics

### Cedar Authorization Policies

Enable fine-grained access control:

```bash
# Enable Cedar integration
provisioning config set secretumvault.authorization.cedar_enabled true

# Define access policies
provisioning policy define-kms-access user@example.com admin
provisioning policy define-kms-access deployer@example.com deploy-only
```

### Key Encryption Keys (KEK)

Configure master key settings:

```bash
# Set KEK rotation interval
provisioning config set secretumvault.rotation.rotation_interval_days 90

# Enable automatic rotation
provisioning config set secretumvault.rotation.auto_rotate true

# Retain old versions for decryption
provisioning config set secretumvault.rotation.retain_old_versions true
```

### Multi-Region Setup

For production deployments across regions:

```bash
# Region 1
export SECRETUMVAULT_URL=https://kms-us-east.example.com
export SECRETUMVAULT_STORAGE=etcd

# Region 2 (for failover)
export SECRETUMVAULT_URL_FALLBACK=https://kms-us-west.example.com
```

## Support and Resources

- **Documentation**: `docs/user/SECRETUMVAULT_KMS_GUIDE.md` (this file)
- **Configuration Template**: `provisioning/config/secretumvault.toml`
- **KMS Configuration**: `provisioning/config/kms.toml`
- **Issues**: Report issues with `provisioning kms debug`
- **Logs**: Check `~/.config/provisioning/logs/secretumvault-*.log`

## See Also

- [Age KMS Guide](guides/AGE_KMS_GUIDE.md) - Simple local encryption
- [Cosmian KMS Guide](guides/COSMIAN_KMS_GUIDE.md) - Enterprise confidential computing
- [RustyVault Guide](RUSTYVAULT_KMS_GUIDE.md) - Self-hosted Vault
- [KMS Overview](KMS_OVERVIEW.md) - KMS backend comparison
