# Hybrid Key Management System (KMS)\n\nA comprehensive hybrid KMS system built for the control center, supporting local/remote/hybrid modes\nwith intelligent caching, failover, and advanced security features.\n\n## Architecture Overview\n\n### Core Components\n\n1. **KMS Backends**\n   - **Local Backend**: SQLite with AES-256-GCM encryption\n   - **Remote Backend**: Cosmian KMS client integration\n   - **Hybrid Backend**: Intelligent combination of local and remote\n\n2. **Caching System**\n   - **Memory Cache**: In-memory LRU cache with TTL\n   - **Redis Cache**: Distributed caching with automatic failover\n   - **Local File Cache**: Persistent file-based cache for offline scenarios\n\n3. **Security Features**\n   - **Encryption**: AES-256-GCM for local storage\n   - **Key Derivation**: HKDF-SHA256 for master key derivation\n   - **Authentication**: Multiple auth methods (certificate, token, basic, OAuth)\n   - **Audit Logging**: Comprehensive audit trail for all operations\n\n4. **Advanced Features**\n   - **Credential Management**: Automatic injection for cloud providers\n   - **Key Rotation**: Configurable automatic key rotation\n   - **HSM Integration**: Hardware Security Module support\n   - **Zero-Knowledge Proofs**: For sensitive operations\n   - **Migration Tools**: Backend-to-backend key migration\n\n## Configuration\n\n### TOML Configuration Example\n\n```{$detected_lang}\n[kms]\n# Operation mode: local, remote, or hybrid\nmode = "hybrid"\n\n# Local SQLite backend configuration\n[kms.local]\ndatabase_path = "./data/kms.db"\n\n[kms.local.master_key]\nderivation_method = "hkdf"\nsource = "generated"\niterations = 100000\n\n[kms.local.encryption]\ndefault_algorithm = "AES-256-GCM"\nkey_size_bits = 256\nauthenticated = true\n\n[kms.local.backup]\nenabled = true\nbackup_dir = "./backups/kms"\ninterval_hours = 24\nmax_backups = 7\ncompress = true\nencrypt = true\n\n# Remote Cosmian KMS configuration\n[kms.remote]\nserver_url = "https://kms.example.com:9998"\nauth_method = "certificate"\nclient_cert_path = "./certs/client.crt"\nclient_key_path = "./certs/client.key"\nca_cert_path = "./certs/ca.crt"\ntimeout_seconds = 30\nverify_ssl = true\n\n[kms.remote.retry]\nmax_attempts = 3\ninitial_delay_ms = 1000\nmax_delay_ms = 30000\nbackoff_multiplier = 2.0\n\n# Cache configuration\n[kms.cache]\nenabled = true\nbackend = "memory"\ndefault_ttl_seconds = 3600\nmax_size_bytes = 104857600  # 100MB\nlocal_dir = "./cache/kms"\n\n# Credential management\n[kms.credentials]\nenabled = true\n\n[kms.credentials.storage]\nstorage_type = "sqlite"\nencryption_key_id = "credential_encryption_key"\ndatabase_path = "./data/credentials.db"\n\n[kms.credentials.providers.aws]\nname = "AWS"\nprovider_type = "aws"\nrefresh_interval_seconds = 3600\nexpiry_warning_seconds = 300\nregions = ["us-east-1", "eu-west-1"]\n\n[kms.credentials.providers.upcloud]\nname = "UpCloud"\nprovider_type = "upcloud"\nrefresh_interval_seconds = 3600\nexpiry_warning_seconds = 300\nregions = ["fi-hel1", "us-nyc1"]\n\n# Key rotation configuration\n[kms.rotation]\nenabled = true\ninterval_seconds = 2592000  # 30 days\nmax_age_seconds = 7776000   # 90 days\nnotice_seconds = 604800     # 7 days\nschedule = "0 2 * * 0"      # Every Sunday at 2 AM\n\n# Audit configuration\n[kms.audit]\nenabled = true\nbackend = "file"\nretention_days = 90\nlog_level = "info"\ninclude_data = false\nmax_file_size_mb = 100\nformat = "json"\n\n# HSM configuration\n[kms.hsm]\nenabled = false\nhsm_type = "pkcs11"\npkcs11_library = "/usr/lib/libpkcs11.so"\nslot_id = 0\n\n# Zero-knowledge proof configuration\n[kms.zkp]\nenabled = false\nproof_system = "groth16"\nsetup_params_path = "./zkp/setup.params"\n\n# Security policy\n[kms.security]\nrequire_strong_auth = true\nmin_key_length_bits = 256\nmax_key_age_days = 90\nenable_pfs = true\nallowed_algorithms = ["AES-256-GCM", "ChaCha20Poly1305", "RSA-4096", "ECDSA-P384"]\nblocked_algorithms = ["DES", "3DES", "RC4", "MD5"]\npolicy_enforcement = "strict"\n```\n\n## Usage Examples\n\n### Basic KMS Operations\n\n```{$detected_lang}\nuse control_center::kms::{KmsManager, KmsConfig, KeyData, KeyType, KeyAlgorithm, KeyUsage};\n\n// Initialize KMS manager\nlet config = KmsConfig::load_from_file("kms.toml").await?;\nlet mut kms = KmsManager::new(&config).await?;\nkms.initialize().await?;\n\n// Create a new encryption key\nlet key_data = KeyData {\n    key_id: "my-encryption-key".to_string(),\n    key_type: KeyType::Symmetric,\n    algorithm: KeyAlgorithm::Aes256Gcm,\n    usage: KeyUsage {\n        encrypt: true,\n        decrypt: true,\n        ..Default::default()\n    },\n    key_size: 256,\n    key_material: SecretBytes::new(generate_random_key(32)),\n    metadata: KeyMetadata {\n        name: Some("Application Encryption Key".to_string()),\n        description: Some("Key for encrypting application data".to_string()),\n        owner: Some("app-service".to_string()),\n        environment: Some("production".to_string()),\n        ..Default::default()\n    },\n    created_at: Utc::now(),\n    last_accessed: None,\n    expires_at: Some(Utc::now() + chrono::Duration::days(90)),\n    status: KeyStatus::Active,\n    tags: HashMap::from([\n        ("purpose".to_string(), "encryption".to_string()),\n        ("service".to_string(), "app".to_string()),\n    ]),\n};\n\n// Store the key\nlet stored_key_id = kms.store_key(key_data).await?;\nprintln!("Key stored with ID: {}", stored_key_id);\n\n// Encrypt data\nlet plaintext = b"sensitive data to encrypt";\nlet context = HashMap::from([\n    ("service".to_string(), "app".to_string()),\n    ("version".to_string(), "1.0".to_string()),\n]);\nlet ciphertext = kms.encrypt(&stored_key_id, plaintext, Some(context.clone())).await?;\n\n// Decrypt data\nlet decrypted = kms.decrypt(&stored_key_id, &ciphertext, Some(context)).await?;\nassert_eq!(plaintext, decrypted.as_slice());\n\n// Get key information\nif let Some(key_info) = kms.get_key(&stored_key_id).await? {\n    println!("Key algorithm: {:?}", key_info.algorithm);\n    println!("Key status: {:?}", key_info.status);\n    println!("Created: {}", key_info.created_at);\n}\n```\n\n### Provider Credential Management\n\n```{$detected_lang}\nuse control_center::kms::{ProviderCredentials, CredentialType};\n\n// Store AWS credentials\nlet aws_creds = ProviderCredentials {\n    provider: "aws".to_string(),\n    credential_type: CredentialType::AccessKey,\n    access_key: "AKIAIOSFODNN7EXAMPLE".to_string(),\n    secret_key: SecretBytes::new(b"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY".to_vec()),\n    session_token: None,\n    region: Some("us-east-1".to_string()),\n    config: HashMap::new(),\n    expires_at: Some(Utc::now() + chrono::Duration::hours(12)),\n    created_at: Utc::now(),\n};\n\nkms.store_provider_credentials("aws", aws_creds).await?;\n\n// Retrieve credentials for automatic injection\nif let Some(creds) = kms.get_provider_credentials("aws").await? {\n    println!("AWS Access Key: {}", creds.access_key);\n    // Credentials are automatically injected into environment variables\n    // or configuration files based on the injection configuration\n}\n```\n\n### Health Monitoring\n\n```{$detected_lang}\n// Check system health\nlet health = kms.health_check().await?;\nprintln!("KMS Health: {}", health.overall);\nprintln!("Backend Status: {}", health.backend.healthy);\nprintln!("Rotation Status: {}", health.rotation.healthy);\nprintln!("Credentials Status: {}", health.credentials.healthy);\n\n// Get cache statistics\nlet cache_stats = kms.cache.stats().await;\nprintln!("Cache hit rate: {:.2}%", cache_stats.hit_rate() * 100.0);\nprintln!("Cache entries: {}", cache_stats.entry_count);\n```\n\n## Integration with Existing System\n\n### Environment Variable Integration\n\nThe KMS system integrates with the existing environment-based configuration:\n\n```{$detected_lang}\n# Set KMS configuration environment\nexport PROVISIONING_KMS_MODE=hybrid\nexport PROVISIONING_KMS_LOCAL_DATABASE_PATH=/var/lib/provisioning/kms.db\nexport PROVISIONING_KMS_REMOTE_SERVER_URL=https://kms.example.com:9998\nexport PROVISIONING_KMS_CACHE_ENABLED=true\n```\n\n### TOML Configuration Integration\n\nAdd KMS configuration to your existing `config.defaults.toml`:\n\n```{$detected_lang}\n# Extend existing configuration with KMS section\n[kms]\nmode = "local"  # Start with local mode\nlocal.database_path = "{{paths.base}}/data/kms.db"\ncache.enabled = true\ncache.local_dir = "{{paths.base}}/cache/kms"\naudit.enabled = true\n```\n\n### Nushell Integration\n\n```{$detected_lang}\n# Load KMS configuration\ndef kms_config [] {\n    let config_path = $"($env.PROVISIONING_BASE)/config.toml"\n    open $config_path | get kms\n}\n\n# Check KMS health\ndef kms_health [] {\n    http get http://localhost:8080/kms/health | from json\n}\n\n# List available keys\ndef kms_keys [] {\n    http get http://localhost:8080/kms/keys | from json\n}\n```\n\n## Security Considerations\n\n### Key Storage Security\n\n1. **Local Storage**: Keys are encrypted using AES-256-GCM with keys derived from a master key using HKDF\n2. **Remote Storage**: Relies on Cosmian KMS security guarantees\n3. **Cache**: Cached keys have TTL and are encrypted in transit\n4. **Memory**: Key material is zeroed on drop using the `zeroize` crate\n\n### Access Control\n\n1. **Authentication**: Multiple authentication methods supported\n2. **Authorization**: Key usage permissions enforced\n3. **Audit**: All operations logged with detailed context\n4. **Network**: TLS encryption for all remote communications\n\n### Compliance Features\n\n1. **Audit Trail**: Comprehensive logging of all KMS operations\n2. **Key Rotation**: Automatic rotation with configurable policies\n3. **Retention**: Configurable key and audit log retention\n4. **Export Controls**: Keys can be marked as non-exportable\n\n## Deployment Guide\n\n### Production Deployment\n\n1. **Configuration**:\n\n   ```toml\n   [kms]\n   mode = "hybrid"\n\n   [kms.security]\n   policy_enforcement = "strict"\n   require_strong_auth = true\n\n   [kms.audit]\n   enabled = true\n   backend = "database"\n   retention_days = 365\n   ```\n\n2. **Infrastructure**:\n   - Dedicated KMS database server\n   - Redis cluster for caching\n   - HSM for high-security keys\n   - Backup and disaster recovery\n\n3. **Monitoring**:\n   - Health check endpoints\n   - Metrics collection\n   - Alert on key expiration\n   - Audit log monitoring\n\n### Development Setup\n\n1. **Local Development**:\n\n   ```toml\n   [kms]\n   mode = "local"\n\n   [kms.security]\n   policy_enforcement = "permissive"\n\n   [kms.audit]\n   enabled = true\n   backend = "stdout"\n   ```\n\n2. **Testing**:\n\n   ```bash\n   # Run with test configuration\n   PROVISIONING_ENV=test ./control-center\n   ```\n\n## Performance Optimization\n\n### Caching Strategy\n\n1. **Multi-level Caching**: Memory → Redis → Local File\n2. **TTL Management**: Configurable per key type\n3. **Cache Warming**: Preload frequently used keys\n4. **Eviction Policies**: LRU with size-based eviction\n\n### Connection Pooling\n\n1. **Database Connections**: Configurable pool size\n2. **HTTP Connections**: Keep-alive and connection reuse\n3. **Batch Operations**: Bulk key operations where supported\n\n## Troubleshooting\n\n### Common Issues\n\n1. **Connection Failures**: Check network connectivity and certificates\n2. **Permission Errors**: Verify file system permissions for local storage\n3. **Cache Misses**: Monitor cache hit rates and adjust TTL\n4. **Key Rotation**: Check rotation scheduler logs\n\n### Debug Mode\n\n```{$detected_lang}\n# Enable debug logging\nexport PROVISIONING_DEBUG=true\nexport PROVISIONING_LOG_LEVEL=debug\n\n# Run with verbose output\n./control-center --debug\n```\n\n### Health Checks\n\n```{$detected_lang}\n# Check KMS health\ncurl http://localhost:8080/kms/health\n\n# Check individual components\ncurl http://localhost:8080/kms/health/backend\ncurl http://localhost:8080/kms/health/cache\ncurl http://localhost:8080/kms/health/rotation\n```\n\n## Future Enhancements\n\n### Planned Features\n\n1. **Multi-Region Support**: Cross-region key replication\n2. **Key Versioning**: Multiple versions per key with rollback\n3. **Policy Engine**: Fine-grained access control policies\n4. **Metrics Dashboard**: Web UI for monitoring and management\n5. **Integration APIs**: REST and gRPC APIs for external systems\n\n### Experimental Features\n\n1. **Zero-Knowledge Proofs**: For privacy-preserving operations\n2. **Quantum-Resistant Algorithms**: Post-quantum cryptography support\n3. **Federated KMS**: Multi-organization key sharing\n4. **Blockchain Integration**: Immutable audit trails\n\nThis hybrid KMS system provides a solid foundation for secure key management in the control center architecture,\nwith room for future enhancements and customization based on specific requirements.