prvng_platform/crates/orchestrator/docs/ssh-key-management.md

SSH Temporal Key Management System

Overview

The SSH Temporal Key Management System provides automated generation, deployment, and cleanup of short-lived SSH keys for secure server access. It eliminates the need for static SSH keys by generating keys on-demand with automatic expiration.

Features

Core Features

• Short-Lived Keys: Keys expire automatically after a configurable TTL (default: 1 hour)
• Multiple Key Types:
  • Dynamic Key Pairs (Ed25519)
  • Vault OTP (One-Time Password)
  • Vault CA-Signed Certificates
• Automatic Cleanup: Background task removes expired keys from servers
• Audit Trail: All key operations are logged
• REST API: HTTP endpoints for integration
• Nushell CLI: User-friendly command-line interface

Security Features

• ✅ Ed25519 keys (modern, secure algorithm)
• ✅ Automatic expiration and cleanup
• ✅ Private keys never stored on disk (only in memory)
• ✅ Vault integration for enterprise scenarios
• ✅ SSH fingerprint tracking
• ✅ Per-key audit logging

Architecture

┌─────────────────────────────────────────────────
────────────┐
│                   SSH Key Manager                           │
├─────────────────────────────────────────────────
────────────┤
│                                                             │
│  ┌──────────────┐  ┌──────────────┐  
┌──────────────┐       │
│  │ Key Generator│  │ Key Deployer │  │   Temporal   │       │
│  │  (Ed25519)   │  │ (SSH Deploy) │  │   Manager    │       │
│  └──────────────┘  └──────────────┘  
└──────────────┘       │
│                                                             │
│  ┌──────────────┐  ┌──────────────┐                         │
│  │    Vault     │  │ Authorized   │                         │
│  │ SSH Engine   │  │ Keys Manager │                         │
│  └──────────────┘  └──────────────┘                         │
│                                                             │
└─────────────────────────────────────────────────
────────────┘
         │                    │                    │
         ▼                    ▼                    ▼
    REST API            Nushell CLI          Background Tasks

Key Types

1. Dynamic Key Pairs (Default)

Generated on-demand Ed25519 keys that are automatically deployed and cleaned up.

Use Case: Quick SSH access without Vault infrastructure

Example:

ssh generate-key server.example.com --user root --ttl 30min

2. Vault OTP (One-Time Password)

Vault generates a one-time password for SSH authentication.

Use Case: Single-use SSH access with centralized authentication

Requirements: Vault with SSH secrets engine in OTP mode

Example:

ssh generate-key server.example.com --type otp --ip 192.168.1.100

3. Vault CA-Signed Certificates

Vault acts as SSH CA, signing user public keys with short TTL.

Use Case: Enterprise scenarios with SSH CA infrastructure

Requirements: Vault with SSH secrets engine in CA mode

Example:

ssh generate-key server.example.com --type ca --principal admin --ttl 1hr

REST API Endpoints

Base URL: http://localhost:9090

Generate SSH Key

POST /api/v1/ssh/generate

{
  "key_type": "dynamickeypair",  // or "otp", "certificate"
  "user": "root",
  "target_server": "server.example.com",
  "ttl_seconds": 3600,
  "allowed_ip": "192.168.1.100",  // optional, for OTP
  "principal": "admin"             // optional, for CA
}

Response:
{
  "success": true,
  "data": {
    "id": "uuid",
    "key_type": "dynamickeypair",
    "public_key": "ssh-ed25519 AAAA...",
    "private_key": "-----BEGIN OPENSSH PRIVATE KEY-----...",
    "fingerprint": "SHA256:...",
    "user": "root",
    "target_server": "server.example.com",
    "created_at": "2024-01-01T00:00:00Z",
    "expires_at": "2024-01-01T01:00:00Z",
    "deployed": false
  }
}

Deploy SSH Key

POST /api/v1/ssh/{key_id}/deploy

Response:
{
  "success": true,
  "data": {
    "key_id": "uuid",
    "server": "server.example.com",
    "success": true,
    "deployed_at": "2024-01-01T00:00:00Z"
  }
}

List SSH Keys

GET /api/v1/ssh/keys

Response:
{
  "success": true,
  "data": [
    {
      "id": "uuid",
      "key_type": "dynamickeypair",
      "user": "root",
      "target_server": "server.example.com",
      "expires_at": "2024-01-01T01:00:00Z",
      "deployed": true
    }
  ]
}

Revoke SSH Key

POST /api/v1/ssh/{key_id}/revoke

Response:
{
  "success": true,
  "data": "Key uuid revoked successfully"
}

Get SSH Key

GET /api/v1/ssh/{key_id}

Response:
{
  "success": true,
  "data": {
    "id": "uuid",
    "key_type": "dynamickeypair",
    ...
  }
}

Cleanup Expired Keys

POST /api/v1/ssh/cleanup

Response:
{
  "success": true,
  "data": {
    "cleaned_count": 5,
    "cleaned_key_ids": ["uuid1", "uuid2", ...]
  }
}

Get Statistics

GET /api/v1/ssh/stats

Response:
{
  "success": true,
  "data": {
    "total_generated": 42,
    "active_keys": 10,
    "expired_keys": 32,
    "keys_by_type": {
      "dynamic": 35,
      "otp": 5,
      "certificate": 2
    },
    "last_cleanup_count": 5,
    "last_cleanup_at": "2024-01-01T00:00:00Z"
  }
}

Nushell CLI Commands

Generate Key

ssh generate-key <server> [options]

Options:
  --user <name>                 SSH user (default: root)
  --ttl <duration>              Key lifetime (default: 1hr)
  --type <ca|otp|dynamic>       Key type (default: dynamic)
  --ip <address>                Allowed IP (OTP mode)
  --principal <name>            Principal (CA mode)

Examples:
  ssh generate-key server.example.com
  ssh generate-key server.example.com --user deploy --ttl 30min
  ssh generate-key server.example.com --type ca --principal admin

Deploy Key

ssh deploy-key <key_id>

Example:
  ssh deploy-key abc-123-def-456

List Keys

ssh list-keys [--expired]

Example:
  ssh list-keys
  ssh list-keys | where deployed == true

Revoke Key

ssh revoke-key <key_id>

Example:
  ssh revoke-key abc-123-def-456

Connect with Auto-Generated Key

ssh connect <server> [options]

Options:
  --user <name>                 SSH user (default: root)
  --ttl <duration>              Key lifetime (default: 1hr)
  --type <ca|otp|dynamic>       Key type (default: dynamic)
  --keep                        Keep key after disconnect

Example:
  ssh connect server.example.com --user deploy

This command:

1. Generates a temporal SSH key
2. Deploys it to the server
3. Opens SSH connection
4. Revokes the key after disconnect (unless --keep is used)

Show Statistics

ssh stats

Example output:
  SSH Key Statistics:
    Total generated: 42
    Active keys: 10
    Expired keys: 32

  Keys by type:
    dynamic: 35
    otp: 5
    certificate: 2

  Last cleanup: 2024-01-01T00:00:00Z
    Cleaned keys: 5

Manual Cleanup

ssh cleanup

Example output:
  ✓ Cleaned up 5 expired keys
  Cleaned key IDs:
    - abc-123
    - def-456
    ...

Configuration

Orchestrator Configuration

Add to orchestrator startup:

use provisioning_orchestrator::{SshKeyManager, SshConfig};

// Create SSH configuration
let ssh_config = SshConfig {
    vault_enabled: false,           // Enable Vault integration
    vault_addr: None,               // Vault address
    vault_token: None,              // Vault token
    vault_mount_point: "ssh".to_string(),
    vault_mode: "ca".to_string(),   // "ca" or "otp"
    default_ttl: Duration::hours(1),
    cleanup_interval: Duration::minutes(5),
    provisioning_key_path: Some("/path/to/provisioning/key".to_string()),
};

// Create SSH key manager
let ssh_manager = Arc::new(SshKeyManager::new(ssh_config).await?);

// Start background cleanup task
Arc::clone(&ssh_manager).start_cleanup_task().await;

Vault SSH Configuration

OTP Mode

# Enable SSH secrets engine
vault secrets enable ssh

# Configure OTP role
vault write ssh/roles/otp_key_role 
    key_type=otp 
    default_user=root 
    cidr_list=0.0.0.0/0

CA Mode

# Enable SSH secrets engine
vault secrets enable ssh

# Generate SSH CA
vault write ssh/config/ca generate_signing_key=true

# Configure CA role
vault write ssh/roles/default 
    key_type=ca 
    ttl=1h 
    max_ttl=24h 
    allow_user_certificates=true 
    allowed_users="*" 
    default_extensions="permit-pty,permit-port-forwarding"

# Get CA public key (add to servers' /etc/ssh/trusted-user-ca-keys.pem)
vault read -field=public_key ssh/config/ca

Server configuration (/etc/ssh/sshd_config):

TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem

Deployment

Prerequisites

1. Orchestrator: Running on port 8080
2. SSH Access: Provisioning key for deploying to servers
3. Vault (optional): For OTP or CA modes

Environment Variables

# Vault integration (optional)
export VAULT_ADDR=https://vault.example.com:8200
export VAULT_TOKEN=your-vault-token

# Provisioning SSH key path
export PROVISIONING_SSH_KEY=/path/to/provisioning/key

Integration with Workflows

The SSH key manager integrates with existing workflows:

# In server creation workflow
let ssh_key = (ssh generate-key $server --ttl 30min)
ssh deploy-key $ssh_key.id

# Execute remote commands
ssh root@$server "install-kubernetes.sh"

# Auto-revoke after workflow
ssh revoke-key $ssh_key.id

Security Considerations

1. Private Key Exposure: Private keys are only shown once during generation
2. Key Storage: Keys stored in memory only, not on disk
3. Cleanup: Automatic cleanup removes expired keys from servers
4. Audit Logging: All operations logged for security audit
5. Vault Integration: Optional Vault integration for enterprise security
6. TTL Limits: Enforce maximum TTL to prevent long-lived keys

Troubleshooting

Key Deployment Fails

Check SSH connectivity:

ssh -i /path/to/provisioning/key root@server.example.com

Verify SSH daemon is running:

systemctl status sshd

Cleanup Not Working

Check orchestrator logs:

tail -f ./data/orchestrator.log | grep SSH

Manual cleanup:

ssh cleanup

Vault Integration Issues

Test Vault connectivity:

vault status
vault token lookup

Check SSH secrets engine:

vault secrets list
vault read ssh/config/ca

Performance

• Key Generation: <100ms (Ed25519)
• Key Deployment: ~1s (depends on SSH latency)
• Cleanup Task: Every 5 minutes (configurable)
• Concurrent Keys: Unlimited (memory bound)

Future Enhancements

• SSH certificate rotation
• Integration with KMS for key encryption
• WebSocket notifications for key expiration
• Prometheus metrics export
• SSH session recording
• Role-based key generation policies

References

• RFC 8709: Ed25519 and Ed448 Public Key Algorithms for SSH
• Vault SSH Secrets Engine: https://www.vaultproject.io/docs/secrets/ssh
• OpenSSH Certificate Authentication: https://man.openbsd.org/ssh-keygen