prvng_platform/crates/orchestrator/docs/ssh-key-management.md

# SSH Temporal Key Management System

## Overview

The SSH Temporal Key Management System provides automated generation, deployment, and cleanup of short-lived SSH keys
for secure server access. It eliminates the need for static SSH keys by generating keys on-demand with automatic expiration.

## Features

### Core Features

- **Short-Lived Keys**: Keys expire automatically after a configurable TTL (default: 1 hour)
- **Multiple Key Types**:
  - Dynamic Key Pairs (Ed25519)
  - Vault OTP (One-Time Password)
  - Vault CA-Signed Certificates
- **Automatic Cleanup**: Background task removes expired keys from servers
- **Audit Trail**: All key operations are logged
- **REST API**: HTTP endpoints for integration
- **Nushell CLI**: User-friendly command-line interface

### Security Features

- ✅ Ed25519 keys (modern, secure algorithm)
- ✅ Automatic expiration and cleanup
- ✅ Private keys never stored on disk (only in memory)
- ✅ Vault integration for enterprise scenarios
- ✅ SSH fingerprint tracking
- ✅ Per-key audit logging

## Architecture

```plaintext
┌─────────────────────────────────────────────────
────────────┐
│                   SSH Key Manager                           │
├─────────────────────────────────────────────────
────────────┤
│                                                             │
│  ┌──────────────┐  ┌──────────────┐  
┌──────────────┐       │
│  │ Key Generator│  │ Key Deployer │  │   Temporal   │       │
│  │  (Ed25519)   │  │ (SSH Deploy) │  │   Manager    │       │
│  └──────────────┘  └──────────────┘  
└──────────────┘       │
│                                                             │
│  ┌──────────────┐  ┌──────────────┐                         │
│  │    Vault     │  │ Authorized   │                         │
│  │ SSH Engine   │  │ Keys Manager │                         │
│  └──────────────┘  └──────────────┘                         │
│                                                             │
└─────────────────────────────────────────────────
────────────┘
         │                    │                    │
         ▼                    ▼                    ▼
    REST API            Nushell CLI          Background Tasks
```text

## Key Types

### 1. Dynamic Key Pairs (Default)

Generated on-demand Ed25519 keys that are automatically deployed and cleaned up.

**Use Case**: Quick SSH access without Vault infrastructure

**Example**:

```bash
ssh generate-key server.example.com --user root --ttl 30min
```text

### 2. Vault OTP (One-Time Password)

Vault generates a one-time password for SSH authentication.

**Use Case**: Single-use SSH access with centralized authentication

**Requirements**: Vault with SSH secrets engine in OTP mode

**Example**:

```bash
ssh generate-key server.example.com --type otp --ip 192.168.1.100
```text

### 3. Vault CA-Signed Certificates

Vault acts as SSH CA, signing user public keys with short TTL.

**Use Case**: Enterprise scenarios with SSH CA infrastructure

**Requirements**: Vault with SSH secrets engine in CA mode

**Example**:

```bash
ssh generate-key server.example.com --type ca --principal admin --ttl 1hr
```text

## REST API Endpoints

Base URL: `http://localhost:9090`

### Generate SSH Key

```http
POST /api/v1/ssh/generate

{
  "key_type": "dynamickeypair",  // or "otp", "certificate"
  "user": "root",
  "target_server": "server.example.com",
  "ttl_seconds": 3600,
  "allowed_ip": "192.168.1.100",  // optional, for OTP
  "principal": "admin"             // optional, for CA
}

Response:
{
  "success": true,
  "data": {
    "id": "uuid",
    "key_type": "dynamickeypair",
    "public_key": "ssh-ed25519 AAAA...",
    "private_key": "-----BEGIN OPENSSH PRIVATE KEY-----...",
    "fingerprint": "SHA256:...",
    "user": "root",
    "target_server": "server.example.com",
    "created_at": "2024-01-01T00:00:00Z",
    "expires_at": "2024-01-01T01:00:00Z",
    "deployed": false
  }
}
```text

### Deploy SSH Key

```http
POST /api/v1/ssh/{key_id}/deploy

Response:
{
  "success": true,
  "data": {
    "key_id": "uuid",
    "server": "server.example.com",
    "success": true,
    "deployed_at": "2024-01-01T00:00:00Z"
  }
}
```text

### List SSH Keys

```http
GET /api/v1/ssh/keys

Response:
{
  "success": true,
  "data": [
    {
      "id": "uuid",
      "key_type": "dynamickeypair",
      "user": "root",
      "target_server": "server.example.com",
      "expires_at": "2024-01-01T01:00:00Z",
      "deployed": true
    }
  ]
}
```text

### Revoke SSH Key

```http
POST /api/v1/ssh/{key_id}/revoke

Response:
{
  "success": true,
  "data": "Key uuid revoked successfully"
}
```text

### Get SSH Key

```http
GET /api/v1/ssh/{key_id}

Response:
{
  "success": true,
  "data": {
    "id": "uuid",
    "key_type": "dynamickeypair",
    ...
  }
}
```text

### Cleanup Expired Keys

```http
POST /api/v1/ssh/cleanup

Response:
{
  "success": true,
  "data": {
    "cleaned_count": 5,
    "cleaned_key_ids": ["uuid1", "uuid2", ...]
  }
}
```text

### Get Statistics

```http
GET /api/v1/ssh/stats

Response:
{
  "success": true,
  "data": {
    "total_generated": 42,
    "active_keys": 10,
    "expired_keys": 32,
    "keys_by_type": {
      "dynamic": 35,
      "otp": 5,
      "certificate": 2
    },
    "last_cleanup_count": 5,
    "last_cleanup_at": "2024-01-01T00:00:00Z"
  }
}
```text

## Nushell CLI Commands

### Generate Key

```bash
ssh generate-key <server> [options]

Options:
  --user <name>                 SSH user (default: root)
  --ttl <duration>              Key lifetime (default: 1hr)
  --type <ca|otp|dynamic>       Key type (default: dynamic)
  --ip <address>                Allowed IP (OTP mode)
  --principal <name>            Principal (CA mode)

Examples:
  ssh generate-key server.example.com
  ssh generate-key server.example.com --user deploy --ttl 30min
  ssh generate-key server.example.com --type ca --principal admin
```text

### Deploy Key

```bash
ssh deploy-key <key_id>

Example:
  ssh deploy-key abc-123-def-456
```text

### List Keys

```bash
ssh list-keys [--expired]

Example:
  ssh list-keys
  ssh list-keys | where deployed == true
```text

### Revoke Key

```bash
ssh revoke-key <key_id>

Example:
  ssh revoke-key abc-123-def-456
```text

### Connect with Auto-Generated Key

```bash
ssh connect <server> [options]

Options:
  --user <name>                 SSH user (default: root)
  --ttl <duration>              Key lifetime (default: 1hr)
  --type <ca|otp|dynamic>       Key type (default: dynamic)
  --keep                        Keep key after disconnect

Example:
  ssh connect server.example.com --user deploy
```text

This command:

1. Generates a temporal SSH key
2. Deploys it to the server
3. Opens SSH connection
4. Revokes the key after disconnect (unless --keep is used)

### Show Statistics

```bash
ssh stats

Example output:
  SSH Key Statistics:
    Total generated: 42
    Active keys: 10
    Expired keys: 32

  Keys by type:
    dynamic: 35
    otp: 5
    certificate: 2

  Last cleanup: 2024-01-01T00:00:00Z
    Cleaned keys: 5
```text

### Manual Cleanup

```bash
ssh cleanup

Example output:
  ✓ Cleaned up 5 expired keys
  Cleaned key IDs:
    - abc-123
    - def-456
    ...
```text

## Configuration

### Orchestrator Configuration

Add to orchestrator startup:

```rust
use provisioning_orchestrator::{SshKeyManager, SshConfig};

// Create SSH configuration
let ssh_config = SshConfig {
    vault_enabled: false,           // Enable Vault integration
    vault_addr: None,               // Vault address
    vault_token: None,              // Vault token
    vault_mount_point: "ssh".to_string(),
    vault_mode: "ca".to_string(),   // "ca" or "otp"
    default_ttl: Duration::hours(1),
    cleanup_interval: Duration::minutes(5),
    provisioning_key_path: Some("/path/to/provisioning/key".to_string()),
};

// Create SSH key manager
let ssh_manager = Arc::new(SshKeyManager::new(ssh_config).await?);

// Start background cleanup task
Arc::clone(&ssh_manager).start_cleanup_task().await;
```text

### Vault SSH Configuration

#### OTP Mode

```bash
# Enable SSH secrets engine
vault secrets enable ssh

# Configure OTP role
vault write ssh/roles/otp_key_role \
    key_type=otp \
    default_user=root \
    cidr_list=0.0.0.0/0
```text

#### CA Mode

```bash
# Enable SSH secrets engine
vault secrets enable ssh

# Generate SSH CA
vault write ssh/config/ca generate_signing_key=true

# Configure CA role
vault write ssh/roles/default \
    key_type=ca \
    ttl=1h \
    max_ttl=24h \
    allow_user_certificates=true \
    allowed_users="*" \
    default_extensions="permit-pty,permit-port-forwarding"

# Get CA public key (add to servers' /etc/ssh/trusted-user-ca-keys.pem)
vault read -field=public_key ssh/config/ca
```text

Server configuration (`/etc/ssh/sshd_config`):

```plaintext
TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem
```text

## Deployment

### Prerequisites

1. **Orchestrator**: Running on port 8080
2. **SSH Access**: Provisioning key for deploying to servers
3. **Vault** (optional): For OTP or CA modes

### Environment Variables

```bash
# Vault integration (optional)
export VAULT_ADDR=https://vault.example.com:8200
export VAULT_TOKEN=your-vault-token

# Provisioning SSH key path
export PROVISIONING_SSH_KEY=/path/to/provisioning/key
```text

### Integration with Workflows

The SSH key manager integrates with existing workflows:

```nushell
# In server creation workflow
let ssh_key = (ssh generate-key $server --ttl 30min)
ssh deploy-key $ssh_key.id

# Execute remote commands
ssh root@$server "install-kubernetes.sh"

# Auto-revoke after workflow
ssh revoke-key $ssh_key.id
```text

## Security Considerations

1. **Private Key Exposure**: Private keys are only shown once during generation
2. **Key Storage**: Keys stored in memory only, not on disk
3. **Cleanup**: Automatic cleanup removes expired keys from servers
4. **Audit Logging**: All operations logged for security audit
5. **Vault Integration**: Optional Vault integration for enterprise security
6. **TTL Limits**: Enforce maximum TTL to prevent long-lived keys

## Troubleshooting

### Key Deployment Fails

Check SSH connectivity:

```bash
ssh -i /path/to/provisioning/key root@server.example.com
```text

Verify SSH daemon is running:

```bash
systemctl status sshd
```text

### Cleanup Not Working

Check orchestrator logs:

```bash
tail -f ./data/orchestrator.log | grep SSH
```text

Manual cleanup:

```bash
ssh cleanup
```text

### Vault Integration Issues

Test Vault connectivity:

```bash
vault status
vault token lookup
```text

Check SSH secrets engine:

```bash
vault secrets list
vault read ssh/config/ca
```text

## Performance

- **Key Generation**: <100ms (Ed25519)
- **Key Deployment**: ~1s (depends on SSH latency)
- **Cleanup Task**: Every 5 minutes (configurable)
- **Concurrent Keys**: Unlimited (memory bound)

## Future Enhancements

- [ ] SSH certificate rotation
- [ ] Integration with KMS for key encryption
- [ ] WebSocket notifications for key expiration
- [ ] Prometheus metrics export
- [ ] SSH session recording
- [ ] Role-based key generation policies

## References

- RFC 8709: Ed25519 and Ed448 Public Key Algorithms for SSH
- Vault SSH Secrets Engine: <https://www.vaultproject.io/docs/secrets/ssh>
- OpenSSH Certificate Authentication: <https://man.openbsd.org/ssh-keygen>