# WebSocket API Reference\n\nThis document provides comprehensive documentation for the WebSocket API used for real-time monitoring, event streaming, and live updates in\nprovisioning.\n\n## Overview\n\nThe WebSocket API enables real-time communication between clients and the provisioning orchestrator, providing:\n\n- Live workflow progress updates\n- System health monitoring\n- Event streaming\n- Real-time metrics\n- Interactive debugging sessions\n\n## WebSocket Endpoints\n\n### Primary WebSocket Endpoint\n\n#### `ws://localhost:9090/ws`\n\nThe main WebSocket endpoint for real-time events and monitoring.\n\n**Connection Parameters:**\n\n- `token`: JWT authentication token (required)\n- `events`: Comma-separated list of event types to subscribe to (optional)\n- `batch_size`: Maximum number of events per message (default: 10)\n- `compression`: Enable message compression (default: false)\n\n**Example Connection:**\n\n```\nconst ws = new WebSocket('ws://localhost:9090/ws?token=jwt-token&events=task,batch,system');\n```\n\n### Specialized WebSocket Endpoints\n\n#### `ws://localhost:9090/metrics`\n\nReal-time metrics streaming endpoint.\n\n**Features:**\n\n- Live system metrics\n- Performance data\n- Resource utilization\n- Custom metric streams\n\n#### `ws://localhost:9090/logs`\n\nLive log streaming endpoint.\n\n**Features:**\n\n- Real-time log tailing\n- Log level filtering\n- Component-specific logs\n- Search and filtering\n\n## Authentication\n\n### JWT Token Authentication\n\nAll WebSocket connections require authentication via JWT token:\n\n```\n// Include token in connection URL\nconst ws = new WebSocket('ws://localhost:9090/ws?token=' + jwtToken);\n\n// Or send token after connection\nws.onopen = function() {\n  ws.send(JSON.stringify({\n    type: 'auth',\n    token: jwtToken\n  }));\n};\n```\n\n### Connection Authentication Flow\n\n1. **Initial Connection**: Client connects with token parameter\n2. **Token Validation**: Server validates JWT token\n3. **Authorization**: Server checks token permissions\n4. **Subscription**: Client subscribes to event types\n5. **Event Stream**: Server begins streaming events\n\n## Event Types and Schemas\n\n### Core Event Types\n\n#### Task Status Changed\n\nFired when a workflow task status changes.\n\n```\n{\n  "event_type": "TaskStatusChanged",\n  "timestamp": "2025-09-26T10:00:00Z",\n  "data": {\n    "task_id": "uuid-string",\n    "name": "create_servers",\n    "status": "Running",\n    "previous_status": "Pending",\n    "progress": 45.5\n  },\n  "metadata": {\n    "task_id": "uuid-string",\n    "workflow_type": "server_creation",\n    "infra": "production"\n  }\n}\n```\n\n#### Batch Operation Update\n\nFired when batch operation status changes.\n\n```\n{\n  "event_type": "BatchOperationUpdate",\n  "timestamp": "2025-09-26T10:00:00Z",\n  "data": {\n    "batch_id": "uuid-string",\n    "name": "multi_cloud_deployment",\n    "status": "Running",\n    "progress": 65.0,\n    "operations": [\n      {\n        "id": "upcloud_servers",\n        "status": "Completed",\n        "progress": 100.0\n      },\n      {\n        "id": "aws_taskservs",\n        "status": "Running",\n        "progress": 30.0\n      }\n    ]\n  },\n  "metadata": {\n    "total_operations": 5,\n    "completed_operations": 2,\n    "failed_operations": 0\n  }\n}\n```\n\n#### System Health Update\n\nFired when system health status changes.\n\n```\n{\n  "event_type": "SystemHealthUpdate",\n  "timestamp": "2025-09-26T10:00:00Z",\n  "data": {\n    "overall_status": "Healthy",\n    "components": {\n      "storage": {\n        "status": "Healthy",\n        "last_check": "2025-09-26T09:59:55Z"\n      },\n      "batch_coordinator": {\n        "status": "Warning",\n        "last_check": "2025-09-26T09:59:55Z",\n        "message": "High memory usage"\n      }\n    },\n    "metrics": {\n      "cpu_usage": 45.2,\n      "memory_usage": 2048,\n      "disk_usage": 75.5,\n      "active_workflows": 5\n    }\n  },\n  "metadata": {\n    "check_interval": 30,\n    "next_check": "2025-09-26T10:00:30Z"\n  }\n}\n```\n\n#### Workflow Progress Update\n\nFired when workflow progress changes.\n\n```\n{\n  "event_type": "WorkflowProgressUpdate",\n  "timestamp": "2025-09-26T10:00:00Z",\n  "data": {\n    "workflow_id": "uuid-string",\n    "name": "kubernetes_deployment",\n    "progress": 75.0,\n    "current_step": "Installing CNI",\n    "total_steps": 8,\n    "completed_steps": 6,\n    "estimated_time_remaining": 120,\n    "step_details": {\n      "step_name": "Installing CNI",\n      "step_progress": 45.0,\n      "step_message": "Downloading Cilium components"\n    }\n  },\n  "metadata": {\n    "infra": "production",\n    "provider": "upcloud",\n    "started_at": "2025-09-26T09:45:00Z"\n  }\n}\n```\n\n#### Log Entry\n\nReal-time log streaming.\n\n```\n{\n  "event_type": "LogEntry",\n  "timestamp": "2025-09-26T10:00:00Z",\n  "data": {\n    "level": "INFO",\n    "message": "Server web-01 created successfully",\n    "component": "server-manager",\n    "task_id": "uuid-string",\n    "details": {\n      "server_id": "server-uuid",\n      "hostname": "web-01",\n      "ip_address": "10.0.1.100"\n    }\n  },\n  "metadata": {\n    "source": "orchestrator",\n    "thread": "worker-1"\n  }\n}\n```\n\n#### Metric Update\n\nReal-time metrics streaming.\n\n```\n{\n  "event_type": "MetricUpdate",\n  "timestamp": "2025-09-26T10:00:00Z",\n  "data": {\n    "metric_name": "workflow_duration",\n    "metric_type": "histogram",\n    "value": 180.5,\n    "labels": {\n      "workflow_type": "server_creation",\n      "status": "completed",\n      "infra": "production"\n    }\n  },\n  "metadata": {\n    "interval": 15,\n    "aggregation": "average"\n  }\n}\n```\n\n### Custom Event Types\n\nApplications can define custom event types:\n\n```\n{\n  "event_type": "CustomApplicationEvent",\n  "timestamp": "2025-09-26T10:00:00Z",\n  "data": {\n    // Custom event data\n  },\n  "metadata": {\n    "custom_field": "custom_value"\n  }\n}\n```\n\n## Client-Side JavaScript API\n\n### Connection Management\n\n```\nclass ProvisioningWebSocket {\n  constructor(baseUrl, token, options = {}) {\n    this.baseUrl = baseUrl;\n    this.token = token;\n    this.options = {\n      reconnect: true,\n      reconnectInterval: 5000,\n      maxReconnectAttempts: 10,\n      ...options\n    };\n    this.ws = null;\n    this.reconnectAttempts = 0;\n    this.eventHandlers = new Map();\n  }\n\n  connect() {\n    const wsUrl = `${this.baseUrl}/ws?token=${this.token}`;\n    this.ws = new WebSocket(wsUrl);\n\n    this.ws.onopen = (event) => {\n      console.log('WebSocket connected');\n      this.reconnectAttempts = 0;\n      this.emit('connected', event);\n    };\n\n    this.ws.onmessage = (event) => {\n      try {\n        const message = JSON.parse(event.data);\n        this.handleMessage(message);\n      } catch (error) {\n        console.error('Failed to parse WebSocket message:', error);\n      }\n    };\n\n    this.ws.onclose = (event) => {\n      console.log('WebSocket disconnected');\n      this.emit('disconnected', event);\n\n      if (this.options.reconnect && this.reconnectAttempts < this.options.maxReconnectAttempts) {\n        setTimeout(() => {\n          this.reconnectAttempts++;\n          console.log(`Reconnecting... (${this.reconnectAttempts}/${this.options.maxReconnectAttempts})`);\n          this.connect();\n        }, this.options.reconnectInterval);\n      }\n    };\n\n    this.ws.onerror = (error) => {\n      console.error('WebSocket error:', error);\n      this.emit('error', error);\n    };\n  }\n\n  handleMessage(message) {\n    if (message.event_type) {\n      this.emit(message.event_type, message);\n      this.emit('message', message);\n    }\n  }\n\n  on(eventType, handler) {\n    if (!this.eventHandlers.has(eventType)) {\n      this.eventHandlers.set(eventType, []);\n    }\n    this.eventHandlers.get(eventType).push(handler);\n  }\n\n  off(eventType, handler) {\n    const handlers = this.eventHandlers.get(eventType);\n    if (handlers) {\n      const index = handlers.indexOf(handler);\n      if (index > -1) {\n        handlers.splice(index, 1);\n      }\n    }\n  }\n\n  emit(eventType, data) {\n    const handlers = this.eventHandlers.get(eventType);\n    if (handlers) {\n      handlers.forEach(handler => {\n        try {\n          handler(data);\n        } catch (error) {\n          console.error(`Error in event handler for ${eventType}:`, error);\n        }\n      });\n    }\n  }\n\n  send(message) {\n    if (this.ws && this.ws.readyState === WebSocket.OPEN) {\n      this.ws.send(JSON.stringify(message));\n    } else {\n      console.warn('WebSocket not connected, message not sent');\n    }\n  }\n\n  disconnect() {\n    this.options.reconnect = false;\n    if (this.ws) {\n      this.ws.close();\n    }\n  }\n\n  subscribe(eventTypes) {\n    this.send({\n      type: 'subscribe',\n      events: Array.isArray(eventTypes) ? eventTypes : [eventTypes]\n    });\n  }\n\n  unsubscribe(eventTypes) {\n    this.send({\n      type: 'unsubscribe',\n      events: Array.isArray(eventTypes) ? eventTypes : [eventTypes]\n    });\n  }\n}\n\n// Usage example\nconst ws = new ProvisioningWebSocket('ws://localhost:9090', 'your-jwt-token');\n\nws.on('TaskStatusChanged', (event) => {\n  console.log(`Task ${event.data.task_id} status: ${event.data.status}`);\n  updateTaskUI(event.data);\n});\n\nws.on('WorkflowProgressUpdate', (event) => {\n  console.log(`Workflow progress: ${event.data.progress}%`);\n  updateProgressBar(event.data.progress);\n});\n\nws.on('SystemHealthUpdate', (event) => {\n  console.log('System health:', event.data.overall_status);\n  updateHealthIndicator(event.data);\n});\n\nws.connect();\n\n// Subscribe to specific events\nws.subscribe(['TaskStatusChanged', 'WorkflowProgressUpdate']);\n```\n\n### Real-Time Dashboard Example\n\n```\nclass ProvisioningDashboard {\n  constructor(wsUrl, token) {\n    this.ws = new ProvisioningWebSocket(wsUrl, token);\n    this.setupEventHandlers();\n    this.connect();\n  }\n\n  setupEventHandlers() {\n    this.ws.on('TaskStatusChanged', this.handleTaskUpdate.bind(this));\n    this.ws.on('BatchOperationUpdate', this.handleBatchUpdate.bind(this));\n    this.ws.on('SystemHealthUpdate', this.handleHealthUpdate.bind(this));\n    this.ws.on('WorkflowProgressUpdate', this.handleProgressUpdate.bind(this));\n    this.ws.on('LogEntry', this.handleLogEntry.bind(this));\n  }\n\n  connect() {\n    this.ws.connect();\n  }\n\n  handleTaskUpdate(event) {\n    const taskCard = document.getElementById(`task-${event.data.task_id}`);\n    if (taskCard) {\n      taskCard.querySelector('.status').textContent = event.data.status;\n      taskCard.querySelector('.status').className = `status ${event.data.status.toLowerCase()}`;\n\n      if (event.data.progress) {\n        const progressBar = taskCard.querySelector('.progress-bar');\n        progressBar.style.width = `${event.data.progress}%`;\n      }\n    }\n  }\n\n  handleBatchUpdate(event) {\n    const batchCard = document.getElementById(`batch-${event.data.batch_id}`);\n    if (batchCard) {\n      batchCard.querySelector('.batch-progress').style.width = `${event.data.progress}%`;\n\n      event.data.operations.forEach(op => {\n        const opElement = batchCard.querySelector(`[data-operation="${op.id}"]`);\n        if (opElement) {\n          opElement.querySelector('.operation-status').textContent = op.status;\n          opElement.querySelector('.operation-progress').style.width = `${op.progress}%`;\n        }\n      });\n    }\n  }\n\n  handleHealthUpdate(event) {\n    const healthIndicator = document.getElementById('health-indicator');\n    healthIndicator.className = `health-indicator ${event.data.overall_status.toLowerCase()}`;\n    healthIndicator.textContent = event.data.overall_status;\n\n    const metricsPanel = document.getElementById('metrics-panel');\n    metricsPanel.innerHTML = `\n      <div class="metric">CPU: ${event.data.metrics.cpu_usage}%</div>\n      <div class="metric">Memory: ${Math.round(event.data.metrics.memory_usage / 1024 / 1024)}MB</div>\n      <div class="metric">Disk: ${event.data.metrics.disk_usage}%</div>\n      <div class="metric">Active Workflows: ${event.data.metrics.active_workflows}</div>\n    `;\n  }\n\n  handleProgressUpdate(event) {\n    const workflowCard = document.getElementById(`workflow-${event.data.workflow_id}`);\n    if (workflowCard) {\n      const progressBar = workflowCard.querySelector('.workflow-progress');\n      const stepInfo = workflowCard.querySelector('.step-info');\n\n      progressBar.style.width = `${event.data.progress}%`;\n      stepInfo.textContent = `${event.data.current_step} (${event.data.completed_steps}/${event.data.total_steps})`;\n\n      if (event.data.estimated_time_remaining) {\n        const timeRemaining = workflowCard.querySelector('.time-remaining');\n        timeRemaining.textContent = `${Math.round(event.data.estimated_time_remaining / 60)} min remaining`;\n      }\n    }\n  }\n\n  handleLogEntry(event) {\n    const logContainer = document.getElementById('log-container');\n    const logEntry = document.createElement('div');\n    logEntry.className = `log-entry log-${event.data.level.toLowerCase()}`;\n    logEntry.innerHTML = `\n      <span class="log-timestamp">${new Date(event.timestamp).toLocaleTimeString()}</span>\n      <span class="log-level">${event.data.level}</span>\n      <span class="log-component">${event.data.component}</span>\n      <span class="log-message">${event.data.message}</span>\n    `;\n\n    logContainer.appendChild(logEntry);\n\n    // Auto-scroll to bottom\n    logContainer.scrollTop = logContainer.scrollHeight;\n\n    // Limit log entries to prevent memory issues\n    const maxLogEntries = 1000;\n    if (logContainer.children.length > maxLogEntries) {\n      logContainer.removeChild(logContainer.firstChild);\n    }\n  }\n}\n\n// Initialize dashboard\nconst dashboard = new ProvisioningDashboard('ws://localhost:9090', jwtToken);\n```\n\n## Server-Side Implementation\n\n### Rust WebSocket Handler\n\nThe orchestrator implements WebSocket support using Axum and Tokio:\n\n```\nuse axum::{\n    extract::{ws::WebSocket, ws::WebSocketUpgrade, Query, State},\n    response::Response,\n};\nuse serde::{Deserialize, Serialize};\nuse std::collections::HashMap;\nuse tokio::sync::broadcast;\n\n#[derive(Debug, Deserialize)]\npub struct WsQuery {\n    token: String,\n    events: Option<String>,\n    batch_size: Option<usize>,\n    compression: Option<bool>,\n}\n\n#[derive(Debug, Clone, Serialize)]\npub struct WebSocketMessage {\n    pub event_type: String,\n    pub timestamp: chrono::DateTime<chrono::Utc>,\n    pub data: serde_json::Value,\n    pub metadata: HashMap<String, String>,\n}\n\npub async fn websocket_handler(\n    ws: WebSocketUpgrade,\n    Query(params): Query<WsQuery>,\n    State(state): State<SharedState>,\n) -> Response {\n    // Validate JWT token\n    let claims = match state.auth_service.validate_token(&params.token) {\n        Ok(claims) => claims,\n        Err(_) => return Response::builder()\n            .status(401)\n            .body("Unauthorized".into())\n            .unwrap(),\n    };\n\n    ws.on_upgrade(move |socket| handle_socket(socket, params, claims, state))\n}\n\nasync fn handle_socket(\n    socket: WebSocket,\n    params: WsQuery,\n    claims: Claims,\n    state: SharedState,\n) {\n    let (mut sender, mut receiver) = socket.split();\n\n    // Subscribe to event stream\n    let mut event_rx = state.monitoring_system.subscribe_to_events().await;\n\n    // Parse requested event types\n    let requested_events: Vec<String> = params.events\n        .unwrap_or_default()\n        .split(',')\n        .map(|s| s.trim().to_string())\n        .filter(|s| !s.is_empty())\n        .collect();\n\n    // Handle incoming messages from client\n    let sender_task = tokio::spawn(async move {\n        while let Some(msg) = receiver.next().await {\n            if let Ok(msg) = msg {\n                if let Ok(text) = msg.to_text() {\n                    if let Ok(client_msg) = serde_json::from_str::<ClientMessage>(text) {\n                        handle_client_message(client_msg, &state).await;\n                    }\n                }\n            }\n        }\n    });\n\n    // Handle outgoing messages to client\n    let receiver_task = tokio::spawn(async move {\n        let mut batch = Vec::new();\n        let batch_size = params.batch_size.unwrap_or(10);\n\n        while let Ok(event) = event_rx.recv().await {\n            // Filter events based on subscription\n            if !requested_events.is_empty() && !requested_events.contains(&event.event_type) {\n                continue;\n            }\n\n            // Check permissions\n            if !has_event_permission(&claims, &event.event_type) {\n                continue;\n            }\n\n            batch.push(event);\n\n            // Send batch when full or after timeout\n            if batch.len() >= batch_size {\n                send_event_batch(&mut sender, &batch).await;\n                batch.clear();\n            }\n        }\n    });\n\n    // Wait for either task to complete\n    tokio::select! {\n        _ = sender_task => {},\n        _ = receiver_task => {},\n    }\n}\n\n#[derive(Debug, Deserialize)]\nstruct ClientMessage {\n    #[serde(rename = "type")]\n    msg_type: String,\n    token: Option<String>,\n    events: Option<Vec<String>>,\n}\n\nasync fn handle_client_message(msg: ClientMessage, state: &SharedState) {\n    match msg.msg_type.as_str() {\n        "subscribe" => {\n            // Handle event subscription\n        },\n        "unsubscribe" => {\n            // Handle event unsubscription\n        },\n        "auth" => {\n            // Handle re-authentication\n        },\n        _ => {\n            // Unknown message type\n        }\n    }\n}\n\nasync fn send_event_batch(sender: &mut SplitSink<WebSocket, Message>, batch: &[WebSocketMessage]) {\n    let batch_msg = serde_json::json!({\n        "type": "batch",\n        "events": batch\n    });\n\n    if let Ok(msg_text) = serde_json::to_string(&batch_msg) {\n        if let Err(e) = sender.send(Message::Text(msg_text)).await {\n            eprintln!("Failed to send WebSocket message: {}", e);\n        }\n    }\n}\n\nfn has_event_permission(claims: &Claims, event_type: &str) -> bool {\n    // Check if user has permission to receive this event type\n    match event_type {\n        "SystemHealthUpdate" => claims.role.contains(&"admin".to_string()),\n        "LogEntry" => claims.role.contains(&"admin".to_string()) ||\n                     claims.role.contains(&"developer".to_string()),\n        _ => true, // Most events are accessible to all authenticated users\n    }\n}\n```\n\n## Event Filtering and Subscriptions\n\n### Client-Side Filtering\n\n```\n// Subscribe to specific event types\nws.subscribe(['TaskStatusChanged', 'WorkflowProgressUpdate']);\n\n// Subscribe with filters\nws.send({\n  type: 'subscribe',\n  events: ['TaskStatusChanged'],\n  filters: {\n    task_name: 'create_servers',\n    status: ['Running', 'Completed', 'Failed']\n  }\n});\n\n// Advanced filtering\nws.send({\n  type: 'subscribe',\n  events: ['LogEntry'],\n  filters: {\n    level: ['ERROR', 'WARN'],\n    component: ['server-manager', 'batch-coordinator'],\n    since: '2025-09-26T10:00:00Z'\n  }\n});\n```\n\n### Server-Side Event Filtering\n\nEvents can be filtered on the server side based on:\n\n- User permissions and roles\n- Event type subscriptions\n- Custom filter criteria\n- Rate limiting\n\n## Error Handling and Reconnection\n\n### Connection Errors\n\n```\nws.on('error', (error) => {\n  console.error('WebSocket error:', error);\n\n  // Handle specific error types\n  if (error.code === 1006) {\n    // Abnormal closure, attempt reconnection\n    setTimeout(() => ws.connect(), 5000);\n  } else if (error.code === 1008) {\n    // Policy violation, check token\n    refreshTokenAndReconnect();\n  }\n});\n\nws.on('disconnected', (event) => {\n  console.log(`WebSocket disconnected: ${event.code} - ${event.reason}`);\n\n  // Handle different close codes\n  switch (event.code) {\n    case 1000: // Normal closure\n      console.log('Connection closed normally');\n      break;\n    case 1001: // Going away\n      console.log('Server is shutting down');\n      break;\n    case 4001: // Custom: Token expired\n      refreshTokenAndReconnect();\n      break;\n    default:\n      // Attempt reconnection for other errors\n      if (shouldReconnect()) {\n        scheduleReconnection();\n      }\n  }\n});\n```\n\n### Heartbeat and Keep-Alive\n\n```\nclass ProvisioningWebSocket {\n  constructor(baseUrl, token, options = {}) {\n    // ... existing code ...\n    this.heartbeatInterval = options.heartbeatInterval || 30000;\n    this.heartbeatTimer = null;\n  }\n\n  connect() {\n    // ... existing connection code ...\n\n    this.ws.onopen = (event) => {\n      console.log('WebSocket connected');\n      this.startHeartbeat();\n      this.emit('connected', event);\n    };\n\n    this.ws.onclose = (event) => {\n      this.stopHeartbeat();\n      // ... existing close handling ...\n    };\n  }\n\n  startHeartbeat() {\n    this.heartbeatTimer = setInterval(() => {\n      if (this.ws && this.ws.readyState === WebSocket.OPEN) {\n        this.send({ type: 'ping' });\n      }\n    }, this.heartbeatInterval);\n  }\n\n  stopHeartbeat() {\n    if (this.heartbeatTimer) {\n      clearInterval(this.heartbeatTimer);\n      this.heartbeatTimer = null;\n    }\n  }\n\n  handleMessage(message) {\n    if (message.type === 'pong') {\n      // Heartbeat response received\n      return;\n    }\n\n    // ... existing message handling ...\n  }\n}\n```\n\n## Performance Considerations\n\n### Message Batching\n\nTo improve performance, the server can batch multiple events into single WebSocket messages:\n\n```\n{\n  "type": "batch",\n  "timestamp": "2025-09-26T10:00:00Z",\n  "events": [\n    {\n      "event_type": "TaskStatusChanged",\n      "data": { ... }\n    },\n    {\n      "event_type": "WorkflowProgressUpdate",\n      "data": { ... }\n    }\n  ]\n}\n```\n\n### Compression\n\nEnable message compression for large events:\n\n```\nconst ws = new WebSocket('ws://localhost:9090/ws?token=jwt&compression=true');\n```\n\n### Rate Limiting\n\nThe server implements rate limiting to prevent abuse:\n\n- Maximum connections per user: 10\n- Maximum messages per second: 100\n- Maximum subscription events: 50\n\n## Security Considerations\n\n### Authentication and Authorization\n\n- All connections require valid JWT tokens\n- Tokens are validated on connection and periodically renewed\n- Event access is controlled by user roles and permissions\n\n### Message Validation\n\n- All incoming messages are validated against schemas\n- Malformed messages are rejected\n- Rate limiting prevents DoS attacks\n\n### Data Sanitization\n\n- All event data is sanitized before transmission\n- Sensitive information is filtered based on user permissions\n- PII and secrets are never transmitted\n\nThis WebSocket API provides a robust, real-time communication channel for monitoring and managing provisioning with comprehensive security and\nperformance features.
