# REST API Reference\n\nThis document provides comprehensive documentation for all REST API endpoints in provisioning.\n\n## Overview\n\nProvisioning exposes two main REST APIs:\n\n- **Orchestrator API** (Port 8080): Core workflow management and batch operations\n- **Control Center API** (Port 9080): Authentication, authorization, and policy management\n\n## Base URLs\n\n- **Orchestrator**: `http://localhost:9090`\n- **Control Center**: `http://localhost:9080`\n\n## Authentication\n\n### JWT Authentication\n\nAll API endpoints (except health checks) require JWT authentication via the Authorization header:\n\n```\nAuthorization: Bearer <jwt_token>\n```\n\n### Getting Access Token\n\n```\nPOST /auth/login\nContent-Type: application/json\n\n{\n  "username": "admin",\n  "password": "password",\n  "mfa_code": "123456"\n}\n```\n\n## Orchestrator API Endpoints\n\n### Health Check\n\n#### GET /health\n\nCheck orchestrator health status.\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": "Orchestrator is healthy"\n}\n```\n\n### Task Management\n\n#### GET /tasks\n\nList all workflow tasks.\n\n**Query Parameters:**\n\n- `status` (optional): Filter by task status (Pending, Running, Completed, Failed, Cancelled)\n- `limit` (optional): Maximum number of results\n- `offset` (optional): Pagination offset\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": [\n    {\n      "id": "uuid-string",\n      "name": "create_servers",\n      "command": "/usr/local/provisioning servers create",\n      "args": ["--infra", "production", "--wait"],\n      "dependencies": [],\n      "status": "Completed",\n      "created_at": "2025-09-26T10:00:00Z",\n      "started_at": "2025-09-26T10:00:05Z",\n      "completed_at": "2025-09-26T10:05:30Z",\n      "output": "Successfully created 3 servers",\n      "error": null\n    }\n  ]\n}\n```\n\n#### GET /tasks/{id}\n\nGet specific task status and details.\n\n**Path Parameters:**\n\n- `id`: Task UUID\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": {\n    "id": "uuid-string",\n    "name": "create_servers",\n    "command": "/usr/local/provisioning servers create",\n    "args": ["--infra", "production", "--wait"],\n    "dependencies": [],\n    "status": "Running",\n    "created_at": "2025-09-26T10:00:00Z",\n    "started_at": "2025-09-26T10:00:05Z",\n    "completed_at": null,\n    "output": null,\n    "error": null\n  }\n}\n```\n\n### Workflow Submission\n\n#### POST /workflows/servers/create\n\nSubmit server creation workflow.\n\n**Request Body:**\n\n```\n{\n  "infra": "production",\n  "settings": "config.ncl",\n  "check_mode": false,\n  "wait": true\n}\n```\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": "uuid-task-id"\n}\n```\n\n#### POST /workflows/taskserv/create\n\nSubmit task service workflow.\n\n**Request Body:**\n\n```\n{\n  "operation": "create",\n  "taskserv": "kubernetes",\n  "infra": "production",\n  "settings": "config.ncl",\n  "check_mode": false,\n  "wait": true\n}\n```\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": "uuid-task-id"\n}\n```\n\n#### POST /workflows/cluster/create\n\nSubmit cluster workflow.\n\n**Request Body:**\n\n```\n{\n  "operation": "create",\n  "cluster_type": "buildkit",\n  "infra": "production",\n  "settings": "config.ncl",\n  "check_mode": false,\n  "wait": true\n}\n```\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": "uuid-task-id"\n}\n```\n\n### Batch Operations\n\n#### POST /batch/execute\n\nExecute batch workflow operation.\n\n**Request Body:**\n\n```\n{\n  "name": "multi_cloud_deployment",\n  "version": "1.0.0",\n  "storage_backend": "surrealdb",\n  "parallel_limit": 5,\n  "rollback_enabled": true,\n  "operations": [\n    {\n      "id": "upcloud_servers",\n      "type": "server_batch",\n      "provider": "upcloud",\n      "dependencies": [],\n      "server_configs": [\n        {"name": "web-01", "plan": "1xCPU-2 GB", "zone": "de-fra1"},\n        {"name": "web-02", "plan": "1xCPU-2 GB", "zone": "us-nyc1"}\n      ]\n    },\n    {\n      "id": "aws_taskservs",\n      "type": "taskserv_batch",\n      "provider": "aws",\n      "dependencies": ["upcloud_servers"],\n      "taskservs": ["kubernetes", "cilium", "containerd"]\n    }\n  ]\n}\n```\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": {\n    "batch_id": "uuid-string",\n    "status": "Running",\n    "operations": [\n      {\n        "id": "upcloud_servers",\n        "status": "Pending",\n        "progress": 0.0\n      },\n      {\n        "id": "aws_taskservs",\n        "status": "Pending",\n        "progress": 0.0\n      }\n    ]\n  }\n}\n```\n\n#### GET /batch/operations\n\nList all batch operations.\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": [\n    {\n      "batch_id": "uuid-string",\n      "name": "multi_cloud_deployment",\n      "status": "Running",\n      "created_at": "2025-09-26T10:00:00Z",\n      "operations": [...]\n    }\n  ]\n}\n```\n\n#### GET /batch/operations/{id}\n\nGet batch operation status.\n\n**Path Parameters:**\n\n- `id`: Batch operation ID\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": {\n    "batch_id": "uuid-string",\n    "name": "multi_cloud_deployment",\n    "status": "Running",\n    "operations": [\n      {\n        "id": "upcloud_servers",\n        "status": "Completed",\n        "progress": 100.0,\n        "results": {...}\n      }\n    ]\n  }\n}\n```\n\n#### POST /batch/operations/{id}/cancel\n\nCancel running batch operation.\n\n**Path Parameters:**\n\n- `id`: Batch operation ID\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": "Operation cancelled"\n}\n```\n\n### State Management\n\n#### GET /state/workflows/{id}/progress\n\nGet real-time workflow progress.\n\n**Path Parameters:**\n\n- `id`: Workflow ID\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": {\n    "workflow_id": "uuid-string",\n    "progress": 75.5,\n    "current_step": "Installing Kubernetes",\n    "total_steps": 8,\n    "completed_steps": 6,\n    "estimated_time_remaining": 180\n  }\n}\n```\n\n#### GET /state/workflows/{id}/snapshots\n\nGet workflow state snapshots.\n\n**Path Parameters:**\n\n- `id`: Workflow ID\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": [\n    {\n      "snapshot_id": "uuid-string",\n      "timestamp": "2025-09-26T10:00:00Z",\n      "state": "running",\n      "details": {...}\n    }\n  ]\n}\n```\n\n#### GET /state/system/metrics\n\nGet system-wide metrics.\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": {\n    "total_workflows": 150,\n    "active_workflows": 5,\n    "completed_workflows": 140,\n    "failed_workflows": 5,\n    "system_load": {\n      "cpu_usage": 45.2,\n      "memory_usage": 2048,\n      "disk_usage": 75.5\n    }\n  }\n}\n```\n\n#### GET /state/system/health\n\nGet system health status.\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": {\n    "overall_status": "Healthy",\n    "components": {\n      "storage": "Healthy",\n      "batch_coordinator": "Healthy",\n      "monitoring": "Healthy"\n    },\n    "last_check": "2025-09-26T10:00:00Z"\n  }\n}\n```\n\n#### GET /state/statistics\n\nGet state manager statistics.\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": {\n    "total_workflows": 150,\n    "active_snapshots": 25,\n    "storage_usage": "245 MB",\n    "average_workflow_duration": 300\n  }\n}\n```\n\n### Rollback and Recovery\n\n#### POST /rollback/checkpoints\n\nCreate new checkpoint.\n\n**Request Body:**\n\n```\n{\n  "name": "before_major_update",\n  "description": "Checkpoint before deploying v2.0.0"\n}\n```\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": "checkpoint-uuid"\n}\n```\n\n#### GET /rollback/checkpoints\n\nList all checkpoints.\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": [\n    {\n      "id": "checkpoint-uuid",\n      "name": "before_major_update",\n      "description": "Checkpoint before deploying v2.0.0",\n      "created_at": "2025-09-26T10:00:00Z",\n      "size": "150 MB"\n    }\n  ]\n}\n```\n\n#### GET /rollback/checkpoints/{id}\n\nGet specific checkpoint details.\n\n**Path Parameters:**\n\n- `id`: Checkpoint ID\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": {\n    "id": "checkpoint-uuid",\n    "name": "before_major_update",\n    "description": "Checkpoint before deploying v2.0.0",\n    "created_at": "2025-09-26T10:00:00Z",\n    "size": "150 MB",\n    "operations_count": 25\n  }\n}\n```\n\n#### POST /rollback/execute\n\nExecute rollback operation.\n\n**Request Body:**\n\n```\n{\n  "checkpoint_id": "checkpoint-uuid"\n}\n```\n\nOr for partial rollback:\n\n```\n{\n  "operation_ids": ["op-1", "op-2", "op-3"]\n}\n```\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": {\n    "rollback_id": "rollback-uuid",\n    "success": true,\n    "operations_executed": 25,\n    "operations_failed": 0,\n    "duration": 45.5\n  }\n}\n```\n\n#### POST /rollback/restore/{id}\n\nRestore system state from checkpoint.\n\n**Path Parameters:**\n\n- `id`: Checkpoint ID\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": "State restored from checkpoint checkpoint-uuid"\n}\n```\n\n#### GET /rollback/statistics\n\nGet rollback system statistics.\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": {\n    "total_checkpoints": 10,\n    "total_rollbacks": 3,\n    "success_rate": 100.0,\n    "average_rollback_time": 30.5\n  }\n}\n```\n\n## Control Center API Endpoints\n\n### Authentication\n\n#### POST /auth/login\n\nAuthenticate user and get JWT token.\n\n**Request Body:**\n\n```\n{\n  "username": "admin",\n  "password": "secure_password",\n  "mfa_code": "123456"\n}\n```\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": {\n    "token": "jwt-token-string",\n    "expires_at": "2025-09-26T18:00:00Z",\n    "user": {\n      "id": "user-uuid",\n      "username": "admin",\n      "email": "admin@example.com",\n      "roles": ["admin", "operator"]\n    }\n  }\n}\n```\n\n#### POST /auth/refresh\n\nRefresh JWT token.\n\n**Request Body:**\n\n```\n{\n  "token": "current-jwt-token"\n}\n```\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": {\n    "token": "new-jwt-token",\n    "expires_at": "2025-09-26T18:00:00Z"\n  }\n}\n```\n\n#### POST /auth/logout\n\nLogout and invalidate token.\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": "Successfully logged out"\n}\n```\n\n### User Management\n\n#### GET /users\n\nList all users.\n\n**Query Parameters:**\n\n- `role` (optional): Filter by role\n- `enabled` (optional): Filter by enabled status\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": [\n    {\n      "id": "user-uuid",\n      "username": "admin",\n      "email": "admin@example.com",\n      "roles": ["admin"],\n      "enabled": true,\n      "created_at": "2025-09-26T10:00:00Z",\n      "last_login": "2025-09-26T12:00:00Z"\n    }\n  ]\n}\n```\n\n#### POST /users\n\nCreate new user.\n\n**Request Body:**\n\n```\n{\n  "username": "newuser",\n  "email": "newuser@example.com",\n  "password": "secure_password",\n  "roles": ["operator"],\n  "enabled": true\n}\n```\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": {\n    "id": "new-user-uuid",\n    "username": "newuser",\n    "email": "newuser@example.com",\n    "roles": ["operator"],\n    "enabled": true\n  }\n}\n```\n\n#### PUT /users/{id}\n\nUpdate existing user.\n\n**Path Parameters:**\n\n- `id`: User ID\n\n**Request Body:**\n\n```\n{\n  "email": "updated@example.com",\n  "roles": ["admin", "operator"],\n  "enabled": false\n}\n```\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": "User updated successfully"\n}\n```\n\n#### DELETE /users/{id}\n\nDelete user.\n\n**Path Parameters:**\n\n- `id`: User ID\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": "User deleted successfully"\n}\n```\n\n### Policy Management\n\n#### GET /policies\n\nList all policies.\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": [\n    {\n      "id": "policy-uuid",\n      "name": "admin_access_policy",\n      "version": "1.0.0",\n      "rules": [...],\n      "created_at": "2025-09-26T10:00:00Z",\n      "enabled": true\n    }\n  ]\n}\n```\n\n#### POST /policies\n\nCreate new policy.\n\n**Request Body:**\n\n```\n{\n  "name": "new_policy",\n  "version": "1.0.0",\n  "rules": [\n    {\n      "effect": "Allow",\n      "resource": "servers:*",\n      "action": ["create", "read"],\n      "condition": "user.role == 'admin'"\n    }\n  ]\n}\n```\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": {\n    "id": "new-policy-uuid",\n    "name": "new_policy",\n    "version": "1.0.0"\n  }\n}\n```\n\n#### PUT /policies/{id}\n\nUpdate policy.\n\n**Path Parameters:**\n\n- `id`: Policy ID\n\n**Request Body:**\n\n```\n{\n  "name": "updated_policy",\n  "rules": [...]\n}\n```\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": "Policy updated successfully"\n}\n```\n\n### Audit Logging\n\n#### GET /audit/logs\n\nGet audit logs.\n\n**Query Parameters:**\n\n- `user_id` (optional): Filter by user\n- `action` (optional): Filter by action\n- `resource` (optional): Filter by resource\n- `from` (optional): Start date (ISO 8601)\n- `to` (optional): End date (ISO 8601)\n- `limit` (optional): Maximum results\n- `offset` (optional): Pagination offset\n\n**Response:**\n\n```\n{\n  "success": true,\n  "data": [\n    {\n      "id": "audit-log-uuid",\n      "timestamp": "2025-09-26T10:00:00Z",\n      "user_id": "user-uuid",\n      "action": "server.create",\n      "resource": "servers/web-01",\n      "result": "success",\n      "details": {...}\n    }\n  ]\n}\n```\n\n## Error Responses\n\nAll endpoints may return error responses in this format:\n\n```\n{\n  "success": false,\n  "error": "Detailed error message"\n}\n```\n\n### HTTP Status Codes\n\n- `200 OK`: Successful request\n- `201 Created`: Resource created successfully\n- `400 Bad Request`: Invalid request parameters\n- `401 Unauthorized`: Authentication required or invalid\n- `403 Forbidden`: Permission denied\n- `404 Not Found`: Resource not found\n- `422 Unprocessable Entity`: Validation error\n- `500 Internal Server Error`: Server error\n\n## Rate Limiting\n\nAPI endpoints are rate-limited:\n\n- Authentication: 5 requests per minute per IP\n- General APIs: 100 requests per minute per user\n- Batch operations: 10 requests per minute per user\n\nRate limit headers are included in responses:\n\n```\nX-RateLimit-Limit: 100\nX-RateLimit-Remaining: 95\nX-RateLimit-Reset: 1632150000\n```\n\n## Monitoring Endpoints\n\n### GET /metrics\n\nPrometheus-compatible metrics endpoint.\n\n**Response:**\n\n```\n# HELP orchestrator_tasks_total Total number of tasks\n# TYPE orchestrator_tasks_total counter\norchestrator_tasks_total{status="completed"} 150\norchestrator_tasks_total{status="failed"} 5\n\n# HELP orchestrator_task_duration_seconds Task execution duration\n# TYPE orchestrator_task_duration_seconds histogram\norchestrator_task_duration_seconds_bucket{le="10"} 50\norchestrator_task_duration_seconds_bucket{le="30"} 120\norchestrator_task_duration_seconds_bucket{le="+Inf"} 155\n```\n\n### WebSocket /ws\n\nReal-time event streaming via WebSocket connection.\n\n**Connection:**\n\n```\nconst ws = new WebSocket('ws://localhost:9090/ws?token=jwt-token');\n\nws.onmessage = function(event) {\n  const data = JSON.parse(event.data);\n  console.log('Event:', data);\n};\n```\n\n**Event Format:**\n\n```\n{\n  "event_type": "TaskStatusChanged",\n  "timestamp": "2025-09-26T10:00:00Z",\n  "data": {\n    "task_id": "uuid-string",\n    "status": "completed"\n  },\n  "metadata": {\n    "task_id": "uuid-string",\n    "status": "completed"\n  }\n}\n```\n\n## SDK Examples\n\n### Python SDK Example\n\n```\nimport requests\n\nclass ProvisioningClient:\n    def __init__(self, base_url, token):\n        self.base_url = base_url\n        self.headers = {\n            'Authorization': f'Bearer {token}',\n            'Content-Type': 'application/json'\n        }\n\n    def create_server_workflow(self, infra, settings, check_mode=False):\n        payload = {\n            'infra': infra,\n            'settings': settings,\n            'check_mode': check_mode,\n            'wait': True\n        }\n        response = requests.post(\n            f'{self.base_url}/workflows/servers/create',\n            json=payload,\n            headers=self.headers\n        )\n        return response.json()\n\n    def get_task_status(self, task_id):\n        response = requests.get(\n            f'{self.base_url}/tasks/{task_id}',\n            headers=self.headers\n        )\n        return response.json()\n\n# Usage\nclient = ProvisioningClient('http://localhost:9090', 'your-jwt-token')\nresult = client.create_server_workflow('production', 'config.ncl')\nprint(f"Task ID: {result['data']}")\n```\n\n### JavaScript/Node.js SDK Example\n\n```\nconst axios = require('axios');\n\nclass ProvisioningClient {\n  constructor(baseUrl, token) {\n    this.client = axios.create({\n      baseURL: baseUrl,\n      headers: {\n        'Authorization': `Bearer ${token}`,\n        'Content-Type': 'application/json'\n      }\n    });\n  }\n\n  async createServerWorkflow(infra, settings, checkMode = false) {\n    const response = await this.client.post('/workflows/servers/create', {\n      infra,\n      settings,\n      check_mode: checkMode,\n      wait: true\n    });\n    return response.data;\n  }\n\n  async getTaskStatus(taskId) {\n    const response = await this.client.get(`/tasks/${taskId}`);\n    return response.data;\n  }\n}\n\n// Usage\nconst client = new ProvisioningClient('http://localhost:9090', 'your-jwt-token');\nconst result = await client.createServerWorkflow('production', 'config.ncl');\nconsole.log(`Task ID: ${result.data}`);\n```\n\n## Webhook Integration\n\nThe system supports webhooks for external integrations:\n\n### Webhook Configuration\n\nConfigure webhooks in the system configuration:\n\n```\n[webhooks]\nenabled = true\nendpoints = [\n  {\n    url = "https://your-system.com/webhook"\n    events = ["task.completed", "task.failed", "batch.completed"]\n    secret = "webhook-secret"\n  }\n]\n```\n\n### Webhook Payload\n\n```\n{\n  "event": "task.completed",\n  "timestamp": "2025-09-26T10:00:00Z",\n  "data": {\n    "task_id": "uuid-string",\n    "status": "completed",\n    "output": "Task completed successfully"\n  },\n  "signature": "sha256=calculated-signature"\n}\n```\n\n## Pagination\n\nFor endpoints that return lists, use pagination parameters:\n\n- `limit`: Maximum number of items per page (default: 50, max: 1000)\n- `offset`: Number of items to skip\n\nPagination metadata is included in response headers:\n\n```\nX-Total-Count: 1500\nX-Limit: 50\nX-Offset: 100\nLink: </api/endpoint?offset=150&limit=50>; rel="next"\n```\n\n## API Versioning\n\nThe API uses header-based versioning:\n\n```\nAccept: application/vnd.provisioning.v1+json\n```\n\nCurrent version: v1\n\n## Testing\n\nUse the included test suite to validate API functionality:\n\n```\n# Run API integration tests\ncd src/orchestrator\ncargo test --test api_tests\n\n# Run load tests\ncargo test --test load_tests --release\n```
