Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

REST API Reference

This document provides comprehensive documentation for all REST API endpoints in provisioning.

Overview

Provisioning exposes two main REST APIs:

  • Orchestrator API (Port 8080): Core workflow management and batch operations
  • Control Center API (Port 9080): Authentication, authorization, and policy management

Base URLs

  • Orchestrator: http://localhost:9090
  • Control Center: http://localhost:9080

Authentication

JWT Authentication

All API endpoints (except health checks) require JWT authentication via the Authorization header:

Authorization: Bearer <jwt_token>

Getting Access Token

POST /auth/login
Content-Type: application/json

{
  "username": "admin",
  "password": "password",
  "mfa_code": "123456"
}

Orchestrator API Endpoints

Health Check

GET /health

Check orchestrator health status.

Response:

{
  "success": true,
  "data": "Orchestrator is healthy"
}

Task Management

GET /tasks

List all workflow tasks.

Query Parameters:

  • status (optional): Filter by task status (Pending, Running, Completed, Failed, Cancelled)
  • limit (optional): Maximum number of results
  • offset (optional): Pagination offset

Response:

{
  "success": true,
  "data": [
    {
      "id": "uuid-string",
      "name": "create_servers",
      "command": "/usr/local/provisioning servers create",
      "args": ["--infra", "production", "--wait"],
      "dependencies": [],
      "status": "Completed",
      "created_at": "2025-09-26T10:00:00Z",
      "started_at": "2025-09-26T10:00:05Z",
      "completed_at": "2025-09-26T10:05:30Z",
      "output": "Successfully created 3 servers",
      "error": null
    }
  ]
}

GET /tasks/

Get specific task status and details.

Path Parameters:

  • id: Task UUID

Response:

{
  "success": true,
  "data": {
    "id": "uuid-string",
    "name": "create_servers",
    "command": "/usr/local/provisioning servers create",
    "args": ["--infra", "production", "--wait"],
    "dependencies": [],
    "status": "Running",
    "created_at": "2025-09-26T10:00:00Z",
    "started_at": "2025-09-26T10:00:05Z",
    "completed_at": null,
    "output": null,
    "error": null
  }
}

Workflow Submission

POST /workflows/servers/create

Submit server creation workflow.

Request Body:

{
  "infra": "production",
  "settings": "config.k",
  "check_mode": false,
  "wait": true
}

Response:

{
  "success": true,
  "data": "uuid-task-id"
}

POST /workflows/taskserv/create

Submit task service workflow.

Request Body:

{
  "operation": "create",
  "taskserv": "kubernetes",
  "infra": "production",
  "settings": "config.k",
  "check_mode": false,
  "wait": true
}

Response:

{
  "success": true,
  "data": "uuid-task-id"
}

POST /workflows/cluster/create

Submit cluster workflow.

Request Body:

{
  "operation": "create",
  "cluster_type": "buildkit",
  "infra": "production",
  "settings": "config.k",
  "check_mode": false,
  "wait": true
}

Response:

{
  "success": true,
  "data": "uuid-task-id"
}

Batch Operations

POST /batch/execute

Execute batch workflow operation.

Request Body:

{
  "name": "multi_cloud_deployment",
  "version": "1.0.0",
  "storage_backend": "surrealdb",
  "parallel_limit": 5,
  "rollback_enabled": true,
  "operations": [
    {
      "id": "upcloud_servers",
      "type": "server_batch",
      "provider": "upcloud",
      "dependencies": [],
      "server_configs": [
        {"name": "web-01", "plan": "1xCPU-2GB", "zone": "de-fra1"},
        {"name": "web-02", "plan": "1xCPU-2GB", "zone": "us-nyc1"}
      ]
    },
    {
      "id": "aws_taskservs",
      "type": "taskserv_batch",
      "provider": "aws",
      "dependencies": ["upcloud_servers"],
      "taskservs": ["kubernetes", "cilium", "containerd"]
    }
  ]
}

Response:

{
  "success": true,
  "data": {
    "batch_id": "uuid-string",
    "status": "Running",
    "operations": [
      {
        "id": "upcloud_servers",
        "status": "Pending",
        "progress": 0.0
      },
      {
        "id": "aws_taskservs",
        "status": "Pending",
        "progress": 0.0
      }
    ]
  }
}

GET /batch/operations

List all batch operations.

Response:

{
  "success": true,
  "data": [
    {
      "batch_id": "uuid-string",
      "name": "multi_cloud_deployment",
      "status": "Running",
      "created_at": "2025-09-26T10:00:00Z",
      "operations": [...]
    }
  ]
}

GET /batch/operations/

Get batch operation status.

Path Parameters:

  • id: Batch operation ID

Response:

{
  "success": true,
  "data": {
    "batch_id": "uuid-string",
    "name": "multi_cloud_deployment",
    "status": "Running",
    "operations": [
      {
        "id": "upcloud_servers",
        "status": "Completed",
        "progress": 100.0,
        "results": {...}
      }
    ]
  }
}

POST /batch/operations/{id}/cancel

Cancel running batch operation.

Path Parameters:

  • id: Batch operation ID

Response:

{
  "success": true,
  "data": "Operation cancelled"
}

State Management

GET /state/workflows/{id}/progress

Get real-time workflow progress.

Path Parameters:

  • id: Workflow ID

Response:

{
  "success": true,
  "data": {
    "workflow_id": "uuid-string",
    "progress": 75.5,
    "current_step": "Installing Kubernetes",
    "total_steps": 8,
    "completed_steps": 6,
    "estimated_time_remaining": 180
  }
}

GET /state/workflows/{id}/snapshots

Get workflow state snapshots.

Path Parameters:

  • id: Workflow ID

Response:

{
  "success": true,
  "data": [
    {
      "snapshot_id": "uuid-string",
      "timestamp": "2025-09-26T10:00:00Z",
      "state": "running",
      "details": {...}
    }
  ]
}

GET /state/system/metrics

Get system-wide metrics.

Response:

{
  "success": true,
  "data": {
    "total_workflows": 150,
    "active_workflows": 5,
    "completed_workflows": 140,
    "failed_workflows": 5,
    "system_load": {
      "cpu_usage": 45.2,
      "memory_usage": 2048,
      "disk_usage": 75.5
    }
  }
}

GET /state/system/health

Get system health status.

Response:

{
  "success": true,
  "data": {
    "overall_status": "Healthy",
    "components": {
      "storage": "Healthy",
      "batch_coordinator": "Healthy",
      "monitoring": "Healthy"
    },
    "last_check": "2025-09-26T10:00:00Z"
  }
}

GET /state/statistics

Get state manager statistics.

Response:

{
  "success": true,
  "data": {
    "total_workflows": 150,
    "active_snapshots": 25,
    "storage_usage": "245MB",
    "average_workflow_duration": 300
  }
}

Rollback and Recovery

POST /rollback/checkpoints

Create new checkpoint.

Request Body:

{
  "name": "before_major_update",
  "description": "Checkpoint before deploying v2.0.0"
}

Response:

{
  "success": true,
  "data": "checkpoint-uuid"
}

GET /rollback/checkpoints

List all checkpoints.

Response:

{
  "success": true,
  "data": [
    {
      "id": "checkpoint-uuid",
      "name": "before_major_update",
      "description": "Checkpoint before deploying v2.0.0",
      "created_at": "2025-09-26T10:00:00Z",
      "size": "150MB"
    }
  ]
}

GET /rollback/checkpoints/

Get specific checkpoint details.

Path Parameters:

  • id: Checkpoint ID

Response:

{
  "success": true,
  "data": {
    "id": "checkpoint-uuid",
    "name": "before_major_update",
    "description": "Checkpoint before deploying v2.0.0",
    "created_at": "2025-09-26T10:00:00Z",
    "size": "150MB",
    "operations_count": 25
  }
}

POST /rollback/execute

Execute rollback operation.

Request Body:

{
  "checkpoint_id": "checkpoint-uuid"
}

Or for partial rollback:

{
  "operation_ids": ["op-1", "op-2", "op-3"]
}

Response:

{
  "success": true,
  "data": {
    "rollback_id": "rollback-uuid",
    "success": true,
    "operations_executed": 25,
    "operations_failed": 0,
    "duration": 45.5
  }
}

POST /rollback/restore/

Restore system state from checkpoint.

Path Parameters:

  • id: Checkpoint ID

Response:

{
  "success": true,
  "data": "State restored from checkpoint checkpoint-uuid"
}

GET /rollback/statistics

Get rollback system statistics.

Response:

{
  "success": true,
  "data": {
    "total_checkpoints": 10,
    "total_rollbacks": 3,
    "success_rate": 100.0,
    "average_rollback_time": 30.5
  }
}

Control Center API Endpoints

Authentication

POST /auth/login

Authenticate user and get JWT token.

Request Body:

{
  "username": "admin",
  "password": "secure_password",
  "mfa_code": "123456"
}

Response:

{
  "success": true,
  "data": {
    "token": "jwt-token-string",
    "expires_at": "2025-09-26T18:00:00Z",
    "user": {
      "id": "user-uuid",
      "username": "admin",
      "email": "admin@example.com",
      "roles": ["admin", "operator"]
    }
  }
}

POST /auth/refresh

Refresh JWT token.

Request Body:

{
  "token": "current-jwt-token"
}

Response:

{
  "success": true,
  "data": {
    "token": "new-jwt-token",
    "expires_at": "2025-09-26T18:00:00Z"
  }
}

POST /auth/logout

Logout and invalidate token.

Response:

{
  "success": true,
  "data": "Successfully logged out"
}

User Management

GET /users

List all users.

Query Parameters:

  • role (optional): Filter by role
  • enabled (optional): Filter by enabled status

Response:

{
  "success": true,
  "data": [
    {
      "id": "user-uuid",
      "username": "admin",
      "email": "admin@example.com",
      "roles": ["admin"],
      "enabled": true,
      "created_at": "2025-09-26T10:00:00Z",
      "last_login": "2025-09-26T12:00:00Z"
    }
  ]
}

POST /users

Create new user.

Request Body:

{
  "username": "newuser",
  "email": "newuser@example.com",
  "password": "secure_password",
  "roles": ["operator"],
  "enabled": true
}

Response:

{
  "success": true,
  "data": {
    "id": "new-user-uuid",
    "username": "newuser",
    "email": "newuser@example.com",
    "roles": ["operator"],
    "enabled": true
  }
}

PUT /users/

Update existing user.

Path Parameters:

  • id: User ID

Request Body:

{
  "email": "updated@example.com",
  "roles": ["admin", "operator"],
  "enabled": false
}

Response:

{
  "success": true,
  "data": "User updated successfully"
}

DELETE /users/

Delete user.

Path Parameters:

  • id: User ID

Response:

{
  "success": true,
  "data": "User deleted successfully"
}

Policy Management

GET /policies

List all policies.

Response:

{
  "success": true,
  "data": [
    {
      "id": "policy-uuid",
      "name": "admin_access_policy",
      "version": "1.0.0",
      "rules": [...],
      "created_at": "2025-09-26T10:00:00Z",
      "enabled": true
    }
  ]
}

POST /policies

Create new policy.

Request Body:

{
  "name": "new_policy",
  "version": "1.0.0",
  "rules": [
    {
      "effect": "Allow",
      "resource": "servers:*",
      "action": ["create", "read"],
      "condition": "user.role == 'admin'"
    }
  ]
}

Response:

{
  "success": true,
  "data": {
    "id": "new-policy-uuid",
    "name": "new_policy",
    "version": "1.0.0"
  }
}

PUT /policies/

Update policy.

Path Parameters:

  • id: Policy ID

Request Body:

{
  "name": "updated_policy",
  "rules": [...]
}

Response:

{
  "success": true,
  "data": "Policy updated successfully"
}

Audit Logging

GET /audit/logs

Get audit logs.

Query Parameters:

  • user_id (optional): Filter by user
  • action (optional): Filter by action
  • resource (optional): Filter by resource
  • from (optional): Start date (ISO 8601)
  • to (optional): End date (ISO 8601)
  • limit (optional): Maximum results
  • offset (optional): Pagination offset

Response:

{
  "success": true,
  "data": [
    {
      "id": "audit-log-uuid",
      "timestamp": "2025-09-26T10:00:00Z",
      "user_id": "user-uuid",
      "action": "server.create",
      "resource": "servers/web-01",
      "result": "success",
      "details": {...}
    }
  ]
}

Error Responses

All endpoints may return error responses in this format:

{
  "success": false,
  "error": "Detailed error message"
}

HTTP Status Codes

  • 200 OK: Successful request
  • 201 Created: Resource created successfully
  • 400 Bad Request: Invalid request parameters
  • 401 Unauthorized: Authentication required or invalid
  • 403 Forbidden: Permission denied
  • 404 Not Found: Resource not found
  • 422 Unprocessable Entity: Validation error
  • 500 Internal Server Error: Server error

Rate Limiting

API endpoints are rate-limited:

  • Authentication: 5 requests per minute per IP
  • General APIs: 100 requests per minute per user
  • Batch operations: 10 requests per minute per user

Rate limit headers are included in responses:

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1632150000

Monitoring Endpoints

GET /metrics

Prometheus-compatible metrics endpoint.

Response:

# HELP orchestrator_tasks_total Total number of tasks
# TYPE orchestrator_tasks_total counter
orchestrator_tasks_total{status="completed"} 150
orchestrator_tasks_total{status="failed"} 5

# HELP orchestrator_task_duration_seconds Task execution duration
# TYPE orchestrator_task_duration_seconds histogram
orchestrator_task_duration_seconds_bucket{le="10"} 50
orchestrator_task_duration_seconds_bucket{le="30"} 120
orchestrator_task_duration_seconds_bucket{le="+Inf"} 155

WebSocket /ws

Real-time event streaming via WebSocket connection.

Connection:

const ws = new WebSocket('ws://localhost:9090/ws?token=jwt-token');

ws.onmessage = function(event) {
  const data = JSON.parse(event.data);
  console.log('Event:', data);
};

Event Format:

{
  "event_type": "TaskStatusChanged",
  "timestamp": "2025-09-26T10:00:00Z",
  "data": {
    "task_id": "uuid-string",
    "status": "completed"
  },
  "metadata": {
    "task_id": "uuid-string",
    "status": "completed"
  }
}

SDK Examples

Python SDK Example

import requests

class ProvisioningClient:
    def __init__(self, base_url, token):
        self.base_url = base_url
        self.headers = {
            'Authorization': f'Bearer {token}',
            'Content-Type': 'application/json'
        }

    def create_server_workflow(self, infra, settings, check_mode=False):
        payload = {
            'infra': infra,
            'settings': settings,
            'check_mode': check_mode,
            'wait': True
        }
        response = requests.post(
            f'{self.base_url}/workflows/servers/create',
            json=payload,
            headers=self.headers
        )
        return response.json()

    def get_task_status(self, task_id):
        response = requests.get(
            f'{self.base_url}/tasks/{task_id}',
            headers=self.headers
        )
        return response.json()

# Usage
client = ProvisioningClient('http://localhost:9090', 'your-jwt-token')
result = client.create_server_workflow('production', 'config.k')
print(f"Task ID: {result['data']}")

JavaScript/Node.js SDK Example

const axios = require('axios');

class ProvisioningClient {
  constructor(baseUrl, token) {
    this.client = axios.create({
      baseURL: baseUrl,
      headers: {
        'Authorization': `Bearer ${token}`,
        'Content-Type': 'application/json'
      }
    });
  }

  async createServerWorkflow(infra, settings, checkMode = false) {
    const response = await this.client.post('/workflows/servers/create', {
      infra,
      settings,
      check_mode: checkMode,
      wait: true
    });
    return response.data;
  }

  async getTaskStatus(taskId) {
    const response = await this.client.get(`/tasks/${taskId}`);
    return response.data;
  }
}

// Usage
const client = new ProvisioningClient('http://localhost:9090', 'your-jwt-token');
const result = await client.createServerWorkflow('production', 'config.k');
console.log(`Task ID: ${result.data}`);

Webhook Integration

The system supports webhooks for external integrations:

Webhook Configuration

Configure webhooks in the system configuration:

[webhooks]
enabled = true
endpoints = [
  {
    url = "https://your-system.com/webhook"
    events = ["task.completed", "task.failed", "batch.completed"]
    secret = "webhook-secret"
  }
]

Webhook Payload

{
  "event": "task.completed",
  "timestamp": "2025-09-26T10:00:00Z",
  "data": {
    "task_id": "uuid-string",
    "status": "completed",
    "output": "Task completed successfully"
  },
  "signature": "sha256=calculated-signature"
}

Pagination

For endpoints that return lists, use pagination parameters:

  • limit: Maximum number of items per page (default: 50, max: 1000)
  • offset: Number of items to skip

Pagination metadata is included in response headers:

X-Total-Count: 1500
X-Limit: 50
X-Offset: 100
Link: </api/endpoint?offset=150&limit=50>; rel="next"

API Versioning

The API uses header-based versioning:

Accept: application/vnd.provisioning.v1+json

Current version: v1

Testing

Use the included test suite to validate API functionality:

# Run API integration tests
cd src/orchestrator
cargo test --test api_tests

# Run load tests
cargo test --test load_tests --release