Jesús Pérez f2be2414e4 core: init repo and codebase

2025-10-07 10:59:52 +01:00

14 KiB

Raw Permalink Blame History

Extension Registry Service

A high-performance Rust microservice that provides a unified REST API for extension discovery, versioning, and download from multiple sources (Gitea releases and OCI registries).

Features

Multi-Backend Support: Fetch extensions from Gitea releases and OCI registries
Unified REST API: Single API for all extension operations
Smart Caching: LRU cache with TTL to reduce backend API calls
Prometheus Metrics: Built-in metrics for monitoring
Health Monitoring: Health checks for all backends
Type-Safe: Strong typing for extension metadata
Async/Await: High-performance async operations with Tokio
Docker Support: Production-ready containerization

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Extension Registry API                    │
│                         (axum)                               │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌────────────────┐  ┌────────────────┐  ┌──────────────┐ │
│  │  Gitea Client  │  │   OCI Client   │  │  LRU Cache   │ │
│  │  (reqwest)     │  │   (reqwest)    │  │  (parking)   │ │
│  └────────────────┘  └────────────────┘  └──────────────┘ │
│         │                    │                    │         │
└─────────┼────────────────────┼────────────────────┼─────────┘
          │                    │                    │
          ▼                    ▼                    ▼
    ┌──────────┐         ┌──────────┐        ┌──────────┐
    │  Gitea   │         │   OCI    │        │  Memory  │
    │ Releases │         │ Registry │        │          │
    └──────────┘         └──────────┘        └──────────┘

Installation

Building from Source

cd provisioning/platform/extension-registry
cargo build --release

Docker Build

docker build -t extension-registry:latest .

Running with Cargo

cargo run -- --config config.toml --port 8082

Running with Docker

docker run -d \
  -p 8082:8082 \
  -v $(pwd)/config.toml:/app/config.toml:ro \
  -v $(pwd)/tokens:/app/tokens:ro \
  extension-registry:latest

Configuration

Create a config.toml file (see config.example.toml):

[server]
host = "0.0.0.0"
port = 8082
workers = 4
enable_cors = true
enable_compression = true

# Gitea backend (optional)
[gitea]
url = "https://gitea.example.com"
organization = "provisioning-extensions"
token_path = "/path/to/gitea-token.txt"
timeout_seconds = 30
verify_ssl = true

# OCI registry backend (optional)
[oci]
registry = "registry.example.com"
namespace = "provisioning"
auth_token_path = "/path/to/oci-token.txt"
timeout_seconds = 30
verify_ssl = true

# Cache configuration
[cache]
capacity = 1000
ttl_seconds = 300
enable_metadata_cache = true
enable_list_cache = true

Note: At least one backend (Gitea or OCI) must be configured.

API Endpoints

Extension Operations

List Extensions

GET /api/v1/extensions

Query parameters:

type (optional): Filter by extension type (provider, taskserv, cluster)
source (optional): Filter by source (gitea, oci)
limit (optional): Maximum results (default: 100)
offset (optional): Pagination offset (default: 0)

Example:

curl http://localhost:8082/api/v1/extensions?type=provider&limit=10

Response:

[
  {
    "name": "aws",
    "type": "provider",
    "version": "1.2.0",
    "description": "AWS provider for provisioning",
    "author": "provisioning-team",
    "repository": "https://gitea.example.com/org/aws_prov",
    "source": "gitea",
    "published_at": "2025-10-06T12:00:00Z",
    "download_url": "https://gitea.example.com/org/aws_prov/releases/download/1.2.0/aws_prov.tar.gz",
    "size": 1024000
  }
]

Get Extension

GET /api/v1/extensions/{type}/{name}

Example:

curl http://localhost:8082/api/v1/extensions/provider/aws

List Versions

GET /api/v1/extensions/{type}/{name}/versions

Example:

curl http://localhost:8082/api/v1/extensions/provider/aws/versions

Response:

[
  {
    "version": "1.2.0",
    "published_at": "2025-10-06T12:00:00Z",
    "download_url": "https://gitea.example.com/org/aws_prov/releases/download/1.2.0/aws_prov.tar.gz",
    "size": 1024000
  },
  {
    "version": "1.1.0",
    "published_at": "2025-09-15T10:30:00Z",
    "download_url": "https://gitea.example.com/org/aws_prov/releases/download/1.1.0/aws_prov.tar.gz",
    "size": 980000
  }
]

Download Extension

GET /api/v1/extensions/{type}/{name}/{version}

Example:

curl -O http://localhost:8082/api/v1/extensions/provider/aws/1.2.0

Returns binary data with Content-Type: application/octet-stream.

Search Extensions

GET /api/v1/extensions/search?q={query}

Query parameters:

q (required): Search query
type (optional): Filter by extension type
limit (optional): Maximum results (default: 50)

Example:

curl http://localhost:8082/api/v1/extensions/search?q=kubernetes&type=taskserv

System Endpoints

Health Check

GET /api/v1/health

Example:

curl http://localhost:8082/api/v1/health

Response:

{
  "status": "healthy",
  "version": "0.1.0",
  "uptime": 3600,
  "backends": {
    "gitea": {
      "enabled": true,
      "healthy": true
    },
    "oci": {
      "enabled": true,
      "healthy": true
    }
  }
}

Metrics

GET /api/v1/metrics

Returns Prometheus-formatted metrics:

# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total 1234

# HELP cache_hits_total Total cache hits
# TYPE cache_hits_total counter
cache_hits_total 567

# HELP cache_misses_total Total cache misses
# TYPE cache_misses_total counter
cache_misses_total 123

Cache Statistics

GET /api/v1/cache/stats

Response:

{
  "list_entries": 45,
  "metadata_entries": 120,
  "version_entries": 80,
  "total_entries": 245
}

Extension Naming Conventions

Gitea Repositories

Extensions in Gitea follow specific naming patterns:

Providers: {name}_prov (e.g., aws_prov, upcloud_prov)
Task Services: {name}_taskserv (e.g., kubernetes_taskserv, postgres_taskserv)
Clusters: {name}_cluster (e.g., buildkit_cluster, ci_cluster)

OCI Artifacts

Extensions in OCI registries follow these patterns:

Providers: {namespace}/{name}-provider (e.g., provisioning/aws-provider)
Task Services: {namespace}/{name}-taskserv (e.g., provisioning/kubernetes-taskserv)
Clusters: {namespace}/{name}-cluster (e.g., provisioning/buildkit-cluster)

Caching Strategy

The service implements a multi-level LRU cache with TTL:

List Cache: Caches extension lists (filtered by type/source)
Metadata Cache: Caches individual extension metadata
Version Cache: Caches version lists per extension

Cache behavior:

Capacity: Configurable (default: 1000 entries)
TTL: Configurable (default: 5 minutes)
Eviction: LRU (Least Recently Used)
Invalidation: Automatic on TTL expiration

Cache keys:

List: list:{type}:{source}
Metadata: {type}/{name}
Versions: {type}/{name}/versions

Error Handling

The API uses standard HTTP status codes:

200 OK: Successful operation
400 Bad Request: Invalid input (e.g., invalid extension type)
401 Unauthorized: Authentication failed
404 Not Found: Extension not found
429 Too Many Requests: Rate limit exceeded
500 Internal Server Error: Server error

Error response format:

{
  "error": "not_found",
  "message": "Extension provider/nonexistent not found"
}

Metrics and Monitoring

Prometheus Metrics

Available metrics:

http_requests_total: Total HTTP requests
http_request_duration_seconds: Request duration histogram
cache_hits_total: Total cache hits
cache_misses_total: Total cache misses
extensions_total: Total extensions served

Health Checks

The health endpoint checks:

Service uptime
Gitea backend connectivity
OCI backend connectivity
Overall service status

Development

Project Structure

extension-registry/
├── Cargo.toml              # Rust dependencies
├── src/
│   ├── main.rs             # Entry point
│   ├── lib.rs              # Library exports
│   ├── config.rs           # Configuration management
│   ├── error.rs            # Error types
│   ├── api/
│   │   ├── handlers.rs     # HTTP handlers
│   │   └── routes.rs       # Route definitions
│   ├── gitea/
│   │   ├── client.rs       # Gitea API client
│   │   └── models.rs       # Gitea data models
│   ├── oci/
│   │   ├── client.rs       # OCI registry client
│   │   └── models.rs       # OCI data models
│   ├── cache/
│   │   └── lru_cache.rs    # LRU caching
│   └── models/
│       └── extension.rs    # Extension models
├── tests/
│   └── integration_test.rs # Integration tests
├── Dockerfile              # Docker build
└── README.md               # This file

Running Tests

# Run all tests
cargo test

# Run with output
cargo test -- --nocapture

# Run specific test
cargo test test_health_check

Code Quality

# Format code
cargo fmt

# Run clippy
cargo clippy

# Check for security vulnerabilities
cargo audit

Deployment

Systemd Service

Create /etc/systemd/system/extension-registry.service:

[Unit]
Description=Extension Registry Service
After=network.target

[Service]
Type=simple
User=registry
WorkingDirectory=/opt/extension-registry
ExecStart=/usr/local/bin/extension-registry --config /etc/extension-registry/config.toml
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl enable extension-registry
sudo systemctl start extension-registry
sudo systemctl status extension-registry

Docker Compose

version: '3.8'

services:
  extension-registry:
    image: extension-registry:latest
    ports:
      - "8082:8082"
    volumes:
      - ./config.toml:/app/config.toml:ro
      - ./tokens:/app/tokens:ro
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8082/api/v1/health"]
      interval: 30s
      timeout: 3s
      retries: 3
      start_period: 5s

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: extension-registry
spec:
  replicas: 3
  selector:
    matchLabels:
      app: extension-registry
  template:
    metadata:
      labels:
        app: extension-registry
    spec:
      containers:
      - name: extension-registry
        image: extension-registry:latest
        ports:
        - containerPort: 8082
        volumeMounts:
        - name: config
          mountPath: /app/config.toml
          subPath: config.toml
        - name: tokens
          mountPath: /app/tokens
        livenessProbe:
          httpGet:
            path: /api/v1/health
            port: 8082
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /api/v1/health
            port: 8082
          initialDelaySeconds: 5
          periodSeconds: 5
      volumes:
      - name: config
        configMap:
          name: extension-registry-config
      - name: tokens
        secret:
          secretName: extension-registry-tokens
---
apiVersion: v1
kind: Service
metadata:
  name: extension-registry
spec:
  selector:
    app: extension-registry
  ports:
  - port: 8082
    targetPort: 8082
  type: ClusterIP

Security

Authentication

Gitea: Token-based authentication via token_path
OCI: Optional token authentication via auth_token_path

Best Practices

Store tokens securely: Use file permissions (600) for token files
Enable SSL verification: Set verify_ssl = true in production
Use HTTPS: Always use HTTPS for Gitea and OCI registries
Limit CORS: Configure CORS appropriately for production
Rate limiting: Consider adding rate limiting for public APIs
Network isolation: Run service in isolated network segments

Performance

Benchmarks

Typical performance characteristics:

Cached requests: <5ms response time
Uncached requests: 50-200ms (depends on backend latency)
Cache hit ratio: ~85-95% in typical workloads
Throughput: 1000+ req/s on modern hardware

Optimization Tips

Increase cache capacity: For large extension catalogs
Tune TTL: Balance freshness vs. performance
Use multiple workers: Scale with CPU cores
Enable compression: Reduce bandwidth usage
Connection pooling: Reuse HTTP connections to backends

Troubleshooting

Common Issues

Service won't start

Check configuration file syntax
Verify token files exist and are readable
Check network connectivity to backends

Extensions not found

Verify backend configuration (URL, organization, namespace)
Check backend connectivity with health endpoint
Review logs for authentication errors

Slow responses

Check backend latency
Increase cache capacity or TTL
Review Prometheus metrics for bottlenecks

Logging

Enable debug logging:

extension-registry --log-level debug

Enable JSON logging for structured logs:

extension-registry --json-log

License

Part of the Provisioning Project.

Contributing

See main project documentation for contribution guidelines.

Support

For issues and questions, please refer to the main provisioning project documentation.

14 KiB Raw Permalink Blame History

Extension Registry Service

Features

Architecture

Installation

Building from Source

Docker Build

Running with Cargo

Running with Docker

Configuration

API Endpoints

Extension Operations

List Extensions

Get Extension

List Versions

Download Extension

Search Extensions

System Endpoints

Health Check

Metrics

Cache Statistics

Extension Naming Conventions

Gitea Repositories

OCI Artifacts

Caching Strategy

Error Handling

Metrics and Monitoring

Prometheus Metrics

Health Checks

Development

Project Structure

Running Tests

Code Quality

Deployment

Systemd Service

Docker Compose

Kubernetes Deployment

Security

Authentication

Best Practices

Performance

Benchmarks

Optimization Tips

Troubleshooting

Common Issues

Service won't start

Extensions not found

Slow responses

Logging

License

Contributing

Support

14 KiB

Raw Permalink Blame History