prvng_platform/extension-registry/README.md

# Extension Registry Service

A high-performance Rust microservice that provides a unified REST API for extension discovery, versioning, and download from multiple sources (Gitea releases and OCI registries).

## Features

- **Multi-Backend Support**: Fetch extensions from Gitea releases and OCI registries
- **Unified REST API**: Single API for all extension operations
- **Smart Caching**: LRU cache with TTL to reduce backend API calls
- **Prometheus Metrics**: Built-in metrics for monitoring
- **Health Monitoring**: Health checks for all backends
- **Type-Safe**: Strong typing for extension metadata
- **Async/Await**: High-performance async operations with Tokio
- **Docker Support**: Production-ready containerization

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    Extension Registry API                    │
│                         (axum)                               │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌────────────────┐  ┌────────────────┐  ┌──────────────┐ │
│  │  Gitea Client  │  │   OCI Client   │  │  LRU Cache   │ │
│  │  (reqwest)     │  │   (reqwest)    │  │  (parking)   │ │
│  └────────────────┘  └────────────────┘  └──────────────┘ │
│         │                    │                    │         │
└─────────┼────────────────────┼────────────────────┼─────────┘
          │                    │                    │
          ▼                    ▼                    ▼
    ┌──────────┐         ┌──────────┐        ┌──────────┐
    │  Gitea   │         │   OCI    │        │  Memory  │
    │ Releases │         │ Registry │        │          │
    └──────────┘         └──────────┘        └──────────┘
```

## Installation

### Building from Source

```bash
cd provisioning/platform/extension-registry
cargo build --release
```

### Docker Build

```bash
docker build -t extension-registry:latest .
```

### Running with Cargo

```bash
cargo run -- --config config.toml --port 8082
```

### Running with Docker

```bash
docker run -d \
  -p 8082:8082 \
  -v $(pwd)/config.toml:/app/config.toml:ro \
  -v $(pwd)/tokens:/app/tokens:ro \
  extension-registry:latest
```

## Configuration

Create a `config.toml` file (see `config.example.toml`):

```toml
[server]
host = "0.0.0.0"
port = 8082
workers = 4
enable_cors = true
enable_compression = true

# Gitea backend (optional)
[gitea]
url = "https://gitea.example.com"
organization = "provisioning-extensions"
token_path = "/path/to/gitea-token.txt"
timeout_seconds = 30
verify_ssl = true

# OCI registry backend (optional)
[oci]
registry = "registry.example.com"
namespace = "provisioning"
auth_token_path = "/path/to/oci-token.txt"
timeout_seconds = 30
verify_ssl = true

# Cache configuration
[cache]
capacity = 1000
ttl_seconds = 300
enable_metadata_cache = true
enable_list_cache = true
```

**Note**: At least one backend (Gitea or OCI) must be configured.

## API Endpoints

### Extension Operations

#### List Extensions

```bash
GET /api/v1/extensions
```

Query parameters:
- `type` (optional): Filter by extension type (`provider`, `taskserv`, `cluster`)
- `source` (optional): Filter by source (`gitea`, `oci`)
- `limit` (optional): Maximum results (default: 100)
- `offset` (optional): Pagination offset (default: 0)

Example:
```bash
curl http://localhost:8082/api/v1/extensions?type=provider&limit=10
```

Response:
```json
[
  {
    "name": "aws",
    "type": "provider",
    "version": "1.2.0",
    "description": "AWS provider for provisioning",
    "author": "provisioning-team",
    "repository": "https://gitea.example.com/org/aws_prov",
    "source": "gitea",
    "published_at": "2025-10-06T12:00:00Z",
    "download_url": "https://gitea.example.com/org/aws_prov/releases/download/1.2.0/aws_prov.tar.gz",
    "size": 1024000
  }
]
```

#### Get Extension

```bash
GET /api/v1/extensions/{type}/{name}
```

Example:
```bash
curl http://localhost:8082/api/v1/extensions/provider/aws
```

#### List Versions

```bash
GET /api/v1/extensions/{type}/{name}/versions
```

Example:
```bash
curl http://localhost:8082/api/v1/extensions/provider/aws/versions
```

Response:
```json
[
  {
    "version": "1.2.0",
    "published_at": "2025-10-06T12:00:00Z",
    "download_url": "https://gitea.example.com/org/aws_prov/releases/download/1.2.0/aws_prov.tar.gz",
    "size": 1024000
  },
  {
    "version": "1.1.0",
    "published_at": "2025-09-15T10:30:00Z",
    "download_url": "https://gitea.example.com/org/aws_prov/releases/download/1.1.0/aws_prov.tar.gz",
    "size": 980000
  }
]
```

#### Download Extension

```bash
GET /api/v1/extensions/{type}/{name}/{version}
```

Example:
```bash
curl -O http://localhost:8082/api/v1/extensions/provider/aws/1.2.0
```

Returns binary data with `Content-Type: application/octet-stream`.

#### Search Extensions

```bash
GET /api/v1/extensions/search?q={query}
```

Query parameters:
- `q` (required): Search query
- `type` (optional): Filter by extension type
- `limit` (optional): Maximum results (default: 50)

Example:
```bash
curl http://localhost:8082/api/v1/extensions/search?q=kubernetes&type=taskserv
```

### System Endpoints

#### Health Check

```bash
GET /api/v1/health
```

Example:
```bash
curl http://localhost:8082/api/v1/health
```

Response:
```json
{
  "status": "healthy",
  "version": "0.1.0",
  "uptime": 3600,
  "backends": {
    "gitea": {
      "enabled": true,
      "healthy": true
    },
    "oci": {
      "enabled": true,
      "healthy": true
    }
  }
}
```

#### Metrics

```bash
GET /api/v1/metrics
```

Returns Prometheus-formatted metrics:
```
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total 1234

# HELP cache_hits_total Total cache hits
# TYPE cache_hits_total counter
cache_hits_total 567

# HELP cache_misses_total Total cache misses
# TYPE cache_misses_total counter
cache_misses_total 123
```

#### Cache Statistics

```bash
GET /api/v1/cache/stats
```

Response:
```json
{
  "list_entries": 45,
  "metadata_entries": 120,
  "version_entries": 80,
  "total_entries": 245
}
```

## Extension Naming Conventions

### Gitea Repositories

Extensions in Gitea follow specific naming patterns:

- **Providers**: `{name}_prov` (e.g., `aws_prov`, `upcloud_prov`)
- **Task Services**: `{name}_taskserv` (e.g., `kubernetes_taskserv`, `postgres_taskserv`)
- **Clusters**: `{name}_cluster` (e.g., `buildkit_cluster`, `ci_cluster`)

### OCI Artifacts

Extensions in OCI registries follow these patterns:

- **Providers**: `{namespace}/{name}-provider` (e.g., `provisioning/aws-provider`)
- **Task Services**: `{namespace}/{name}-taskserv` (e.g., `provisioning/kubernetes-taskserv`)
- **Clusters**: `{namespace}/{name}-cluster` (e.g., `provisioning/buildkit-cluster`)

## Caching Strategy

The service implements a multi-level LRU cache with TTL:

1. **List Cache**: Caches extension lists (filtered by type/source)
2. **Metadata Cache**: Caches individual extension metadata
3. **Version Cache**: Caches version lists per extension

Cache behavior:
- **Capacity**: Configurable (default: 1000 entries)
- **TTL**: Configurable (default: 5 minutes)
- **Eviction**: LRU (Least Recently Used)
- **Invalidation**: Automatic on TTL expiration

Cache keys:
- List: `list:{type}:{source}`
- Metadata: `{type}/{name}`
- Versions: `{type}/{name}/versions`

## Error Handling

The API uses standard HTTP status codes:

- `200 OK`: Successful operation
- `400 Bad Request`: Invalid input (e.g., invalid extension type)
- `401 Unauthorized`: Authentication failed
- `404 Not Found`: Extension not found
- `429 Too Many Requests`: Rate limit exceeded
- `500 Internal Server Error`: Server error

Error response format:
```json
{
  "error": "not_found",
  "message": "Extension provider/nonexistent not found"
}
```

## Metrics and Monitoring

### Prometheus Metrics

Available metrics:
- `http_requests_total`: Total HTTP requests
- `http_request_duration_seconds`: Request duration histogram
- `cache_hits_total`: Total cache hits
- `cache_misses_total`: Total cache misses
- `extensions_total`: Total extensions served

### Health Checks

The health endpoint checks:
- Service uptime
- Gitea backend connectivity
- OCI backend connectivity
- Overall service status

## Development

### Project Structure

```
extension-registry/
├── Cargo.toml              # Rust dependencies
├── src/
│   ├── main.rs             # Entry point
│   ├── lib.rs              # Library exports
│   ├── config.rs           # Configuration management
│   ├── error.rs            # Error types
│   ├── api/
│   │   ├── handlers.rs     # HTTP handlers
│   │   └── routes.rs       # Route definitions
│   ├── gitea/
│   │   ├── client.rs       # Gitea API client
│   │   └── models.rs       # Gitea data models
│   ├── oci/
│   │   ├── client.rs       # OCI registry client
│   │   └── models.rs       # OCI data models
│   ├── cache/
│   │   └── lru_cache.rs    # LRU caching
│   └── models/
│       └── extension.rs    # Extension models
├── tests/
│   └── integration_test.rs # Integration tests
├── Dockerfile              # Docker build
└── README.md               # This file
```

### Running Tests

```bash
# Run all tests
cargo test

# Run with output
cargo test -- --nocapture

# Run specific test
cargo test test_health_check
```

### Code Quality

```bash
# Format code
cargo fmt

# Run clippy
cargo clippy

# Check for security vulnerabilities
cargo audit
```

## Deployment

### Systemd Service

Create `/etc/systemd/system/extension-registry.service`:

```ini
[Unit]
Description=Extension Registry Service
After=network.target

[Service]
Type=simple
User=registry
WorkingDirectory=/opt/extension-registry
ExecStart=/usr/local/bin/extension-registry --config /etc/extension-registry/config.toml
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
```

Enable and start:
```bash
sudo systemctl enable extension-registry
sudo systemctl start extension-registry
sudo systemctl status extension-registry
```

### Docker Compose

```yaml
version: '3.8'

services:
  extension-registry:
    image: extension-registry:latest
    ports:
      - "8082:8082"
    volumes:
      - ./config.toml:/app/config.toml:ro
      - ./tokens:/app/tokens:ro
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8082/api/v1/health"]
      interval: 30s
      timeout: 3s
      retries: 3
      start_period: 5s
```

### Kubernetes Deployment

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: extension-registry
spec:
  replicas: 3
  selector:
    matchLabels:
      app: extension-registry
  template:
    metadata:
      labels:
        app: extension-registry
    spec:
      containers:
      - name: extension-registry
        image: extension-registry:latest
        ports:
        - containerPort: 8082
        volumeMounts:
        - name: config
          mountPath: /app/config.toml
          subPath: config.toml
        - name: tokens
          mountPath: /app/tokens
        livenessProbe:
          httpGet:
            path: /api/v1/health
            port: 8082
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /api/v1/health
            port: 8082
          initialDelaySeconds: 5
          periodSeconds: 5
      volumes:
      - name: config
        configMap:
          name: extension-registry-config
      - name: tokens
        secret:
          secretName: extension-registry-tokens
---
apiVersion: v1
kind: Service
metadata:
  name: extension-registry
spec:
  selector:
    app: extension-registry
  ports:
  - port: 8082
    targetPort: 8082
  type: ClusterIP
```

## Security

### Authentication

- Gitea: Token-based authentication via `token_path`
- OCI: Optional token authentication via `auth_token_path`

### Best Practices

1. **Store tokens securely**: Use file permissions (600) for token files
2. **Enable SSL verification**: Set `verify_ssl = true` in production
3. **Use HTTPS**: Always use HTTPS for Gitea and OCI registries
4. **Limit CORS**: Configure CORS appropriately for production
5. **Rate limiting**: Consider adding rate limiting for public APIs
6. **Network isolation**: Run service in isolated network segments

## Performance

### Benchmarks

Typical performance characteristics:
- **Cached requests**: <5ms response time
- **Uncached requests**: 50-200ms (depends on backend latency)
- **Cache hit ratio**: ~85-95% in typical workloads
- **Throughput**: 1000+ req/s on modern hardware

### Optimization Tips

1. **Increase cache capacity**: For large extension catalogs
2. **Tune TTL**: Balance freshness vs. performance
3. **Use multiple workers**: Scale with CPU cores
4. **Enable compression**: Reduce bandwidth usage
5. **Connection pooling**: Reuse HTTP connections to backends

## Troubleshooting

### Common Issues

#### Service won't start

- Check configuration file syntax
- Verify token files exist and are readable
- Check network connectivity to backends

#### Extensions not found

- Verify backend configuration (URL, organization, namespace)
- Check backend connectivity with health endpoint
- Review logs for authentication errors

#### Slow responses

- Check backend latency
- Increase cache capacity or TTL
- Review Prometheus metrics for bottlenecks

### Logging

Enable debug logging:
```bash
extension-registry --log-level debug
```

Enable JSON logging for structured logs:
```bash
extension-registry --json-log
```

## License

Part of the Provisioning Project.

## Contributing

See main project documentation for contribution guidelines.

## Support

For issues and questions, please refer to the main provisioning project documentation.