syntaxis/docs/service-catalog-module.md

# Service Catalog Module: Reusable Abstraction

## Overview

The Service Catalog system (implemented in syntaxis) can be **extracted and reused by any application** that needs to manage services, presets, and provisioning integration.

This document describes how to:
1. **Understand the abstraction** - What makes it reusable
2. **Implement in your app** - How to adapt it to other projects
3. **Integrate with provisioners** - How to connect with project-provisioning or similar systems

---

## What Is a Service Catalog?

A **Service Catalog** is a **standardized TOML configuration** that completely describes:

1. **Available Services**: What can be installed/deployed
2. **Service Metadata**: How to run, health check, port, dependencies
3. **Presets**: Combinations of services for different contexts
4. **Patterns**: Common deployment topologies
5. **Networking**: Port assignments and protocols

**Key Principle**: One source of truth for service definitions that both local and remote systems can read independently.

---

## The Abstraction

### Core Components

```
Service Catalog System
├── Configuration Schema (TOML)
│   ├── Service definitions
│   ├── Preset compositions
│   ├── Service groups/patterns
│   └── Port mappings
├── Discovery Tool
│   ├── List services
│   ├── Show usage examples
│   ├── View preset compositions
│   └── Export for provisioners
└── Integration Bridge
    ├── Export presets as YAML
    ├── Generate provisioning configs
    └── Validate deployments
```

### TOML Schema (Simplified)

```toml
[version]
catalog_version = "1.0.0"

[service.my-service]
name = "my-service"
display_name = "My Service"
description = "Service description"
type = "server"  # or "cli", "database", "web", etc.

[service.my-service.server]
default_host = "127.0.0.1"
default_port = 3000
scheme = "http"

[service.my-service.usage]
basic_command = "my-service --bind 127.0.0.1:3000"
health_check_endpoint = "/health"
examples = [...]

[service.my-service.dependencies]
requires = []
conflicts = []

[preset.my-preset]
name = "My Preset"
description = "Preset description"
database_backend = "sqlite"

[preset.my-preset.services]
my_service = { enabled = true, auto_start = true, port = 3000 }
```

---

## Implementation Steps

### Step 1: Create the Catalog Schema

Create `configs/services-catalog.toml` in your project with this structure:

```toml
[version]
catalog_version = "1.0.0"
app_version = "1.0.0"

# Define all services
[service.app-api]
name = "app-api"
display_name = "API Server"
description = "REST API for the application"
type = "server"
required = false

[service.app-api.server]
default_host = "127.0.0.1"
default_port = 3000
scheme = "http"

[service.app-api.usage]
basic_command = "app-api --bind 127.0.0.1:3000"
health_check_endpoint = "/health"
examples = ["app-api --bind 127.0.0.1:3000"]

[service.app-api.dependencies]
requires = []
conflicts = []

[service.app-api.metadata]
platform_support = ["linux", "macos"]
min_memory_mb = 256
min_disk_space_mb = 100

# Define presets
[preset.local]
name = "Local Development"
description = "Single-machine development"

[preset.local.services]
app_api = { enabled = true, auto_start = false, port = 3000 }

[preset.production]
name = "Production"
description = "Production deployment"

[preset.production.services]
app_api = { enabled = true, auto_start = true, port = 3000 }
```

### Step 2: Create Discovery Tool

Create a simple script to help users explore the catalog:

**Option A: Simple Bash**
```bash
#!/bin/bash
# scripts/service-catalog.sh

echo "Available Services:"
grep '^\[service\.' configs/services-catalog.toml | sed 's/\[service\.//' | sed 's/\]//'

echo ""
echo "Available Presets:"
grep '^\[preset\.' configs/services-catalog.toml | sed 's/\[preset\.//' | sed 's/\]//'
```

**Option B: NuShell** (See syntax-catalog.nu for examples)

### Step 3: Integrate with Your Installation

Update your installation script to read from the catalog:

```bash
# scripts/install.sh
CATALOG=$(cat configs/services-catalog.toml)
PRESET=${1:-local}

# Extract services from preset
SERVICES=$(grep -A 10 "^\[preset.$PRESET.services\]" configs/services-catalog.toml)

# Install each enabled service
echo "$SERVICES" | while read line; do
    if [[ $line == *"enabled = true"* ]]; then
        SERVICE=$(echo $line | cut -d= -f1)
        echo "Installing $SERVICE..."
        # ... installation logic
    fi
done
```

### Step 4: Export for Provisioning

Create an export function that converts the catalog to YAML for provisioners:

```bash
# scripts/export-catalog.sh
PRESET=$1

cat > /tmp/export.yaml << EOF
preset: $PRESET
services:
EOF

grep -A 20 "^\[preset.$PRESET.services\]" configs/services-catalog.toml | \
  while read line; do
    if [[ $line == *"enabled = true"* ]]; then
      echo "  - $(echo $line | cut -d= -f1)" >> /tmp/export.yaml
    fi
  done

cat /tmp/export.yaml
```

---

## Using with Provisioners

### Integration Pattern 1: Direct File Reading

```bash
# provisioner reads catalog directly
provisioner \
  --catalog myapp/configs/services-catalog.toml \
  --preset production \
  --output terraform/main.tf
```

### Integration Pattern 2: Export and Read

```bash
# export from app
./scripts/export-catalog.sh production > /tmp/myapp-services.yaml

# provisioner reads export
provisioner \
  --config /tmp/myapp-services.yaml \
  --backend kubernetes \
  --output k8s/deployment.yaml
```

### Integration Pattern 3: API/Bridge

```bash
# Create a simple HTTP bridge
python -m http.server --directory configs/

# Provisioner fetches
curl http://localhost:8000/services-catalog.toml
```

---

## Reusing in Other Applications

### Example 1: Microservices App

```toml
[service.auth-service]
name = "auth-service"
type = "server"
[service.auth-service.server]
default_port = 4000

[service.user-service]
name = "user-service"
type = "server"
[service.user-service.server]
default_port = 4001

[service.data-service]
name = "data-service"
type = "database"
[service.data-service.server]
default_port = 5432

[preset.dev]
[preset.dev.services]
auth_service = { enabled = true }
user_service = { enabled = true }
data_service = { enabled = true }

[preset.production]
[preset.production.services]
auth_service = { enabled = true, replicas = 3 }
user_service = { enabled = true, replicas = 2 }
data_service = { enabled = true, replicas = 1 }
```

### Example 2: Data Pipeline

```toml
[service.data-ingress]
name = "data-ingress"
type = "server"
[service.data-ingress.server]
default_port = 8000

[service.processor]
name = "processor"
type = "worker"

[service.storage]
name = "storage"
type = "database"

[preset.local-dev]
[preset.local-dev.services]
data_ingress = { enabled = true }
processor = { enabled = true }
storage = { enabled = true }

[preset.cloud]
[preset.cloud.services]
data_ingress = { enabled = true, scale = "auto" }
processor = { enabled = true, scale = "auto" }
storage = { enabled = true, backup = true }
```

---

## Key Design Principles

### 1. Configuration-First

Everything is configuration, no hardcoding:
- Service definitions in TOML
- Preset combinations in TOML
- Ports, health checks, dependencies all declarative

### 2. Single Source of Truth

One file (`services-catalog.toml`) defines everything:
- Local tools read it for discovery
- Remote provisioners read it for deployment
- No duplication between systems

### 3. Minimal Tooling Required

Works with standard tools:
- TOML is text-based (git-friendly)
- Can be read by bash, Python, Go, NuShell, etc.
- No special infrastructure needed

### 4. Self-Documenting

The catalog is both specification and documentation:
- Service definitions include examples
- Health checks documented
- Dependencies explicit
- Usage patterns clear

---

## Extending the Module

### Adding Custom Fields

You can extend the schema for your specific needs:

```toml
[service.my-service]
# Standard fields
name = "my-service"
type = "server"

# Custom fields for your app
custom_field_1 = "value"
tags = ["tag1", "tag2"]
compliance = "hipaa"
cost_per_month = 100
```

### Adding Custom Patterns

Create application-specific deployment patterns:

```toml
[pattern.ml-training]
name = "ML Training Pipeline"
description = "For batch ML training jobs"
services = ["data-ingress", "preprocessor", "trainer", "model-storage"]

[pattern.ml-serving]
name = "ML Model Serving"
description = "For ML inference in production"
services = ["model-service", "feature-store", "metrics-collector"]
```

### Adding Custom Validators

Extend validation to your needs:

```bash
# scripts/validate-catalog.sh
validate_compliance_tags() {
    grep '^\[service\.' configs/services-catalog.toml | while read svc; do
        if ! grep -A 10 "$svc" configs/services-catalog.toml | grep -q "compliance ="; then
            echo "❌ $svc missing compliance tag"
        fi
    done
}
```

---

## Best Practices

### 1. Keep It Maintainable

- One service = one [service.xxx] block
- Group related fields together
- Use consistent naming

### 2. Document Services

```toml
[service.my-service]
# Always include usage examples
[service.my-service.usage]
examples = [
    "Command to run locally",
    "With config file: my-service --config config.toml",
    "In Docker: docker run myapp/my-service",
]
```

### 3. Version Your Catalog

```toml
[version]
catalog_version = "1.0.0"  # schema version
app_version = "1.0.0"      # your app version
updated_at = "2025-11-19"
```

### 4. Make It Testable

```bash
# scripts/test-catalog.sh
test_valid_toml() {
    nu -c "open configs/services-catalog.toml" > /dev/null
}

test_required_services() {
    # Your app might require certain services
    grep -q "app-api" configs/services-catalog.toml
}

test_valid_ports() {
    # Check no duplicate ports
    grep "default_port" configs/services-catalog.toml | sort | uniq -d | [ -z "$(cat)" ]
}
```

---

## Integration Checklist

- [ ] Create `configs/services-catalog.toml`
- [ ] Define all services with metadata
- [ ] Define presets for different contexts
- [ ] Create simple discovery tool (bash/NuShell)
- [ ] Update installation script to read from catalog
- [ ] Add export function for provisioners
- [ ] Document how to extend the catalog
- [ ] Add validation tests
- [ ] Version control the catalog
- [ ] Document in your project README

---

## Real-World Example: Using with project-provisioning

### Step 1: Create Catalog (myapp/configs/)
```toml
[service.my-api]
display_name = "My API"
type = "server"
[service.my-api.server]
default_port = 3000

[preset.production]
[preset.production.services]
my_api = { enabled = true }
```

### Step 2: Export (myapp/scripts/)
```bash
#!/bin/bash
nu -c "open configs/services-catalog.toml | get preset | get production"
```

### Step 3: Integrate (project-provisioning/scripts/)
```bash
#!/bin/bash
# Read from myapp
SERVICES=$(myapp/scripts/export-preset.sh production)

# Generate infrastructure
echo "$SERVICES" | while read service; do
    # Create terraform resource
    cat >> terraform/main.tf << EOF
resource "docker_container" "$service" {
    name = "$service"
    image = "myapp/$service:latest"
    ports = [...]  # extracted from catalog
}
EOF
done
```

---

## Conclusion

The Service Catalog abstraction is **language-agnostic** and **tool-agnostic**. It's a **simple but powerful pattern** for:

- Centralizing service definitions
- Enabling local discovery
- Integrating with remote provisioners
- Documenting infrastructure
- Maintaining single source of truth

Use it in any application where you need to manage multiple services or deployment contexts.

---

**Document Version**: 1.0.0
**Last Updated**: 2025-11-19
**Status**: Ready for implementation in external projects