syntaxis/docs/service-catalog-module.md

Service Catalog Module: Reusable Abstraction

Overview

The Service Catalog system (implemented in syntaxis) can be extracted and reused by any application that needs to manage services, presets, and provisioning integration.

This document describes how to:

1. Understand the abstraction - What makes it reusable
2. Implement in your app - How to adapt it to other projects
3. Integrate with provisioners - How to connect with project-provisioning or similar systems

---

What Is a Service Catalog?

A Service Catalog is a standardized TOML configuration that completely describes:

1. Available Services: What can be installed/deployed
2. Service Metadata: How to run, health check, port, dependencies
3. Presets: Combinations of services for different contexts
4. Patterns: Common deployment topologies
5. Networking: Port assignments and protocols

Key Principle: One source of truth for service definitions that both local and remote systems can read independently.

---

The Abstraction

Core Components

Service Catalog System
├── Configuration Schema (TOML)
│   ├── Service definitions
│   ├── Preset compositions
│   ├── Service groups/patterns
│   └── Port mappings
├── Discovery Tool
│   ├── List services
│   ├── Show usage examples
│   ├── View preset compositions
│   └── Export for provisioners
└── Integration Bridge
    ├── Export presets as YAML
    ├── Generate provisioning configs
    └── Validate deployments

TOML Schema (Simplified)

[version]
catalog_version = "1.0.0"

[service.my-service]
name = "my-service"
display_name = "My Service"
description = "Service description"
type = "server"  # or "cli", "database", "web", etc.

[service.my-service.server]
default_host = "127.0.0.1"
default_port = 3000
scheme = "http"

[service.my-service.usage]
basic_command = "my-service --bind 127.0.0.1:3000"
health_check_endpoint = "/health"
examples = [...]

[service.my-service.dependencies]
requires = []
conflicts = []

[preset.my-preset]
name = "My Preset"
description = "Preset description"
database_backend = "sqlite"

[preset.my-preset.services]
my_service = { enabled = true, auto_start = true, port = 3000 }

---

Implementation Steps

Step 1: Create the Catalog Schema

Create configs/services-catalog.toml in your project with this structure:

[version]
catalog_version = "1.0.0"
app_version = "1.0.0"

# Define all services
[service.app-api]
name = "app-api"
display_name = "API Server"
description = "REST API for the application"
type = "server"
required = false

[service.app-api.server]
default_host = "127.0.0.1"
default_port = 3000
scheme = "http"

[service.app-api.usage]
basic_command = "app-api --bind 127.0.0.1:3000"
health_check_endpoint = "/health"
examples = ["app-api --bind 127.0.0.1:3000"]

[service.app-api.dependencies]
requires = []
conflicts = []

[service.app-api.metadata]
platform_support = ["linux", "macos"]
min_memory_mb = 256
min_disk_space_mb = 100

# Define presets
[preset.local]
name = "Local Development"
description = "Single-machine development"

[preset.local.services]
app_api = { enabled = true, auto_start = false, port = 3000 }

[preset.production]
name = "Production"
description = "Production deployment"

[preset.production.services]
app_api = { enabled = true, auto_start = true, port = 3000 }

Step 2: Create Discovery Tool

Create a simple script to help users explore the catalog:

Option A: Simple Bash

#!/bin/bash
# scripts/service-catalog.sh

echo "Available Services:"
grep '^\[service\.' configs/services-catalog.toml | sed 's/\[service\.//' | sed 's/\]//'

echo ""
echo "Available Presets:"
grep '^\[preset\.' configs/services-catalog.toml | sed 's/\[preset\.//' | sed 's/\]//'

Option B: NuShell (See syntax-catalog.nu for examples)

Step 3: Integrate with Your Installation

Update your installation script to read from the catalog:

# scripts/install.sh
CATALOG=$(cat configs/services-catalog.toml)
PRESET=${1:-local}

# Extract services from preset
SERVICES=$(grep -A 10 "^\[preset.$PRESET.services\]" configs/services-catalog.toml)

# Install each enabled service
echo "$SERVICES" | while read line; do
    if [[ $line == *"enabled = true"* ]]; then
        SERVICE=$(echo $line | cut -d= -f1)
        echo "Installing $SERVICE..."
        # ... installation logic
    fi
done

Step 4: Export for Provisioning

Create an export function that converts the catalog to YAML for provisioners:

# scripts/export-catalog.sh
PRESET=$1

cat > /tmp/export.yaml << EOF
preset: $PRESET
services:
EOF

grep -A 20 "^\[preset.$PRESET.services\]" configs/services-catalog.toml | \
  while read line; do
    if [[ $line == *"enabled = true"* ]]; then
      echo "  - $(echo $line | cut -d= -f1)" >> /tmp/export.yaml
    fi
  done

cat /tmp/export.yaml

---

Using with Provisioners

Integration Pattern 1: Direct File Reading

# provisioner reads catalog directly
provisioner \
  --catalog myapp/configs/services-catalog.toml \
  --preset production \
  --output terraform/main.tf

Integration Pattern 2: Export and Read

# export from app
./scripts/export-catalog.sh production > /tmp/myapp-services.yaml

# provisioner reads export
provisioner \
  --config /tmp/myapp-services.yaml \
  --backend kubernetes \
  --output k8s/deployment.yaml

Integration Pattern 3: API/Bridge

# Create a simple HTTP bridge
python -m http.server --directory configs/

# Provisioner fetches
curl http://localhost:8000/services-catalog.toml

---

Reusing in Other Applications

Example 1: Microservices App

[service.auth-service]
name = "auth-service"
type = "server"
[service.auth-service.server]
default_port = 4000

[service.user-service]
name = "user-service"
type = "server"
[service.user-service.server]
default_port = 4001

[service.data-service]
name = "data-service"
type = "database"
[service.data-service.server]
default_port = 5432

[preset.dev]
[preset.dev.services]
auth_service = { enabled = true }
user_service = { enabled = true }
data_service = { enabled = true }

[preset.production]
[preset.production.services]
auth_service = { enabled = true, replicas = 3 }
user_service = { enabled = true, replicas = 2 }
data_service = { enabled = true, replicas = 1 }

Example 2: Data Pipeline

[service.data-ingress]
name = "data-ingress"
type = "server"
[service.data-ingress.server]
default_port = 8000

[service.processor]
name = "processor"
type = "worker"

[service.storage]
name = "storage"
type = "database"

[preset.local-dev]
[preset.local-dev.services]
data_ingress = { enabled = true }
processor = { enabled = true }
storage = { enabled = true }

[preset.cloud]
[preset.cloud.services]
data_ingress = { enabled = true, scale = "auto" }
processor = { enabled = true, scale = "auto" }
storage = { enabled = true, backup = true }

---

Key Design Principles

1. Configuration-First

Everything is configuration, no hardcoding:

• Service definitions in TOML
• Preset combinations in TOML
• Ports, health checks, dependencies all declarative

2. Single Source of Truth

One file (services-catalog.toml) defines everything:

• Local tools read it for discovery
• Remote provisioners read it for deployment
• No duplication between systems

3. Minimal Tooling Required

Works with standard tools:

• TOML is text-based (git-friendly)
• Can be read by bash, Python, Go, NuShell, etc.
• No special infrastructure needed

4. Self-Documenting

The catalog is both specification and documentation:

• Service definitions include examples
• Health checks documented
• Dependencies explicit
• Usage patterns clear

---

Extending the Module

Adding Custom Fields

You can extend the schema for your specific needs:

[service.my-service]
# Standard fields
name = "my-service"
type = "server"

# Custom fields for your app
custom_field_1 = "value"
tags = ["tag1", "tag2"]
compliance = "hipaa"
cost_per_month = 100

Adding Custom Patterns

Create application-specific deployment patterns:

[pattern.ml-training]
name = "ML Training Pipeline"
description = "For batch ML training jobs"
services = ["data-ingress", "preprocessor", "trainer", "model-storage"]

[pattern.ml-serving]
name = "ML Model Serving"
description = "For ML inference in production"
services = ["model-service", "feature-store", "metrics-collector"]

Adding Custom Validators

Extend validation to your needs:

# scripts/validate-catalog.sh
validate_compliance_tags() {
    grep '^\[service\.' configs/services-catalog.toml | while read svc; do
        if ! grep -A 10 "$svc" configs/services-catalog.toml | grep -q "compliance ="; then
            echo "❌ $svc missing compliance tag"
        fi
    done
}

---

Best Practices

1. Keep It Maintainable

• One service = one [service.xxx] block
• Group related fields together
• Use consistent naming

2. Document Services

[service.my-service]
# Always include usage examples
[service.my-service.usage]
examples = [
    "Command to run locally",
    "With config file: my-service --config config.toml",
    "In Docker: docker run myapp/my-service",
]

3. Version Your Catalog

[version]
catalog_version = "1.0.0"  # schema version
app_version = "1.0.0"      # your app version
updated_at = "2025-11-19"

4. Make It Testable

# scripts/test-catalog.sh
test_valid_toml() {
    nu -c "open configs/services-catalog.toml" > /dev/null
}

test_required_services() {
    # Your app might require certain services
    grep -q "app-api" configs/services-catalog.toml
}

test_valid_ports() {
    # Check no duplicate ports
    grep "default_port" configs/services-catalog.toml | sort | uniq -d | [ -z "$(cat)" ]
}

---

Integration Checklist

• Create configs/services-catalog.toml
• Define all services with metadata
• Define presets for different contexts
• Create simple discovery tool (bash/NuShell)
• Update installation script to read from catalog
• Add export function for provisioners
• Document how to extend the catalog
• Add validation tests
• Version control the catalog
• Document in your project README

---

Real-World Example: Using with project-provisioning

Step 1: Create Catalog (myapp/configs/)

[service.my-api]
display_name = "My API"
type = "server"
[service.my-api.server]
default_port = 3000

[preset.production]
[preset.production.services]
my_api = { enabled = true }

Step 2: Export (myapp/scripts/)

#!/bin/bash
nu -c "open configs/services-catalog.toml | get preset | get production"

Step 3: Integrate (project-provisioning/scripts/)

#!/bin/bash
# Read from myapp
SERVICES=$(myapp/scripts/export-preset.sh production)

# Generate infrastructure
echo "$SERVICES" | while read service; do
    # Create terraform resource
    cat >> terraform/main.tf << EOF
resource "docker_container" "$service" {
    name = "$service"
    image = "myapp/$service:latest"
    ports = [...]  # extracted from catalog
}
EOF
done

---

Conclusion

The Service Catalog abstraction is language-agnostic and tool-agnostic. It's a simple but powerful pattern for:

• Centralizing service definitions
• Enabling local discovery
• Integrating with remote provisioners
• Documenting infrastructure
• Maintaining single source of truth

Use it in any application where you need to manage multiple services or deployment contexts.

---

Document Version: 1.0.0 Last Updated: 2025-11-19 Status: Ready for implementation in external projects