syntaxis/docs/provisioning-architecture.md

# Syntaxis Provisioning Architecture

## Overview

This document describes how **syntaxis** (binary distribution) and **project-provisioning** (infrastructure orchestration) integrate together to provide a complete deployment solution.

**Key Principle**: Clear separation of concerns with syntaxis handling **what services are available** and project-provisioning handling **how to deploy them**.

---

## Architecture Diagram

```
┌─────────────────────────────────────────────────────────────────┐
│                         DEVELOPER/OPERATOR                       │
└────────────────────────┬────────────────────────────────────────┘
                         ↓
         ┌───────────────────────────────────┐
         │   1. LOCAL SETUP (laptop)         │
         │   ─────────────────────────────   │
         │   syntaxis installer:             │
         │  • Build binaries locally          │
         │  • Install to ~/.local/bin         │
         │  • Start services locally          │
         │  • Access via localhost            │
         └───────────────────┬───────────────┘
                             ↓
                  Outputs: binaries, configs
                             ↓
         ┌───────────────────────────────────┐
         │   2. REMOTE SETUP (staging/prod)  │
         │   ─────────────────────────────── │
         │   project-provisioning:           │
         │  • Reads service definitions      │
         │  • Provisions infrastructure      │
         │  • Deploys services              │
         │  • Manages multi-host            │
         │  • Handles observability         │
         └───────────────────┬───────────────┘
                             ↓
              Infrastructure running syntaxis
```

---

## Component Responsibilities

### SYNTAXIS (Binary Distribution Layer)

**Location**: `/Users/Akasha/Development/syntaxis`

**Responsibilities:**
1. ✅ Build and package binaries for multiple platforms
2. ✅ Define what services are available (`configs/services-catalog.toml`)
3. ✅ Show how to use each service (examples, ports, health checks)
4. ✅ Manage local installation and setup
5. ✅ Provide service metadata for integration

**Does NOT do:**
- ❌ Infrastructure provisioning (VMs, networks, storage)
- ❌ Multi-host deployment orchestration
- ❌ Service registry and discovery (beyond local)
- ❌ Secrets management
- ❌ Observability setup
- ❌ Complex deployment workflows

**Key Files:**
```
configs/
├── services-catalog.toml      ← Central service registry
├── installation.toml           ← Installation presets
└── provisioning/services/      ← Service definitions

scripts/provisioning/
├── service-catalog.nu          ← Service discovery tool
├── install-with-presets.nu     ← LOCAL installer only
├── pack.nu                      ← Bundle creation
├── install.nu                   ← Binary installation
└── deploy.nu                    ← Config deployment

docs/
├── INSTALLATION_CONTEXTS.md    ← User guide for LOCAL setup
└── PROVISIONING_ARCHITECTURE.md ← This file
```

### PROJECT-PROVISIONING (Orchestration Layer)

**Location**: `/Users/Akasha/project-provisioning`

**Responsibilities:**
1. ✅ Read service definitions from syntaxis
2. ✅ Provision infrastructure (Terraform, CloudFormation, etc.)
3. ✅ Deploy services to multiple hosts
4. ✅ Manage service registry and discovery
5. ✅ Setup secrets and configuration management
6. ✅ Configure monitoring and observability
7. ✅ Implement deployment workflows (rolling, canary, blue-green)
8. ✅ Handle multi-environment management
9. ✅ Integration via `provctl-bridge`

**Does NOT do:**
- ❌ Build binaries (syntaxis does that)
- ❌ Package distribution (syntaxis does that)
- ❌ User-facing installation UI (syntaxis does that)

**Integration Point:**
```
project-provisioning/platform/provctl-bridge/
├── Reads: configs/services-catalog.toml (from syntaxis)
├── Reads: configs/installation.toml (presets)
├── Reads: configs/provisioning/services/*.toml
└── Calls: provisioning/platform/* (deployment backends)
```

---

## Service Catalog: The Integration Contract

### What is the Service Catalog?

**File**: `configs/services-catalog.toml`

The service catalog is a **complete specification** of:
1. What services syntaxis provides
2. How to use each service
3. Service dependencies and compatibility
4. Networking requirements (ports, protocols)
5. Resource requirements (CPU, memory, disk)
6. Configuration and data locations

### Example: syntaxis-api

```toml
[service.syntaxis-api]
name = "syntaxis-api"
display_name = "syntaxis REST API"
description = "REST API server for programmatic access"
type = "server"

[service.syntaxis-api.server]
default_host = "127.0.0.1"
default_port = 3000
scheme = "http"

[service.syntaxis-api.usage]
health_check_endpoint = "/health"
examples = [
    "syntaxis-api --bind 127.0.0.1:3000",
    "curl http://127.0.0.1:3000/health",
]

[service.syntaxis-api.dependencies]
requires = ["syntaxis-cli"]
```

### How project-provisioning Uses It

```bash
# 1. Read service definitions
nu scripts/provisioning/service-catalog.nu export dev

# 2. Get service details
cat configs/services-catalog.toml | grep -A 20 '\[service.syntaxis-api\]'

# 3. Generate infrastructure as code
# For Terraform:
resource "docker_container" "syntaxis_api" {
    image = "syntaxis:latest"
    ports = {
        "3000" = "3000"  # from services-catalog.toml
    }
    healthcheck = {
        test = ["CMD", "curl", "-f", "http://localhost:3000/health"]
        # from service definition
    }
}

# 4. Deploy to infrastructure
provctl provision --preset dev --backend terraform
```

---

## How to Use the System

### Scenario 1: Local Development (Use syntaxis)

```bash
# 1. List available presets
nu scripts/provisioning/install-with-presets.nu --list-presets

# 2. View what's in a preset
nu scripts/provisioning/service-catalog.nu preset dev

# 3. Install locally
nu scripts/provisioning/install-with-presets.nu --preset dev

# 4. Start services
syntaxis-api --bind 127.0.0.1:3000
surreal start --bind 127.0.0.1:8000 memory

# 5. Check health
curl http://127.0.0.1:3000/health

# 6. Access dashboard
open http://127.0.0.1:8080  # requires syntaxis-api running
```

**Responsibility**: syntaxis
- Shows what's available (via service catalog)
- Installs binaries and configs locally
- Provides usage examples

### Scenario 2: Remote Deployment (Use project-provisioning)

```bash
# 1. Export preset for infrastructure deployment
nu scripts/provisioning/service-catalog.nu export staging

# 2. Use project-provisioning to deploy
cd /Users/Akasha/project-provisioning

# 3. Deploy using provctl-bridge
provctl provision \
    --preset staging \
    --version 0.1.0 \
    --environment staging \
    --backend terraform

# 4. Monitor deployment
provctl status staging

# 5. Access deployed services
curl https://api.staging.example.com/health
open https://dashboard.staging.example.com
```

**Responsibility**: project-provisioning
- Reads service definitions from syntaxis
- Provisions infrastructure
- Deploys services to remote infrastructure
- Manages secrets, monitoring, load balancing
- Handles multi-environment and high availability

### Scenario 3: Discovering Available Services

```bash
# List all services
nu scripts/provisioning/service-catalog.nu list

# Get details about a specific service
nu scripts/provisioning/service-catalog.nu info syntaxis-api

# See how to use a service
nu scripts/provisioning/service-catalog.nu usage syntaxis-api

# View port mappings
nu scripts/provisioning/service-catalog.nu ports

# See deployment patterns
nu scripts/provisioning/service-catalog.nu patterns
```

**Responsibility**: syntaxis (service catalog)
- Maintains complete service metadata
- Provides discovery and documentation
- Shows examples and usage patterns
- Enables integration with project-provisioning

---

## Integration Points

### 1. Service Definitions Contract

**What project-provisioning reads from syntaxis:**

```
configs/services-catalog.toml
├── Service metadata (name, description, type)
├── Network configuration (host, port, scheme)
├── Health checks (endpoint, method, status)
├── Dependencies (requires, conflicts)
├── Resource requirements (CPU, memory, disk)
├── Configuration locations
├── Usage examples
└── Deployment patterns
```

### 2. Preset Composition Contract

**How presets define service combinations:**

```toml
[preset.staging]
name = "Staging"
database_backend = "surrealdb"

[preset.staging.services]
syntaxis_cli = { enabled = false }
syntaxis_tui = { enabled = false }
syntaxis_api = { enabled = true, auto_start = true, port = 3000 }
surrealdb = { enabled = true, auto_start = true, port = 8000 }
nats = { enabled = true, auto_start = true, port = 4222 }
```

Project-provisioning reads this to understand:
- Which services to deploy
- Service dependencies and ordering
- Port assignments
- Auto-start behavior

### 3. Binary Distribution Contract

**What project-provisioning receives from syntaxis:**

```
Distribution bundles:
├── syntaxis-v0.1.0.tar.gz  ← Contains all binaries
├── SHA256SUMS              ← Integrity verification
└── manifest.toml           ← Installation tracking
```

Project-provisioning:
1. Verifies bundle integrity
2. Extracts binaries
3. Deploys to infrastructure
4. Starts services per preset

### 4. provctl-bridge Contract

**How project-provisioning calls syntaxis infrastructure code:**

```bash
# project-provisioning calls provctl-bridge to:
provctl-bridge read-services \
    --preset dev \
    --from ~/Development/syntaxis/configs

provctl-bridge generate-deployment \
    --preset staging \
    --backend kubernetes \
    --output kubernetes/manifest.yaml

provctl-bridge export-config \
    --preset production \
    --output terraform/variables.tfvars
```

---

## Data Flow Examples

### Example 1: User wants to deploy staging environment

```
Step 1: syntaxis provides service definitions
  ├─ User runs: nu service-catalog.nu preset staging
  └─ Output: YAML/JSON with service list, ports, health checks

Step 2: User exports for project-provisioning
  ├─ User runs: nu service-catalog.nu export staging > /tmp/staging.yaml
  └─ File contains: service names, ports, dependencies, examples

Step 3: project-provisioning reads service definitions
  ├─ Read: configs/services-catalog.toml
  ├─ Read: configs/installation.toml [preset.staging]
  └─ Output: Infrastructure as Code (Terraform)

Step 4: project-provisioning deploys
  ├─ Provision VMs for each service
  ├─ Install syntaxis binaries
  ├─ Start services per preset
  ├─ Setup monitoring and health checks
  └─ Configure load balancers and DNS

Step 5: User accesses deployed system
  ├─ curl https://api.staging.example.com/health
  ├─ open https://dashboard.staging.example.com
  └─ View metrics in Grafana
```

### Example 2: User adds a new service to syntaxis

```
Step 1: Define service in syntaxis
  ├─ Edit: configs/services-catalog.toml
  ├─ Add: [service.my-new-service]
  └─ Specify: binary, port, health check, dependencies

Step 2: Add to preset
  ├─ Edit: configs/installation.toml
  └─ Add to [preset.X.services]: my_new_service = { enabled = true }

Step 3: Build binary
  ├─ cargo build --release
  └─ nu scripts/provisioning/pack.nu

Step 4: Test service discovery
  ├─ nu service-catalog.nu info my-new-service
  ├─ nu service-catalog.nu preset dev
  └─ nu service-catalog.nu usage my-new-service

Step 5: project-provisioning auto-discovers
  ├─ Next deployment automatically includes new service
  ├─ Health checks configured automatically
  ├─ Port assignments automatic
  └─ No changes needed to provisioning code!
```

---

## Architecture Principles

### 1. Configuration Driven

Both systems use **TOML configuration** as the source of truth:
- No hardcoded service definitions
- Extensible without code changes
- Easy to audit and version control

### 2. Separation of Concerns

**syntaxis**: "What is available?"
- Binary distribution
- Service metadata
- Usage documentation

**project-provisioning**: "How to deploy it?"
- Infrastructure provisioning
- Multi-host orchestration
- Monitoring and observability

### 3. Delegation Pattern

When a request doesn't match a system's responsibility:
- syntaxis redirects to project-provisioning for remote deployments
- project-provisioning reads metadata from syntaxis

### 4. Contract-Based Integration

Systems communicate via:
- Configuration files (TOML)
- Exported data (YAML/JSON)
- Well-defined interfaces (provctl-bridge)
- No tight coupling or hardcoded paths

---

## Future Extensions

### For syntaxis:

1. **Enhanced Service Discovery**
   - Expose service catalog via REST API
   - Service availability check: `nu service-catalog.nu validate-local`
   - Real-time service status monitoring

2. **Advanced Usage Patterns**
   - Service dependency graphing
   - Automatic startup ordering
   - Service interaction examples

3. **Configuration Generation**
   - Generate nginx/caddy configs from port definitions
   - Generate docker-compose.yml from preset
   - Generate systemd unit files

### For project-provisioning integration:

1. **Bidirectional Updates**
   - project-provisioning exports deployment status back to syntaxis
   - Service health propagates from infrastructure to local
   - Metrics flow back to developer environment

2. **Plugin System**
   - Custom validators for service dependencies
   - Custom deployment backends
   - Custom health check definitions

3. **Template Generation**
   - Helm charts from presets
   - Terraform modules for each service
   - Kubernetes manifests from presets

---

## Implementation Status

### Current (Implemented)

✅ Service catalog with complete metadata
✅ Preset composition system
✅ Service discovery tool (service-catalog.nu)
✅ Usage documentation and examples
✅ Port and networking specifications
✅ Dependency tracking

### Planned

🔲 REST API for service catalog
🔲 Real-time service status monitoring
🔲 Advanced deployment patterns (canary, blue-green)
🔲 Automatic nginx/docker-compose generation
🔲 Bidirectional integration with project-provisioning

---

## Quick Reference

### For Developers (Local Setup)

```bash
# Discover what's available
nu scripts/provisioning/service-catalog.nu list

# See what's in a preset
nu scripts/provisioning/service-catalog.nu preset dev

# Install and setup locally
nu scripts/provisioning/install-with-presets.nu --preset dev

# Start services
syntaxis-api --bind 127.0.0.1:3000
```

### For DevOps (Remote Deployment)

```bash
# Export preset for infrastructure
nu scripts/provisioning/service-catalog.nu export staging > /tmp/staging.yaml

# Use project-provisioning to deploy
cd /Users/Akasha/project-provisioning
provctl provision --preset staging --backend kubernetes
```

### For Integration

```bash
# Read service metadata
cat configs/services-catalog.toml

# Check preset composition
cat configs/installation.toml | grep -A 20 '\[preset.production\]'

# Export for external systems
nu scripts/provisioning/service-catalog.nu export dev
```

---

## Related Documentation

- **[INSTALLATION_CONTEXTS.md](INSTALLATION_CONTEXTS.md)** - User guide for LOCAL setup
- **[ADVANCED_INSTALLATION.md](ADVANCED_INSTALLATION.md)** - Advanced scenarios and troubleshooting
- **[IMPLEMENTATION_SUMMARY.md](../IMPLEMENTATION_SUMMARY.md)** - Complete file manifest
- **[CLAUDE.md](../CLAUDE.md)** - Development reference
- **project-provisioning documentation** - Infrastructure orchestration details

---

**Document Version**: 1.0.0
**Last Updated**: 2025-11-19
**Architecture**: syntaxis (distribution) + project-provisioning (orchestration)