kogral/docs/architecture/config-driven.md

# Config-Driven Architecture

The KOGRAL follows a **config-driven architecture** where all behavior is defined through Nickel configuration files rather than hardcoded in Rust.

## Philosophy

**"Configuration, not code, defines behavior"**

Instead of hardcoding storage backends, embedding providers, or query parameters, KOGRAL uses a layered configuration system that composes settings from multiple sources:

1. **Schema contracts** (type definitions)
2. **Defaults** (base values)
3. **Mode overlays** (dev/prod/test optimizations)
4. **User customizations** (project-specific overrides)

This approach provides:

- ✅ **Type safety** - Nickel contracts validate configuration before runtime
- ✅ **Composability** - Mix and match configurations for different environments
- ✅ **Discoverability** - Self-documenting schemas with inline documentation
- ✅ **Hot-reload** - Change behavior without recompiling Rust code
- ✅ **Double validation** - Nickel contracts + serde ensure correctness

## Configuration Composition Flow

![Configuration Composition](../diagrams/config-composition.svg)

The configuration system uses a **four-layer composition pattern**:

### Layer 1: Schema Contracts

**Location**: `schemas/contracts.ncl`

**Purpose**: Define types and validation rules using Nickel contracts.

**Example**:

```nickel
{
  StorageType = [| 'filesystem, 'memory, 'surrealdb |],

  StorageConfig = {
    primary | StorageType
      | doc "Primary storage backend"
      | default = 'filesystem,

    secondary | SecondaryStorageConfig
      | doc "Optional secondary storage"
      | default = { enabled = false },
  },
}
```

**Benefits**:
- Enum validation (only valid storage types accepted)
- Required vs optional fields
- Default values for optional fields
- Documentation attached to types

### Layer 2: Defaults

**Location**: `schemas/defaults.ncl`

**Purpose**: Provide sensible base values for all configuration options.

**Example**:

```nickel
{
  base = {
    storage = {
      primary = 'filesystem,
      secondary = {
        enabled = false,
        type = 'surrealdb,
        url = "ws://localhost:8000",
      },
    },
    embeddings = {
      enabled = true,
      provider = 'fastembed,
      model = "BAAI/bge-small-en-v1.5",
      dimensions = 384,
    },
  } | contracts.KbConfig,
}
```

**Validated by**: `contracts.KbConfig` contract ensures defaults are valid.

### Layer 3: Mode Overlays

**Location**: `schemas/modes/{dev,prod,test}.ncl`

**Purpose**: Environment-specific optimizations and tuning.

#### Development Mode (`dev.ncl`)

Optimized for: Fast iteration, local development, debugging

```nickel
{
  storage = {
    primary = 'filesystem,
    secondary = { enabled = false },  # No database overhead
  },
  embeddings = {
    provider = 'fastembed,  # Local, no API costs
  },
  sync = {
    auto_index = false,  # Manual control
  },
}
```

#### Production Mode (`prod.ncl`)

Optimized for: Performance, reliability, scalability

```nickel
{
  storage = {
    secondary = { enabled = true },  # SurrealDB for scale
  },
  embeddings = {
    provider = 'openai,  # High-quality cloud embeddings
    model = "text-embedding-3-small",
    dimensions = 1536,
  },
  sync = {
    auto_index = true,
    debounce_ms = 300,  # Fast response
  },
}
```

#### Test Mode (`test.ncl`)

Optimized for: Fast tests, isolation, determinism

```nickel
{
  storage = {
    primary = 'memory,  # Ephemeral, no disk I/O
  },
  embeddings = {
    enabled = false,  # Disable for speed
  },
  sync = {
    auto_index = false,
    debounce_ms = 0,  # No delays in tests
  },
}
```

### Layer 4: User Customizations

**Location**: `.kogral-config/core/kogral.ncl` or `.kogral-config/platform/{dev,prod,test}.ncl`

**Purpose**: Project-specific or deployment-specific overrides.

**Example** (user project config):

```nickel
let mode = import "../../schemas/modes/dev.ncl" in

let user_custom = {
  graph = {
    name = "my-project",
  },
  embeddings = {
    provider = 'claude,  # Override to use Claude
    model = "claude-3-haiku-20240307",
  },
  query = {
    similarity_threshold = 0.7,  # Stricter threshold
  },
} in

helpers.compose_config defaults.base mode user_custom
  | contracts.KbConfig
```

## Composition Mechanism

The `helpers.ncl` module provides the composition function:

```nickel
{
  # Recursively merge with override precedence
  merge_with_override = fun base override => /* ... */,

  # Compose three layers
  compose_config = fun defaults mode_config user_custom =>
    let with_mode = merge_with_override defaults mode_config in
    merge_with_override with_mode user_custom,
}
```

**Merge behavior**:
- Records are merged recursively
- Override values take precedence over base values
- Arrays are not merged, override replaces base
- Null in override keeps base value

**Example merge**:

```nickel
base = { storage = { primary = 'filesystem }, embeddings = { enabled = true } }
override = { storage = { primary = 'memory } }
# Result: { storage = { primary = 'memory }, embeddings = { enabled = true } }
```

## Export to JSON

Once composed, the Nickel configuration is exported to JSON for Rust consumption:

```bash
nickel export --format json .kogral-config/core/kogral.ncl > .kogral-config/targets/kogral-core.json
```

**Output** (`.kogral-config/targets/kogral-core.json`):

```json
{
  "graph": {
    "name": "my-project",
    "version": "1.0.0"
  },
  "storage": {
    "primary": "memory",
    "secondary": {
      "enabled": false
    }
  },
  "embeddings": {
    "enabled": true,
    "provider": "claude",
    "model": "claude-3-haiku-20240307",
    "dimensions": 768
  }
}
```

## Rust Integration

The Rust code deserializes the JSON into typed structs using serde:

```rust
use serde::{Deserialize, Serialize};

#[derive(Debug, Deserialize, Serialize)]
pub struct KbConfig {
    pub graph: GraphConfig,
    pub storage: StorageConfig,
    pub embeddings: EmbeddingConfig,
    pub templates: TemplateConfig,
    pub query: QueryConfig,
    pub mcp: McpConfig,
    pub sync: SyncConfig,
}

impl KbConfig {
    pub fn from_file(path: &Path) -> Result<Self> {
        let json = std::fs::read_to_string(path)?;
        let config: KbConfig = serde_json::from_str(&json)?;
        Ok(config)
    }
}
```

**Usage in kogral-core**:

```rust
let config = KbConfig::from_file(".kogral-config/targets/kogral-core.json")?;

// Config drives behavior
let storage: Box<dyn Storage> = match config.storage.primary {
    StorageType::Filesystem => Box::new(FilesystemStorage::new(&config)?),
    StorageType::Memory => Box::new(MemoryStorage::new()),
    StorageType::SurrealDb => Box::new(SurrealDbStorage::new(&config).await?),
};

let embeddings: Box<dyn EmbeddingProvider> = match config.embeddings.provider {
    EmbeddingProviderType::FastEmbed => Box::new(FastEmbedProvider::new()?),
    EmbeddingProviderType::OpenAI => Box::new(RigEmbeddingProvider::openai(&config)?),
    EmbeddingProviderType::Claude => Box::new(RigEmbeddingProvider::claude(&config)?),
    EmbeddingProviderType::Ollama => Box::new(RigEmbeddingProvider::ollama(&config)?),
};
```

## Double Validation

Configuration is validated **twice**:

### 1. Nickel Contract Validation

At export time, Nickel validates:
- ✅ Types match contracts (e.g., `primary | StorageType`)
- ✅ Required fields are present
- ✅ Enums have valid values
- ✅ Nested structure is correct

**Error example**:

```text
error: contract broken by a value
  ┌─ .kogral-config/core/kogral.ncl:15:5
  │
15│     primary = 'invalid,
  │     ^^^^^^^^^^^^^^^^^^^ applied to this expression
  │
  = This value is not in the enum ['filesystem, 'memory, 'surrealdb]
```

### 2. Serde Deserialization Validation

At runtime, serde validates:
- ✅ JSON structure matches Rust types
- ✅ Field names match (with rename support)
- ✅ Values can be converted to Rust types
- ✅ Required fields are not null

**Error example**:

```rust
Error: missing field `graph` at line 1 column 123
```

## Benefits of Config-Driven Architecture

### 1. Zero Hardcoding

**Bad** (hardcoded):

```rust
// Hardcoded - requires recompilation to change
let storage = FilesystemStorage::new("/fixed/path");
let threshold = 0.6;
```

**Good** (config-driven):

```rust
// Config-driven - change via .ncl file
let storage = create_storage(&config)?;
let threshold = config.query.similarity_threshold;
```

### 2. Environment Flexibility

Same codebase, different behavior:

```bash
# Development
nickel export .kogral-config/platform/dev.ncl > targets/kogral-core.json
# → Filesystem storage, fastembed, no auto-sync

# Production
nickel export .kogral-config/platform/prod.ncl > targets/kogral-core.json
# → SurrealDB enabled, OpenAI embeddings, auto-sync

# Testing
nickel export .kogral-config/platform/test.ncl > targets/kogral-core.json
# → In-memory storage, no embeddings, isolated
```

### 3. Self-Documenting

Nickel contracts include inline documentation:

```nickel
StorageType = [| 'filesystem, 'memory, 'surrealdb |]
  | doc "Storage backend type: filesystem (git-tracked), memory (ephemeral), surrealdb (scalable)",
```

IDEs can show this documentation when editing `.ncl` files.

### 4. Type-Safe Evolution

When adding new features:

1. Update contract in `contracts.ncl`
2. Add default in `defaults.ncl`
3. Export validates existing configs
4. Rust compilation validates deserialization

Breaking changes are caught **before runtime**.

### 5. Testability

Different test scenarios without code changes:

```nickel
# test-semantic-search.ncl
let test_config = defaults.base & {
  embeddings = { enabled = true, provider = 'fastembed },
  query = { similarity_threshold = 0.3 },
} in test_config
```

```rust
#[test]
fn test_semantic_search() {
    let config = KbConfig::from_file("test-semantic-search.json")?;
    // Config drives test behavior
}
```

## Configuration Discovery

KOGRAL tools automatically discover configuration:

1. **Check `.kogral-config/targets/kogral-core.json`** (pre-exported)
2. **Check `.kogral-config/core/kogral.ncl`** (export on-demand)
3. **Check environment variable `KOGRAL_CONFIG`**
4. **Fall back to embedded defaults**

```rust
impl KbConfig {
    pub fn discover() -> Result<Self> {
        if let Ok(config) = Self::from_file(".kogral-config/targets/kogral-core.json") {
            return Ok(config);
        }

        if Path::new(".kogral-config/core/kogral.ncl").exists() {
            // Export and load
            let output = Command::new("nickel")
                .args(["export", "--format", "json", ".kogral-config/core/kogral.ncl"])
                .output()?;
            return serde_json::from_slice(&output.stdout)?;
        }

        if let Ok(path) = std::env::var("KOGRAL_CONFIG") {
            return Self::from_file(&path);
        }

        Ok(Self::default())  // Embedded defaults
    }
}
```

## Integration with justfile

The `justfile` integrates configuration validation:

```just
# Validate all Nickel configs
nickel-validate-all:
    @echo "Validating Nickel schemas..."
    nickel typecheck schemas/contracts.ncl
    nickel typecheck schemas/defaults.ncl
    nickel typecheck schemas/helpers.ncl
    nickel typecheck schemas/modes/dev.ncl
    nickel typecheck schemas/modes/prod.ncl
    nickel typecheck schemas/modes/test.ncl
    nickel typecheck .kogral-config/core/kogral.ncl

# Export all platform configs
nickel-export-all:
    @echo "Exporting platform configs to JSON..."
    @mkdir -p .kogral-config/targets
    nickel export --format json .kogral-config/platform/dev.ncl > .kogral-config/targets/kogral-dev.json
    nickel export --format json .kogral-config/platform/prod.ncl > .kogral-config/targets/kogral-prod.json
    nickel export --format json .kogral-config/platform/test.ncl > .kogral-config/targets/kogral-test.json
    @echo "Exported 3 configurations to .kogral-config/targets/"
```

Usage:

```bash
just nickel::validate-all  # Check configs are valid
just nickel::export-all    # Generate JSON for all environments
```

## Best Practices

### 1. Never Hardcode

If it's behavior, it's config:
- Storage paths
- API endpoints
- Thresholds
- Timeouts
- Feature flags
- Provider selection

### 2. Use Modes for Environment Differences

Don't put environment-specific values in user config:

**Bad**:

```nickel
# user config with env-specific values
{
  storage = {
    url = if env == "prod" then "prod-url" else "dev-url"  # Don't do this
  }
}
```

**Good**:

```nickel
# modes/prod.ncl
{ storage = { url = "prod-url" } }

# modes/dev.ncl
{ storage = { url = "dev-url" } }

# user config is environment-agnostic
{ graph = { name = "my-project" } }
```

### 3. Document Complex Fields

Use Nickel's `doc` metadata:

```nickel
similarity_threshold | Number
  | doc "Minimum cosine similarity (0-1) for semantic search matches. Higher = stricter."
  | default = 0.6,
```

### 4. Validate Early

Run `nickel typecheck` in CI/CD before building Rust code.

### 5. Version Configs

Track `.ncl` files in git, ignore `.kogral-config/targets/*.json` (generated).

## See Also

- **Schema Reference**: [Configuration Schema](../config/schema.md)
- **User Guide**: [Configuration Guide](../config/overview.md)
- **ADR**: [Why Nickel vs TOML](adrs/001-nickel-vs-toml.md)
- **Examples**: `.kogral-config/core/kogral.ncl`, `.kogral-config/platform/*.ncl`