kogral/docs/architecture/config-driven.md

Config-Driven Architecture

The KOGRAL follows a config-driven architecture where all behavior is defined through Nickel configuration files rather than hardcoded in Rust.

Philosophy

"Configuration, not code, defines behavior"

Instead of hardcoding storage backends, embedding providers, or query parameters, KOGRAL uses a layered configuration system that composes settings from multiple sources:

1. Schema contracts (type definitions)
2. Defaults (base values)
3. Mode overlays (dev/prod/test optimizations)
4. User customizations (project-specific overrides)

This approach provides:

• ✅ Type safety - Nickel contracts validate configuration before runtime
• ✅ Composability - Mix and match configurations for different environments
• ✅ Discoverability - Self-documenting schemas with inline documentation
• ✅ Hot-reload - Change behavior without recompiling Rust code
• ✅ Double validation - Nickel contracts + serde ensure correctness

Configuration Composition Flow

The configuration system uses a four-layer composition pattern:

Layer 1: Schema Contracts

Location: schemas/contracts.ncl

Purpose: Define types and validation rules using Nickel contracts.

Example:

{
  StorageType = [| 'filesystem, 'memory, 'surrealdb |],

  StorageConfig = {
    primary | StorageType
      | doc "Primary storage backend"
      | default = 'filesystem,

    secondary | SecondaryStorageConfig
      | doc "Optional secondary storage"
      | default = { enabled = false },
  },
}

Benefits:

• Enum validation (only valid storage types accepted)
• Required vs optional fields
• Default values for optional fields
• Documentation attached to types

Layer 2: Defaults

Location: schemas/defaults.ncl

Purpose: Provide sensible base values for all configuration options.

Example:

{
  base = {
    storage = {
      primary = 'filesystem,
      secondary = {
        enabled = false,
        type = 'surrealdb,
        url = "ws://localhost:8000",
      },
    },
    embeddings = {
      enabled = true,
      provider = 'fastembed,
      model = "BAAI/bge-small-en-v1.5",
      dimensions = 384,
    },
  } | contracts.KbConfig,
}

Validated by: contracts.KbConfig contract ensures defaults are valid.

Layer 3: Mode Overlays

Location: schemas/modes/{dev,prod,test}.ncl

Purpose: Environment-specific optimizations and tuning.

Development Mode (dev.ncl)

Optimized for: Fast iteration, local development, debugging

{
  storage = {
    primary = 'filesystem,
    secondary = { enabled = false },  # No database overhead
  },
  embeddings = {
    provider = 'fastembed,  # Local, no API costs
  },
  sync = {
    auto_index = false,  # Manual control
  },
}

Production Mode (prod.ncl)

Optimized for: Performance, reliability, scalability

{
  storage = {
    secondary = { enabled = true },  # SurrealDB for scale
  },
  embeddings = {
    provider = 'openai,  # High-quality cloud embeddings
    model = "text-embedding-3-small",
    dimensions = 1536,
  },
  sync = {
    auto_index = true,
    debounce_ms = 300,  # Fast response
  },
}

Test Mode (test.ncl)

Optimized for: Fast tests, isolation, determinism

{
  storage = {
    primary = 'memory,  # Ephemeral, no disk I/O
  },
  embeddings = {
    enabled = false,  # Disable for speed
  },
  sync = {
    auto_index = false,
    debounce_ms = 0,  # No delays in tests
  },
}

Layer 4: User Customizations

Location: .kogral-config/core/kogral.ncl or .kogral-config/platform/{dev,prod,test}.ncl

Purpose: Project-specific or deployment-specific overrides.

Example (user project config):

let mode = import "../../schemas/modes/dev.ncl" in

let user_custom = {
  graph = {
    name = "my-project",
  },
  embeddings = {
    provider = 'claude,  # Override to use Claude
    model = "claude-3-haiku-20240307",
  },
  query = {
    similarity_threshold = 0.7,  # Stricter threshold
  },
} in

helpers.compose_config defaults.base mode user_custom
  | contracts.KbConfig

Composition Mechanism

The helpers.ncl module provides the composition function:

{
  # Recursively merge with override precedence
  merge_with_override = fun base override => /* ... */,

  # Compose three layers
  compose_config = fun defaults mode_config user_custom =>
    let with_mode = merge_with_override defaults mode_config in
    merge_with_override with_mode user_custom,
}

Merge behavior:

• Records are merged recursively
• Override values take precedence over base values
• Arrays are not merged, override replaces base
• Null in override keeps base value

Example merge:

base = { storage = { primary = 'filesystem }, embeddings = { enabled = true } }
override = { storage = { primary = 'memory } }
# Result: { storage = { primary = 'memory }, embeddings = { enabled = true } }

Export to JSON

Once composed, the Nickel configuration is exported to JSON for Rust consumption:

nickel export --format json .kogral-config/core/kogral.ncl > .kogral-config/targets/kogral-core.json

Output (.kogral-config/targets/kogral-core.json):

{
  "graph": {
    "name": "my-project",
    "version": "1.0.0"
  },
  "storage": {
    "primary": "memory",
    "secondary": {
      "enabled": false
    }
  },
  "embeddings": {
    "enabled": true,
    "provider": "claude",
    "model": "claude-3-haiku-20240307",
    "dimensions": 768
  }
}

Rust Integration

The Rust code deserializes the JSON into typed structs using serde:

use serde::{Deserialize, Serialize};

#[derive(Debug, Deserialize, Serialize)]
pub struct KbConfig {
    pub graph: GraphConfig,
    pub storage: StorageConfig,
    pub embeddings: EmbeddingConfig,
    pub templates: TemplateConfig,
    pub query: QueryConfig,
    pub mcp: McpConfig,
    pub sync: SyncConfig,
}

impl KbConfig {
    pub fn from_file(path: &Path) -> Result<Self> {
        let json = std::fs::read_to_string(path)?;
        let config: KbConfig = serde_json::from_str(&json)?;
        Ok(config)
    }
}

Usage in kogral-core:

let config = KbConfig::from_file(".kogral-config/targets/kogral-core.json")?;

// Config drives behavior
let storage: Box<dyn Storage> = match config.storage.primary {
    StorageType::Filesystem => Box::new(FilesystemStorage::new(&config)?),
    StorageType::Memory => Box::new(MemoryStorage::new()),
    StorageType::SurrealDb => Box::new(SurrealDbStorage::new(&config).await?),
};

let embeddings: Box<dyn EmbeddingProvider> = match config.embeddings.provider {
    EmbeddingProviderType::FastEmbed => Box::new(FastEmbedProvider::new()?),
    EmbeddingProviderType::OpenAI => Box::new(RigEmbeddingProvider::openai(&config)?),
    EmbeddingProviderType::Claude => Box::new(RigEmbeddingProvider::claude(&config)?),
    EmbeddingProviderType::Ollama => Box::new(RigEmbeddingProvider::ollama(&config)?),
};

Double Validation

Configuration is validated twice:

1. Nickel Contract Validation

At export time, Nickel validates:

• ✅ Types match contracts (e.g., primary | StorageType)
• ✅ Required fields are present
• ✅ Enums have valid values
• ✅ Nested structure is correct

Error example:

error: contract broken by a value
  ┌─ .kogral-config/core/kogral.ncl:15:5
  │
15│     primary = 'invalid,
  │     ^^^^^^^^^^^^^^^^^^^ applied to this expression
  │
  = This value is not in the enum ['filesystem, 'memory, 'surrealdb]

2. Serde Deserialization Validation

At runtime, serde validates:

• ✅ JSON structure matches Rust types
• ✅ Field names match (with rename support)
• ✅ Values can be converted to Rust types
• ✅ Required fields are not null

Error example:

Error: missing field `graph` at line 1 column 123

Benefits of Config-Driven Architecture

1. Zero Hardcoding

Bad (hardcoded):

// Hardcoded - requires recompilation to change
let storage = FilesystemStorage::new("/fixed/path");
let threshold = 0.6;

Good (config-driven):

// Config-driven - change via .ncl file
let storage = create_storage(&config)?;
let threshold = config.query.similarity_threshold;

2. Environment Flexibility

Same codebase, different behavior:

# Development
nickel export .kogral-config/platform/dev.ncl > targets/kogral-core.json
# → Filesystem storage, fastembed, no auto-sync

# Production
nickel export .kogral-config/platform/prod.ncl > targets/kogral-core.json
# → SurrealDB enabled, OpenAI embeddings, auto-sync

# Testing
nickel export .kogral-config/platform/test.ncl > targets/kogral-core.json
# → In-memory storage, no embeddings, isolated

3. Self-Documenting

Nickel contracts include inline documentation:

StorageType = [| 'filesystem, 'memory, 'surrealdb |]
  | doc "Storage backend type: filesystem (git-tracked), memory (ephemeral), surrealdb (scalable)",

IDEs can show this documentation when editing .ncl files.

4. Type-Safe Evolution

When adding new features:

1. Update contract in contracts.ncl
2. Add default in defaults.ncl
3. Export validates existing configs
4. Rust compilation validates deserialization

Breaking changes are caught before runtime.

5. Testability

Different test scenarios without code changes:

# test-semantic-search.ncl
let test_config = defaults.base & {
  embeddings = { enabled = true, provider = 'fastembed },
  query = { similarity_threshold = 0.3 },
} in test_config

#[test]
fn test_semantic_search() {
    let config = KbConfig::from_file("test-semantic-search.json")?;
    // Config drives test behavior
}

Configuration Discovery

KOGRAL tools automatically discover configuration:

1. Check .kogral-config/targets/kogral-core.json (pre-exported)
2. Check .kogral-config/core/kogral.ncl (export on-demand)
3. Check environment variable KOGRAL_CONFIG
4. Fall back to embedded defaults

impl KbConfig {
    pub fn discover() -> Result<Self> {
        if let Ok(config) = Self::from_file(".kogral-config/targets/kogral-core.json") {
            return Ok(config);
        }

        if Path::new(".kogral-config/core/kogral.ncl").exists() {
            // Export and load
            let output = Command::new("nickel")
                .args(["export", "--format", "json", ".kogral-config/core/kogral.ncl"])
                .output()?;
            return serde_json::from_slice(&output.stdout)?;
        }

        if let Ok(path) = std::env::var("KOGRAL_CONFIG") {
            return Self::from_file(&path);
        }

        Ok(Self::default())  // Embedded defaults
    }
}

Integration with justfile

The justfile integrates configuration validation:

# Validate all Nickel configs
nickel-validate-all:
    @echo "Validating Nickel schemas..."
    nickel typecheck schemas/contracts.ncl
    nickel typecheck schemas/defaults.ncl
    nickel typecheck schemas/helpers.ncl
    nickel typecheck schemas/modes/dev.ncl
    nickel typecheck schemas/modes/prod.ncl
    nickel typecheck schemas/modes/test.ncl
    nickel typecheck .kogral-config/core/kogral.ncl

# Export all platform configs
nickel-export-all:
    @echo "Exporting platform configs to JSON..."
    @mkdir -p .kogral-config/targets
    nickel export --format json .kogral-config/platform/dev.ncl > .kogral-config/targets/kogral-dev.json
    nickel export --format json .kogral-config/platform/prod.ncl > .kogral-config/targets/kogral-prod.json
    nickel export --format json .kogral-config/platform/test.ncl > .kogral-config/targets/kogral-test.json
    @echo "Exported 3 configurations to .kogral-config/targets/"

Usage:

just nickel::validate-all  # Check configs are valid
just nickel::export-all    # Generate JSON for all environments

Best Practices

1. Never Hardcode

If it's behavior, it's config:

• Storage paths
• API endpoints
• Thresholds
• Timeouts
• Feature flags
• Provider selection

2. Use Modes for Environment Differences

Don't put environment-specific values in user config:

Bad:

# user config with env-specific values
{
  storage = {
    url = if env == "prod" then "prod-url" else "dev-url"  # Don't do this
  }
}

Good:

# modes/prod.ncl
{ storage = { url = "prod-url" } }

# modes/dev.ncl
{ storage = { url = "dev-url" } }

# user config is environment-agnostic
{ graph = { name = "my-project" } }

3. Document Complex Fields

Use Nickel's doc metadata:

similarity_threshold | Number
  | doc "Minimum cosine similarity (0-1) for semantic search matches. Higher = stricter."
  | default = 0.6,

4. Validate Early

Run nickel typecheck in CI/CD before building Rust code.

5. Version Configs

Track .ncl files in git, ignore .kogral-config/targets/*.json (generated).

See Also

• Schema Reference: Configuration Schema
• User Guide: Configuration Guide
• ADR: Why Nickel vs TOML
• Examples: .kogral-config/core/kogral.ncl, .kogral-config/platform/*.ncl