stratumiops/crates/stratum-llm/README.md

stratum-llm

Unified LLM abstraction for the stratumiops ecosystem with automatic provider detection, fallback chains, and smart caching.

Features

• Credential Auto-detection: Automatically finds CLI credentials (Claude, OpenAI) and API keys
• Provider Fallback: Circuit breaker pattern with automatic failover across providers
• Smart Caching: xxHash-based request deduplication reduces duplicate API calls
• Kogral Integration: Inject project context from knowledge base (optional)
• Cost Tracking: Transparent cost estimation across all providers
• Multiple Providers: Anthropic Claude, OpenAI, DeepSeek, Ollama

Quick Start

use stratum_llm::{UnifiedClient, Message, Role};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = UnifiedClient::auto()?;

    let messages = vec![
        Message {
            role: Role::User,
            content: "What is Rust?".to_string(),
        }
    ];

    let response = client.generate(&messages, None).await?;
    println!("{}", response.content);

    Ok(())
}

Provider Priority

1. CLI credentials (subscription-based, no per-token cost) - preferred
2. API keys from environment variables
3. Local models (Ollama)

The client automatically detects available credentials and builds a fallback chain.

Features

Default Features

[dependencies]
stratum-llm = "0.1"

Includes: Anthropic, OpenAI, Ollama

All Features

[dependencies]
stratum-llm = { version = "0.1", features = ["all"] }

Includes: All providers, CLI detection, Kogral integration, Prometheus metrics

Custom Feature Set

[dependencies]
stratum-llm = { version = "0.1", features = ["anthropic", "deepseek", "kogral"] }

Available features:

• anthropic - Anthropic Claude API
• openai - OpenAI API
• deepseek - DeepSeek API
• ollama - Ollama local models
• claude-cli - Claude CLI credential detection
• kogral - Kogral knowledge base integration
• metrics - Prometheus metrics

Advanced Usage

With Kogral Context

let client = UnifiedClient::builder()
    .auto_detect()?
    .with_kogral()
    .build()?;

let response = client
    .generate_with_kogral(&messages, None, Some("rust"), None)
    .await?;

Custom Fallback Strategy

use stratum_llm::{FallbackStrategy, ProviderChain};

let chain = ProviderChain::from_detected()?
    .with_strategy(FallbackStrategy::OnRateLimitOrUnavailable);

let client = UnifiedClient::builder()
    .with_chain(chain)
    .build()?;

Cost Budget

let chain = ProviderChain::from_detected()?
    .with_strategy(FallbackStrategy::OnBudgetExceeded {
        budget_cents: 10.0,
    });

Examples

Run examples with:

cargo run --example basic_usage
cargo run --example with_kogral --features kogral
cargo run --example fallback_demo

License

MIT OR Apache-2.0