provisioning/provisioning-platform

Fork 0

Jesús Pérez 584dd0e079

platform: clean-start baseline — Rust workspace: binaries/daemon (constellation materialization)

2026-07-09 22:04:43 +01:00

11 KiB

Raw Permalink Blame History

ADR-001: KCL and Nickel Rendering Architecture - CLI Wrapper Approach

Status

✅ IMPLEMENTED - 2025-12-15

Context

The daemon-cli crate provides unified orchestration for the provisioning ecosystem. Configuration rendering is a core capability: transforming templates and configuration files into final configurations.

The project needs to support:

KCL rendering - Evaluate KCL configuration files with module resolution
Nickel rendering - Evaluate Nickel configuration files with module resolution
Template rendering - Already using Tera (pure Rust template engine)

Discovery: Why Pure Rust Libraries Don't Work

Initial investigation found that pure Rust libraries (kcl-lib v0.2.116, nickel-lang-core v0.16.1) exist. However, they cannot be used directly without the official CLI because:

KCL Module System Complexity:

KCL files use imports: import provisioning.lib as lib
Requires parsing kcl.mod (similar to Go's go.mod)
Module resolution is non-trivial (path mapping, dependency handling)
Standard library discovery and loading
Build context configuration

Key Finding: The kcl-lib crate is incomplete - using it without the CLI would require:

Manually implementing KCL module resolution in Rust
Duplicating all the logic the official CLI already handles
High risk of subtle differences from CLI behavior
Maintenance burden as KCL evolves

The Same Problem: Nickel has identical issues with its module/import system and standard library resolution.

Architectural Pattern from Nushell Plugins

Investigation of the production nushell-plugins repository revealed the established pattern:

Why Nushell Plugins Use CLI Wrappers:

✅ Module system guaranteed correct (delegates to official implementation)
✅ Zero maintenance as language evolves
✅ No risk of subtle differences
✅ Single source of truth (the official CLI)
⚠️ Process overhead acceptable (~100-200ms)

This is the industry standard pattern used by other tool plugins (Prettier, ESLint, Terraform, etc.)

Decision

Implement KCL and Nickel rendering in daemon-cli using CLI wrappers, NOT pure Rust libraries.

Rationale: Both KCL and Nickel have complex module systems that are NOT cleanly exposed through Rust crates. The authoritative implementation of module resolution, imports, and standard library access is in the official CLI tools. Delegating to these CLIs is the only way to guarantee correct behavior.

Why This Is Different from Pure Template Engines

Tera (template engine):

Simple template → rendered text transformation
No complex features (imports/modules)
Pure Rust tera-rs crate is complete implementation
✅ Can reimplement entirely in Rust

KCL/Nickel (configuration languages):

Full programming languages with validation
Module system with dependency resolution
Standard library with complex interactions
Type system and validation rules
❌ Cannot feasibly reimplement in Rust

Architecture Vision

daemon-cli (Rust)
    │
    ├── KCL Rendering
    │   ├── Call: kcl run - (via stdin)
    │   ├── CLI handles: Module resolution, imports, stdlib
    │   └── Daemon manages: Caching, result parsing
    │
    ├── Nickel Rendering
    │   ├── Call: nickel eval - (via stdin)
    │   ├── CLI handles: Module resolution, imports, stdlib
    │   └── Daemon manages: Caching, result parsing
    │
    └── Template Rendering
        └── tera-rs (already implemented, pure Rust)

Implementation Approach

1. CLI Wrapper Pattern

Spawn kcl run - process with content via stdin
Spawn nickel eval - process with content via stdin
Parse output and handle errors
Proper error handling and context mapping

2. Module System

Delegated to official CLI
No custom implementation needed
Guarantees correct behavior with kcl.mod, imports, stdlib

3. Performance Optimization

Application-level caching layer
Reuse processes if beneficial
Process overhead acceptable (~100-200ms per call)

Consequences

Positive

Correctness: Module resolution guaranteed correct (delegates to official CLI)
Maintenance: Zero maintenance burden as KCL/Nickel evolve
Reliability: No risk of subtle differences from CLI behavior
Features: All language features automatically supported
Standards: Follows industry pattern (Prettier, ESLint, Terraform)
Simplicity: Clean architecture, clear separation of concerns
Testing: Easy to test (just mock subprocess calls)

Negative

External Dependency: Requires kcl and nickel CLIs installed on system
Process Overhead: ~100-200ms per call (vs 1-5ms for pure Rust)
Complexity: Subprocess communication (stdin/stdout/stderr handling)
Error Handling: Need to parse CLI error messages

Mitigations

For Process Overhead:

Implement application-level caching
Batch operations when possible
Consider persistent process pool (future optimization)

For External Dependencies:

Clear documentation about CLI requirements
Graceful error messages when CLIs not found
Feature flags to disable when CLIs not available

For Error Handling:

Comprehensive error type mapping
Detailed error context from CLI stderr
Clear error messages to users

Alternatives Considered

Alternative 1: Pure Rust Libraries (kcl-lib, nickel-lang-core)

Pros: No external dependencies, potential performance improvement Cons:

❌ Module system not cleanly exposed in Rust crates
❌ Would require reimplementing KCL/Nickel module resolution
❌ High maintenance burden
❌ Risk of subtle differences from CLI behavior Decision: REJECTED - Incomplete Rust crate APIs

Alternative 2: Use Only Tera (No KCL/Nickel)

Pros: Simpler, one template engine, pure Rust Cons: Limits configuration capabilities, less type safety, doesn't meet requirements Decision: REJECTED - Project requires KCL/Nickel support

Alternative 3: Implement Custom Parsers

Pros: Complete control, no external dependencies Cons: Massive effort, duplication of work, hard to maintain Decision: REJECTED - Infeasible

Implementation Plan

✅ Step 1: CLI Wrapper Structure - COMPLETE

// src/config_renderer/kcl.rs
#[cfg(feature = "kcl")]
pub async fn render(content: &str, _variables: &HashMap<String, Value>) -> Result<String> {
    let mut child = Command::new("kcl")
        .arg("run")
        .arg("-")
        .stdin(Stdio::piped())
        .stdout(Stdio::piped())
        .stderr(Stdio::piped())
        .spawn()?;

    // Write content, collect output, return result
}

// src/config_renderer/nickel.rs
#[cfg(feature = "nickel")]
pub async fn render(content: &str, _variables: &HashMap<String, Value>) -> Result<String> {
    let mut child = Command::new("nickel")
        .arg("eval")
        .arg("-")
        .stdin(Stdio::piped())
        .stdout(Stdio::piped())
        .stderr(Stdio::piped())
        .spawn()?;

    // Write content, collect output, return result
}

✅ Step 2: Error Handling - COMPLETE

Maps IO errors to DaemonError::io_error
Maps validation errors for non-success exit codes
Maps UTF-8 errors to serialization errors
Proper error context from CLI stderr

✅ Step 3: Feature Flags - COMPLETE

kcl feature enables KCL rendering via CLI
nickel feature enables Nickel rendering via CLI
No optional Rust dependencies needed
Both in default feature set

Step 4: Integration Testing (Future)

Test with actual provisioning KCL files
Test with actual provisioning Nickel files
Verify module resolution works correctly
Performance profiling

Step 5: Caching Layer (Future Optimization)

Application-level result caching
Consider persistent process pool for repeated calls
Measure real-world performance impact

Configuration Integration

Both renderers integrate with daemon-cli configuration:

[kcl]
enabled = true
cache_size = 1000
search_paths = ["./kcl", "/usr/share/kcl"]
max_evaluation_time_ms = 5000

[nickel]
enabled = true
cache_size = 1000
search_paths = ["./nickel", "/usr/share/nickel"]
max_evaluation_time_ms = 5000

Testing Strategy

Unit Tests:

#[test]
fn test_kcl_simple_eval() {
    let result = render("name = \"test\"", &context)?;
    assert_eq!(result, "{ name: \"test\" }");
}

Integration Tests:

#[test]
fn test_kcl_with_modules() {
    // Use real kcl.mod and config files
    let result = render(include_str!("../../tests/fixtures/complex.k"), &context)?;
    // Verify module resolution worked
}

Performance Benchmarks:

#[bench]
fn bench_kcl_eval(b: &mut Bencher) {
    b.iter(|| render("name = \"test\"", &context))
}

Risk Assessment

Risk 1: Undocumented kcl-lib API

Probability: Medium Impact: Implementation delays Mitigation: Study source code, start simple, incremental complexity

Risk 2: nickel-lang-core API instability

Probability: Medium Impact: Version compatibility issues Mitigation: Pin versions, monitor updates, test compatibility

Risk 3: Module system complexity

Probability: High Impact: Implementation time, correctness risks Mitigation: Implement incrementally, thorough testing

Risk 4: Performance degradation

Probability: Low Impact: Users affected Mitigation: Profile, optimize hot paths, cache aggressively

Success Criteria

✅ Basic KCL/Nickel evaluation works without modules
✅ Module resolution implemented and tested
✅ Performance meets or exceeds CLI approach
✅ Integration tests pass with real project files
✅ No external CLI dependencies required
✅ Documented with examples

Implementation Phases

Phase 1: Research and Setup

Deliverables:

Study kcl-lib and nickel-lang-core APIs
Create basic rendering modules
Implement simple evaluation without imports
Document findings and API learnings

Completion Criteria:

Basic evaluation works for simple expressions
No module/import handling yet

Phase 2: Module System

Deliverables:

kcl.mod file parsing
Import path resolution
Module caching mechanism
Comprehensive error handling

Completion Criteria:

Module imports resolve correctly
Path resolution tested with real project files
Cache hits/misses working properly

Phase 3: Integration

Deliverables:

Integration tests with provisioning KCL files
Integration tests with provisioning Nickel files
Performance measurements
Documentation and examples

Completion Criteria:

Real project files evaluate correctly
Performance comparable to CLI approach
All tests passing

Phase 4: Advanced Features

Deliverables (future):

Standard library support
Build context configuration
Advanced caching strategies
Type checking integration

Completion Criteria:

All advanced features working
Performance optimized
Comprehensive documentation

References

daemon-cli Cargo.toml (kcl-lib and nickel-lang-core dependency declarations)
kcl-lib Documentation
nickel-lang-core Documentation
KCL Language Specification
Nickel Documentation
Related: ADR-013 and ADR-014 (Nushell Plugins - CLI Wrapper Approach)

Author: Architecture Team Date: 2025-12-15 Status: Proposed Next Steps: Team review and approval for implementation start

11 KiB Raw Permalink Blame History

ADR-001: KCL and Nickel Rendering Architecture - CLI Wrapper Approach

Status

Context

Discovery: Why Pure Rust Libraries Don't Work

Architectural Pattern from Nushell Plugins

Decision

Why This Is Different from Pure Template Engines

Architecture Vision

Implementation Approach

Consequences

Positive

Negative

Mitigations

Alternatives Considered

Alternative 1: Pure Rust Libraries (kcl-lib, nickel-lang-core)

Alternative 2: Use Only Tera (No KCL/Nickel)

Alternative 3: Implement Custom Parsers

Implementation Plan

✅ Step 1: CLI Wrapper Structure - COMPLETE

✅ Step 2: Error Handling - COMPLETE

✅ Step 3: Feature Flags - COMPLETE

Step 4: Integration Testing (Future)

Step 5: Caching Layer (Future Optimization)

Configuration Integration

Testing Strategy

Risk Assessment

Risk 1: Undocumented kcl-lib API

Risk 2: nickel-lang-core API instability

Risk 3: Module system complexity

Risk 4: Performance degradation

Success Criteria

Implementation Phases

Phase 1: Research and Setup

Phase 2: Module System

Phase 3: Integration

Phase 4: Advanced Features

References

11 KiB

Raw Permalink Blame History