provisioning-platform/daemon-cli/docs/ADR-001-KCL-NICKEL-RENDERING-ARCHITECTURE.md

# ADR-001: KCL and Nickel Rendering Architecture - CLI Wrapper Approach

## Status
**✅ IMPLEMENTED** - 2025-12-15

## Context

The daemon-cli crate provides unified orchestration for the provisioning ecosystem. Configuration rendering is a core capability: transforming templates and configuration files into final configurations.

The project needs to support:
1. **KCL rendering** - Evaluate KCL configuration files with module resolution
2. **Nickel rendering** - Evaluate Nickel configuration files with module resolution
3. **Template rendering** - Already using Tera (pure Rust template engine)

### Discovery: Why Pure Rust Libraries Don't Work

Initial investigation found that pure Rust libraries (`kcl-lib` v0.2.116, `nickel-lang-core` v0.16.1) exist. However, they **cannot be used directly** without the official CLI because:

**KCL Module System Complexity**:
- KCL files use imports: `import provisioning.lib as lib`
- Requires parsing `kcl.mod` (similar to Go's `go.mod`)
- Module resolution is non-trivial (path mapping, dependency handling)
- Standard library discovery and loading
- Build context configuration

**Key Finding**: The `kcl-lib` crate is incomplete - using it without the CLI would require:
1. Manually implementing KCL module resolution in Rust
2. Duplicating all the logic the official CLI already handles
3. High risk of subtle differences from CLI behavior
4. Maintenance burden as KCL evolves

**The Same Problem**: Nickel has identical issues with its module/import system and standard library resolution.

### Architectural Pattern from Nushell Plugins

Investigation of the production `nushell-plugins` repository revealed the established pattern:

**Why Nushell Plugins Use CLI Wrappers**:
- ✅ Module system guaranteed correct (delegates to official implementation)
- ✅ Zero maintenance as language evolves
- ✅ No risk of subtle differences
- ✅ Single source of truth (the official CLI)
- ⚠️ Process overhead acceptable (~100-200ms)

This is the industry standard pattern used by other tool plugins (Prettier, ESLint, Terraform, etc.)

## Decision

**Implement KCL and Nickel rendering in daemon-cli using CLI wrappers**, NOT pure Rust libraries.

**Rationale**: Both KCL and Nickel have complex module systems that are NOT cleanly exposed through Rust crates. The authoritative implementation of module resolution, imports, and standard library access is in the official CLI tools. Delegating to these CLIs is the only way to guarantee correct behavior.

### Why This Is Different from Pure Template Engines

**Tera** (template engine):
- Simple template → rendered text transformation
- No complex features (imports/modules)
- Pure Rust `tera-rs` crate is complete implementation
- ✅ Can reimplement entirely in Rust

**KCL/Nickel** (configuration languages):
- Full programming languages with validation
- Module system with dependency resolution
- Standard library with complex interactions
- Type system and validation rules
- ❌ Cannot feasibly reimplement in Rust

### Architecture Vision

```
daemon-cli (Rust)
    │
    ├── KCL Rendering
    │   ├── Call: kcl run - (via stdin)
    │   ├── CLI handles: Module resolution, imports, stdlib
    │   └── Daemon manages: Caching, result parsing
    │
    ├── Nickel Rendering
    │   ├── Call: nickel eval - (via stdin)
    │   ├── CLI handles: Module resolution, imports, stdlib
    │   └── Daemon manages: Caching, result parsing
    │
    └── Template Rendering
        └── tera-rs (already implemented, pure Rust)
```

### Implementation Approach

**1. CLI Wrapper Pattern**
- Spawn `kcl run -` process with content via stdin
- Spawn `nickel eval -` process with content via stdin
- Parse output and handle errors
- Proper error handling and context mapping

**2. Module System**
- Delegated to official CLI
- No custom implementation needed
- Guarantees correct behavior with kcl.mod, imports, stdlib

**3. Performance Optimization**
- Application-level caching layer
- Reuse processes if beneficial
- Process overhead acceptable (~100-200ms per call)

## Consequences

### Positive
- **Correctness**: Module resolution guaranteed correct (delegates to official CLI)
- **Maintenance**: Zero maintenance burden as KCL/Nickel evolve
- **Reliability**: No risk of subtle differences from CLI behavior
- **Features**: All language features automatically supported
- **Standards**: Follows industry pattern (Prettier, ESLint, Terraform)
- **Simplicity**: Clean architecture, clear separation of concerns
- **Testing**: Easy to test (just mock subprocess calls)

### Negative
- **External Dependency**: Requires `kcl` and `nickel` CLIs installed on system
- **Process Overhead**: ~100-200ms per call (vs 1-5ms for pure Rust)
- **Complexity**: Subprocess communication (stdin/stdout/stderr handling)
- **Error Handling**: Need to parse CLI error messages

### Mitigations

**For Process Overhead**:
- Implement application-level caching
- Batch operations when possible
- Consider persistent process pool (future optimization)

**For External Dependencies**:
- Clear documentation about CLI requirements
- Graceful error messages when CLIs not found
- Feature flags to disable when CLIs not available

**For Error Handling**:
- Comprehensive error type mapping
- Detailed error context from CLI stderr
- Clear error messages to users

## Alternatives Considered

### Alternative 1: Pure Rust Libraries (kcl-lib, nickel-lang-core)
**Pros**: No external dependencies, potential performance improvement
**Cons**:
- ❌ Module system not cleanly exposed in Rust crates
- ❌ Would require reimplementing KCL/Nickel module resolution
- ❌ High maintenance burden
- ❌ Risk of subtle differences from CLI behavior
**Decision**: REJECTED - Incomplete Rust crate APIs

### Alternative 2: Use Only Tera (No KCL/Nickel)
**Pros**: Simpler, one template engine, pure Rust
**Cons**: Limits configuration capabilities, less type safety, doesn't meet requirements
**Decision**: REJECTED - Project requires KCL/Nickel support

### Alternative 3: Implement Custom Parsers
**Pros**: Complete control, no external dependencies
**Cons**: Massive effort, duplication of work, hard to maintain
**Decision**: REJECTED - Infeasible

## Implementation Plan

### ✅ Step 1: CLI Wrapper Structure - COMPLETE
```rust
// src/config_renderer/kcl.rs
#[cfg(feature = "kcl")]
pub async fn render(content: &str, _variables: &HashMap<String, Value>) -> Result<String> {
    let mut child = Command::new("kcl")
        .arg("run")
        .arg("-")
        .stdin(Stdio::piped())
        .stdout(Stdio::piped())
        .stderr(Stdio::piped())
        .spawn()?;

    // Write content, collect output, return result
}

// src/config_renderer/nickel.rs
#[cfg(feature = "nickel")]
pub async fn render(content: &str, _variables: &HashMap<String, Value>) -> Result<String> {
    let mut child = Command::new("nickel")
        .arg("eval")
        .arg("-")
        .stdin(Stdio::piped())
        .stdout(Stdio::piped())
        .stderr(Stdio::piped())
        .spawn()?;

    // Write content, collect output, return result
}
```

### ✅ Step 2: Error Handling - COMPLETE
- Maps IO errors to `DaemonError::io_error`
- Maps validation errors for non-success exit codes
- Maps UTF-8 errors to serialization errors
- Proper error context from CLI stderr

### ✅ Step 3: Feature Flags - COMPLETE
- `kcl` feature enables KCL rendering via CLI
- `nickel` feature enables Nickel rendering via CLI
- No optional Rust dependencies needed
- Both in default feature set

### Step 4: Integration Testing (Future)
- Test with actual provisioning KCL files
- Test with actual provisioning Nickel files
- Verify module resolution works correctly
- Performance profiling

### Step 5: Caching Layer (Future Optimization)
- Application-level result caching
- Consider persistent process pool for repeated calls
- Measure real-world performance impact

## Configuration Integration

Both renderers integrate with daemon-cli configuration:

```toml
[kcl]
enabled = true
cache_size = 1000
search_paths = ["./kcl", "/usr/share/kcl"]
max_evaluation_time_ms = 5000

[nickel]
enabled = true
cache_size = 1000
search_paths = ["./nickel", "/usr/share/nickel"]
max_evaluation_time_ms = 5000
```

## Testing Strategy

**Unit Tests**:
```rust
#[test]
fn test_kcl_simple_eval() {
    let result = render("name = \"test\"", &context)?;
    assert_eq!(result, "{ name: \"test\" }");
}
```

**Integration Tests**:
```rust
#[test]
fn test_kcl_with_modules() {
    // Use real kcl.mod and config files
    let result = render(include_str!("../../tests/fixtures/complex.k"), &context)?;
    // Verify module resolution worked
}
```

**Performance Benchmarks**:
```rust
#[bench]
fn bench_kcl_eval(b: &mut Bencher) {
    b.iter(|| render("name = \"test\"", &context))
}
```

## Risk Assessment

### Risk 1: Undocumented kcl-lib API
**Probability**: Medium
**Impact**: Implementation delays
**Mitigation**: Study source code, start simple, incremental complexity

### Risk 2: nickel-lang-core API instability
**Probability**: Medium
**Impact**: Version compatibility issues
**Mitigation**: Pin versions, monitor updates, test compatibility

### Risk 3: Module system complexity
**Probability**: High
**Impact**: Implementation time, correctness risks
**Mitigation**: Implement incrementally, thorough testing

### Risk 4: Performance degradation
**Probability**: Low
**Impact**: Users affected
**Mitigation**: Profile, optimize hot paths, cache aggressively

## Success Criteria

- ✅ Basic KCL/Nickel evaluation works without modules
- ✅ Module resolution implemented and tested
- ✅ Performance meets or exceeds CLI approach
- ✅ Integration tests pass with real project files
- ✅ No external CLI dependencies required
- ✅ Documented with examples

## Implementation Phases

### Phase 1: Research and Setup
**Deliverables**:
- Study kcl-lib and nickel-lang-core APIs
- Create basic rendering modules
- Implement simple evaluation without imports
- Document findings and API learnings

**Completion Criteria**:
- Basic evaluation works for simple expressions
- No module/import handling yet

### Phase 2: Module System
**Deliverables**:
- kcl.mod file parsing
- Import path resolution
- Module caching mechanism
- Comprehensive error handling

**Completion Criteria**:
- Module imports resolve correctly
- Path resolution tested with real project files
- Cache hits/misses working properly

### Phase 3: Integration
**Deliverables**:
- Integration tests with provisioning KCL files
- Integration tests with provisioning Nickel files
- Performance measurements
- Documentation and examples

**Completion Criteria**:
- Real project files evaluate correctly
- Performance comparable to CLI approach
- All tests passing

### Phase 4: Advanced Features
**Deliverables** (future):
- Standard library support
- Build context configuration
- Advanced caching strategies
- Type checking integration

**Completion Criteria**:
- All advanced features working
- Performance optimized
- Comprehensive documentation

## References

- daemon-cli Cargo.toml (kcl-lib and nickel-lang-core dependency declarations)
- [kcl-lib Documentation](https://docs.rs/kcl-lib/)
- [nickel-lang-core Documentation](https://docs.rs/nickel-lang-core/)
- [KCL Language Specification](https://kcl-lang.io/)
- [Nickel Documentation](https://nickel-lang.org/)
- Related: ADR-013 and ADR-014 (Nushell Plugins - CLI Wrapper Approach)

---

**Author**: Architecture Team
**Date**: 2025-12-15
**Status**: Proposed
**Next Steps**: Team review and approval for implementation start