# ADR-010: Configuration File Format Strategy

**Status**: Accepted
**Date**: 2025-12-03
**Decision Makers**: Architecture Team
**Implementation**: Multi-phase migration (KCL workspace configs + template reorganization)

---

## Context

The provisioning project historically used a single configuration format (YAML/TOML environment variables) for all purposes. As the system evolved, different parts naturally adopted different formats:

- **TOML** for modular provider and platform configurations (`providers/*.toml`, `platform/*.toml`)
- **KCL** for infrastructure-as-code definitions with type safety
- **YAML** for workspace metadata

However, the workspace configuration remained in **YAML** (`provisioning.yaml`), creating inconsistency and leaving type-unsafe configuration handling. Meanwhile, complete KCL schemas for workspace configuration were designed but unused.

**Problem**: Three different formats in the same system without documented rationale or consistent patterns.

---

## Decision

Adopt a **three-format strategy** with clear separation of concerns:

| Format | Purpose | Use Cases |
|--------|---------|-----------|
| **KCL** | Infrastructure as Code & Schemas | Workspace config, infrastructure definitions, type-safe validation |
| **TOML** | Application Configuration & Settings | System defaults, provider settings, user preferences, interpolation |
| **YAML** | Metadata & Kubernetes Resources | K8s manifests, tool metadata, version tracking, CI/CD resources |

---

## Implementation Strategy

### Phase 1: Documentation (Complete)

Define and document the three-format approach through:

1. **ADR-010** (this document) - Rationale and strategy
2. **CLAUDE.md updates** - Quick reference for developers
3. **Configuration hierarchy** - Explicit precedence rules

### Phase 2: Workspace Config Migration (In Progress)

**Migrate workspace configuration from YAML to KCL**:

1. Create comprehensive workspace configuration schema in KCL
2. Implement backward-compatible config loader (KCL first, fallback to YAML)
3. Provide migration script to convert existing workspaces
4. Update workspace initialization to generate KCL configs

**Expected Outcome**:

- `workspace/config/provisioning.ncl` (KCL, type-safe, validated)
- Full schema validation with semantic versioning checks
- Automatic validation at config load time

### Phase 3: Template File Reorganization (In Progress)

**Move template files to proper directory structure and correct extensions**:

```plaintext
Previous (KCL):
  provisioning/kcl/templates/*.k  (had Nushell/Jinja2 code, not KCL)

Current (Nickel):
  provisioning/templates/
    ├── nushell/*.nu.j2
    ├── config/*.toml.j2
    ├── nickel/*.ncl.j2
    └── README.md
```plaintext

**Expected Outcome**:

- Templates properly classified and discoverable
- KCL validation passes (15/16 errors eliminated)
- Template system clean and maintainable

---

## Rationale for Each Format

### KCL for Workspace Configuration

**Why KCL over YAML or TOML?**

1. **Type Safety**: Catch configuration errors at schema validation time, not runtime

   ```kcl
   schema WorkspaceDeclaration:
       metadata: Metadata
       check:
           regex.match(metadata.version, r"^\d+\.\d+\.\d+$"), \
               "Version must be semantic versioning"
   ```

1. **Schema-First Development**: Schemas are first-class citizens
   - Document expected structure upfront
   - IDE support for auto-completion
   - Enforce required fields and value ranges

2. **Immutable by Default**: Infrastructure configurations are immutable
   - Prevents accidental mutations
   - Better for reproducible deployments
   - Aligns with PAP principle: "configuration-driven, not hardcoded"

3. **Complex Validation**: KCL supports sophisticated validation rules
   - Semantic versioning validation
   - Dependency checking
   - Cross-field validation
   - Range constraints on numeric values

4. **Ecosystem Consistency**: KCL is already used for infrastructure definitions
   - Server configurations use KCL
   - Cluster definitions use KCL
   - Taskserv definitions use KCL
   - Using KCL for workspace config maintains consistency

5. **Existing Schemas**: `provisioning/kcl/generator/declaration.ncl` already defines complete workspace schemas
   - No design work needed
   - Production-ready schemas
   - Well-tested patterns

### TOML for Application Configuration

**Why TOML for settings?**

1. **Hierarchical Structure**: Native support for nested configurations

   ```toml
   [http]
   use_curl = false
   timeout = 30

   [debug]
   enabled = false
   log_level = "info"
   ```

2. **Interpolation Support**: Dynamic variable substitution

   ```toml
   base_path = "/Users/home/provisioning"
   cache_path = "{{base_path}}/.cache"
   ```

3. **Industry Standard**: Widely used for application configuration (Rust, Python, Go)

4. **Human Readable**: Clear, explicit, easy to edit

5. **Validation Support**: Schema files (`.schema.toml`) for validation

**Use Cases**:

- System defaults: `provisioning/config/config.defaults.toml`
- Provider settings: `workspace/config/providers/*.toml`
- Platform services: `workspace/config/platform/*.toml`
- User preferences: User config files

### YAML for Metadata and Kubernetes Resources

**Why YAML for metadata?**

1. **Kubernetes Compatibility**: YAML is K8s standard
   - K8s manifests use YAML
   - Consistent with ecosystem
   - Familiar to DevOps engineers

2. **Lightweight**: Good for simple data structures

   ```yaml
   workspace:
     name: "librecloud"
     version: "1.0.0"
     created: "2025-10-06T12:29:43Z"
   ```

3. **Version Control**: Human-readable format
   - Diffs are clear and meaningful
   - Git-friendly
   - Comments supported

**Use Cases**:

- K8s resource definitions
- Tool metadata (versions, sources, tags)
- CI/CD configuration files
- User workspace metadata (during transition)

---

## Configuration Hierarchy (Priority)

**When loading configuration, use this precedence (highest to lowest)**:

1. **Runtime Arguments** (highest priority)
   - CLI flags passed to commands
   - Explicit user input

2. **Environment Variables** (PROVISIONING_*)
   - Override system settings
   - Deployment-specific overrides
   - Secrets via env vars

3. **User Configuration** (Centralized)
   - User preferences: `~/.config/provisioning/user_config.yaml`
   - User workspace overrides: `workspace/config/local-overrides.toml`

4. **Infrastructure Configuration**
   - Workspace KCL config: `workspace/config/provisioning.ncl`
   - Platform services: `workspace/config/platform/*.toml`
   - Provider configs: `workspace/config/providers/*.toml`

5. **System Defaults** (lowest priority)
   - System config: `provisioning/config/config.defaults.toml`
   - Schema defaults: defined in KCL schemas

---

## Migration Path

### For Existing Workspaces

1. **Migration Path**: Config loader checks for `.ncl` first, then falls back to `.yaml` for legacy systems

   ```nushell
   # Try Nickel first (current)
   if ($config_nickel | path exists) {
       let config = (load_nickel_workspace_config $config_nickel)
   } else if ($config_yaml | path exists) {
       # Legacy YAML support (from pre-migration)
       let config = (open $config_yaml)
   }
   ```

2. **Automatic Migration**: Migration script converts YAML/KCL → Nickel

   ```bash
   provisioning workspace migrate-config --all
   ```

3. **Validation**: New KCL configs validated against schemas

### For New Workspaces

1. **Generate KCL**: Workspace initialization creates `.k` files

   ```bash
   provisioning workspace create my-workspace
   # Creates: workspace/my-workspace/config/provisioning.ncl
   ```

2. **Use Existing Schemas**: Leverage `provisioning/kcl/generator/declaration.ncl`

3. **Schema Validation**: Automatic validation during config load

---

## File Format Guidelines for Developers

### When to Use Each Format

**Use KCL for**:

- Infrastructure definitions (servers, clusters, taskservs)
- Configuration with type requirements
- Schema definitions
- Any config that needs validation rules
- Workspace configuration

**Use TOML for**:

- Application settings (HTTP client, logging, timeouts)
- Provider-specific settings
- Platform service configuration
- User preferences and overrides
- System defaults with interpolation

**Use YAML for**:

- Kubernetes manifests
- CI/CD configuration (GitHub Actions, GitLab CI)
- Tool metadata
- Human-readable documentation files
- Version control metadata

---

## Consequences

### Benefits

✅ **Type Safety**: KCL schema validation catches config errors early
✅ **Consistency**: Infrastructure definitions and configs use same language
✅ **Maintainability**: Clear separation of concerns (IaC vs settings vs metadata)
✅ **Validation**: Semantic versioning, required fields, range checks
✅ **Tooling**: IDE support for KCL auto-completion
✅ **Documentation**: Self-documenting schemas with descriptions
✅ **Ecosystem Alignment**: TOML for settings (Rust standard), YAML for K8s

### Trade-offs

⚠️ **Learning Curve**: Developers must understand three formats
⚠️ **Migration Effort**: Existing YAML configs need conversion
⚠️ **Tooling Requirements**: KCL compiler needed (already a dependency)

### Risk Mitigation

1. **Documentation**: Clear guidelines in CLAUDE.md
2. **Backward Compatibility**: YAML support maintained during transition
3. **Automation**: Migration scripts for existing workspaces
4. **Gradual Migration**: No hard cutoff, both formats supported for extended period

---

## Template File Reorganization

### Problem

Currently, 15/16 files in `provisioning/kcl/templates/` have `.k` extension but contain Nushell/Jinja2 code, not KCL:

```plaintext
provisioning/kcl/templates/
├── server.ncl          # Actually Nushell/Jinja2 template
├── taskserv.ncl        # Actually Nushell/Jinja2 template
└── ...               # 15 more template files
```plaintext

This causes:

- KCL validation failures (96.6% of errors)
- Misclassification (templates in KCL directory)
- Confusing directory structure

### Solution

Reorganize into type-specific directories:

```plaintext
provisioning/templates/
├── nushell/           # Nushell code generation (*.nu.j2)
│   ├── server.nu.j2
│   ├── taskserv.nu.j2
│   └── ...
├── config/            # Config file generation (*.toml.j2, *.yaml.j2)
│   ├── provider.toml.j2
│   └── ...
├── kcl/               # KCL file generation (*.k.j2)
│   ├── workspace.ncl.j2
│   └── ...
└── README.md
```plaintext

### Outcome

✅ Correct file classification
✅ KCL validation passes completely
✅ Clear template organization
✅ Easier to discover and maintain templates

---

## References

### Existing KCL Schemas

1. **Workspace Declaration**: `provisioning/kcl/generator/declaration.ncl`
   - `WorkspaceDeclaration` - Complete workspace specification
   - `Metadata` - Name, version, author, timestamps
   - `DeploymentConfig` - Deployment modes, servers, HA settings
   - Includes validation rules and semantic versioning

2. **Workspace Layer**: `provisioning/workspace/layers/workspace.layer.ncl`
   - `WorkspaceLayer` - Template paths, priorities, metadata

3. **Core Settings**: `provisioning/kcl/settings.ncl`
   - `Settings` - Main provisioning settings
   - `SecretProvider` - SOPS/KMS configuration
   - `AIProvider` - AI provider configuration

### Related ADRs

- **ADR-001**: Project Structure
- **ADR-005**: Extension Framework
- **ADR-006**: Provisioning CLI Refactoring
- **ADR-009**: Security System Complete

---

## Decision Status

**Status**: Accepted

**Next Steps**:

1. ✅ Document strategy (this ADR)
2. ⏳ Create workspace configuration KCL schema
3. ⏳ Implement backward-compatible config loader
4. ⏳ Create migration script for YAML → KCL
5. ⏳ Move template files to proper directories
6. ⏳ Update documentation with examples
7. ⏳ Migrate workspace_librecloud to KCL

---

**Last Updated**: 2025-12-03
