# Changes

## [Unreleased]

### Changed - Nickel as primary form and config format

All form definitions and backend configs migrated from TOML to Nickel (`.ncl`).

**Examples** (`examples/01-basic/` through `examples/17-advanced-i18n/`): all `.toml` form files
replaced with type-checked `.ncl` equivalents using Nickel contracts and `| default` annotations.

**Backend configs** (`config/cli/`, `config/tui/`, `config/web/`, `config/ai/`, `config/ag/`,
`config/prov-gen/`): `default.toml`, `dev.toml`, `production.toml` replaced with `config.ncl`,
`dev.ncl`, `production.ncl`. Each expresses schema constraints inline (`| default`, `| String`,
`| Bool`) rather than raw TOML values.

**Config loader** (`crates/typedialog-core/src/config/cli_loader.rs`): search order updated to
prefer `.ncl` over `.toml`. NCL files are loaded via `nickel export --format json` and deserialised
with `serde_json`; TOML path retained as fallback. New order:

1. `{backend}/{TYPEDIALOG_ENV}.ncl`
2. `{backend}/{TYPEDIALOG_ENV}.toml`
3. `{backend}/config.ncl`
4. `{backend}/config.toml`

TOML form definitions remain supported — no breaking change to the parser or any backend.

### Refactored - Eliminate duplicated field execution logic

**Single source of truth for CLI field dispatch**

Three independent copies of the `match field.field_type` dispatch existed across
`form_parser/executor.rs`, `backends/cli.rs`, and `nickel/roundtrip.rs`. Each had
drifted in behavior: `executor.rs` lacked `RepeatingGroup`, `cli.rs` lacked
`options_from` filtering, `roundtrip.rs` had both but also its own `filter_options_from`.

- Consolidated into `InquireBackend::execute_field_sync` (`backends/cli.rs`) as the
  single canonical implementation
- Added `previous_results: &HashMap<String, Value>` parameter to `execute_field_sync`
  (now `pub(crate)`) — unlocks `options_from` filtering for dependent `Select` fields
- Moved `filter_options_from` to `backends/cli.rs` as `pub(crate)`, removed the two
  duplicate copies from `executor.rs` and `nickel/roundtrip.rs`
- `FormBackend::execute_field` trait impl now passes `context.results` directly to
  `execute_field_sync` instead of ignoring the accumulated results
- Legacy sync path (`execute_with_base_dir`, `execute`, `load_and_execute_from_file`)
  gated on `#[cfg(feature = "cli")]` — semantically correct, previously they compiled
  against non-CLI feature combinations without the Inquire dependency being present
- Net: −346 lines across 4 files, 0 behavior regressions

**Fixed - Doctest compilation failures**

Five doctests in `typedialog-core` failed to compile against the current API:

- `lib.rs` / `prelude.rs`: `FormDefinition { title, display_items }` → fields renamed
  to `name` / `elements` in a prior refactor; replaced with `form_parser::parse_toml()`
- `lib.rs` / `prompt_api.rs`: `confirm(prompt, default)` → takes 3 args
  (`prompt, default, formatter`); added missing `None`
- `advanced.rs`: example implemented `render_text`, `render_confirm`, etc. — methods
  that no longer exist in `FormBackend`; replaced with the 7 real trait methods
- `config/cli_loader.rs`: `ignore` annotation with fictional `MyBackendConfig` type;
  replaced with a concrete inline struct and changed to `no_run`

**Refactored - Eliminate O(n) HashMap clone on every field boundary**

`RenderContext.results` changed from owned `HashMap<String, Value>` to
`Arc<HashMap<String, Value>>`. The executor accumulator was also changed to `Arc`
so that `RenderContext` construction costs one atomic ref-count increment instead of
a full HashMap deep clone. After each backend call, `Arc::make_mut` mutates in place
since the ref count drops to 1 (the context is dropped before the insert). The
`execute_form_complete` default impl, all three executor functions
(`execute_with_backend_complete`, `execute_with_backend_two_phase_with_defaults`,
`execute_with_backend_i18n_with_defaults`), and the TUI fragment executor were updated.

- All backends receive `&context.results` unchanged — `Arc<T>` deref coercion to `&T`
  is handled by the compiler with no source changes required in most call sites
- Net: eliminates O(n) allocations per field in the sequential execution hot path
- Resolves last open technical debt item documented in `docs/architecture.md`

**Fixed - Unbounded retry recursion in required-field validation**

`InquireBackend::execute_field_sync` retried invalid required-field input by calling itself
recursively with no depth limit. A user repeatedly entering empty values for a required field
would grow the call stack unboundedly until stack overflow.

- Wrapped the `match field.field_type` dispatch in a `loop {}`
- Replaced `return self.execute_field_sync(field, previous_results)` with `continue`
- Changed all non-retry exit points from `Ok(...)` to `return Ok(...)`
- `RepeatingGroup` arm retains `return self.execute_repeating_group(...)` — it carries its
  own internal loop and returns a terminal `Result`
- No behavior change for the happy path; retry now uses O(1) stack instead of O(n)

**Added - Architecture documentation**

- New `docs/architecture.md`: technical positioning (TypeDialog vs Inquire/Ratatui/Axum),
  `BackendFactory` mechanics, three-phase execution model, `CLI Field Execution` section
  documenting `execute_field_sync` + `filter_options_from` contract, legacy sync path,
  and remaining known technical debt

### Added - Web Backend Browser Auto-Launch

**`--open` Flag for typedialog-web**
- Added `--open` flag to auto-launch form in default web browser
- Supported in both `typedialog-web` command and `nickel-roundtrip` subcommand
- Graceful error handling: warnings if browser open fails, execution continues
- Cross-platform support: macOS (`open`), Linux (`xdg-open`), Windows (`start`)

**Usage Examples**
```bash
typedialog-web form.toml --open
typedialog-web form.toml --port 8080 --open
typedialog-web nickel-roundtrip input.ncl form.toml --output output.ncl --open
```text

**Benefits**
- Improves UX: eliminates manual URL copy/paste steps
- Aligns with industry standards (mdbook serve --open, http-server, etc.)
- Faster development workflow: edit → view changes without manual browser navigation
- Perfect for demos and user onboarding workflows

**Documentation**
- Updated `docs/web/README.md` with `--open` examples in "Basic Usage" section
- Added "Browser Auto-Launch" subsection explaining use cases and behavior

### Added - Unified Command Interface (Dispatcher)

**Dispatcher for Backend Access**
- Added dispatcher to `typedialog` command allowing access to all backends via single entry point
- Users can now invoke: `typedialog web`, `typedialog tui`, `typedialog ai`, `typedialog ag`, `typedialog prov-gen`
- Each backend command dispatches to corresponding binary (`typedialog-web`, `typedialog-tui`, etc.)
- All arguments are passed through transparently to backend binaries
- Help available via: `typedialog web -h`, `typedialog tui -h`, etc.
- Maintains backward compatibility - specialized binaries still work directly

**Updated Documentation**
- README.md: Added "Unified Command Interface" section with dispatcher examples
- Backend sections now show both dispatcher and direct binary usage
- Consistent command examples across all backend documentation

**Benefits**
- Simplified UX - Single entry point instead of 6 separate binaries
- No need to know about internal binary separation
- Easier discoverability: `typedialog -h` shows all backends as subcommands
- Transparent argument passing maintains full feature access

### Added - Professional Quality Improvements

**Community Health Files**
- Created `CONTRIBUTING.md` with contribution guidelines, development workflow, and coding standards
- Created `SECURITY.md` with vulnerability reporting process and security best practices
- Created `CODE_OF_CONDUCT.md` with Contributor Covenant 2.1

**Security & Compliance**
- Created `deny.toml` with cargo-deny configuration for license/security policies
- Configured Renovate for automated dependency updates with automerge for minor/patch
- Added pre-commit hooks (fmt, clippy, shellcheck, markdownlint, trailing-whitespace, check-yaml/toml/json)

**Testing Infrastructure**
- Configured cargo-llvm-cov for code coverage tracking
- Integrated Codecov for coverage reporting in CI
- Added coverage badge to README
- Created benchmark tests with criterion for performance baselines
- Added property-based testing with proptest for validation functions
- Justfile recipes: `just dev::coverage`, `just dev::bench`, `just dev::coverage-open`

**CI/CD Enhancements**
- Multi-OS CI matrix: Linux, macOS, Windows (was Linux only)
- Cargo cache with rust-cache action (faster CI builds)
- Code coverage job uploading to Codecov
- Benchmark job running on pull requests
- cargo-deny checks in CI pipeline

**Release Automation**
- Full automated release workflow: tag → build multi-OS → publish GitHub + crates.io
- Builds for 5 targets: Linux x86_64/aarch64, macOS x86_64/aarch64, Windows x86_64
- Automatic tarball + SHA256 generation and upload
- Automatic crates.io publishing in dependency order
- cargo-binstall metadata for easy installation: `cargo binstall typedialog`

**Impact**
- Code coverage now tracked (was unknown)
- CI tests on 3 operating systems (was 1)
- Release process fully automated (was manual)
- 7 pre-commit hooks validate code quality
- Benchmark baselines established for performance tracking
- Property tests validate edge cases
- Security scanning with cargo-deny
- Automated dependency updates with Renovate

### Added - Multi-Language Linting System

**Linting Infrastructure**
- Created `.cargo/config.toml` with strict rustflags (warnings, future-incompatible, unused, etc.)
- Extended `justfiles/dev.just` with language-specific linters:
  - `just dev::lint-rust` - Clippy linting
  - `just dev::lint-bash` - Shellcheck for 142 bash scripts
  - `just dev::lint-nickel` - Nickel typecheck for 63 files
  - `just dev::lint-nushell` - Nushell validation for 9 scripts
  - `just dev::lint-markdown` - Markdownlint for docs/ folder only
  - `just dev::lint-all` - Execute all linters
- Extended `justfiles/ci.just` with CI-compatible linter recipes
- Updated `just check-all` to include multi-language linting

**GitHub Actions CI/CD**
- Created `.github/workflows/ci.yml` with automated linting matrix
- Parallel linting jobs per language (rust, bash, nickel, nushell, markdown)
- Integrated with test and build stages

**Linter Tools**
- Added shellcheck for bash script analysis
- Added markdownlint-cli2 for docs/ validation
- Leveraged existing nickel CLI for typecheck
- Leveraged nushell built-in --check flag

**Configuration**
- Created `.markdownlintrc` with basic rules (MD013, MD033, MD041 disabled)
- Updated `scripts/check_deps.sh` to verify all linter dependencies
- Updated `docs/build.md` with shellcheck and markdownlint installation

**Impact**
- 142 bash scripts now validated with shellcheck
- 63 Nickel files now typechecked
- 9 Nushell scripts validated
- docs/ markdown files (subset of 8,530 total) validated
- Rust code enforces stricter compilation standards via rustflags

### Added - TypeDialog Agent System

**Multi-Provider LLM Support**
- Complete LLM provider abstraction supporting Claude, OpenAI, Gemini, and Ollama
- Unified trait-based architecture for consistent API across providers
- Streaming support with two protocols: SSE (Claude/OpenAI) and JSON (Gemini/Ollama)
- Token usage tracking across all providers (except OpenAI streaming limitation)
- Local execution with Ollama for privacy-sensitive workloads

**Agent Execution System**
- New `.agent.mdx` file format for defining AI agents
- YAML frontmatter configuration with `@agent`, `@input`, `@import`, `@shell`, `@validate` directives
- CLI tool `typedialog-ag` for executing agent files
- Server component `typedialog-ag-server` for HTTP API access

**Template System**
- Tera template engine (Jinja2-compatible) for variable substitution and conditionals
- Template variables with `{{ var }}` syntax
- Conditional logic with `{% if %}...{% endif %}`
- File imports with glob pattern support: `@import "src/**/*.rs" as code`
- Shell command integration: `@shell "git status" as status`
- Environment variable loading

**Output Validation**
- Configurable validation rules for agent outputs
- Format validation (markdown, JSON, YAML)
- Content requirements (`must_contain` checks)
- Length constraints (`min_length`, `max_length`)

**Provider-Specific Features**
- **Claude (Anthropic)**: 3 models (Haiku, Sonnet, Opus), excellent code/reasoning, SSE streaming
- **OpenAI (GPT)**: gpt-4o, gpt-4o-mini, o1, o3 models, broad capabilities, SSE streaming
- **Gemini (Google)**: Creative tasks, 2M context window, free tier, JSON streaming
- **Ollama (Local)**: Complete privacy, offline operation, no API costs, supports llama2/mistral/codellama/etc.

### Added - Organization & Documentation

**Project Structure**
- Created `demos/agent/` with 4 provider comparison demos
- Created `tests/agent/` with validation tests
- Created `examples/12-agent-execution/` with 8 practical CLI examples
- Organized docs into backend-specific folders: `docs/cli/`, `docs/tui/`, `docs/web/`, `docs/ai/`, `docs/agent/`, `docs/encryption/`

**User Documentation**
- `docs/agent/README.md` - Agent system overview and navigation
- `docs/agent/getting_started.md` - Installation and first agent guide
- `docs/agent/llm_providers.md` - Complete provider comparison (400+ lines)
- `docs/cli/README.md` - CLI backend documentation
- `docs/tui/README.md` - TUI backend documentation
- `docs/web/README.md` - Web backend documentation
- `docs/ai/README.md` - AI backend documentation
- `docs/encryption/README.md` - Encryption documentation

**Agent Examples**
- `basic-greeting.agent.mdx` - Simple introduction (Claude Haiku)
- `code-review.agent.mdx` - Security & performance analysis (Claude Sonnet)
- `documentation.agent.mdx` - Generate docs from code (GPT-4o)
- `task-planner.agent.mdx` - Project planning (Claude Sonnet)
- `local-privacy.agent.mdx` - Privacy-first analysis (Ollama/llama2)
- `creative-writer.agent.mdx` - Creative content (Gemini 2.0)
- `refactoring-assistant.agent.mdx` - Code refactoring (GPT-4o-mini)
- `architect.agent.mdx` - High-level design (Claude Opus)

**Cross-References**
- Extensive linking between examples, tests, demos, and documentation
- Navigation structure: examples → demos → tests → core → technical docs

### Added - Configuration & Provisioning

**Configuration Structure**
- Renamed `config/agent/` → `config/ag/` for consistency with `typedialog-ag` binary name
- Added configuration profiles for all 6 backends: `cli/`, `tui/`, `web/`, `ai/`, `ag/`, `prov-gen/`
- Each backend has 3 environments: `default.toml`, `dev.toml`, `production.toml`
- Total: 18 configuration files (6 backends × 3 environments)

**config/ag/** - Agent configuration (LLM execution)
- `default.toml` - Claude Haiku default, streaming enabled, Tera templates
- `dev.toml` - Ollama (local) for development, debug logging, temperature 0.7
- `production.toml` - Claude Sonnet, lower temperature (0.3), rate limiting enabled

**config/prov-gen/** - Provisioning generator configuration (IaC)
- `default.toml` - AWS + Hetzner providers, Nickel validation, AI disabled
- `dev.toml` - Hetzner + LXD (cheaper), Ollama for AI, verbose logging
- `production.toml` - AWS + GCP, strict validation, encryption required, tests required

**Provisioning Generator Templates**
- Created `provider-gcp-section.toml` (256 lines, 15 elements)
  - Google Cloud Platform: 24 regions, 16 machine types (e2/n1/n2/c2)
  - Project ID + Service Account authentication
  - Disk types: Standard, Balanced SSD, Premium SSD, Ultra SSD
  - VPC networking, preemptible instances, deletion protection

- Created `provider-azure-section.toml` (296 lines, 17 elements)
  - Microsoft Azure: 30 regions globally
  - Service Principal authentication (Tenant ID, Client ID, Secret)
  - 12 VM sizes (B-series burstable, D/E-series, F-series compute)
  - Managed disks, Virtual Networks, accelerated networking

- Created `provider-lxd-section.toml` (207 lines, 16 elements)
  - LXD containers and virtual machines (local/remote)
  - 10 base images (Ubuntu, Debian, Alpine, Rocky, Arch)
  - CPU/Memory/Disk limits, 5 storage pools (dir, zfs, btrfs, lvm)
  - Network modes: lxdbr0 (NAT), host, macvlan
  - Nesting support (Docker inside LXD), cloud-init integration

**Total provider templates**: 6 providers (AWS, GCP, Azure, Hetzner, UpCloud, LXD) - 1,191 lines

**Dependency Management**
- Created `scripts/check_deps.sh` - Automated dependency verification script
- Checks critical dependencies: `just`, `nickel`, `cargo`, `prov-ecosystem`, `secretumvault`
- Exit status 0 if all found, 1 if any missing
- Provides installation instructions for missing dependencies

**Provisioning Generator Command Enhancements**
- Added `-f, --format <FORMAT>` flag to `typedialog-prov-gen templates` command
  - Supported formats: `json`, `yaml`, `toml`, `text` (default)
  - Enables machine-readable template listing for automation and integration
  - JSON/YAML for CI/CD pipelines, TOML for configuration files
- Template path display: Shows resolved absolute path before listing
  - Output: "From: /absolute/path/to/templates"
  - Helps users verify which template directory is loaded
- TemplateCategory structure now serializable across all formats
  - Consistent structure: `name`, `description`, `templates[]`
  - TOML output uses `[[categories]]` array-of-tables format

### Changed - Documentation Updates

**docs/build.md** - Critical Dependencies Section (320+ lines added)
- **Internal Dependencies**: `prov-ecosystem` (for typedialog-prov-gen), `secretumvault` (for encryption)
  - Installation: sibling directory or git submodule
  - Path configuration: Cargo.toml paths, environment variables
  - Verification commands

- **External Dependencies**: `just` (command runner), `Nickel` (configuration language)
  - Multi-platform installation: Homebrew, cargo, from source
  - PATH configuration
  - Verification tests

- **Dependency Quick Check**: Bash script to verify all dependencies
- **Troubleshooting**: 3 common issues with solutions each
  - prov-ecosystem/secretumvault not found (3 solutions)
  - Nickel binary not found (3 solutions)
  - just command not found (3 solutions)

**docs/README.md** - Navigation Updates
- Added "Critical Dependencies" to Building section
- Updated "Building & Distribution" with dependencies reference

**config/README.md** - Complete Backend Documentation (550+ lines)
- Documented all 6 backends: CLI, TUI, Web, AI, Agent (ag), Prov-gen
- Each backend has: configuration table, usage examples, features list
- **AI Backend**: RAG pipeline, embeddings (OpenAI/Ollama), vector store, knowledge graphs
- **Agent Backend**: Multi-provider LLM, Tera templates, streaming, variable substitution
- **Prov-gen**: Multi-cloud IaC, 7-layer validation, Nickel integration, AI-assisted generation
- Added configuration details for AI, Agent, and Prov-gen backends
- Updated development and production configuration examples
- Distribution structure updated to show 6 backends

**README.md** - Main Project Documentation
- Updated Features: "6 Backends" (was "3 Backends")
- Added **AI Backend section** (lines 95-113)
  - RAG system with semantic search
  - Multi-provider embeddings (OpenAI, Ollama)
  - Vector store (in-memory, Redis)
  - Knowledge graph generation

- Added **Provisioning Generator section** (lines 182-223)
  - Infrastructure as Code generation
  - Quick example with 3 command variants
  - 6 features: Multi-cloud, 7-layer validation, Nickel, AI-assist, templates, profiles
  - Supported providers table: AWS, GCP, Azure, Hetzner, UpCloud, LXD

- Updated Configuration structure: 6 backends listed
- Updated Project Structure: All 9 crates documented
- Updated Key Technologies: Organized into Core, Backends, AI & Agent, Infrastructure
- Updated Examples: Added `11-prov-gen/` and `12-agent-execution/` categories

**docs/prov-gen/README.md**
- Updated Multi-Provider Support: AWS, GCP, Azure, Hetzner, UpCloud, LXD (was AWS, GCP, Azure, Hetzner, LXD, bare metal)
- Changed "generates complete provisioning" → "generates provisioning" (modesty)

### Changed - Naming Consistency

**Package Renaming**
- `typeagent-core` → `typedialog-ag-core`
- `typeagent-cli` → `typedialog-ag`
- `typeagent-server` → `typedialog-ag-server`
- `config/agent/` → `config/ag/`

**Rationale**: Align with project-wide `typedialog-*` naming convention and binary names

**Impact**
- Updated 50+ files (imports, docs, scripts, configs)
- Updated workspace `Cargo.toml` dependencies
- Updated all crate manifests
- Updated justfile recipes
- Updated configuration directory structure
- 53 tests passing, 0 clippy warnings after refactor

### Changed - Build & Distribution

**Justfile Updates**
- Added `just build::agent` - Build Agent CLI
- Added `just build::agent-server` - Build Agent server
- Added `just build::prov-gen` - Build Provisioning Generator
- Added `just test::agent` - Test Agent system (53 tests)
- Added `just test::prov-gen` - Test Provisioning Generator (11 tests)
- Updated `just build::all` to include Agent and Prov-gen builds
- Fixed `just test::agent` - Removed invalid lib test for binary-only `typedialog-ag` crate

**Script Updates**
- `scripts/create_distro.sh` - Include all binaries: `typedialog-ag`, `typedialog-ag-server`, `typedialog-prov-gen`
- `scripts/check_deps.sh` - New dependency verification script (67 lines)
- `installers/bootstrap/install.sh` - Install Agent and Prov-gen binaries
- `installers/bootstrap/install.ps1` - Install Agent and Prov-gen binaries (Windows)
- `.woodpecker/Dockerfile.cross` - Cross-compile Agent and Prov-gen binaries

**Build Features**
- New feature flag `FEATURES_AGENT := "markup,nickel,cache"`
- Agent and Prov-gen builds integrated into full CI/CD pipeline
- All builds verified: `just build::prov-gen` ✓, `just build::agent` ✓
- All tests passing: `just test::prov-gen` ✓ (11 tests), `just test::agent` ✓ (53 tests, 5 ignored)

## [Previous]

### Added

**RepeatingGroup Duplicate Detection**
- Implemented duplicate item validation across all backends (CLI, TUI, Web)
- Added `unique = true` attribute support to prevent duplicate items
- Added item counter display updates in Web backend
- New `is_duplicate()` function in CLI and TUI backends

**Web Backend Improvements**
- Refactored JavaScript: Replaced global constants with closure-based state management
- Added `render_global_repeating_group_script()` for generic repeating group handling
- Configuration stored in `data-*` attributes: `data-min-items`, `data-max-items`, `data-has-unique`, `data-unique-key`, `data-field-names`
- Live item counter updates after add/delete operations
- Single document-level event delegation for all repeating group interactions

**Documentation**
- Updated `docs/field_types.md` with unique validation details
- Updated `examples/README.md` with RepeatingGroup testing commands
- Added `docs/field_types.md` (new file)
- Updated example forms with `unique = true` attribute

### Fixed

- **CLI Backend**: Fixed validation of duplicate items in add/edit operations
- **TUI Backend**: Fixed validation of duplicate items with error overlay feedback
- **Web Backend**: Fixed `has_unique` flag reading from field definition (was incorrectly scanning fragment fields)
- **Web Backend**: Fixed item counter not updating when adding/deleting items

### Changed

**Architecture**
- Each repeating group now maintains independent state via closure
- Removed ~600 lines of dead code in Web JavaScript
- Event handling now context-aware (finds correct repeating group controller)

**Examples**
- `examples/05-fragments/array-trackers.toml`: Added `unique = true` to UDP and HTTP tracker arrays
- `examples/07-nickel-generation/arrays-form.toml`: Added `unique = true` to all RepeatingGroup fields

### Technical Notes

- Duplicate detection compares ALL field values when `unique = true`
- Works consistently across CLI, TUI, and Web backends
- Backwards compatible: repeating groups without `unique = true` unaffected
- max_items limit already enforced in all backends (no changes needed)

### Testing

- All 174 unit tests passing
- No clippy warnings
- Build verified with `--all-features` flag
- Manual testing in Web backend: duplicate detection working correctly
- Item counter updates verified on add/delete operations
