TypeDialog/CHANGELOG.md

# Changes

## [Unreleased]

### Added - Professional Quality Improvements

**Community Health Files**
- Created `CONTRIBUTING.md` with contribution guidelines, development workflow, and coding standards
- Created `SECURITY.md` with vulnerability reporting process and security best practices
- Created `CODE_OF_CONDUCT.md` with Contributor Covenant 2.1

**Security & Compliance**
- Created `deny.toml` with cargo-deny configuration for license/security policies
- Configured Renovate for automated dependency updates with automerge for minor/patch
- Added pre-commit hooks (fmt, clippy, shellcheck, markdownlint, trailing-whitespace, check-yaml/toml/json)

**Testing Infrastructure**
- Configured cargo-llvm-cov for code coverage tracking
- Integrated Codecov for coverage reporting in CI
- Added coverage badge to README
- Created benchmark tests with criterion for performance baselines
- Added property-based testing with proptest for validation functions
- Justfile recipes: `just dev::coverage`, `just dev::bench`, `just dev::coverage-open`

**CI/CD Enhancements**
- Multi-OS CI matrix: Linux, macOS, Windows (was Linux only)
- Cargo cache with rust-cache action (faster CI builds)
- Code coverage job uploading to Codecov
- Benchmark job running on pull requests
- cargo-deny checks in CI pipeline

**Release Automation**
- Full automated release workflow: tag → build multi-OS → publish GitHub + crates.io
- Builds for 5 targets: Linux x86_64/aarch64, macOS x86_64/aarch64, Windows x86_64
- Automatic tarball + SHA256 generation and upload
- Automatic crates.io publishing in dependency order
- cargo-binstall metadata for easy installation: `cargo binstall typedialog`

**Impact**
- Code coverage now tracked (was unknown)
- CI tests on 3 operating systems (was 1)
- Release process fully automated (was manual)
- 7 pre-commit hooks validate code quality
- Benchmark baselines established for performance tracking
- Property tests validate edge cases
- Security scanning with cargo-deny
- Automated dependency updates with Renovate

### Added - Multi-Language Linting System

**Linting Infrastructure**
- Created `.cargo/config.toml` with strict rustflags (warnings, future-incompatible, unused, etc.)
- Extended `justfiles/dev.just` with language-specific linters:
  - `just dev::lint-rust` - Clippy linting
  - `just dev::lint-bash` - Shellcheck for 142 bash scripts
  - `just dev::lint-nickel` - Nickel typecheck for 63 files
  - `just dev::lint-nushell` - Nushell validation for 9 scripts
  - `just dev::lint-markdown` - Markdownlint for docs/ folder only
  - `just dev::lint-all` - Execute all linters
- Extended `justfiles/ci.just` with CI-compatible linter recipes
- Updated `just check-all` to include multi-language linting

**GitHub Actions CI/CD**
- Created `.github/workflows/ci.yml` with automated linting matrix
- Parallel linting jobs per language (rust, bash, nickel, nushell, markdown)
- Integrated with test and build stages

**Linter Tools**
- Added shellcheck for bash script analysis
- Added markdownlint-cli2 for docs/ validation
- Leveraged existing nickel CLI for typecheck
- Leveraged nushell built-in --check flag

**Configuration**
- Created `.markdownlintrc` with basic rules (MD013, MD033, MD041 disabled)
- Updated `scripts/check_deps.sh` to verify all linter dependencies
- Updated `docs/build.md` with shellcheck and markdownlint installation

**Impact**
- 142 bash scripts now validated with shellcheck
- 63 Nickel files now typechecked
- 9 Nushell scripts validated
- docs/ markdown files (subset of 8,530 total) validated
- Rust code enforces stricter compilation standards via rustflags

### Added - TypeDialog Agent System

**Multi-Provider LLM Support**
- Complete LLM provider abstraction supporting Claude, OpenAI, Gemini, and Ollama
- Unified trait-based architecture for consistent API across providers
- Streaming support with two protocols: SSE (Claude/OpenAI) and JSON (Gemini/Ollama)
- Token usage tracking across all providers (except OpenAI streaming limitation)
- Local execution with Ollama for privacy-sensitive workloads

**Agent Execution System**
- New `.agent.mdx` file format for defining AI agents
- YAML frontmatter configuration with `@agent`, `@input`, `@import`, `@shell`, `@validate` directives
- CLI tool `typedialog-ag` for executing agent files
- Server component `typedialog-ag-server` for HTTP API access

**Template System**
- Tera template engine (Jinja2-compatible) for variable substitution and conditionals
- Template variables with `{{ var }}` syntax
- Conditional logic with `{% if %}...{% endif %}`
- File imports with glob pattern support: `@import "src/**/*.rs" as code`
- Shell command integration: `@shell "git status" as status`
- Environment variable loading

**Output Validation**
- Configurable validation rules for agent outputs
- Format validation (markdown, JSON, YAML)
- Content requirements (`must_contain` checks)
- Length constraints (`min_length`, `max_length`)

**Provider-Specific Features**
- **Claude (Anthropic)**: 3 models (Haiku, Sonnet, Opus), excellent code/reasoning, SSE streaming
- **OpenAI (GPT)**: gpt-4o, gpt-4o-mini, o1, o3 models, broad capabilities, SSE streaming
- **Gemini (Google)**: Creative tasks, 2M context window, free tier, JSON streaming
- **Ollama (Local)**: Complete privacy, offline operation, no API costs, supports llama2/mistral/codellama/etc.

### Added - Organization & Documentation

**Project Structure**
- Created `demos/agent/` with 4 provider comparison demos
- Created `tests/agent/` with validation tests
- Created `examples/12-agent-execution/` with 8 practical CLI examples
- Organized docs into backend-specific folders: `docs/cli/`, `docs/tui/`, `docs/web/`, `docs/ai/`, `docs/agent/`, `docs/encryption/`

**User Documentation**
- `docs/agent/README.md` - Agent system overview and navigation
- `docs/agent/getting_started.md` - Installation and first agent guide
- `docs/agent/llm_providers.md` - Complete provider comparison (400+ lines)
- `docs/cli/README.md` - CLI backend documentation
- `docs/tui/README.md` - TUI backend documentation
- `docs/web/README.md` - Web backend documentation
- `docs/ai/README.md` - AI backend documentation
- `docs/encryption/README.md` - Encryption documentation

**Agent Examples**
- `basic-greeting.agent.mdx` - Simple introduction (Claude Haiku)
- `code-review.agent.mdx` - Security & performance analysis (Claude Sonnet)
- `documentation.agent.mdx` - Generate docs from code (GPT-4o)
- `task-planner.agent.mdx` - Project planning (Claude Sonnet)
- `local-privacy.agent.mdx` - Privacy-first analysis (Ollama/llama2)
- `creative-writer.agent.mdx` - Creative content (Gemini 2.0)
- `refactoring-assistant.agent.mdx` - Code refactoring (GPT-4o-mini)
- `architect.agent.mdx` - High-level design (Claude Opus)

**Cross-References**
- Extensive linking between examples, tests, demos, and documentation
- Navigation structure: examples → demos → tests → core → technical docs

### Added - Configuration & Provisioning

**Configuration Structure**
- Renamed `config/agent/` → `config/ag/` for consistency with `typedialog-ag` binary name
- Added configuration profiles for all 6 backends: `cli/`, `tui/`, `web/`, `ai/`, `ag/`, `prov-gen/`
- Each backend has 3 environments: `default.toml`, `dev.toml`, `production.toml`
- Total: 18 configuration files (6 backends × 3 environments)

**config/ag/** - Agent configuration (LLM execution)
- `default.toml` - Claude Haiku default, streaming enabled, Tera templates
- `dev.toml` - Ollama (local) for development, debug logging, temperature 0.7
- `production.toml` - Claude Sonnet, lower temperature (0.3), rate limiting enabled

**config/prov-gen/** - Provisioning generator configuration (IaC)
- `default.toml` - AWS + Hetzner providers, Nickel validation, AI disabled
- `dev.toml` - Hetzner + LXD (cheaper), Ollama for AI, verbose logging
- `production.toml` - AWS + GCP, strict validation, encryption required, tests required

**Provisioning Generator Templates**
- Created `provider-gcp-section.toml` (256 lines, 15 elements)
  - Google Cloud Platform: 24 regions, 16 machine types (e2/n1/n2/c2)
  - Project ID + Service Account authentication
  - Disk types: Standard, Balanced SSD, Premium SSD, Ultra SSD
  - VPC networking, preemptible instances, deletion protection

- Created `provider-azure-section.toml` (296 lines, 17 elements)
  - Microsoft Azure: 30 regions globally
  - Service Principal authentication (Tenant ID, Client ID, Secret)
  - 12 VM sizes (B-series burstable, D/E-series, F-series compute)
  - Managed disks, Virtual Networks, accelerated networking

- Created `provider-lxd-section.toml` (207 lines, 16 elements)
  - LXD containers and virtual machines (local/remote)
  - 10 base images (Ubuntu, Debian, Alpine, Rocky, Arch)
  - CPU/Memory/Disk limits, 5 storage pools (dir, zfs, btrfs, lvm)
  - Network modes: lxdbr0 (NAT), host, macvlan
  - Nesting support (Docker inside LXD), cloud-init integration

**Total provider templates**: 6 providers (AWS, GCP, Azure, Hetzner, UpCloud, LXD) - 1,191 lines

**Dependency Management**
- Created `scripts/check_deps.sh` - Automated dependency verification script
- Checks critical dependencies: `just`, `nickel`, `cargo`, `prov-ecosystem`, `secretumvault`
- Exit status 0 if all found, 1 if any missing
- Provides installation instructions for missing dependencies

**Provisioning Generator Command Enhancements**
- Added `-f, --format <FORMAT>` flag to `typedialog-prov-gen templates` command
  - Supported formats: `json`, `yaml`, `toml`, `text` (default)
  - Enables machine-readable template listing for automation and integration
  - JSON/YAML for CI/CD pipelines, TOML for configuration files
- Template path display: Shows resolved absolute path before listing
  - Output: "From: /absolute/path/to/templates"
  - Helps users verify which template directory is loaded
- TemplateCategory structure now serializable across all formats
  - Consistent structure: `name`, `description`, `templates[]`
  - TOML output uses `[[categories]]` array-of-tables format

### Changed - Documentation Updates

**docs/build.md** - Critical Dependencies Section (320+ lines added)
- **Internal Dependencies**: `prov-ecosystem` (for typedialog-prov-gen), `secretumvault` (for encryption)
  - Installation: sibling directory or git submodule
  - Path configuration: Cargo.toml paths, environment variables
  - Verification commands

- **External Dependencies**: `just` (command runner), `Nickel` (configuration language)
  - Multi-platform installation: Homebrew, cargo, from source
  - PATH configuration
  - Verification tests

- **Dependency Quick Check**: Bash script to verify all dependencies
- **Troubleshooting**: 3 common issues with solutions each
  - prov-ecosystem/secretumvault not found (3 solutions)
  - Nickel binary not found (3 solutions)
  - just command not found (3 solutions)

**docs/README.md** - Navigation Updates
- Added "Critical Dependencies" to Building section
- Updated "Building & Distribution" with dependencies reference

**config/README.md** - Complete Backend Documentation (550+ lines)
- Documented all 6 backends: CLI, TUI, Web, AI, Agent (ag), Prov-gen
- Each backend has: configuration table, usage examples, features list
- **AI Backend**: RAG pipeline, embeddings (OpenAI/Ollama), vector store, knowledge graphs
- **Agent Backend**: Multi-provider LLM, Tera templates, streaming, variable substitution
- **Prov-gen**: Multi-cloud IaC, 7-layer validation, Nickel integration, AI-assisted generation
- Added configuration details for AI, Agent, and Prov-gen backends
- Updated development and production configuration examples
- Distribution structure updated to show 6 backends

**README.md** - Main Project Documentation
- Updated Features: "6 Backends" (was "3 Backends")
- Added **AI Backend section** (lines 95-113)
  - RAG system with semantic search
  - Multi-provider embeddings (OpenAI, Ollama)
  - Vector store (in-memory, Redis)
  - Knowledge graph generation

- Added **Provisioning Generator section** (lines 182-223)
  - Infrastructure as Code generation
  - Quick example with 3 command variants
  - 6 features: Multi-cloud, 7-layer validation, Nickel, AI-assist, templates, profiles
  - Supported providers table: AWS, GCP, Azure, Hetzner, UpCloud, LXD

- Updated Configuration structure: 6 backends listed
- Updated Project Structure: All 9 crates documented
- Updated Key Technologies: Organized into Core, Backends, AI & Agent, Infrastructure
- Updated Examples: Added `11-prov-gen/` and `12-agent-execution/` categories

**docs/prov-gen/README.md**
- Updated Multi-Provider Support: AWS, GCP, Azure, Hetzner, UpCloud, LXD (was AWS, GCP, Azure, Hetzner, LXD, bare metal)
- Changed "generates complete provisioning" → "generates provisioning" (modesty)

### Changed - Naming Consistency

**Package Renaming**
- `typeagent-core` → `typedialog-ag-core`
- `typeagent-cli` → `typedialog-ag`
- `typeagent-server` → `typedialog-ag-server`
- `config/agent/` → `config/ag/`

**Rationale**: Align with project-wide `typedialog-*` naming convention and binary names

**Impact**
- Updated 50+ files (imports, docs, scripts, configs)
- Updated workspace `Cargo.toml` dependencies
- Updated all crate manifests
- Updated justfile recipes
- Updated configuration directory structure
- 53 tests passing, 0 clippy warnings after refactor

### Changed - Build & Distribution

**Justfile Updates**
- Added `just build::agent` - Build Agent CLI
- Added `just build::agent-server` - Build Agent server
- Added `just build::prov-gen` - Build Provisioning Generator
- Added `just test::agent` - Test Agent system (53 tests)
- Added `just test::prov-gen` - Test Provisioning Generator (11 tests)
- Updated `just build::all` to include Agent and Prov-gen builds
- Fixed `just test::agent` - Removed invalid lib test for binary-only `typedialog-ag` crate

**Script Updates**
- `scripts/create_distro.sh` - Include all binaries: `typedialog-ag`, `typedialog-ag-server`, `typedialog-prov-gen`
- `scripts/check_deps.sh` - New dependency verification script (67 lines)
- `installers/bootstrap/install.sh` - Install Agent and Prov-gen binaries
- `installers/bootstrap/install.ps1` - Install Agent and Prov-gen binaries (Windows)
- `.woodpecker/Dockerfile.cross` - Cross-compile Agent and Prov-gen binaries

**Build Features**
- New feature flag `FEATURES_AGENT := "markup,nickel,cache"`
- Agent and Prov-gen builds integrated into full CI/CD pipeline
- All builds verified: `just build::prov-gen` ✓, `just build::agent` ✓
- All tests passing: `just test::prov-gen` ✓ (11 tests), `just test::agent` ✓ (53 tests, 5 ignored)

## [Previous]

### Added

**RepeatingGroup Duplicate Detection**
- Implemented duplicate item validation across all backends (CLI, TUI, Web)
- Added `unique = true` attribute support to prevent duplicate items
- Added item counter display updates in Web backend
- New `is_duplicate()` function in CLI and TUI backends

**Web Backend Improvements**
- Refactored JavaScript: Replaced global constants with closure-based state management
- Added `render_global_repeating_group_script()` for generic repeating group handling
- Configuration stored in `data-*` attributes: `data-min-items`, `data-max-items`, `data-has-unique`, `data-unique-key`, `data-field-names`
- Live item counter updates after add/delete operations
- Single document-level event delegation for all repeating group interactions

**Documentation**
- Updated `docs/field_types.md` with unique validation details
- Updated `examples/README.md` with RepeatingGroup testing commands
- Added `docs/field_types.md` (new file)
- Updated example forms with `unique = true` attribute

### Fixed

- **CLI Backend**: Fixed validation of duplicate items in add/edit operations
- **TUI Backend**: Fixed validation of duplicate items with error overlay feedback
- **Web Backend**: Fixed `has_unique` flag reading from field definition (was incorrectly scanning fragment fields)
- **Web Backend**: Fixed item counter not updating when adding/deleting items

### Changed

**Architecture**
- Each repeating group now maintains independent state via closure
- Removed ~600 lines of dead code in Web JavaScript
- Event handling now context-aware (finds correct repeating group controller)

**Examples**
- `examples/05-fragments/array-trackers.toml`: Added `unique = true` to UDP and HTTP tracker arrays
- `examples/07-nickel-generation/arrays-form.toml`: Added `unique = true` to all RepeatingGroup fields

### Technical Notes

- Duplicate detection compares ALL field values when `unique = true`
- Works consistently across CLI, TUI, and Web backends
- Backwards compatible: repeating groups without `unique = true` unaffected
- max_items limit already enforced in all backends (no changes needed)

### Testing

- All 174 unit tests passing
- No clippy warnings
- Build verified with `--all-features` flag
- Manual testing in Web backend: duplicate detection working correctly
- Item counter updates verified on add/delete operations