TypeDialog/CHANGELOG.md

Changes

[Unreleased]

Added - Professional Quality Improvements

Community Health Files

• Created CONTRIBUTING.md with contribution guidelines, development workflow, and coding standards
• Created SECURITY.md with vulnerability reporting process and security best practices
• Created CODE_OF_CONDUCT.md with Contributor Covenant 2.1

Security & Compliance

• Created deny.toml with cargo-deny configuration for license/security policies
• Configured Renovate for automated dependency updates with automerge for minor/patch
• Added pre-commit hooks (fmt, clippy, shellcheck, markdownlint, trailing-whitespace, check-yaml/toml/json)

Testing Infrastructure

• Configured cargo-llvm-cov for code coverage tracking
• Integrated Codecov for coverage reporting in CI
• Added coverage badge to README
• Created benchmark tests with criterion for performance baselines
• Added property-based testing with proptest for validation functions
• Justfile recipes: just dev::coverage, just dev::bench, just dev::coverage-open

CI/CD Enhancements

• Multi-OS CI matrix: Linux, macOS, Windows (was Linux only)
• Cargo cache with rust-cache action (faster CI builds)
• Code coverage job uploading to Codecov
• Benchmark job running on pull requests
• cargo-deny checks in CI pipeline

Release Automation

• Full automated release workflow: tag → build multi-OS → publish GitHub + crates.io
• Builds for 5 targets: Linux x86_64/aarch64, macOS x86_64/aarch64, Windows x86_64
• Automatic tarball + SHA256 generation and upload
• Automatic crates.io publishing in dependency order
• cargo-binstall metadata for easy installation: cargo binstall typedialog

Impact

• Code coverage now tracked (was unknown)
• CI tests on 3 operating systems (was 1)
• Release process fully automated (was manual)
• 7 pre-commit hooks validate code quality
• Benchmark baselines established for performance tracking
• Property tests validate edge cases
• Security scanning with cargo-deny
• Automated dependency updates with Renovate

Added - Multi-Language Linting System

Linting Infrastructure

• Created .cargo/config.toml with strict rustflags (warnings, future-incompatible, unused, etc.)
• Extended justfiles/dev.just with language-specific linters:
  • just dev::lint-rust - Clippy linting
  • just dev::lint-bash - Shellcheck for 142 bash scripts
  • just dev::lint-nickel - Nickel typecheck for 63 files
  • just dev::lint-nushell - Nushell validation for 9 scripts
  • just dev::lint-markdown - Markdownlint for docs/ folder only
  • just dev::lint-all - Execute all linters
• Extended justfiles/ci.just with CI-compatible linter recipes
• Updated just check-all to include multi-language linting

GitHub Actions CI/CD

• Created .github/workflows/ci.yml with automated linting matrix
• Parallel linting jobs per language (rust, bash, nickel, nushell, markdown)
• Integrated with test and build stages

Linter Tools

• Added shellcheck for bash script analysis
• Added markdownlint-cli2 for docs/ validation
• Leveraged existing nickel CLI for typecheck
• Leveraged nushell built-in --check flag

Configuration

• Created .markdownlintrc with basic rules (MD013, MD033, MD041 disabled)
• Updated scripts/check_deps.sh to verify all linter dependencies
• Updated docs/build.md with shellcheck and markdownlint installation

Impact

• 142 bash scripts now validated with shellcheck
• 63 Nickel files now typechecked
• 9 Nushell scripts validated
• docs/ markdown files (subset of 8,530 total) validated
• Rust code enforces stricter compilation standards via rustflags

Added - TypeDialog Agent System

Multi-Provider LLM Support

• Complete LLM provider abstraction supporting Claude, OpenAI, Gemini, and Ollama
• Unified trait-based architecture for consistent API across providers
• Streaming support with two protocols: SSE (Claude/OpenAI) and JSON (Gemini/Ollama)
• Token usage tracking across all providers (except OpenAI streaming limitation)
• Local execution with Ollama for privacy-sensitive workloads

Agent Execution System

• New .agent.mdx file format for defining AI agents
• YAML frontmatter configuration with @agent, @input, @import, @shell, @validate directives
• CLI tool typedialog-ag for executing agent files
• Server component typedialog-ag-server for HTTP API access

Template System

• Tera template engine (Jinja2-compatible) for variable substitution and conditionals
• Template variables with {{ var }} syntax
• Conditional logic with {% if %}...{% endif %}
• File imports with glob pattern support: @import "src/**/*.rs" as code
• Shell command integration: @shell "git status" as status
• Environment variable loading

Output Validation

• Configurable validation rules for agent outputs
• Format validation (markdown, JSON, YAML)
• Content requirements (must_contain checks)
• Length constraints (min_length, max_length)

Provider-Specific Features

• Claude (Anthropic): 3 models (Haiku, Sonnet, Opus), excellent code/reasoning, SSE streaming
• OpenAI (GPT): gpt-4o, gpt-4o-mini, o1, o3 models, broad capabilities, SSE streaming
• Gemini (Google): Creative tasks, 2M context window, free tier, JSON streaming
• Ollama (Local): Complete privacy, offline operation, no API costs, supports llama2/mistral/codellama/etc.

Added - Organization & Documentation

Project Structure

• Created demos/agent/ with 4 provider comparison demos
• Created tests/agent/ with validation tests
• Created examples/12-agent-execution/ with 8 practical CLI examples
• Organized docs into backend-specific folders: docs/cli/, docs/tui/, docs/web/, docs/ai/, docs/agent/, docs/encryption/

User Documentation

• docs/agent/README.md - Agent system overview and navigation
• docs/agent/getting_started.md - Installation and first agent guide
• docs/agent/llm_providers.md - Complete provider comparison (400+ lines)
• docs/cli/README.md - CLI backend documentation
• docs/tui/README.md - TUI backend documentation
• docs/web/README.md - Web backend documentation
• docs/ai/README.md - AI backend documentation
• docs/encryption/README.md - Encryption documentation

Agent Examples

• basic-greeting.agent.mdx - Simple introduction (Claude Haiku)
• code-review.agent.mdx - Security & performance analysis (Claude Sonnet)
• documentation.agent.mdx - Generate docs from code (GPT-4o)
• task-planner.agent.mdx - Project planning (Claude Sonnet)
• local-privacy.agent.mdx - Privacy-first analysis (Ollama/llama2)
• creative-writer.agent.mdx - Creative content (Gemini 2.0)
• refactoring-assistant.agent.mdx - Code refactoring (GPT-4o-mini)
• architect.agent.mdx - High-level design (Claude Opus)

Cross-References

• Extensive linking between examples, tests, demos, and documentation
• Navigation structure: examples → demos → tests → core → technical docs

Added - Configuration & Provisioning

Configuration Structure

• Renamed config/agent/ → config/ag/ for consistency with typedialog-ag binary name
• Added configuration profiles for all 6 backends: cli/, tui/, web/, ai/, ag/, prov-gen/
• Each backend has 3 environments: default.toml, dev.toml, production.toml
• Total: 18 configuration files (6 backends × 3 environments)

config/ag/ - Agent configuration (LLM execution)

• default.toml - Claude Haiku default, streaming enabled, Tera templates
• dev.toml - Ollama (local) for development, debug logging, temperature 0.7
• production.toml - Claude Sonnet, lower temperature (0.3), rate limiting enabled

config/prov-gen/ - Provisioning generator configuration (IaC)

• default.toml - AWS + Hetzner providers, Nickel validation, AI disabled
• dev.toml - Hetzner + LXD (cheaper), Ollama for AI, verbose logging
• production.toml - AWS + GCP, strict validation, encryption required, tests required

Provisioning Generator Templates

• Created provider-gcp-section.toml (256 lines, 15 elements)

  • Google Cloud Platform: 24 regions, 16 machine types (e2/n1/n2/c2)
  • Project ID + Service Account authentication
  • Disk types: Standard, Balanced SSD, Premium SSD, Ultra SSD
  • VPC networking, preemptible instances, deletion protection

• Created provider-azure-section.toml (296 lines, 17 elements)

  • Microsoft Azure: 30 regions globally
  • Service Principal authentication (Tenant ID, Client ID, Secret)
  • 12 VM sizes (B-series burstable, D/E-series, F-series compute)
  • Managed disks, Virtual Networks, accelerated networking

• Created provider-lxd-section.toml (207 lines, 16 elements)

  • LXD containers and virtual machines (local/remote)
  • 10 base images (Ubuntu, Debian, Alpine, Rocky, Arch)
  • CPU/Memory/Disk limits, 5 storage pools (dir, zfs, btrfs, lvm)
  • Network modes: lxdbr0 (NAT), host, macvlan
  • Nesting support (Docker inside LXD), cloud-init integration

Total provider templates: 6 providers (AWS, GCP, Azure, Hetzner, UpCloud, LXD) - 1,191 lines

Dependency Management

• Created scripts/check_deps.sh - Automated dependency verification script
• Checks critical dependencies: just, nickel, cargo, prov-ecosystem, secretumvault
• Exit status 0 if all found, 1 if any missing
• Provides installation instructions for missing dependencies

Provisioning Generator Command Enhancements

• Added -f, --format <FORMAT> flag to typedialog-prov-gen templates command
  • Supported formats: json, yaml, toml, text (default)
  • Enables machine-readable template listing for automation and integration
  • JSON/YAML for CI/CD pipelines, TOML for configuration files
• Template path display: Shows resolved absolute path before listing
  • Output: "From: /absolute/path/to/templates"
  • Helps users verify which template directory is loaded
• TemplateCategory structure now serializable across all formats
  • Consistent structure: name, description, templates[]
  • TOML output uses [[categories]] array-of-tables format

Changed - Documentation Updates

docs/build.md - Critical Dependencies Section (320+ lines added)

• Internal Dependencies: prov-ecosystem (for typedialog-prov-gen), secretumvault (for encryption)

  • Installation: sibling directory or git submodule
  • Path configuration: Cargo.toml paths, environment variables
  • Verification commands

• External Dependencies: just (command runner), Nickel (configuration language)

  • Multi-platform installation: Homebrew, cargo, from source
  • PATH configuration
  • Verification tests

• Dependency Quick Check: Bash script to verify all dependencies

• Troubleshooting: 3 common issues with solutions each

  • prov-ecosystem/secretumvault not found (3 solutions)
  • Nickel binary not found (3 solutions)
  • just command not found (3 solutions)

docs/README.md - Navigation Updates

• Added "Critical Dependencies" to Building section
• Updated "Building & Distribution" with dependencies reference

config/README.md - Complete Backend Documentation (550+ lines)

• Documented all 6 backends: CLI, TUI, Web, AI, Agent (ag), Prov-gen
• Each backend has: configuration table, usage examples, features list
• AI Backend: RAG pipeline, embeddings (OpenAI/Ollama), vector store, knowledge graphs
• Agent Backend: Multi-provider LLM, Tera templates, streaming, variable substitution
• Prov-gen: Multi-cloud IaC, 7-layer validation, Nickel integration, AI-assisted generation
• Added configuration details for AI, Agent, and Prov-gen backends
• Updated development and production configuration examples
• Distribution structure updated to show 6 backends

README.md - Main Project Documentation

• Updated Features: "6 Backends" (was "3 Backends")

• Added AI Backend section (lines 95-113)

  • RAG system with semantic search
  • Multi-provider embeddings (OpenAI, Ollama)
  • Vector store (in-memory, Redis)
  • Knowledge graph generation

• Added Provisioning Generator section (lines 182-223)

  • Infrastructure as Code generation
  • Quick example with 3 command variants
  • 6 features: Multi-cloud, 7-layer validation, Nickel, AI-assist, templates, profiles
  • Supported providers table: AWS, GCP, Azure, Hetzner, UpCloud, LXD

• Updated Configuration structure: 6 backends listed

• Updated Project Structure: All 9 crates documented

• Updated Key Technologies: Organized into Core, Backends, AI & Agent, Infrastructure

• Updated Examples: Added 11-prov-gen/ and 12-agent-execution/ categories

docs/prov-gen/README.md

• Updated Multi-Provider Support: AWS, GCP, Azure, Hetzner, UpCloud, LXD (was AWS, GCP, Azure, Hetzner, LXD, bare metal)
• Changed "generates complete provisioning" → "generates provisioning" (modesty)

Changed - Naming Consistency

Package Renaming

• typeagent-core → typedialog-ag-core
• typeagent-cli → typedialog-ag
• typeagent-server → typedialog-ag-server
• config/agent/ → config/ag/

Rationale: Align with project-wide typedialog-* naming convention and binary names

Impact

• Updated 50+ files (imports, docs, scripts, configs)
• Updated workspace Cargo.toml dependencies
• Updated all crate manifests
• Updated justfile recipes
• Updated configuration directory structure
• 53 tests passing, 0 clippy warnings after refactor

Changed - Build & Distribution

Justfile Updates

• Added just build::agent - Build Agent CLI
• Added just build::agent-server - Build Agent server
• Added just build::prov-gen - Build Provisioning Generator
• Added just test::agent - Test Agent system (53 tests)
• Added just test::prov-gen - Test Provisioning Generator (11 tests)
• Updated just build::all to include Agent and Prov-gen builds
• Fixed just test::agent - Removed invalid lib test for binary-only typedialog-ag crate

Script Updates

• scripts/create_distro.sh - Include all binaries: typedialog-ag, typedialog-ag-server, typedialog-prov-gen
• scripts/check_deps.sh - New dependency verification script (67 lines)
• installers/bootstrap/install.sh - Install Agent and Prov-gen binaries
• installers/bootstrap/install.ps1 - Install Agent and Prov-gen binaries (Windows)
• .woodpecker/Dockerfile.cross - Cross-compile Agent and Prov-gen binaries

Build Features

• New feature flag FEATURES_AGENT := "markup,nickel,cache"
• Agent and Prov-gen builds integrated into full CI/CD pipeline
• All builds verified: just build::prov-gen ✓, just build::agent ✓
• All tests passing: just test::prov-gen ✓ (11 tests), just test::agent ✓ (53 tests, 5 ignored)

[Previous]

Added

RepeatingGroup Duplicate Detection

• Implemented duplicate item validation across all backends (CLI, TUI, Web)
• Added unique = true attribute support to prevent duplicate items
• Added item counter display updates in Web backend
• New is_duplicate() function in CLI and TUI backends

Web Backend Improvements

• Refactored JavaScript: Replaced global constants with closure-based state management
• Added render_global_repeating_group_script() for generic repeating group handling
• Configuration stored in data-* attributes: data-min-items, data-max-items, data-has-unique, data-unique-key, data-field-names
• Live item counter updates after add/delete operations
• Single document-level event delegation for all repeating group interactions

Documentation

• Updated docs/field_types.md with unique validation details
• Updated examples/README.md with RepeatingGroup testing commands
• Added docs/field_types.md (new file)
• Updated example forms with unique = true attribute

Fixed

• CLI Backend: Fixed validation of duplicate items in add/edit operations
• TUI Backend: Fixed validation of duplicate items with error overlay feedback
• Web Backend: Fixed has_unique flag reading from field definition (was incorrectly scanning fragment fields)
• Web Backend: Fixed item counter not updating when adding/deleting items

Changed

Architecture

• Each repeating group now maintains independent state via closure
• Removed ~600 lines of dead code in Web JavaScript
• Event handling now context-aware (finds correct repeating group controller)

Examples

• examples/05-fragments/array-trackers.toml: Added unique = true to UDP and HTTP tracker arrays
• examples/07-nickel-generation/arrays-form.toml: Added unique = true to all RepeatingGroup fields

Technical Notes

• Duplicate detection compares ALL field values when unique = true
• Works consistently across CLI, TUI, and Web backends
• Backwards compatible: repeating groups without unique = true unaffected
• max_items limit already enforced in all backends (no changes needed)

Testing

• All 174 unit tests passing
• No clippy warnings
• Build verified with --all-features flag
• Manual testing in Web backend: duplicate detection working correctly
• Item counter updates verified on add/delete operations