# Nushell Update Automation Guide **Version**: 1.0 **Date**: 2025-10-18 **Purpose**: Automate Nushell version updates for the plugins repository --- ## Overview This guide documents the semi-automated system for updating Nushell versions in the nushell-plugins repository. The system reduces manual work by ~80% while maintaining safety through strategic manual approval checkpoints. ### Design Philosophy - **Semi-Automated**: Automate repetitive tasks, require human approval for critical decisions - **Safe by Default**: Create backups, validate before proceeding, allow rollback - **Comprehensive**: Download, analyze, update, build, test, and document - **Idempotent**: Can be run multiple times safely - **Observable**: Clear logging and status reporting at every step --- ## Automation Architecture ### System Components ``` ┌─────────────────────────────────────────────────────────────┐ │ Update Orchestrator │ │ (update_nushell_version.nu) │ └─────────────────┬───────────────────────────────────────────┘ │ ┌─────────────┼─────────────┬──────────────┬──────────────┐ │ │ │ │ │ ▼ ▼ ▼ ▼ ▼ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌──────────┐ ┌─────────┐ │Download │ │Analyze │ │ Audit │ │ Detect │ │ Build │ │Nushell │ │Features │ │ Deps │ │Breaking │ │Nushell │ └─────────┘ └─────────┘ └─────────┘ └──────────┘ └─────────┘ │ │ │ │ │ └─────────────┴─────────────┴────────────┴──────────────┘ │ ▼ ┌──────────────────┐ │ Update Versions │ │ Test Plugins │ │Generate Report │ └──────────────────┘ ``` ### Script Hierarchy 1. **update_nushell_version.nu** (Main Orchestrator) - Entry point for all updates - Coordinates all other scripts - Implements manual approval checkpoints - Generates comprehensive reports 2. **download_nushell.nu** (Download Manager) - Downloads Nushell tarball from GitHub - Verifies extraction and workspace - Supports `--latest` flag 3. **analyze_nushell_features.nu** (Feature Analyzer) - Parses Cargo.toml for available features - Validates desired features exist - Shows dependency trees - Exports JSON analysis 4. **audit_crate_dependencies.nu** (Dependency Auditor) - Scans all plugin dependencies - Detects version mismatches - Shows dependency matrix - Identifies update requirements 5. **detect_breaking_changes.nu** (Breaking Change Detector) - Database of known breaking changes - Scans plugin code for affected patterns - Generates migration reports - Suggests fixes 6. **build_nushell.nu** (Build Manager) - Builds Nushell with selected features - Handles workspace builds - Manages build artifacts 7. **test_plugin_compatibility.nu** (Compatibility Tester) - Tests plugins against new Nushell version - Validates plugin registration - Checks command execution - Reports compatibility issues 8. **validate_code_rules.nu** (Code Validator) - Validates best_nushell_code.md against actual binary - Tests syntax rules - Verifies patterns work - Documents syntax changes --- ## Quick Start ### Basic Update ```bash # Update to specific version (recommended) ./scripts/update_nushell_version.nu 0.108.0 # Update to latest release ./scripts/update_nushell_version.nu --latest # Check current update status ./scripts/update_nushell_version.nu status ``` ### Advanced Options ```bash # Skip build step (faster for testing) ./scripts/update_nushell_version.nu 0.108.0 --skip-build # Auto-approve all steps (use with extreme caution!) ./scripts/update_nushell_version.nu 0.108.0 --auto-approve # Combination (testing workflow) ./scripts/update_nushell_version.nu 0.108.0 --skip-build --auto-approve ``` --- ## Detailed Workflow ### Step-by-Step Process #### Step 1: Download Nushell Source (Automated) ```nushell ./scripts/download_nushell.nu 0.108.0 --clean ``` **What It Does**: - Downloads tarball from GitHub tags - Extracts to `./nushell/` directory - Verifies workspace structure - Checks Cargo.toml version **Output**: ``` [INFO] Downloading Nushell 0.108.0 from GitHub... [INFO] Download URL: https://github.com/nushell/nushell/archive/refs/tags/0.108.0.tar.gz [SUCCESS] Downloaded: 15.2 MB [INFO] Extracting tarball... [SUCCESS] Extracted 43 crates [INFO] Verifying workspace... [SUCCESS] Nushell 0.108.0 ready at: ./nushell/ ``` **Options**: - `--latest`: Download latest release automatically - `--clean`: Remove existing nushell directory first - `--verify`: Verify checksum (future feature) #### Step 2: Analyze Features (Automated) ```nushell ./scripts/analyze_nushell_features.nu --validate --export ``` **What It Does**: - Parses nushell Cargo.toml features section - Validates desired features exist - Shows feature dependency tree - Exports analysis to JSON **Desired Features** (configurable in script): ```nushell const DESIRED_FEATURES = [ "mcp", # Model Context Protocol "plugin", # Plugin system "sqlite", # SQLite support "trash-support", # Trash bin support "system-clipboard" # System clipboard ] ``` **Output**: ``` [INFO] Analyzing Nushell version: 0.108.0 [INFO] Found 37 features === Desired Features Analysis === [SUCCESS] ✓ mcp Dependencies: none [SUCCESS] ✓ plugin Dependencies: nu-plugin, nu-plugin-engine [SUCCESS] ✓ sqlite Dependencies: nu-command/sqlite ... [SUCCESS] All desired features are valid! [SUCCESS] Feature analysis exported to: ./tmp/feature_analysis.json ``` **Subcommands**: ```bash # Show all available features ./scripts/analyze_nushell_features.nu --show-all # Show dependency tree for specific feature ./scripts/analyze_nushell_features.nu tree mcp # Generate build command ./scripts/analyze_nushell_features.nu build-cmd # Categorize features ./scripts/analyze_nushell_features.nu categorize ``` #### Step 3: Audit Dependencies (Automated) ```nushell ./scripts/audit_crate_dependencies.nu --export ``` **What It Does**: - Scans all plugin Cargo.toml files - Extracts nu-* dependencies - Compares versions with nushell core - Reports mismatches **Output**: ``` [INFO] Nushell Plugin Dependency Audit [INFO] Found 8 system plugins [INFO] Found 3 custom plugins === Dependency Audit Results === Total plugins: 11 Clean: 8 With issues: 3 --- System Plugins --- ✅ nu_plugin_query (v0.108.0) nu-* dependencies: 3 • nu-plugin: 0.108.0 • nu-protocol: 0.108.0 [plugin] • nu-engine: 0.108.0 ... --- Custom Plugins --- ⚠️ nu_plugin_image (v0.1.0) nu-* dependencies: 2 • nu-plugin: 0.107.1 (path: ../nushell/crates/nu-plugin) • nu-protocol: 0.107.1 (path: ../nushell/crates/nu-protocol) === Plugins with Version Issues === [ERROR] nu_plugin_image • nu-plugin: found 0.107.1, expected 0.108.0 • nu-protocol: found 0.107.1, expected 0.108.0 ``` **Subcommands**: ```bash # Audit specific plugin ./scripts/audit_crate_dependencies.nu --plugin nu_plugin_image # Audit only custom plugins ./scripts/audit_crate_dependencies.nu --custom-only # Show dependency matrix ./scripts/audit_crate_dependencies.nu matrix ``` #### Step 4: Detect Breaking Changes (Automated) ```nushell ./scripts/detect_breaking_changes.nu --scan-plugins --export ``` **What It Does**: - Maintains database of known breaking changes - Scans plugin source code for affected patterns - Reports potential issues - Suggests migration steps **Breaking Change Database**: ```nushell const BREAKING_CHANGES = { "0.108.0": [ { type: "command_rename" old_name: "into value" new_name: "detect type" description: "Command renamed and behavior changed" impact: "high" migration: "Replace 'into value' with 'detect type'" }, { type: "behavior_change" component: "stream_error_handling" description: "Collecting streams with errors now raises errors" impact: "medium" migration: "Add explicit error handling when collecting streams" } ] } ``` **Output**: ``` [INFO] Detecting breaking changes for 0.108.0... [INFO] Found 2 breaking changes in database [INFO] Scanning plugin source code... === Breaking Changes Report === Breaking Change #1: Command Rename Old: into value New: detect type Impact: high Affected Plugins: • nu_plugin_image: 2 occurrences in src/main.rs • nu_plugin_hashes: 1 occurrence in src/lib.rs Migration: Replace 'into value' with 'detect type' Breaking Change #2: Stream Error Handling Component: stream_error_handling Impact: medium Affected Plugins: • nu_plugin_image: 5 stream collections need error handling Migration: Add explicit error handling when collecting streams [WARN] Breaking changes detected! [SUCCESS] Report exported to: ./tmp/breaking_changes_report.json ``` #### Step 5: Manual Approval Checkpoint #1 ⚠️ The orchestrator pauses and displays: ``` ═══ MANUAL REVIEW REQUIRED ═══ Breaking changes detected. Please review the report above. Affected plugins: 2 • nu_plugin_image (3 issues) • nu_plugin_hashes (1 issue) Continue with update? (yes/no): ``` **What to Review**: - Breaking change impact on each plugin - Number of affected code locations - Migration complexity - Risk level **Decision**: - Type `yes` to continue with updates - Type `no` to abort and investigate manually #### Step 6: Update Plugin Versions (Semi-Automated) ```nushell ./scripts/update_nu_versions.nu update ``` **What It Does**: - Updates all plugin Cargo.toml files - Changes nu-* dependency versions - Preserves other dependencies - Creates backup before changes **Output**: ``` [INFO] Current Plugin Versions: Plugin | nu-plugin | nu-protocol | Status ------------------------|-----------|-------------|-------- nu_plugin_image | 0.107.1 | 0.107.1 | Needs update nu_plugin_hashes | 0.107.1 | 0.107.1 | Needs update nu_plugin_clipboard | 0.108.0 | 0.108.0 | OK ... Update all plugin versions? (yes/no): yes [INFO] Updating plugin versions to 0.108.0... [SUCCESS] Updated: nu_plugin_image/Cargo.toml [SUCCESS] Updated: nu_plugin_hashes/Cargo.toml [SUCCESS] Updated 8 plugins to version 0.108.0 ``` #### Step 7: Build Nushell (Automated, Optional) ```nushell ./scripts/build_nushell.nu ``` **What It Does**: - Builds Nushell with selected features - Builds workspace (includes system plugins) - Reports build time and artifacts **Build Command Generated**: ```bash cd nushell && cargo build --release --workspace \ --features "mcp,plugin,sqlite,trash-support,system-clipboard,rustls-tls" ``` **Output**: ``` [INFO] Building Nushell 0.108.0 with MCP features... [WARN] This will take 10-20 minutes... [INFO] Build started at: 2025-10-18 10:00:00 ... [Cargo build output] ... [SUCCESS] Build completed at: 2025-10-18 10:15:23 [INFO] Build time: 15m 23s Artifacts Generated: • nushell/target/release/nu (42.3 MB) • nushell/target/release/nu_plugin_* (8 plugins) ``` #### Step 8: Manual Approval Checkpoint #2 ⚠️ ``` ═══ BUILD REVIEW REQUIRED ═══ Please review build results above. Build status: Success Build time: 15m 23s Artifacts: 9 binaries Proceed with plugin compatibility tests? (yes/no): ``` **What to Review**: - Build completed successfully - No critical warnings - Expected artifacts present - Build time reasonable #### Step 9: Test Plugin Compatibility (Automated) ```nushell ./scripts/test_plugin_compatibility.nu ``` **What It Does**: - Registers each plugin with new nushell - Tests plugin commands - Verifies output format - Reports any failures **Output**: ``` [INFO] Testing plugin compatibility with Nushell 0.108.0... Testing: nu_plugin_clipboard [SUCCESS] Plugin registered [SUCCESS] Command 'clipboard' works [SUCCESS] Output format correct Testing: nu_plugin_image [SUCCESS] Plugin registered [WARN] Command 'to ansi' slow (2.3s) [SUCCESS] Output format correct ... === Compatibility Summary === Total plugins tested: 11 Passed: 10 Failed: 0 Warnings: 1 (performance) ``` #### Step 10: Generate Update Report (Automated) ```nushell # Generates comprehensive markdown report ``` **Output**: `./tmp/update_report_0.108.0.md` **Contents**: - Update summary - Files modified - Breaking changes encountered - Test results - Next steps - Links to detailed reports #### Step 11: Final Manual Approval ⚠️ ``` ═══ UPDATE COMPLETE ═══ Review the update report above. Next steps: 1. Review changes with: git status 2. Test the updated plugins 3. Commit changes when ready To create a commit: git add -A git commit -m "chore: update to Nushell 0.108.0" ``` --- ## Manual Approval Checkpoints ### Checkpoint #1: Breaking Changes Review **When**: After detecting breaking changes **Purpose**: Verify impact is acceptable **What to Check**: - [ ] Number of breaking changes manageable - [ ] Affected plugins are known - [ ] Migration path is clear - [ ] No unexpected critical changes **Response Options**: - `yes`: Continue with automatic updates - `no`: Stop to manually investigate ### Checkpoint #2: Build Review **When**: After building Nushell **Purpose**: Verify build succeeded properly **What to Check**: - [ ] Build completed without errors - [ ] No suspicious warnings - [ ] All expected binaries created - [ ] Build time is reasonable **Response Options**: - `yes`: Proceed with plugin tests - `no`: Skip plugin tests, review manually ### Checkpoint #3: Final Approval **When**: After all steps complete **Purpose**: Review before committing **What to Check**: - [ ] All tests passed - [ ] Documentation updated - [ ] Changes look correct in git status - [ ] Ready to commit **Response Options**: - Commit changes: `git add -A && git commit` - Rollback: `git restore .` - Manual fixes: Edit files as needed --- ## Configuration ### Script Configuration Files #### etc/plugin_registry.toml ```toml [plugins.nu_plugin_image] upstream = "https://github.com/fdncred/nu_plugin_image" status = "ok" auto_merge = false [plugins.nu_plugin_clipboard] upstream = "https://github.com/FMotalleb/nu_plugin_clipboard" status = "ok" auto_merge = true # Auto-merge nu-* dependency changes ``` #### scripts/lib/common_lib.nu ```nushell # Shared configuration for all scripts export const LOG_LEVEL = "INFO" export const NUSHELL_DIR = "./nushell" export const TMP_DIR = "./tmp" export const BACKUP_DIR = "./backups" # Feature configuration export const DESIRED_FEATURES = [ "mcp", "plugin", "sqlite", "trash-support", "system-clipboard" ] ``` ### Environment Variables ```bash # Override default directories export NUSHELL_SOURCE_DIR="/custom/path/nushell" export UPDATE_TMP_DIR="/custom/tmp" # Control logging export UPDATE_LOG_LEVEL="DEBUG" # DEBUG, INFO, WARN, ERROR # Skip checkpoints (dangerous!) export UPDATE_AUTO_APPROVE="false" ``` --- ## Error Handling ### Automatic Recovery The system implements automatic recovery for common errors: #### Download Failures ```nushell # Automatically retries with exponential backoff def download_with_retry [url: string]: nothing -> binary { mut attempts = 0 while $attempts < 3 { try { return (http get $url) } catch {|err| $attempts = $attempts + 1 log_warn $"Download failed, retry ($attempts)/3" sleep (2sec * $attempts) } } error make {msg: "Download failed after 3 attempts"} } ``` #### Build Failures ```nushell # Cleans and retries once try { cargo build --release } catch {|err| log_warn "Build failed, cleaning and retrying..." cargo clean cargo build --release } ``` #### Network Errors ```nushell # Falls back to git clone if tarball download fails try { download_tarball $version } catch { log_warn "Tarball download failed, trying git clone..." git clone --depth 1 --branch $version https://github.com/nushell/nushell } ``` ### Manual Intervention Some errors require manual intervention: #### Merge Conflicts ``` [ERROR] Merge conflict in nu_plugin_image/Cargo.toml [INFO] Manual resolution required: 1. Edit the file to resolve conflicts 2. Run: git add nu_plugin_image/Cargo.toml 3. Continue with: ./scripts/update_nushell_version.nu --continue ``` #### Missing Dependencies ``` [ERROR] Missing system dependency: libssl-dev [INFO] Install with: Ubuntu/Debian: sudo apt install libssl-dev macOS: brew install openssl Then retry the build ``` --- ## Logging and Debugging ### Log Levels ```nushell # Log functions in common_lib.nu export def log_debug [msg: string]: nothing -> nothing { if $env.LOG_LEVEL? == "DEBUG" { print $"[(ansi grey)(date now | format date '%H:%M:%S')(ansi reset)] [DEBUG] ($msg)" } } export def log_info [msg: string]: nothing -> nothing { print $"[(ansi blue)(date now | format date '%H:%M:%S')(ansi reset)] [INFO] ($msg)" } export def log_success [msg: string]: nothing -> nothing { print $"[(ansi green)(date now | format date '%H:%M:%S')(ansi reset)] [SUCCESS] ($msg)" } export def log_warn [msg: string]: nothing -> nothing { print $"[(ansi yellow)(date now | format date '%H:%M:%S')(ansi reset)] [WARN] ($msg)" } export def log_error [msg: string]: nothing -> nothing { print $"[(ansi red)(date now | format date '%H:%M:%S')(ansi reset)] [ERROR] ($msg)" } ``` ### Debug Mode ```bash # Enable debug logging export LOG_LEVEL=DEBUG ./scripts/update_nushell_version.nu 0.108.0 # Enable Nushell tracing RUST_LOG=debug ./scripts/update_nushell_version.nu 0.108.0 # Save complete log ./scripts/update_nushell_version.nu 0.108.0 2>&1 | tee update.log ``` ### Generated Reports All operations generate detailed reports: ``` ./tmp/ ├── feature_analysis.json # Feature analysis results ├── dependency_audit.json # Dependency audit results ├── breaking_changes_report.json # Breaking changes found ├── test_results.json # Plugin test results └── update_report_0.108.0.md # Comprehensive summary ``` --- ## Performance Optimization ### Parallel Execution ```nushell # Analyze and audit in parallel [ (do { ./scripts/analyze_nushell_features.nu --export }) (do { ./scripts/audit_crate_dependencies.nu --export }) ] | par-each {|cmd| $cmd } ``` ### Caching ```nushell # Cache GitHub API responses def get_latest_version_cached []: nothing -> string { let cache_file = "./tmp/latest_version.txt" let cache_age = (date now) - (ls $cache_file | get modified | first) if $cache_age < 1hr { open $cache_file } else { let version = get_latest_from_github $version | save -f $cache_file $version } } ``` ### Incremental Builds ```nushell # Only rebuild changed plugins def build_changed_plugins []: nothing -> list { git diff --name-only HEAD~1 | lines | where {|f| $f | str starts-with "nu_plugin_"} | each {|plugin| cd $plugin cargo build --release } } ``` --- ## Testing the Automation ### Unit Tests ```nushell # Test download script def test_download []: nothing -> nothing { ./scripts/download_nushell.nu 0.108.0 --clean assert (("./nushell/Cargo.toml" | path exists) == true) assert ((open ./nushell/Cargo.toml | get package.version) == "0.108.0") } # Test feature analysis def test_analyze []: nothing -> nothing { let result = (./scripts/analyze_nushell_features.nu --validate | complete) assert ($result.exit_code == 0) assert (("./tmp/feature_analysis.json" | path exists) == true) } ``` ### Integration Tests ```bash # Test full workflow with skip-build ./scripts/update_nushell_version.nu 0.108.0 --skip-build --auto-approve # Verify results test -f ./nushell/Cargo.toml test -f ./tmp/feature_analysis.json test -f ./tmp/dependency_audit.json test -f ./tmp/breaking_changes_report.json ``` ### Dry Run Mode ```bash # Future feature: test without making changes ./scripts/update_nushell_version.nu 0.108.0 --dry-run ``` --- ## Future Enhancements ### Planned Features 1. **Automated Rollback** ```nushell ./scripts/rollback_version.nu # Automatic rollback on failure ``` 2. **CI/CD Integration** ```yaml # .github/workflows/update-nushell.yml on: schedule: - cron: '0 0 * * 1' # Check weekly jobs: update: runs-on: ubuntu-latest steps: - run: ./scripts/update_nushell_version.nu --latest --auto-approve ``` 3. **Version Validation** ```nushell ./scripts/validate_code_rules.nu # Test rules against binary ``` 4. **Interactive Mode** ```nushell ./scripts/update_nushell_version.nu --interactive # Shows progress, allows step-by-step execution ``` 5. **Email Notifications** ```nushell # Notify on completion or errors send_email "Update to 0.108.0 completed" $report ``` ### Known Limitations 1. **Directory Naming**: Download script creates `nushell-X.Y.Z/` instead of `nushell/` - Workaround: Manual move or symlink - Fix planned: Next script version 2. **Private Repos**: SSH authentication not automated - Workaround: Set up SSH keys manually - Enhancement: Key management automation 3. **Windows Support**: Scripts tested on Unix-like systems - Workaround: Use WSL or Git Bash - Enhancement: Native Windows support --- ## Best Practices ### Do's ✅ 1. **Always Review Breaking Changes**: Don't auto-approve blindly 2. **Test Before Committing**: Run quality checks 3. **Keep Backups**: Git stash before major updates 4. **Read Generated Reports**: Contains important details 5. **Update Documentation**: Keep best_nushell_code.md in sync ### Don'ts ❌ 1. **Don't Use --auto-approve in Production**: Manual review is critical 2. **Don't Skip Testing**: Plugin tests catch issues 3. **Don't Ignore Warnings**: They often indicate real problems 4. **Don't Modify Scripts During Update**: Can cause inconsistencies 5. **Don't Delete tmp/ During Update**: Contains important state --- ## Troubleshooting ### Common Issues **Issue**: Download fails with network error ```bash # Solution: Use git clone fallback cd nushell && git fetch --tags && git checkout 0.108.0 ``` **Issue**: Build takes too long ```bash # Solution: Use --skip-build for faster iteration ./scripts/update_nushell_version.nu 0.108.0 --skip-build ``` **Issue**: Version mismatch after update ```bash # Solution: Run update_nu_versions.nu again ./scripts/update_nu_versions.nu update ``` **Issue**: Plugin test failures ```bash # Solution: Test manually with new nushell ./nushell/target/release/nu > plugin add target/release/nu_plugin_NAME > plugin list ``` --- ## Appendix ### Script Reference | Script | Purpose | Time | Required | |--------|---------|------|----------| | update_nushell_version.nu | Main orchestrator | 3-4h | Yes | | download_nushell.nu | Download source | 2m | Yes | | analyze_nushell_features.nu | Analyze features | 1m | Yes | | audit_crate_dependencies.nu | Audit deps | 1m | Yes | | detect_breaking_changes.nu | Find changes | 1m | Yes | | build_nushell.nu | Build nushell | 15m | Optional | | test_plugin_compatibility.nu | Test plugins | 5m | Optional | | validate_code_rules.nu | Validate docs | 2m | Optional | ### File Locations ``` project-root/ ├── scripts/ │ ├── update_nushell_version.nu # Main orchestrator │ ├── download_nushell.nu # Download manager │ ├── analyze_nushell_features.nu # Feature analyzer │ ├── audit_crate_dependencies.nu # Dependency auditor │ ├── detect_breaking_changes.nu # Change detector │ ├── build_nushell.nu # Build manager │ ├── test_plugin_compatibility.nu # Compatibility tester │ ├── validate_code_rules.nu # Code validator │ └── lib/ │ └── common_lib.nu # Shared utilities ├── etc/ │ └── plugin_registry.toml # Plugin configuration ├── tmp/ # Generated reports │ ├── feature_analysis.json │ ├── dependency_audit.json │ ├── breaking_changes_report.json │ └── update_report_*.md └── nushell/ # Downloaded nushell source ``` --- **Automation Guide Version**: 1.0 **Last Updated**: 2025-10-18 **Maintained By**: Nushell Plugins Team