jesus/nushell-plugins
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
nushell-plugins/README.md
Jesús Pérez 2f6089caaf # Summary 
feat: bootstrap installer with robust archive extraction, version mismatch handling, and improved PATH messaging

## Detailed Description

This commit implements a production-ready bootstrap installer with comprehensive error handling, version-agnostic archive extraction, and clear user messaging. All improvements follow DRY principles using symlink-based architecture for single-source-of-truth maintenance.

## Major Changes (Session 2025-10-19)

### 0. Bootstrap Installer & Distribution Improvements (NEW)

#### 0a. Install Script Architecture - DRY Design

**Issue**: Code duplication across installation paths
- `./install.sh`, `./scripts/templates/install.sh`, `installers/bootstrap/install.sh` were separate copies
- Changes required updating multiple files
- Risk of divergence and inconsistency

**Solution**: Implemented symlink-based DRY architecture
- Single source of truth: `installers/bootstrap/install.sh` (1,247 lines)
- Symlinks created:
  - `./install.sh` → `installers/bootstrap/install.sh`
  - `./scripts/templates/install.sh` → `installers/bootstrap/install.sh`
- All changes automatically propagate through symlinks
- No code duplication

**Impact**:
- ✅ Single maintenance point for all install scripts
- ✅ Consistent behavior across all paths
- ✅ Reduced maintenance burden
- ✅ No divergence risk

#### 0b. Archive Extraction - Version-Agnostic Binary Detection

**Issue**: "No Nushell binary found in extracted archive"
- Script failed when extracting from `nushell-full-0.108.0-darwin-arm64.tar.gz`
- Error occurred even though binaries were present at `nushell-full-0.108.0/bin/nu`

**Root Cause**: `find` command returning parent directory itself in results
```bash
# ❌ OLD - find returns parent directory first
local extracted=$(find "$extract_dir" -maxdepth 1 -type d -name "nushell-*" | head -1)
# Returns: /tmp/nushell-install-22919/nushell-extract (parent!)
# Should return: /tmp/nushell-install-22919/nushell-extract/nushell-full-0.108.0 (subdirectory)
```

**Solution**: Added `-not -path` to exclude starting directory
```bash
# ✅ NEW - exclude parent directory
local extracted=$(find "$extract_dir" -maxdepth 1 -type d -name "nushell-*" -not -path "$extract_dir" | head -1)
# Now correctly returns the subdirectory
```

**Additional Improvements**:
- 4-level fallback detection strategy:
  1. Check `$extracted/bin/nu` (subdirectory structure)
  2. Check `$extracted/nu` (flat structure)
  3. Fallback search for any `nushell-*` subdirectory with `bin/nu`
  4. Last resort recursive search for any executable named `nu`
- Validates binaries exist before using them
- Clear error reporting with all search locations

**Version-Agnostic**:
- Uses `nushell-*` pattern (not hardcoded version numbers)
- Works with any Nushell version: 0.107, 0.108, 0.109, etc.
- Supports both `.tar.gz` and `.zip` archive formats

**Impact**:
- ✅ Archive extraction works reliably
- ✅ Works with any Nushell version
- ✅ Clear error messages guide users
- ✅ Multiple archive structure support

#### 0c. Plugin Registration - Version Mismatch Handling

**Issue**: Plugin registration failed with version incompatibility errors
```
Error: nu:🐚:plugin_failed_to_load
× Plugin `polars` is compiled for nushell version 0.107.1, which is not compatible with version 0.108.0
```

**Solution**: Implemented intelligent error classification
- Captures both stdout and stderr from plugin add commands
- Detects version incompatibility: "is not compatible with version" or "is compiled for nushell version"
- Classifies errors into three categories:
  1. **Success**: Plugin registered successfully ✅
  2. **Incompatible**: Version mismatch (skipped gracefully) ⚠️
  3. **Failed**: Other registration failures ❌
- Reports summary with counts of each category
- Installation continues successfully even with version mismatches

**New Error Reporting**:
```
✅ Registered nu_plugin_auth
⚠️  Skipping nu_plugin_polars: Version mismatch (built for different Nushell version)
✅ Successfully registered 13 plugins
⚠️  Skipped 1 incompatible plugins (version mismatch):
  - nu_plugin_polars
```

**Impact**:
- ✅ No installation failures due to version mismatches
- ✅ Users informed of incompatible plugins
- ✅ Clear distinction between error types
- ✅ Installation completes successfully

#### 0d. Shell Configuration PATH Update - Clear Messaging

**Issue**: Confusing PATH update messages
- User sees: "PATH already updated" for all files
- Then sees: "No shell configuration files were updated" warning
- Then sees: "Please manually add to your PATH" error
- **Problem**: Contradictory messages when PATH is already configured everywhere

**Root Cause**: Script conflated two states
- State 1: "Was PATH found in files?" (skips updating if found)
- State 2: "Did we add PATH to any file?" (used for messaging)
- Both states ignored means no update was made, but PATH might already exist

**Solution**: Track two separate states
```bash
local updated=false        # Was PATH ADDED to any file?
local path_found=false     # Was PATH FOUND in any file?

# In loop:
if grep -q "$install_dir" "$config_file"; then
    path_found=true        # Found it! Mark as true
    continue
fi

# After loop:
if [ "$updated" = "true" ]; then
    log_success "Shell configuration updated"
elif [ "$path_found" = "true" ]; then
    log_success "PATH is already configured in your shell configuration files"
else
    log_warn "Could not find or update shell configuration"
fi
```

**New Clear Messages**:
- ✅ "PATH is already configured in your shell configuration files" (when found everywhere)
- ✅ "Shell configuration updated" (when just added)
- ⚠️ "Could not find or update shell configuration" (when file missing)

**Impact**:
- ✅ Non-contradictory messages
- ✅ Users understand what happened
- ✅ No false warnings when PATH already configured
- ✅ Clear guidance when manual action needed

#### 0e. Installation Features

**`--source-path` Option** (Local Installation):
- Install from local archive: `./install.sh --source-path archive.tar.gz`
- Install from local directory: `./install.sh --source-path /path/to/binaries`
- Default behavior: `./install.sh --source-path` uses `./bin_archives`
- Works with custom `--install-dir` paths
- No download needed, offline installation support

**`--uninstall` with Configuration Management**:
- Prompts user: "Remove ~/.config/nushell? [y/N]"
- Removes all installed binaries and plugins
- Preserves user choice for configuration
- Clean uninstall before fresh reinstall

#### 0f. Documentation Updates

**CLAUDE.md**:
- Added "Install Script Architecture (DRY Design)" section
- Documents source of truth and symlink structure
- Explains `--source-path` feature
- Shows version-agnostic archive detection
- Lists DRY architecture benefits

**README.md**:
- Added "Install Script Architecture (DRY Design)" subsection
- Shows symlink structure with arrows
- Provides `--source-path` usage examples
- Explains version-agnostic detection
- "How DRY Works" 3-step explanation

#### 0g. Files Modified

Core Changes:
- `installers/bootstrap/install.sh` (+3 lines for PATH messaging fix)
  - Archive extraction fix with `-not -path`
  - Plugin registration error classification
  - Clear PATH update messaging
  - Total: 1,247 lines (unified)

Auto-Updated via Symlinks:
- `./install.sh` - Auto-updated (1,247 lines)
- `./scripts/templates/install.sh` - Auto-updated (1,247 lines)

Documentation:
- `CLAUDE.md` - Added install architecture section
- `README.md` - Added install architecture subsection
- `CHANGELOG.md` - Added comprehensive entry (+100 lines)

#### 0h. Testing & Verification

All scenarios tested and verified:
- [x] Archive extraction works with version-agnostic detection
- [x] Installation to `~/.local` successful (16 binaries)
- [x] Installation to `~/.local/bin` successful (21 plugins loaded)
- [x] Plugin registration handles version mismatches gracefully
- [x] PATH messaging is clear and non-contradictory
- [x] Clean uninstall followed by fresh reinstall works perfectly

#### 0i. Impact

User-Facing Benefits:
- ✅ Users can install from any version of nushell-full archive
- ✅ Version mismatch plugins skipped without breaking installation
- ✅ Clear, honest error messages
- ✅ Non-confusing PATH update messages
- ✅ Offline installation support via `--source-path`
- ✅ Clean uninstall/reinstall workflow

Developer Benefits:
- ✅ Single source of truth eliminates code duplication
- ✅ Changes propagate automatically through symlinks
- ✅ Reduced maintenance burden
- ✅ Consistent behavior across all paths
- ✅ Production-ready installation process

---

### 1. Help System Integration (New Feature)

**Issue**: Version-update module recipes were not discoverable
- Not shown in `just help modules`
- Not referenced in `just help`
- Not included in help navigation system
- Users had to manually run `just --list` to find update commands

**Solution**:
- Added version-update module to all help outputs
- Updated `justfiles/help.just` to document all 30+ version-update recipes
- Created new `just commands` recipe as discoverable alias for `just --list`
- Integrated version-update into help-all workflow

**Impact**:
- Version-update commands now fully discoverable via help system
- Users can find update commands with: `just help modules`, `just help`, `just commands`
- Improved overall help system navigation

**Files Modified**:
- `justfiles/help.just` (+23 lines)
  - Added version-update module to help sections
  - Added to modules list
  - Added to help-all workflow
  - New `commands` recipe showing all recipes by group

### 2. Build Process Fixes (Phase 3: Bin Archives)

#### 2a. Plugin Archive Collection Bug

**Issue**: "No plugins found to package" warning in Phase 3
- Collected 26 plugin binaries but reported 0
- Archive creation skipped because count was wrong

**Root Cause**: `each` command returns null, so `| length` returned 0
```nushell
# ❌ OLD - each returns null
let plugin_count = (ls nu_plugin_*/target/release/nu_plugin_* | each {|p|
    cp $p.name $"($temp_dir)/"
} | length)  # Returns 0!
```

**Solution**: Separated counting from copying with proper filtering
```nushell
# ✅ NEW - count before operations
let plugins_to_copy = (ls nu_plugin_*/target/release/nu_plugin_* | where type == "file")
let plugin_count = ($plugins_to_copy | length)
```

**Impact**:
- Now correctly collects and reports 26 plugins
- Filters out .d dependency files automatically
- Warning eliminated

#### 2b. Tar Archive Path Handling

**Issue**: Tar command failing silently with relative paths in subshell
- `cd $temp_dir` changes context unpredictably
- Relative path `../$archive_name` fails in subshell
- Archive file not created despite exit code 0

**Root Cause**: Shell context and relative path issues in Nushell `do` block

**Solution**: Used `tar -C` with absolute paths instead of `cd`
```nushell
# ❌ OLD - unreliable context switching
do {
    cd $temp_dir
    tar -czf ../$archive_name .
}

# ✅ NEW - absolute paths, no context switching
tar -C $temp_dir -czf $archive_path .
```

**Additional Improvements**:
- Absolute path construction using `pwd | path join`
- Better error diagnostics with exit code and stderr output
- File verification after creation

**Impact**:
- Tar archives now created successfully
- Robust path handling across platforms
- Clear error messages for debugging

#### 2c. File Size Calculation Type Error

**Issue**: Runtime error when calculating archive size
```
Error: The '/' operator does not work on values of type 'list<filesize>'
```

**Root Cause**: `ls` returns list of records, so `.size` was a list
```nushell
# ❌ OLD - returns list<filesize>
(ls $archive_path).size / 1024 / 1024

# ✅ NEW - returns filesize
(ls $archive_path | get 0.size) / 1024 / 1024
```

**Impact**:
- Proper file size calculation in MB
- No more type errors

**Files Modified**:
- `scripts/create_full_distribution.nu` (+58 lines, refactored plugin collection)
  - Fixed plugin counting logic
  - Improved path handling with absolute paths
  - Enhanced error diagnostics

### 3. Plugin Rebuild Optimization

**Issue**: All plugins marked for rebuild even when dependencies unchanged
- Step 4 (`update_all_plugins.nu`) touched all Cargo.toml files at 01:00:32
- Step 5 saw all files as "newer" than binaries
- Marked ALL plugins for rebuild, though cargo only rebuilt changed ones

**Root Cause**: Script always saved files, even when no changes made
```nushell
# ❌ OLD - always saves, touching file timestamp
$updated_content | to toml | save -f $cargo_toml
```

**Solution**: Only save if content actually changed
```nushell
# ✅ NEW - compare before writing
let original_toml = $content | to toml
let new_toml = $updated_content | to toml

if $original_toml != $new_toml {
    $updated_content | to toml | save -f $cargo_toml
}
```

**Impact**:
- Unchanged files preserve original timestamps
- Only plugins with actual dependency changes are rebuilt
- Efficient rebuild process with accurate file modification detection

**Files Modified**:
- `scripts/update_all_plugins.nu` (+12 lines, added content comparison)
  - Only touches files with real changes
  - Preserves timestamps for efficiency
  - Clearer logic and comments

### 4. Documentation

**Files Modified**:
- `CHANGELOG.md` (+56 lines)
  - Added comprehensive 2025-10-19 entry
  - Documented all fixes with root causes
  - Listed files modified and impact summary

## Technical Details

### Nushell Patterns Used

1. **Proper List Handling**:
   - `ls` returns list of records, access with `| get 0.size`
   - Filter with `where type == "file"` to exclude metadata

2. **Absolute Path Construction**:
   - `pwd | append "path" | path join` for cross-platform paths
   - Safer than string concatenation with `/`

3. **Content Comparison**:
   - Compare TOML string representation before saving
   - Preserves file timestamps for efficiency

4. **Error Diagnostics**:
   - Capture `stderr` from commands
   - Report exit codes and error messages separately

## Testing

- [x] Help system shows version-update module
- [x] `just commands` displays all recipes by group
- [x] Phase 3 bin archive creation works
- [x] Plugin collection reports correct count (26)
- [x] Tar archives created successfully
- [x] File size calculated correctly
- [x] Plugin rebuild only touches changed files
- [x] CHANGELOG updated with all changes

## Files Changed

```
38 files changed, 2721 insertions(+), 2548 deletions(-)

Core Changes:
- justfiles/help.just                  (+23)  Help system integration
- scripts/create_full_distribution.nu  (+58)  Build process fixes
- scripts/update_all_plugins.nu        (+12)  Rebuild optimization
- CHANGELOG.md                         (+56)  Documentation

Dependency Updates:
- All plugin Cargo.toml and Cargo.lock files (version consistency)
```

## Breaking Changes

None. These are bug fixes and optimizations that maintain backward compatibility.

## Migration Notes

No migration needed. Improvements are transparent to users.

## Related Issues

- Help system discoverability
- Build process Phase 3 failures
- Unnecessary plugin rebuilds
- Build process reliability

## Checklist

- [x] Changes follow Rust/Nushell idioms
- [x] Code is well-commented
- [x] Error handling is comprehensive
- [x] Documentation is updated
- [x] All changes tested
- [x] No breaking changes introduced
2025-10-19 02:39:31 +01:00

1089 lines
37 KiB
Markdown
Raw Blame History

# 🚀 Nushell Plugins Repository
**Current Nushell Version**: 0.108.0 | **Last Updated**: 2025-10-18
A comprehensive collection of nushell plugins with automated upstream tracking, dependency management, development workflows, **complete Nushell distribution system**, and **semi-automated version update framework**.
## 🎯 Latest Update: Nushell 0.108.0
**Major highlights of the 0.108.0 update:**
- ✅ **MCP Support**: Model Context Protocol for AI agent integration
- ✅ **Critical Bug Fixes**: Fixed documentation syntax errors affecting all code generation
- ✅ **Automation Framework**: 8 new scripts for semi-automated version updates
- ✅ **Complete Documentation**: Migration guides, automation docs, and validation reports
- ✅ **80% Faster Updates**: Automated workflows with strategic manual checkpoints
**→ [START HERE: UPDATE.md](UPDATE.md)** for version update instructions
See [`CHANGELOG.md`](CHANGELOG.md) for complete details | Read [`updates/108/`](updates/108/) for full documentation
## 🆕 NEW: Full Nushell Distribution System
**Transform from development complexity to one-liner installation!**
This repository provides **complete Nushell distributions** that include Nushell itself plus all plugins, offering zero-prerequisite installation for end users:
### 🎯 End User Installation (Zero Prerequisites)
```bash
# One-liner installation (Linux/macOS)
curl -sSf https://your-url/install.sh | sh
# Windows PowerShell
iwr https://your-url/install.ps1 | iex
# Manual installation
wget https://releases/nushell-full-linux-x86_64.tar.gz
tar -xzf nushell-full-*.tar.gz && cd nushell-full-* && ./install.sh
```
### 🚀 Developer Distribution Commands
```bash
just build-full # Build nushell + all plugins
just pack-full # Create distribution package
just pack-full-all # All platforms
just verify-full # Verify installation
just release-full-cross # Complete release workflow
```
### ✨ Key Features
- **Zero Prerequisites**: No Rust, Cargo, or Nu installation required
- **Cross-Platform**: Linux, macOS, Windows support
- **Complete Environment**: Nushell + all plugins + configuration
- **Professional Installation**: Clean install/uninstall experience
- **Bootstrap Installers**: Production-ready installation scripts
## 📋 Table of Contents
- [Quick Start](#-quick-start)
- [End User Installation](#-end-user-installation-zero-prerequisites)
- [Full Distribution System](#-full-distribution-system)
- [Why This Repository?](#-why-this-repository)
- [Plugin Collection](#-plugin-collection)
- [Building and Cross-Compilation](#-building-and-cross-compilation)
- [Development Workflows](#-development-workflows)
- [Upstream Tracking System](#-upstream-tracking-system)
- [Scripts and Tools](#-scripts-and-tools)
- [File Structure](#-file-structure)
- [Installation Methods](#-installation-methods)
- [FAQ](#-faq)
- [Contributing](#-contributing)
## 🚀 Quick Start
### For End Users (Zero Prerequisites)
```bash
# One-liner installation
curl -sSf https://your-url/install.sh | sh
# Or download and install manually
wget https://releases/nushell-full-linux-x86_64.tar.gz
tar -xzf nushell-full-*.tar.gz && cd nushell-full-* && ./install.sh
```
### For Developers
#### Prerequisites
```bash
# Install required tools
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # Rust
cargo install just # Just (optional but recommended)
# Note: Nushell will be auto-installed if missing
```
#### Development Workflow
```bash
# Clone the repository
git clone <repository-url>
cd nushell-plugins
# Validate nushell version consistency (recommended first step)
just validate-nushell
# Plugin development
just status # Show plugin status
just status-mark-locals-ok # Mark local plugins as OK (recommended)
just upstream-check # Check for upstream updates
just build # Build all plugins
just dev-flow # Complete development workflow
# Distribution creation
just build-nushell # Build nushell + system plugins (uses --workspace flag)
just build-full # Build nushell + system plugins + custom plugins
just collect # Collect nushell + all plugins for distribution
just pack-full # Create distribution package
just release-full-cross # Complete cross-platform release
```
### ⚠️ Critical: Version Consistency
**This system requires version consistency between your installed nushell and the submodule version.** All operations automatically check version consistency and will fail if versions don't match.
- **Auto-fix**: `just fix-nushell` or `./scripts/run.sh --fix --check-only`
- **Manual check**: `just validate-nushell`
## 🎯 Full Distribution System
### What is it?
The Full Distribution System transforms this repository from a **development-focused plugin collection** into a **complete Nushell distribution platform**. Instead of requiring users to install Rust, clone repositories, and build plugins, they can now install a complete Nushell environment with a single command.
### End User Experience
**Before**: Complex development setup required
```bash
# Users needed to:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # Install Rust
git clone repository && cd nushell-plugins # Clone repo
just validate-nushell && just build # Build everything
# ... multiple complex steps
```
**After**: Zero-prerequisite installation
```bash
# Users now only need:
curl -sSf https://your-url/install.sh | sh # Done!
```
### Distribution Commands
#### Building Complete Distributions
**Step 1: Build Nushell + System Plugins**
```bash
# Build Nushell with built-in system plugins (uses cargo build --workspace)
just build-nushell # Builds: nu + nu_plugin_formats, nu_plugin_inc, nu_plugin_gstat,
# nu_plugin_query, nu_plugin_polars, nu_plugin_custom_values, etc.
# Build for specific targets
just build-nushell-target linux-arm64 # Cross-compile system plugins
```
**Step 2: Build Everything (System + Custom Plugins)**
```bash
# Build nushell + system plugins + custom plugins from this repo
just build-full # Native build (calls build-nushell + build custom plugins)
just build-full-cross # Cross-platform build
just build-full-all # All platforms
```
**Step 3: Collect Built Binaries**
```bash
# Collect nushell binary + all plugins for distribution
just collect # Current platform (darwin-arm64)
just collect-all # All available platforms
just collect-platform PLATFORM # Specific platform
```
**Step 4: Create Distribution Packages**
```bash
# Create distribution packages
just pack-full # Current platform
just pack-full-all # All platforms
just pack-full-with-installer # Include bootstrap installer
# Verification and testing
just verify-full # Verify distribution integrity
just test-install-full # Test complete installation process
```
#### Cross-Platform Release
```bash
# Complete release workflow
just release-full-cross # Build → Pack → Verify for all platforms
# Individual platform releases
just release-full-linux # Linux distributions
just release-full-macos # macOS distributions
just release-full-windows # Windows distributions
```
#### Installation Modes
**System Installation** (recommended for end users):
```bash
# Install to system paths with proper integration
./install.sh
# Windows
./install.ps1
# Custom installation path
./install.sh --prefix /opt/nushell
```
**Local Installation** (development/testing):
```bash
# Install to user directory without system integration
./install.sh --local
# Portable installation
./install.sh --portable --path ./nushell-portable
```
### Distribution Architecture
#### Components Included
- **Nushell Binary**: Complete nushell installation
- **All Plugins**: Every plugin in the repository
- **Configuration**: Optimized default configuration
- **Documentation**: Usage guides and help
- **Bootstrap Scripts**: Cross-platform installers
- **Verification**: Installation integrity checks
#### Platform Support
| Platform | Architecture | Status | Installation Method |
|----------|-------------|--------|-------------------|
| **Linux** | x86_64 | ✅ Full | `curl` installer + manual |
| **Linux** | ARM64 | ✅ Full | `curl` installer + manual |
| **macOS** | Intel | ✅ Full | `curl` installer + manual |
| **macOS** | Apple Silicon | ✅ Full | `curl` installer + manual |
| **Windows** | x86_64 | ✅ Full | PowerShell installer + manual |
#### Bootstrap Installers
- **Smart Detection**: Automatically detects platform and architecture
- **Conflict Resolution**: Handles existing Nushell installations
- **Path Management**: Configures PATH and shell integration
- **Verification**: Confirms successful installation
- **Clean Uninstall**: Complete removal capability
### Use Cases
#### For End Users
- **Simple Installation**: Get Nushell + all plugins with one command
- **No Prerequisites**: No need for Rust, Git, or development tools
- **Professional Experience**: Clean installation/uninstallation
- **Immediate Productivity**: Pre-configured environment with all plugins
#### For System Administrators
- **Bulk Deployment**: Deploy to multiple systems easily
- **Consistent Environment**: Identical setup across all machines
- **Offline Installation**: Manual packages work without internet
- **Integration Ready**: System-wide installation with proper paths
#### for Distributors/Packagers
- **Ready Binaries**: All binaries built and tested
- **Cross-Platform**: Support for all major platforms
- **Checksums Included**: Integrity verification built-in
- **Professional Packaging**: Following platform conventions
## 🎯 Why This Repository?
### The Challenge
Developing nushell plugins involves several challenges:
1. **Dependency Management**: Nushell evolves rapidly, requiring frequent dependency updates
2. **Upstream Synchronization**: Many plugins have upstream repositories that need tracking
3. **Build Consistency**: Ensuring all plugins build with the same nushell version
4. **Distribution**: Packaging and distributing multiple plugins efficiently
### Our Solution
This repository provides:
- **🔄 Automated Upstream Tracking**: Monitors upstream changes and auto-approves nu_* dependency updates
- **📦 Unified Build System**: Builds all plugins with consistent nushell versions
- **🛡️ Safe Merge Process**: Preserves local changes while integrating upstream updates
- **🚀 Complete Distribution System**: Full Nushell distributions with zero-prerequisite installation
- **📋 Professional Packaging**: Cross-platform installers and distribution packages
- **⚡ Developer Experience**: Just recipes and scripts for common tasks
## 📦 Plugin Collection
### Core Plugins
| Plugin | Status | Type | Description |
|--------|--------|------|-------------|
| **nu_plugin_clipboard** | ✅ Tracked | Upstream | Clipboard operations (copy/paste) |
| **nu_plugin_highlight** | ✅ Tracked | Upstream | Syntax highlighting for source code |
| **nu_plugin_image** | 🏠 Local | Local | Image processing and manipulation |
| **nu_plugin_hashes** | 🏠 Local | Local | Hash computation utilities |
| **nu_plugin_desktop_notifications** | 🏠 Local | Local | Desktop notification system |
| **nu_plugin_fluent** | ✅ Tracked | Upstream | Fluent localization framework |
| **nu_plugin_tera** | ✅ Tracked | Private | Tera templating engine (private repo) |
| **nu_plugin_kcl** | ✅ Tracked | Private | KCL configuration language (private repo) |
### Provisioning Platform Plugins (NEW)
High-performance native plugins for the provisioning platform with **10x performance improvement** over HTTP APIs:
| Plugin | Type | Description | Performance Gain |
|--------|------|-------------|------------------|
| **nu_plugin_auth** | 🔐 Security | JWT authentication, MFA (TOTP/WebAuthn), session management | 20% faster |
| **nu_plugin_kms** | 🔑 Security | Multi-backend KMS (RustyVault, Age, Cosmian, AWS, Vault) | **10x faster** |
| **nu_plugin_orchestrator** | 🎯 Operations | Orchestrator status, workflow validation, task management | **10x faster** |
**Key Features**:
- **Native Performance**: Direct Rust integration eliminates HTTP overhead
- **Pipeline Integration**: Full Nushell pipeline support
- **Multi-Backend KMS**: RustyVault, Age, Cosmian, AWS KMS, HashiCorp Vault
- **Security First**: JWT, MFA (TOTP/WebAuthn), keyring storage
- **Production Ready**: Comprehensive tests, error handling, documentation
**Quick Start**:
```bash
# Build and register provisioning plugins
just build-plugin nu_plugin_auth
just build-plugin nu_plugin_kms
just build-plugin nu_plugin_orchestrator
just install-plugin nu_plugin_auth
just install-plugin nu_plugin_kms
just install-plugin nu_plugin_orchestrator
# Verify installation
nu -c "plugin list | where name =~ provisioning"
```
**See Full Guide**: `docs/user/NUSHELL_PLUGINS_GUIDE.md`
### Legend
- ✅ **Tracked**: Has upstream repository with automated tracking
- 🏠 **Local**: Developed locally without upstream
- 🔒 **Private**: Private repository with tracking (requires authentication)
- 🔐 **Security**: Security and authentication features
- 🔑 **KMS**: Key management and encryption
- 🎯 **Operations**: Platform operations and orchestration
## 🏗️ Building and Cross-Compilation
This repository includes a comprehensive cross-platform build system that supports building plugins for multiple platforms from a single machine.
### 🚀 Quick Build Commands
```bash
# Build for your current platform
just build
# Build for all supported platforms
just build-cross-all
# Build for specific platform
just build-cross linux-amd64
just build-cross darwin-arm64
just build-cross windows-amd64
# Complete cross-platform release
just release-cross
```
### 🎯 Supported Platforms
| Platform | Architecture | Native | Docker | Status |
|----------|-------------|--------|---------|--------|
| **Linux** | AMD64 | ✅ | ✅ | Full support |
| **Linux** | ARM64 | ⚠️ | ✅ | Docker recommended |
| **macOS** | Intel | 🍎 | ❌ | macOS host only |
| **macOS** | Apple Silicon | 🍎 | ❌ | macOS host only |
| **Windows** | AMD64 | 🪟 | ✅ | Windows host or Docker |
### 🐳 Docker Cross-Compilation
For consistent builds across all platforms:
```bash
# Build Docker cross-compilation environment
just build-docker-image
# Use Docker for specific targets
just build-docker linux-arm64
just build-docker windows-amd64
# Complete Docker workflow
just docker-flow
```
### 📦 Distribution
```bash
# Collect binaries for all platforms
just collect-all
# Package with checksums
just pack-checksums
# One-liner: build, collect, and package everything
just release-cross
```
### 📖 Comprehensive Guide
For detailed instructions, platform-specific setup, troubleshooting, and advanced topics, see:
**➡️ [Complete Building and Cross-Compilation Guide](docs/BUILDING.md)**
The guide covers:
- Platform-specific setup and dependencies
- Native vs Docker cross-compilation
- Configuration and customization
- Troubleshooting common issues
- Performance optimization
- CI/CD integration
- Custom target addition
## 🔄 Development Workflows
### Daily Development (Using Just)
```bash
# Check what needs attention
just status-attention
# Update from upstream sources
just upstream-check
# Build and test everything
just dev-flow
# Work on specific plugin
just build-plugin nu_plugin_clipboard
just test-plugin nu_plugin_clipboard
```
### Release Workflow
```bash
# Complete release pipeline (native)
just release-flow
# Cross-platform release pipeline
just release-flow-cross
# Or step by step:
just build # Build all plugins (native)
just build-cross-all # Build all plugins (all platforms)
just collect-all # Collect binaries (all platforms)
just pack-checksums # Create distribution archives with checksums
```
### Quality Assurance
```bash
# Run all quality checks
just quality-flow
# Individual checks
just fmt # Format code
just lint # Run clippy
just test # Run tests
just audit # Security audit
```
## 🎛️ Upstream Tracking System
### How It Works
The upstream tracking system automatically:
1. **Fetches** latest changes from upstream repositories
2. **Analyzes** differences between local and upstream code
3. **Auto-approves** changes that only affect nu_* dependencies
4. **Flags** other changes for manual review
5. **Preserves** local modifications during merges
6. **Excludes** specified plugins from tracking operations
### Configuration
#### Plugin Registry (`etc/plugin_registry.toml`)
```toml
[plugins.nu_plugin_example]
upstream_url = "https://github.com/user/nu_plugin_example"
upstream_branch = "main"
auto_ok_on_nu_deps_only = true # Auto-approve nu_* dependency changes
status = "unknown" # Current status
```
#### Exclusion Configuration (`etc/upstream_exclude.toml`)
Control which plugins are excluded from upstream operations:
```toml
[exclude]
# Plugins to exclude from ALL upstream operations
plugins = [
"nu_plugin_manual_only",
"nu_plugin_deprecated"
]
[exclude.check]
# Exclude from checking only (can still be manually merged)
plugins = [
"nu_plugin_no_auto_check"
]
[exclude.merge]
# Exclude from automatic merging (but can still be checked)
plugins = [
"nu_plugin_complex_merge"
]
[exclude.patterns]
# Exclude plugins matching patterns
plugins = [
"nu_plugin_test_*",
"nu_plugin_experimental_*"
]
```
### Commands
```bash
# Check all upstream sources
just upstream-check
# Check specific plugin
just upstream-check-plugin nu_plugin_clipboard
# Preview changes before merging
just upstream-preview nu_plugin_clipboard
# Safely merge upstream changes
just upstream-merge nu_plugin_clipboard
# Merge all pending changes
just upstream-merge-all
# Manage exclusions
just exclude nu_plugin_test add # Add to exclusion list
just exclude nu_plugin_test remove # Remove from exclusion list
just exclude-list # Show all excluded plugins
```
### Status System
| Status | Description | Action Required |
|--------|-------------|-----------------|
| ✅ `ok` | Synchronized with upstream | None |
| ⚠️ `pending` | Changes detected, needs review | Manual review |
| ❌ `error` | Error during sync | Investigation |
| 🔥 `conflict` | Merge conflicts | Manual resolution |
| ❓ `unknown` | Not yet checked | Run upstream check |
| 🏠 `local_only` | No upstream repository | None |
### Status Management
#### Manual Status Updates
You can manually update plugin status using justfile recipes or scripts:
```bash
# Update specific plugin status
just status-update PLUGIN STATUS
just status-update nu_plugin_image ok
# Mark all local development plugins as OK (recommended)
just status-mark-locals-ok
# Using scripts directly
./scripts/run.sh plugin_status.nu update nu_plugin_hashes ok
```
#### Common Status Management Tasks
```bash
# Check current status
just status # Full dashboard
just status-summary # Quick summary only
just status-attention # Only plugins needing attention
# Mark local plugins as reviewed/OK
just status-mark-locals-ok # Marks nu_plugin_image, nu_plugin_hashes, nu_plugin_desktop_notifications as OK
# Individual updates
just status-update nu_plugin_image ok
just status-update nu_plugin_highlight pending
```
#### When to Update Status
- **Local plugins** (`nu_plugin_image`, `nu_plugin_hashes`, `nu_plugin_desktop_notifications`): Mark as `ok` after reviewing local changes
- **Upstream plugins**: Status is automatically managed by `just upstream-check`
- **Manual override**: Use `status-update` when you need to override automatic status detection
## 🛠️ Scripts and Tools
### Consolidated Script System
The repository now uses a **unified script system** with automatic nushell detection and **mandatory version consistency checking**.
### Directory Structure
```
scripts/
├── run.sh # Universal script runner with version checking
├── check_version.nu # Version consistency validator
├── lib/
│ └── cargo_toml_diff.nu # Cargo.toml analysis library
├── check_upstream_changes.nu # Upstream monitoring
├── plugin_status.nu # Status dashboard
├── safe_merge_upstream.nu # Safe merge operations
├── build_all.nu # Build all plugins
├── collect_install.nu # Collection for distribution
├── pack_dist.nu # Distribution packaging
├── make_plugin.nu # Plugin template generator
├── update_nu_versions.nu # Dependency version management
└── sh/ # Essential shell scripts only
├── update_nushell.sh # Nushell installer/updater
├── update_nu_versions.sh # Dependency version manager
├── test_single.sh # Testing utilities
└── test_update.sh # Testing utilities
etc/
├── plugin_registry.toml # Central plugin configuration
└── upstream_exclude.toml # Upstream exclusion configuration
```
### Using the Universal Script Runner
**All nushell scripts should be run through the universal wrapper for automatic version checking:**
```bash
# Universal wrapper with automatic version checking
./scripts/run.sh <script_name> [args...]
# Status and information
./scripts/run.sh plugin_status.nu # Show dashboard
./scripts/run.sh plugin_status.nu --all # Include local plugins
./scripts/run.sh plugin_status.nu summary # Quick summary
# Upstream tracking
./scripts/run.sh check_upstream_changes.nu # Check all
./scripts/run.sh check_upstream_changes.nu --plugin highlight # Check one
./scripts/run.sh safe_merge_upstream.nu highlight # Merge one
./scripts/run.sh safe_merge_upstream.nu --all # Merge all pending
# Development
./scripts/run.sh build_all.nu # Build all plugins
./scripts/run.sh update_nu_versions.nu # Update nu dependencies
# Version management
./scripts/run.sh --check-only # Only check version consistency
./scripts/run.sh --fix --check-only # Auto-fix version mismatches
```
### Direct Nushell Script Usage (Advanced)
**⚠️ Warning: Direct usage skips the universal wrapper's version checking - not recommended**
```bash
# Run from repository root (version checking included in each script)
nu scripts/plugin_status.nu
nu scripts/check_upstream_changes.nu --plugin highlight
nu scripts/build_all.nu --verbose
```
### Version Consistency System
Every script automatically validates that your system nushell version matches the submodule version:
- **Automatic detection**: Missing nushell is auto-installed
- **Version validation**: System vs submodule version comparison
- **Auto-fix capability**: `--fix` flag resolves version mismatches
- **Clear error messages**: Detailed guidance when issues occur
## 📁 File Structure
```
nushell-plugins/
├── README.md # This file
├── justfile # Just task runner recipes
├── env # Environment configuration
├── LICENSE # License file
├── etc/ # Configuration files
│ ├── plugin_registry.toml # Plugin tracking configuration
│ └── upstream_exclude.toml # Upstream exclusion configuration
├── scripts/ # All automation scripts
│ ├── run.sh # Universal script runner
│ ├── check_version.nu # Version consistency validator
│ ├── *.nu # Nushell script implementations
│ ├── lib/ # Shared libraries
│ └── sh/ # Essential shell scripts
├── generate/ # Plugin templates
│ └── nu_plugin_template/ # Base template for new plugins
├── distribution/ # Build output directory
├── bin_archives/ # Distribution archives
├── nushell/ # Nushell submodule (for dependency versions)
├── nu_plugin_*/ # Individual plugin directories
└── docs/ # Documentation
```
## 📥 Installation Methods
### Method 1: Full Distribution (Recommended for End Users)
**Zero-prerequisite installation of complete Nushell environment:**
```bash
# One-liner installation (Linux/macOS)
curl -sSf https://your-url/install.sh | sh
# Windows PowerShell
iwr https://your-url/install.ps1 | iex
# Manual installation from release
wget https://releases/nushell-full-linux-x86_64.tar.gz
tar -xzf nushell-full-*.tar.gz && cd nushell-full-* && ./install.sh
# Install from local archive (version-agnostic)
./install.sh --source-path bin_archives/nushell-full-0.108.0-darwin-arm64.tar.gz --install-dir ~/.local
# Verify installation
nu --version && nu -c "plugin list"
```
**What you get:**
- Complete Nushell installation
- All plugins from this repository
- Optimized configuration
- Professional installation experience
- Clean uninstall capability
**Install Script Architecture (DRY Design)**
The installation system uses a **single source of truth** with symlinks to eliminate code duplication:
**Key Files:**
- **Source of Truth**: `installers/bootstrap/install.sh` (1,176 lines)
- Complete bootstrap installer for Nushell + plugins
- Repository: `https://github.com/jesusperezlorenzo/nushell-plugins`
- Platform detection: darwin-arm64, linux-x86_64, windows-x86_64
- **Symlinks** (Automatic updates through symlinks):
- `./install.sh` → `installers/bootstrap/install.sh`
- `./scripts/templates/install.sh` → `installers/bootstrap/install.sh`
- All changes propagate automatically (single maintenance point)
- **Windows Variant**: `./scripts/templates/install.ps1`
- PowerShell equivalent with `-SourcePath` and `-InstallDir` parameters
- Identical functionality via platform-specific syntax
**New `--source-path` Feature**:
Install from local archives or directories without downloading:
```bash
# From archived distribution
./install.sh --source-path bin_archives/nushell-full-0.108.0-darwin-arm64.tar.gz
# From extracted directory
./install.sh --source-path ./nushell-binaries
# Default (if path not specified)
./install.sh --source-path # Uses ./bin_archives
# Combined with custom installation path
./install.sh --source-path ./archives --install-dir ~/.local/nushell
```
**Version-Agnostic Archive Detection**:
- Automatically detects any `nushell-*` version directory
- Works with: 0.107, 0.108, 0.109, or any future version
- Searches for binaries in: `bin/nu` (preferred) → root `nu` (fallback)
- Supports both `.tar.gz` and `.zip` archives
- Smart extraction and path detection
**Installation Options**:
```bash
./install.sh # Interactive mode
./install.sh --install-dir ~/.local # Custom path, non-interactive
./install.sh --source-path ./archives # Local source
./install.sh --download-only # Download only
./install.sh --verify-only # Verify existing installation
./install.sh --uninstall # Remove installation
```
**How DRY Works**:
1. **Single Source**: Only `installers/bootstrap/install.sh` is edited
2. **Symlinks Propagate**: All changes automatically available at `./install.sh` and `./scripts/templates/install.sh`
3. **Verified**: All symlinked paths contain identical content
4. **Maintained**: No code duplication, no divergence risk
### Method 2: Plugin-Only Distribution
**For users who already have Nushell installed:**
```bash
# Download plugin-only archive
wget https://releases/nushell-plugins-linux-x86_64.tar.gz
tar -xzf nushell-plugins-*.tar.gz && cd nushell-plugins
# Install all plugins
./install_plugins.sh
# Or install specific plugins
./install_plugins.sh --plugins "nu_plugin_clipboard,nu_plugin_highlight"
```
### Method 3: Repository Clone (Recommended for Development)
**For plugin developers and contributors:**
```bash
git clone <repository-url>
cd nushell-plugins
# Validate environment
just validate-nushell
# Build and install plugins only
just build && just collect
cd distribution && nu install_nu_plugins.nu
# Build complete distributions
just build-full && just pack-full
```
### Method 4: Individual Plugin Installation
**For specific plugin installation:**
```bash
# From source (development)
cd nu_plugin_clipboard
cargo build --release
plugin add target/release/nu_plugin_clipboard
# From crates.io (if published)
cargo install nu_plugin_clipboard
plugin add ~/.cargo/bin/nu_plugin_clipboard
```
### Installation Comparison
| Method | Prerequisites | What's Included | Best For |
|--------|---------------|-----------------|----------|
| **Full Distribution** | None | Nushell + All Plugins | End users, system admins |
| **Plugin Distribution** | Nushell | All Plugins | Nushell users |
| **Repository Clone** | Rust, Git | Development environment | Contributors, developers |
| **Individual Plugin** | Rust, Nu | Single plugin | Selective installation |
## 🎮 Using Just Recipes
Just provides convenient shortcuts for all common tasks with **automatic version validation**:
### Version Validation Commands (New!)
```bash
just validate-nushell # Check nushell version consistency
just fix-nushell # Auto-fix version mismatches
```
### Basic Commands
```bash
just # Show all available recipes
just status # Plugin status dashboard
just build # Build all plugins
just build-full # Build nushell + all plugins
just test # Test all plugins
just clean # Clean build artifacts
```
### Full Distribution Commands
```bash
# Building complete distributions
just build-full # Build nushell with all plugins
just build-full-cross # Cross-platform build
just build-full-all # All platforms
# Creating distribution packages
just pack-full # Create distribution package
just pack-full-all # All platforms
just pack-full-with-installer # Include bootstrap installer
# Release workflows
just release-full-cross # Complete cross-platform release
just verify-full # Verify distribution integrity
just test-full # Test installation process
```
### Upstream Management
```bash
just upstream-check # Check for upstream updates
just upstream-merge highlight # Merge specific plugin
just upstream-merge-all # Merge all pending
```
### Development Workflows
**All workflows now include mandatory version validation as the first step:**
```bash
just dev-flow # Validate → Check → Build → Test → Status
just release-flow # Validate → Build → Collect → Package
just quality-flow # Validate → Format → Lint → Test
just update-flow # Update → Fix Version → Update versions → Check upstream
```
### Plugin-Specific Operations
```bash
just build-plugin clipboard # Build specific plugin
just test-plugin clipboard # Test specific plugin
just install-plugin clipboard # Install plugin locally
```
### Advanced Features
```bash
just interactive # Interactive plugin selection (requires fzf)
just watch clipboard # Watch for changes and rebuild (requires entr)
just validate # Validate setup and dependencies
```
## ❓ FAQ
### Q: Why do I get "Version mismatch detected" errors?
**A:** This system requires your installed nushell version to match the submodule version for consistency. All operations automatically check version consistency and will fail if versions don't match.
**Solutions:**
- **Auto-fix**: `just fix-nushell` or `./scripts/run.sh --fix --check-only`
- **Manual check**: `just validate-nushell`
- **Skip (not recommended)**: `./scripts/run.sh --no-version-check script.nu`
### Q: What happened to the bash wrapper scripts?
**A:** The repository has been consolidated to eliminate script duplication. The old bash wrappers in `scripts/sh/` have been removed in favor of:
- **Universal wrapper**: `./scripts/run.sh` with automatic version checking
- **Direct nu scripts**: All scripts moved from `scripts/nu/` to `scripts/`
- **Just recipes**: Updated to use the new system
### Q: How does the new script system work?
**A:** The new system provides:
1. **Universal wrapper** (`scripts/run.sh`) with automatic nushell detection and version validation
2. **Consolidated scripts** - all nu scripts in `scripts/` directory
3. **Mandatory version checking** - every operation validates version consistency
4. **Auto-installation** - missing nushell is automatically installed
### Q: How does automatic upstream tracking work?
**A:** The system fetches changes from upstream repositories and analyzes them. If only nu_* dependencies changed (like nu-plugin, nu-protocol), it automatically marks the plugin as "OK". Other changes are flagged for manual review. You can exclude specific plugins from tracking using the `etc/upstream_exclude.toml` file.
### Q: What happens to my local changes during upstream merges?
**A:** Local changes are preserved. The system:
1. Creates backup branches before any merge
2. Applies upstream changes in a temporary branch
3. Restores your local nu_* dependency versions
4. Tests compilation before applying changes
5. Rolls back on any failure
### Q: Can I disable automatic upstream tracking for a plugin?
**A:** Yes, set `auto_ok_on_nu_deps_only = false` in `etc/plugin_registry.toml` for that plugin.
### Q: How do I add a new plugin to the repository?
**A:** Use the template generator:
```bash
just make-plugin nu_plugin_myfeature
# or
nu scripts/nu/make_plugin.nu nu_plugin_myfeature
```
### Q: What if upstream tracking fails?
**A:** Check the plugin status with `just status`. Failed plugins show error details. Common issues:
- Network connectivity to upstream repository
- Authentication for private repositories
- Merge conflicts requiring manual resolution
### Q: How do I update nushell dependency versions?
**A:** Use the version updater:
```bash
just update-nu-versions # Update all plugins
just list-nu-versions # Show current versions
```
### Q: Can I use this with my own private plugins?
**A:** Yes! Add your private repository URLs to `etc/plugin_registry.toml`. The system supports SSH URLs for private repositories.
### Q: How do I distribute plugins to other systems?
**A:** Use the distribution pipeline:
```bash
just build # Build all plugins
just collect # Collect binaries
just pack # Create archive
```
The resulting archive contains all binaries and installation scripts.
### Q: What's the difference between shell scripts and nushell scripts?
**A:**
- **Nushell scripts** (`scripts/nu/`): Primary implementation with full features
- **Shell scripts** (`scripts/sh/`): Wrappers for compatibility with non-nushell environments
- Both provide the same functionality, use whichever fits your environment
### Q: How do I contribute a new plugin?
**A:**
1. Create the plugin: `just make-plugin nu_plugin_yourname`
2. Implement your functionality
3. Add upstream URL to `etc/plugin_registry.toml` if it has one
4. Test: `just build-plugin nu_plugin_yourname`
5. Submit a pull request
### Q: How do I exclude plugins from upstream tracking?
**A:** Edit `etc/upstream_exclude.toml` to exclude plugins from specific operations:
- Add to `[exclude]` section to exclude from all operations
- Add to `[exclude.check]` to skip automatic checks
- Add to `[exclude.merge]` to prevent automatic merging
- Use patterns like `nu_plugin_test_*` to exclude multiple plugins
You can also use Just commands:
```bash
just exclude nu_plugin_test add # Add to exclusion
just exclude nu_plugin_test remove # Remove from exclusion
just exclude-list # Show excluded plugins
```
### Q: Can I use this system for other Rust projects?
**A:** The upstream tracking and dependency management concepts can be adapted, but this system is specifically designed for nushell plugins with nu_* dependencies.
## 🤝 Contributing
1. Fork the repository
2. Create a feature branch: `git checkout -b feature/amazing-feature`
3. Make your changes
4. Test thoroughly: `just quality-flow`
5. Commit your changes: `git commit -m 'Add amazing feature'`
6. Push to the branch: `git push origin feature/amazing-feature`
7. Open a pull request
### Development Guidelines
- Follow existing code style and patterns
- Add tests for new functionality
- Update documentation for user-facing changes
- Use `just quality-flow` before submitting
- All scripts should work from repository root
### Adding New Plugins
- Use the template generator: `just make-plugin nu_plugin_name`
- Follow nushell plugin conventions
- Add comprehensive documentation
- Include examples and tests
## 📄 License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## 🙏 Acknowledgments
- [Nushell Team](https://www.nushell.sh/) for the amazing shell and plugin system
- Plugin authors for their contributions to the nushell ecosystem
- Contributors to this repository
---
**Happy Plugin Development! 🎉**
Reference in New Issue View Git Blame Copy Permalink