nushell-plugins/README.md

# 🚀 Nushell Plugins Repository

A comprehensive collection of nushell plugins with automated upstream tracking, dependency management, and development workflows.

## 📋 Table of Contents

- [Quick Start](#-quick-start)
- [Why This Repository?](#-why-this-repository)
- [Plugin Collection](#-plugin-collection)
- [Building and Cross-Compilation](#-building-and-cross-compilation)
- [Development Workflows](#-development-workflows)
- [Upstream Tracking System](#-upstream-tracking-system)
- [Scripts and Tools](#-scripts-and-tools)
- [File Structure](#-file-structure)
- [Installation Methods](#-installation-methods)
- [FAQ](#-faq)
- [Contributing](#-contributing)

## 🚀 Quick Start

### Prerequisites

```bash
# Install required tools
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh  # Rust
cargo install just                                              # Just (optional but recommended)
# Note: Nushell will be auto-installed if missing
```

### Basic Usage

```bash
# Clone the repository
git clone <repository-url>
cd nushell-plugins

# Validate nushell version consistency (recommended first step)
just validate-nushell

# Using Just (recommended)
just status              # Show plugin status
just status-mark-locals-ok  # Mark local plugins as OK (recommended)
just upstream-check      # Check for upstream updates
just build              # Build all plugins
just dev-flow           # Complete development workflow

# Using scripts directly (with automatic version checking)
./scripts/run.sh plugin_status.nu
./scripts/run.sh check_upstream_changes.nu
./scripts/run.sh build_all.nu
```

### ⚠️ Critical: Version Consistency

**This system requires version consistency between your installed nushell and the submodule version.** All operations automatically check version consistency and will fail if versions don't match.

- **Auto-fix**: `just fix-nushell` or `./scripts/run.sh --fix --check-only`
- **Manual check**: `just validate-nushell`

## 🎯 Why This Repository?

### The Challenge

Developing nushell plugins involves several challenges:

1. **Dependency Management**: Nushell evolves rapidly, requiring frequent dependency updates
2. **Upstream Synchronization**: Many plugins have upstream repositories that need tracking
3. **Build Consistency**: Ensuring all plugins build with the same nushell version
4. **Distribution**: Packaging and distributing multiple plugins efficiently

### Our Solution

This repository provides:

- **🔄 Automated Upstream Tracking**: Monitors upstream changes and auto-approves nu_* dependency updates
- **📦 Unified Build System**: Builds all plugins with consistent nushell versions
- **🛡️ Safe Merge Process**: Preserves local changes while integrating upstream updates
- **🚀 Distribution Pipeline**: Automated packaging and installation scripts
- **⚡ Developer Experience**: Just recipes and scripts for common tasks

## 📦 Plugin Collection

| Plugin | Status | Type | Description |
|--------|--------|------|-------------|
| **nu_plugin_clipboard** | ✅ Tracked | Upstream | Clipboard operations (copy/paste) |
| **nu_plugin_highlight** | ✅ Tracked | Upstream | Syntax highlighting for source code |
| **nu_plugin_image** | 🏠 Local | Local | Image processing and manipulation |
| **nu_plugin_hashes** | 🏠 Local | Local | Hash computation utilities |
| **nu_plugin_desktop_notifications** | 🏠 Local | Local | Desktop notification system |
| **nu_plugin_fluent** | ✅ Tracked | Upstream | Fluent localization framework |
| **nu_plugin_tera** | ✅ Tracked | Private | Tera templating engine (private repo) |
| **nu_plugin_kcl** | ✅ Tracked | Private | KCL configuration language (private repo) |

### Legend
- ✅ **Tracked**: Has upstream repository with automated tracking
- 🏠 **Local**: Developed locally without upstream
- 🔒 **Private**: Private repository with tracking (requires authentication)

## 🏗️ Building and Cross-Compilation

This repository includes a comprehensive cross-platform build system that supports building plugins for multiple platforms from a single machine.

### 🚀 Quick Build Commands

```bash
# Build for your current platform
just build

# Build for all supported platforms
just build-cross-all

# Build for specific platform
just build-cross linux-amd64
just build-cross darwin-arm64
just build-cross windows-amd64

# Complete cross-platform release
just release-cross
```

### 🎯 Supported Platforms

| Platform | Architecture | Native | Docker | Status |
|----------|-------------|--------|---------|--------|
| **Linux** | AMD64 | ✅ | ✅ | Full support |
| **Linux** | ARM64 | ⚠️ | ✅ | Docker recommended |
| **macOS** | Intel | 🍎 | ❌ | macOS host only |
| **macOS** | Apple Silicon | 🍎 | ❌ | macOS host only |
| **Windows** | AMD64 | 🪟 | ✅ | Windows host or Docker |

### 🐳 Docker Cross-Compilation

For consistent builds across all platforms:

```bash
# Build Docker cross-compilation environment
just build-docker-image

# Use Docker for specific targets
just build-docker linux-arm64
just build-docker windows-amd64

# Complete Docker workflow
just docker-flow
```

### 📦 Distribution

```bash
# Collect binaries for all platforms
just collect-all

# Package with checksums
just pack-checksums

# One-liner: build, collect, and package everything
just release-cross
```

### 📖 Comprehensive Guide

For detailed instructions, platform-specific setup, troubleshooting, and advanced topics, see:

**➡️ [Complete Building and Cross-Compilation Guide](docs/BUILDING.md)**

The guide covers:
- Platform-specific setup and dependencies
- Native vs Docker cross-compilation
- Configuration and customization
- Troubleshooting common issues
- Performance optimization
- CI/CD integration
- Custom target addition

## 🔄 Development Workflows

### Daily Development (Using Just)

```bash
# Check what needs attention
just status-attention

# Update from upstream sources
just upstream-check

# Build and test everything
just dev-flow

# Work on specific plugin
just build-plugin nu_plugin_clipboard
just test-plugin nu_plugin_clipboard
```

### Release Workflow

```bash
# Complete release pipeline (native)
just release-flow

# Cross-platform release pipeline
just release-flow-cross

# Or step by step:
just build           # Build all plugins (native)
just build-cross-all # Build all plugins (all platforms)
just collect-all     # Collect binaries (all platforms)
just pack-checksums  # Create distribution archives with checksums
```

### Quality Assurance

```bash
# Run all quality checks
just quality-flow

# Individual checks
just fmt            # Format code
just lint           # Run clippy
just test           # Run tests
just audit          # Security audit
```

## 🎛️ Upstream Tracking System

### How It Works

The upstream tracking system automatically:

1. **Fetches** latest changes from upstream repositories
2. **Analyzes** differences between local and upstream code
3. **Auto-approves** changes that only affect nu_* dependencies
4. **Flags** other changes for manual review
5. **Preserves** local modifications during merges
6. **Excludes** specified plugins from tracking operations

### Configuration

#### Plugin Registry (`etc/plugin_registry.toml`)

```toml
[plugins.nu_plugin_example]
upstream_url = "https://github.com/user/nu_plugin_example"
upstream_branch = "main"
auto_ok_on_nu_deps_only = true  # Auto-approve nu_* dependency changes
status = "unknown"              # Current status
```

#### Exclusion Configuration (`etc/upstream_exclude.toml`)

Control which plugins are excluded from upstream operations:

```toml
[exclude]
# Plugins to exclude from ALL upstream operations
plugins = [
    "nu_plugin_manual_only",
    "nu_plugin_deprecated"
]

[exclude.check]
# Exclude from checking only (can still be manually merged)
plugins = [
    "nu_plugin_no_auto_check"
]

[exclude.merge]
# Exclude from automatic merging (but can still be checked)
plugins = [
    "nu_plugin_complex_merge"
]

[exclude.patterns]
# Exclude plugins matching patterns
plugins = [
    "nu_plugin_test_*",
    "nu_plugin_experimental_*"
]
```

### Commands

```bash
# Check all upstream sources
just upstream-check

# Check specific plugin
just upstream-check-plugin nu_plugin_clipboard

# Preview changes before merging
just upstream-preview nu_plugin_clipboard

# Safely merge upstream changes
just upstream-merge nu_plugin_clipboard

# Merge all pending changes
just upstream-merge-all

# Manage exclusions
just exclude nu_plugin_test add      # Add to exclusion list
just exclude nu_plugin_test remove   # Remove from exclusion list
just exclude-list                    # Show all excluded plugins
```

### Status System

| Status | Description | Action Required |
|--------|-------------|-----------------|
| ✅ `ok` | Synchronized with upstream | None |
| ⚠️ `pending` | Changes detected, needs review | Manual review |
| ❌ `error` | Error during sync | Investigation |
| 🔥 `conflict` | Merge conflicts | Manual resolution |
| ❓ `unknown` | Not yet checked | Run upstream check |
| 🏠 `local_only` | No upstream repository | None |

### Status Management

#### Manual Status Updates

You can manually update plugin status using justfile recipes or scripts:

```bash
# Update specific plugin status
just status-update PLUGIN STATUS
just status-update nu_plugin_image ok

# Mark all local development plugins as OK (recommended)
just status-mark-locals-ok

# Using scripts directly
./scripts/run.sh plugin_status.nu update nu_plugin_hashes ok
```

#### Common Status Management Tasks

```bash
# Check current status
just status                    # Full dashboard
just status-summary           # Quick summary only
just status-attention         # Only plugins needing attention

# Mark local plugins as reviewed/OK
just status-mark-locals-ok     # Marks nu_plugin_image, nu_plugin_hashes, nu_plugin_desktop_notifications as OK

# Individual updates
just status-update nu_plugin_image ok
just status-update nu_plugin_highlight pending
```

#### When to Update Status

- **Local plugins** (`nu_plugin_image`, `nu_plugin_hashes`, `nu_plugin_desktop_notifications`): Mark as `ok` after reviewing local changes
- **Upstream plugins**: Status is automatically managed by `just upstream-check`
- **Manual override**: Use `status-update` when you need to override automatic status detection

## 🛠️ Scripts and Tools

### Consolidated Script System

The repository now uses a **unified script system** with automatic nushell detection and **mandatory version consistency checking**.

### Directory Structure

```
scripts/
├── run.sh                     # Universal script runner with version checking
├── check_version.nu           # Version consistency validator
├── lib/
│   └── cargo_toml_diff.nu     # Cargo.toml analysis library
├── check_upstream_changes.nu  # Upstream monitoring
├── plugin_status.nu           # Status dashboard
├── safe_merge_upstream.nu     # Safe merge operations
├── build_all.nu              # Build all plugins
├── collect_install.nu        # Collection for distribution
├── pack_dist.nu              # Distribution packaging
├── make_plugin.nu            # Plugin template generator
├── update_nu_versions.nu     # Dependency version management
└── sh/                       # Essential shell scripts only
    ├── update_nushell.sh      # Nushell installer/updater
    ├── update_nu_versions.sh  # Dependency version manager
    ├── test_single.sh         # Testing utilities
    └── test_update.sh         # Testing utilities
etc/
├── plugin_registry.toml       # Central plugin configuration
└── upstream_exclude.toml      # Upstream exclusion configuration
```

### Using the Universal Script Runner

**All nushell scripts should be run through the universal wrapper for automatic version checking:**

```bash
# Universal wrapper with automatic version checking
./scripts/run.sh <script_name> [args...]

# Status and information
./scripts/run.sh plugin_status.nu           # Show dashboard
./scripts/run.sh plugin_status.nu --all     # Include local plugins
./scripts/run.sh plugin_status.nu summary   # Quick summary

# Upstream tracking
./scripts/run.sh check_upstream_changes.nu                    # Check all
./scripts/run.sh check_upstream_changes.nu --plugin highlight # Check one
./scripts/run.sh safe_merge_upstream.nu highlight             # Merge one
./scripts/run.sh safe_merge_upstream.nu --all                 # Merge all pending

# Development
./scripts/run.sh build_all.nu              # Build all plugins
./scripts/run.sh update_nu_versions.nu     # Update nu dependencies

# Version management
./scripts/run.sh --check-only               # Only check version consistency
./scripts/run.sh --fix --check-only        # Auto-fix version mismatches
```

### Direct Nushell Script Usage (Advanced)

**⚠️ Warning: Direct usage skips the universal wrapper's version checking - not recommended**

```bash
# Run from repository root (version checking included in each script)
nu scripts/plugin_status.nu
nu scripts/check_upstream_changes.nu --plugin highlight
nu scripts/build_all.nu --verbose
```

### Version Consistency System

Every script automatically validates that your system nushell version matches the submodule version:

- **Automatic detection**: Missing nushell is auto-installed
- **Version validation**: System vs submodule version comparison
- **Auto-fix capability**: `--fix` flag resolves version mismatches
- **Clear error messages**: Detailed guidance when issues occur

## 📁 File Structure

```
nushell-plugins/
├── README.md                   # This file
├── justfile                   # Just task runner recipes
├── env                        # Environment configuration
├── LICENSE                    # License file
├── etc/                       # Configuration files
│   ├── plugin_registry.toml   # Plugin tracking configuration
│   └── upstream_exclude.toml  # Upstream exclusion configuration
├── scripts/                   # All automation scripts
│   ├── run.sh                 # Universal script runner
│   ├── check_version.nu       # Version consistency validator
│   ├── *.nu                   # Nushell script implementations
│   ├── lib/                   # Shared libraries
│   └── sh/                    # Essential shell scripts
├── generate/                  # Plugin templates
│   └── nu_plugin_template/    # Base template for new plugins
├── distribution/              # Build output directory
├── bin_archives/              # Distribution archives
├── nushell/                   # Nushell submodule (for dependency versions)
├── nu_plugin_*/               # Individual plugin directories
└── docs/                      # Documentation
```

## 📥 Installation Methods

### Method 1: Repository Clone (Recommended for Development)

```bash
git clone <repository-url>
cd nushell-plugins
just build                    # Build all plugins
just collect                  # Prepare for installation

# Install specific plugins
cd distribution
nu install_nu_plugins.nu --plugins [nu_plugin_clipboard, nu_plugin_highlight]
```

### Method 2: Pre-built Archive

```bash
# Download and extract archive
wget https://releases.../linux-amd64-nushell-plugins.tar.gz
tar -xzf linux-amd64-nushell-plugins.tar.gz
cd nushell-plugins

# Install all plugins
nu install_nu_plugins.nu

# Or install specific plugins
nu install_nu_plugins.nu --plugins [nu_plugin_clipboard]
```

### Method 3: Individual Plugin Installation

```bash
# Build and install single plugin
cd nu_plugin_clipboard
cargo build --release
plugin add target/release/nu_plugin_clipboard
```

### Method 4: Using Cargo

```bash
# For plugins published to crates.io
cargo install nu_plugin_clipboard
plugin add ~/.cargo/bin/nu_plugin_clipboard
```

## 🎮 Using Just Recipes

Just provides convenient shortcuts for all common tasks with **automatic version validation**:

### Version Validation Commands (New!)

```bash
just validate-nushell         # Check nushell version consistency
just fix-nushell             # Auto-fix version mismatches
```

### Basic Commands

```bash
just                          # Show all available recipes
just status                   # Plugin status dashboard
just build                    # Build all plugins
just test                     # Test all plugins
just clean                    # Clean build artifacts
```

### Upstream Management

```bash
just upstream-check           # Check for upstream updates
just upstream-merge highlight # Merge specific plugin
just upstream-merge-all       # Merge all pending
```

### Development Workflows

**All workflows now include mandatory version validation as the first step:**

```bash
just dev-flow                 # Validate → Check → Build → Test → Status
just release-flow             # Validate → Build → Collect → Package
just quality-flow             # Validate → Format → Lint → Test
just update-flow              # Update → Fix Version → Update versions → Check upstream
```

### Plugin-Specific Operations

```bash
just build-plugin clipboard   # Build specific plugin
just test-plugin clipboard    # Test specific plugin
just install-plugin clipboard # Install plugin locally
```

### Advanced Features

```bash
just interactive              # Interactive plugin selection (requires fzf)
just watch clipboard          # Watch for changes and rebuild (requires entr)
just validate                 # Validate setup and dependencies
```

## ❓ FAQ

### Q: Why do I get "Version mismatch detected" errors?

**A:** This system requires your installed nushell version to match the submodule version for consistency. All operations automatically check version consistency and will fail if versions don't match.

**Solutions:**
- **Auto-fix**: `just fix-nushell` or `./scripts/run.sh --fix --check-only`
- **Manual check**: `just validate-nushell`
- **Skip (not recommended)**: `./scripts/run.sh --no-version-check script.nu`

### Q: What happened to the bash wrapper scripts?

**A:** The repository has been consolidated to eliminate script duplication. The old bash wrappers in `scripts/sh/` have been removed in favor of:
- **Universal wrapper**: `./scripts/run.sh` with automatic version checking
- **Direct nu scripts**: All scripts moved from `scripts/nu/` to `scripts/`
- **Just recipes**: Updated to use the new system

### Q: How does the new script system work?

**A:** The new system provides:
1. **Universal wrapper** (`scripts/run.sh`) with automatic nushell detection and version validation
2. **Consolidated scripts** - all nu scripts in `scripts/` directory
3. **Mandatory version checking** - every operation validates version consistency
4. **Auto-installation** - missing nushell is automatically installed

### Q: How does automatic upstream tracking work?

**A:** The system fetches changes from upstream repositories and analyzes them. If only nu_* dependencies changed (like nu-plugin, nu-protocol), it automatically marks the plugin as "OK". Other changes are flagged for manual review. You can exclude specific plugins from tracking using the `etc/upstream_exclude.toml` file.

### Q: What happens to my local changes during upstream merges?

**A:** Local changes are preserved. The system:
1. Creates backup branches before any merge
2. Applies upstream changes in a temporary branch
3. Restores your local nu_* dependency versions
4. Tests compilation before applying changes
5. Rolls back on any failure

### Q: Can I disable automatic upstream tracking for a plugin?

**A:** Yes, set `auto_ok_on_nu_deps_only = false` in `etc/plugin_registry.toml` for that plugin.

### Q: How do I add a new plugin to the repository?

**A:** Use the template generator:
```bash
just make-plugin nu_plugin_myfeature
# or
nu scripts/nu/make_plugin.nu nu_plugin_myfeature
```

### Q: What if upstream tracking fails?

**A:** Check the plugin status with `just status`. Failed plugins show error details. Common issues:
- Network connectivity to upstream repository
- Authentication for private repositories
- Merge conflicts requiring manual resolution

### Q: How do I update nushell dependency versions?

**A:** Use the version updater:
```bash
just update-nu-versions       # Update all plugins
just list-nu-versions         # Show current versions
```

### Q: Can I use this with my own private plugins?

**A:** Yes! Add your private repository URLs to `etc/plugin_registry.toml`. The system supports SSH URLs for private repositories.

### Q: How do I distribute plugins to other systems?

**A:** Use the distribution pipeline:
```bash
just build                    # Build all plugins
just collect                  # Collect binaries
just pack                     # Create archive
```

The resulting archive contains all binaries and installation scripts.

### Q: What's the difference between shell scripts and nushell scripts?

**A:**
- **Nushell scripts** (`scripts/nu/`): Primary implementation with full features
- **Shell scripts** (`scripts/sh/`): Wrappers for compatibility with non-nushell environments
- Both provide the same functionality, use whichever fits your environment

### Q: How do I contribute a new plugin?

**A:**
1. Create the plugin: `just make-plugin nu_plugin_yourname`
2. Implement your functionality
3. Add upstream URL to `etc/plugin_registry.toml` if it has one
4. Test: `just build-plugin nu_plugin_yourname`
5. Submit a pull request

### Q: How do I exclude plugins from upstream tracking?

**A:** Edit `etc/upstream_exclude.toml` to exclude plugins from specific operations:
- Add to `[exclude]` section to exclude from all operations
- Add to `[exclude.check]` to skip automatic checks
- Add to `[exclude.merge]` to prevent automatic merging
- Use patterns like `nu_plugin_test_*` to exclude multiple plugins

You can also use Just commands:
```bash
just exclude nu_plugin_test add      # Add to exclusion
just exclude nu_plugin_test remove   # Remove from exclusion
just exclude-list                    # Show excluded plugins
```

### Q: Can I use this system for other Rust projects?

**A:** The upstream tracking and dependency management concepts can be adapted, but this system is specifically designed for nushell plugins with nu_* dependencies.

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch: `git checkout -b feature/amazing-feature`
3. Make your changes
4. Test thoroughly: `just quality-flow`
5. Commit your changes: `git commit -m 'Add amazing feature'`
6. Push to the branch: `git push origin feature/amazing-feature`
7. Open a pull request

### Development Guidelines

- Follow existing code style and patterns
- Add tests for new functionality
- Update documentation for user-facing changes
- Use `just quality-flow` before submitting
- All scripts should work from repository root

### Adding New Plugins

- Use the template generator: `just make-plugin nu_plugin_name`
- Follow nushell plugin conventions
- Add comprehensive documentation
- Include examples and tests

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- [Nushell Team](https://www.nushell.sh/) for the amazing shell and plugin system
- Plugin authors for their contributions to the nushell ecosystem
- Contributors to this repository

---

**Happy Plugin Development! 🎉**