nushell-plugins/docs/BUILDING.md

# Building and Cross-Compilation Guide

This comprehensive guide covers everything you need to know about building nushell plugins, including cross-compilation for multiple platforms.

## Table of Contents

- [Quick Start](#quick-start)
- [Prerequisites](#prerequisites)
- [Build Methods](#build-methods)
- [Cross-Compilation](#cross-compilation)
- [Docker Builds](#docker-builds)
- [Platform Support](#platform-support)
- [Configuration](#configuration)
- [Workflows](#workflows)
- [Distribution](#distribution)
- [Troubleshooting](#troubleshooting)
- [FAQ](#faq)
- [Advanced Topics](#advanced-topics)

## Quick Start

### Native Build (Host Platform Only)
```bash
# Clone repository
git clone https://github.com/YOUR_ORG/nushell-plugins.git
cd nushell-plugins

# Build all plugins for your platform
just build

# Or using the script directly
./scripts/run.sh build_all.nu
```

### Cross-Platform Build
```bash
# Build for all supported platforms
just build-cross-all

# Build for specific platform
just build-cross linux-amd64

# Build and create distribution packages
just release-cross
```

## Prerequisites

### Required Tools

1. **Rust Toolchain** (1.70.0+)
   ```bash
   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
   ```

2. **Nushell** (0.107.1)
   ```bash
   # Install via cargo
   cargo install nu

   # Or download from https://github.com/nushell/nushell/releases
   ```

3. **Just** (optional but recommended)
   ```bash
   cargo install just
   ```

4. **Git** with submodules
   ```bash
   git clone --recursive https://github.com/YOUR_ORG/nushell-plugins.git
   ```

### Optional Tools

- **Docker** - For Docker-based cross-compilation
- **Cross-compilation toolchains** - For native cross-compilation

## Build Methods

### 1. Native Compilation

Build plugins for your current platform using the standard Rust toolchain.

```bash
# Basic build
just build

# Verbose output
just build-verbose

# Parallel build (experimental)
just build-parallel

# Build specific plugins
./scripts/run.sh build_all.nu --plugins nu_plugin_clipboard,nu_plugin_image
```

**Pros:**
- Fast compilation
- Native debugging support
- Simple setup

**Cons:**
- Only builds for current platform
- Limited cross-compilation support

### 2. Cross-Compilation

Build plugins for different platforms using Rust's cross-compilation features.

```bash
# List available targets
just build-targets

# Build for specific target
just build-cross linux-amd64
just build-cross darwin-arm64
just build-cross windows-amd64

# Build for all targets
just build-cross-all

# Parallel cross-compilation
just build-cross-parallel
```

**Pros:**
- Multiple platforms from single machine
- Fast execution
- No Docker dependency

**Cons:**
- May require target-specific toolchains
- Some targets may not work on all hosts

### 3. Docker-Based Cross-Compilation

Use Docker containers with pre-configured cross-compilation environments.

```bash
# Build Docker image
just build-docker-image

# Build specific target with Docker
just build-docker linux-arm64

# Force Docker for all builds
./scripts/run.sh build_cross.nu --all-targets --docker
```

**Pros:**
- Consistent build environment
- All toolchains pre-installed
- Works on any Docker-capable host

**Cons:**
- Slower than native compilation
- Requires Docker
- Larger resource usage

## Cross-Compilation

### Supported Platforms

| Platform | Target Triple | Native | Docker | Notes |
|----------|---------------|--------|---------|-------|
| **Linux AMD64** | `x86_64-unknown-linux-gnu` | ✅ | ✅ | Most common Linux |
| **Linux ARM64** | `aarch64-unknown-linux-gnu` | ⚠️ | ✅ | Requires cross toolchain |
| **macOS Intel** | `x86_64-apple-darwin` | 🍎 | ❌ | macOS host only |
| **macOS Apple Silicon** | `aarch64-apple-darwin` | 🍎 | ❌ | macOS host only |
| **Windows AMD64** | `x86_64-pc-windows-msvc` | 🪟 | ✅ | Windows host or Docker |

**Legend:**
- ✅ Fully supported
- ⚠️ Requires additional setup
- 🍎 macOS host required
- 🪟 Windows host required
- ❌ Not supported

### Setting Up Cross-Compilation

#### Linux → Linux ARM64
```bash
# Install cross-compilation toolchain (Ubuntu/Debian)
sudo apt-get install gcc-aarch64-linux-gnu

# Add Rust target
rustup target add aarch64-unknown-linux-gnu

# Configure environment
export CC_aarch64_unknown_linux_gnu=aarch64-linux-gnu-gcc
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc

# Build
just build-cross linux-arm64
```

#### Any → Windows (Docker)
```bash
# Build Docker image with Windows toolchain
just build-docker-image

# Cross-compile to Windows
just build-docker windows-amd64
```

### Configuration Files

#### Build Targets (`etc/build_targets.toml`)

```toml
[targets.linux-amd64]
rust_target = "x86_64-unknown-linux-gnu"
platform_name = "linux-amd64"
archive_format = "tar.gz"
docker_required = false
native_build = true
description = "Linux x86_64 (64-bit Intel/AMD)"

[targets.linux-arm64]
rust_target = "aarch64-unknown-linux-gnu"
platform_name = "linux-arm64"
archive_format = "tar.gz"
docker_required = true
native_build = false
description = "Linux ARM64 (64-bit ARM)"
linker = "aarch64-linux-gnu-gcc"
```

#### Environment Variables
```bash
# Cross-compilation linkers
export CC_aarch64_unknown_linux_gnu=aarch64-linux-gnu-gcc
export CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++
export AR_aarch64_unknown_linux_gnu=aarch64-linux-gnu-ar

# Cargo configuration
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc
```

## Docker Builds

### Docker Image

The cross-compilation Docker image includes:
- **Rust toolchain** (1.75.0)
- **Cross-compilation targets** for all supported platforms
- **System toolchains** (GCC, MinGW, etc.)
- **Build tools** (cargo, rustfmt, clippy)

### Docker Commands

```bash
# Build Docker image
just build-docker-image

# Rebuild from scratch
just build-docker-image-fresh

# Show Docker environment info
just docker-info

# Build specific plugin with Docker
just build-docker-plugin nu_plugin_clipboard linux-arm64

# Clean up Docker artifacts
just docker-cleanup
```

### Docker Workflow

```bash
# Complete Docker-based workflow
just docker-flow

# This runs:
# 1. just validate-nushell
# 2. just build-docker-image
# 3. just build-cross-all --docker
# 4. just collect-all
# 5. just pack-all
```

### Manual Docker Usage

```bash
# Run interactive Docker container
docker run -it --rm \
  -v $(pwd):/workspace \
  -w /workspace \
  nushell-plugins-cross:latest \
  bash

# Build specific target in container
docker run --rm \
  -v $(pwd):/workspace \
  -w /workspace/nu_plugin_clipboard \
  nushell-plugins-cross:latest \
  cargo build --release --target aarch64-unknown-linux-gnu
```

## Platform Support

### Linux

**Supported Architectures:**
- x86_64 (AMD64) - Native and Docker
- ARM64 (AArch64) - Docker recommended
- ARM32 (ARMv7) - Docker only, disabled by default

**Dependencies:**
```bash
# Ubuntu/Debian
sudo apt-get install build-essential libssl-dev pkg-config

# Cross-compilation toolchains
sudo apt-get install gcc-aarch64-linux-gnu gcc-arm-linux-gnueabihf
```

### macOS

**Supported Architectures:**
- Intel (x86_64) - Native only
- Apple Silicon (ARM64) - Native only

**Dependencies:**
```bash
# Install Xcode command line tools
xcode-select --install

# Add Rust targets
rustup target add x86_64-apple-darwin aarch64-apple-darwin
```

**Cross-compilation:**
```bash
# On Apple Silicon, build for Intel
just build-cross darwin-amd64

# On Intel, build for Apple Silicon (may not work)
just build-cross darwin-arm64
```

### Windows

**Supported Architectures:**
- x86_64 (AMD64) - Native and Docker

**Native Dependencies:**
- Visual Studio Build Tools or Visual Studio Community
- Windows SDK

**Docker Alternative:**
```bash
# Use Docker for Windows builds on any platform
just build-docker windows-amd64
```

## Configuration

### Global Configuration

The main configuration file is `etc/build_targets.toml`:

```toml
[metadata]
version = "1.0.0"
description = "Cross-compilation targets for nushell plugins"

[defaults]
docker_image = "nushell-plugins-cross:latest"
strip_binaries = true
optimize_size = true

[targets.custom-target]
rust_target = "x86_64-unknown-linux-musl"
platform_name = "linux-amd64-musl"
archive_format = "tar.gz"
docker_required = false
description = "Linux x86_64 with musl libc"
enabled = true
```

### Environment Configuration

Create an `env` file for local overrides:

```bash
# env file
TARGET_PATH=distribution
INSTALL_BIN_PATH=/usr/local/bin
BIN_ARCHIVES_DIR_PATH=bin_archives
APP_NAME=nushell-plugins
```

### Plugin-Specific Configuration

Some plugins may require special build configurations:

```toml
# In plugin's Cargo.toml
[package.metadata.cross]
image = "ghcr.io/cross-rs/cross:aarch64-unknown-linux-gnu"

[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-gcc"
```

## Workflows

### Development Workflows

#### Standard Development
```bash
just dev-flow
# 1. Validate nushell version
# 2. Check upstream changes
# 3. Build for host platform
# 4. Run tests
# 5. Show status
```

#### Cross-Platform Development
```bash
just dev-flow-cross
# 1. Validate nushell version
# 2. Check upstream changes
# 3. Build for all platforms
# 4. Show status
```

#### Quality Assurance
```bash
just quality-flow
# 1. Validate nushell version
# 2. Format code
# 3. Run clippy linting
# 4. Run tests
```

### Release Workflows

#### Standard Release
```bash
just release-flow
# 1. Validate nushell version
# 2. Build for host platform
# 3. Collect binaries
# 4. Create packages
```

#### Cross-Platform Release
```bash
just release-flow-cross
# 1. Validate nushell version
# 2. Build for all platforms
# 3. Collect all binaries
# 4. Create packages with checksums
```

#### CI Simulation
```bash
just ci-flow
# Simulates exactly what GitHub Actions will run
```

### Custom Workflows

You can create custom workflows by combining individual commands:

```bash
# Custom workflow for testing
just validate-nushell
just build-cross linux-amd64
just build-cross darwin-arm64
just test
just collect-platform linux-amd64
just pack-platform linux-amd64
```

## Distribution

### Collection

Collect built binaries for distribution:

```bash
# Collect all platforms
just collect-all

# Collect specific platform
just collect-platform linux-amd64

# List what can be collected
just collect-platforms
```

### Packaging

Create distribution archives:

```bash
# Package all platforms
just pack-all

# Package with checksums
just pack-checksums

# Package specific platform
just pack-platform darwin-arm64

# List packaging options
just pack-platforms
```

### Directory Structure

After building and packaging:

```
distribution/
├── linux-amd64/
│   ├── nu_plugin_clipboard
│   ├── nu_plugin_image
│   ├── install_nu_plugins.nu
│   ├── LICENSE
│   └── README
├── darwin-arm64/
│   └── [same structure]
└── windows-amd64/
    ├── nu_plugin_clipboard.exe
    └── [same structure]

bin_archives/
├── linux-amd64-nushell-plugins.tar.gz
├── darwin-arm64-nushell-plugins.tar.gz
├── windows-amd64-nushell-plugins.zip
└── checksums.txt
```

### Checksums

All distribution packages include SHA256 checksums:

```bash
# Generate checksums
just pack-checksums

# Verify downloaded archive
sha256sum -c checksums.txt

# Manual verification
sha256sum linux-amd64-nushell-plugins.tar.gz
```

## Troubleshooting

### Common Issues

#### Version Mismatch
```bash
# Error: Nushell version mismatch detected
# Fix: Update versions automatically
just fix-nushell

# Or manually check and fix
./scripts/run.sh check_version.nu --fix
```

#### Missing Cross-Compilation Target
```bash
# Error: target 'aarch64-unknown-linux-gnu' not found
# Fix: Install the target
rustup target add aarch64-unknown-linux-gnu
```

#### Cross-Compilation Linker Error
```bash
# Error: linker 'aarch64-linux-gnu-gcc' not found
# Fix: Install cross-compilation toolchain
sudo apt-get install gcc-aarch64-linux-gnu

# Set environment variable
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc
```

#### Docker Build Failures
```bash
# Error: Docker daemon not running
# Fix: Start Docker service
sudo systemctl start docker

# Error: Docker image not found
# Fix: Build the image
just build-docker-image

# Error: Permission denied in Docker
# Fix: Add user to docker group
sudo usermod -aG docker $USER
```

#### Build Failures

**Plugin-specific failures:**
```bash
# Debug specific plugin
cd nu_plugin_clipboard
cargo build --release --verbose

# Check plugin dependencies
cargo tree
```

**Cross-compilation failures:**
```bash
# Use Docker instead of native cross-compilation
just build-docker linux-arm64

# Check target support
rustup target list | grep installed
```

### Debug Mode

Enable verbose output for troubleshooting:

```bash
# Verbose build output
just build-verbose

# Verbose cross-compilation
./scripts/run.sh build_cross.nu --all-targets --verbose

# Debug Docker builds
./scripts/run.sh build_docker_cross.nu --verbose
```

### Log Files

Build logs are preserved for debugging:

```bash
# View recent build logs
ls -la target/*/build/*/out/

# Search for specific errors
grep -r "error" target/*/build/*/stderr
```

## FAQ

### General Questions

**Q: What platforms are supported?**
A: Linux (AMD64, ARM64), macOS (Intel, Apple Silicon), and Windows (AMD64). See the [Platform Support](#platform-support) section for details.

**Q: Do I need Docker for cross-compilation?**
A: Not always. Native cross-compilation works for many targets, but Docker provides a more consistent environment and supports more target combinations.

**Q: How long does cross-compilation take?**
A: Building all platforms takes 10-30 minutes depending on your hardware. Parallel builds (`just build-cross-parallel`) can significantly reduce this time.

**Q: Can I add new platforms?**
A: Yes! Edit `etc/build_targets.toml` to add new targets. See [Configuration](#configuration) for details.

### Build Questions

**Q: Why do some builds fail on my platform?**
A: Cross-compilation has limitations. macOS targets can only be built on macOS, and some Linux ARM64 builds require specific toolchains. Use Docker builds as a fallback.

**Q: How can I speed up builds?**
A: Use parallel builds (`--parallel`), enable incremental compilation, and use Docker layer caching. Consider building only the platforms you need.

**Q: Can I build just one plugin?**
A: Yes! Use `./scripts/run.sh build_all.nu --plugins nu_plugin_clipboard` or build manually in the plugin directory.

**Q: Why are Windows binaries so large?**
A: Windows binaries include the MSVC runtime. You can reduce size by using the GNU toolchain or enabling link-time optimization.

### Docker Questions

**Q: How much disk space does the Docker image use?**
A: The cross-compilation image is approximately 2-3 GB, including all toolchains and dependencies.

**Q: Can I use my own Docker image?**
A: Yes! Modify the `docker.image_tag` setting in `etc/build_targets.toml` to use a custom image.

**Q: How do I update the Docker image?**
A: Run `just build-docker-image-fresh` to rebuild from scratch with the latest dependencies.

### Distribution Questions

**Q: How do I install the plugins?**
A: Use the universal installer: `curl -L https://github.com/YOUR_ORG/nushell-plugins/releases/latest/download/install.sh | sh`

**Q: Can I distribute individual plugins?**
A: Yes! Each plugin binary is self-contained. Just copy the `nu_plugin_*` files to the target system.

**Q: What's the difference between tar.gz and zip archives?**
A: Unix-like systems (Linux, macOS) use tar.gz for better compression and permission preservation. Windows uses zip for compatibility.

### Troubleshooting Questions

**Q: Build fails with "permission denied" errors**
A: Check that you have write permissions to the target directory and that antivirus software isn't blocking the builds.

**Q: Cross-compilation fails with "unsupported target"**
A: Install the target with `rustup target add TARGET_NAME` or use Docker builds instead.

**Q: Docker builds are very slow**
A: Make sure Docker has sufficient resources allocated. Consider using Docker BuildKit for faster builds.

## Advanced Topics

### Custom Build Targets

Add support for new platforms by editing `etc/build_targets.toml`:

```toml
[targets.linux-musl]
rust_target = "x86_64-unknown-linux-musl"
platform_name = "linux-amd64-musl"
archive_format = "tar.gz"
docker_required = false
description = "Linux x86_64 with musl libc"
strip_binaries = true

[environment.linux-musl]
CC = "musl-gcc"
```

### Build Optimization

#### Link-Time Optimization (LTO)
```toml
# In Cargo.toml
[profile.release]
lto = true
codegen-units = 1
panic = "abort"
strip = "symbols"
```

#### Size Optimization
```toml
[profile.release]
opt-level = "z"  # Optimize for size
lto = true
strip = "symbols"
```

#### Build Scripts

Create custom build scripts for complex scenarios:

```bash
#!/bin/bash
# scripts/custom_build.sh

# Custom build with specific optimizations
export RUSTFLAGS="-C target-cpu=native"
cargo build --release --target x86_64-unknown-linux-gnu

# Strip debug symbols
strip target/x86_64-unknown-linux-gnu/release/nu_plugin_*
```

### Continuous Integration

The repository includes GitHub Actions workflows for:

- **Build & Test** (`.github/workflows/build.yml`)
- **Release** (`.github/workflows/release.yml`)
- **Nightly Builds** (`.github/workflows/nightly.yml`)

To set up in your own repository:

1. Copy the `.github/workflows/` directory
2. Update the repository references
3. Configure GitHub secrets if needed
4. Customize the build matrix for your needs

### Plugin Development

When developing new plugins:

1. **Use the template:**
   ```bash
   just make-plugin nu_plugin_yourname
   ```

2. **Follow the plugin structure:**
   ```
   nu_plugin_yourname/
   ├── Cargo.toml
   ├── src/
   │   ├── main.rs
   │   └── lib.rs
   └── README.md
   ```

3. **Test cross-compilation early:**
   ```bash
   just build-cross-all
   ```

4. **Add to the registry:**
   Update `etc/plugin_registry.toml` if the plugin has an upstream repository.

### Performance Tuning

#### Parallel Compilation
```bash
# Set number of parallel jobs
export CARGO_BUILD_JOBS=8

# Use parallel cross-compilation
just build-cross-parallel
```

#### Caching
```bash
# Use sccache for distributed compilation caching
cargo install sccache
export RUSTC_WRAPPER=sccache
```

#### Memory Usage
```bash
# Limit memory usage for large builds
export CARGO_BUILD_RUSTFLAGS="-C link-arg=-Wl,--no-keep-memory"
```

---

## Getting Help

- **Issues**: [GitHub Issues](https://github.com/YOUR_ORG/nushell-plugins/issues)
- **Discussions**: [GitHub Discussions](https://github.com/YOUR_ORG/nushell-plugins/discussions)
- **Documentation**: [Repository Wiki](https://github.com/YOUR_ORG/nushell-plugins/wiki)
- **Nushell Community**: [Discord](https://discord.gg/NtAbbGn) | [Reddit](https://www.reddit.com/r/Nushell/)

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on contributing to the build system and adding new platforms or features.

---

*This documentation is maintained by the nushell-plugins team. Last updated: 2024-09-20*