9cef9b8d57
Merge _configs/ into config/ for single configuration directory. Update all path references. Changes: - Move _configs/* to config/ - Update .gitignore for new patterns - No code references to _configs/ found Impact: -1 root directory (layout_conventions.md compliance)
37 KiB
37 KiB
Bootstrap Compiler: Visual Diagrams and Flowcharts
Diagram 1: The Bootstrap Problem and Solution
The Circular Dependency Problem
┌──────────────────────────────────────────────────────────────┐
│ THE BOOTSTRAP PROBLEM │
└──────────────────────────────────────────────────────────────┘
WITHOUT BOOTSTRAP STRATEGY (Impossible Loop):
To compile Rust source
↓
Need rustc compiler
↓
But rustc is written in Rust
↓
Need rustc compiler to compile rustc
↓
🔄 INFINITE LOOP - How do we escape?
WITH BOOTSTRAP STRATEGY (Break the cycle):
Need a working compiler
↓
Download pre-compiled Rust binary
↓
Run pre-compiled rustc
↓
Use rustc to compile more Rust
↓
🎉 LOOP BROKEN - Now we can compile anything!
Diagram 2: Installation Timeline with Timestamps
┌───────────────────────────────────────────────────────────────────┐
│ curl https://sh.rustup.rs | bash │
│ ⏱️ START TIME │
└───────────────────────────────────────────────────────────────────┘
│
↓
⏱️ T+0s ┌─────────────────────────┐
│ Download installer │
│ (50KB bash script) │
│ via curl │
└────────┬────────────────┘
↓
⏱️ T+1s ┌─────────────────────────┐
│ Execute bash script │
│ (runs in memory) │
└────────┬────────────────┘
↓
⏱️ T+2s ┌─────────────────────────┐
│ Detect system: │
│ uname -s → Darwin │
│ uname -m → arm64 │
│ Target: aarch64-... │
└────────┬────────────────┘
↓
⏱️ T+30s ┌─────────────────────────┐
│ Download pre-compiled │
│ Rust (100-150MB) │
│ Network bandwidth limit │
└────────┬────────────────┘
↓
⏱️ T+35s ┌─────────────────────────┐
│ Verify SHA256 checksum │
│ Ensure no corruption │
└────────┬────────────────┘
↓
⏱️ T+40s ┌─────────────────────────┐
│ Extract tarball │
│ Install to ~/.rustup/ │
│ Run installer.sh │
└────────┬────────────────┘
↓
⏱️ T+45s ┌─────────────────────────┐
│ Configure PATH │
│ Export to ~/.bashrc │
│ or ~/.zshrc │
└────────┬────────────────┘
↓
⏱️ T+46s ┌─────────────────────────┐
│ ✅ Rust is installed! │
│ rustc + cargo ready │
└─────────────────────────┘
Total time: ~45 seconds (mostly network download)
Most of that: Downloading pre-compiled binary (100MB)
Diagram 3: Dependency Chain
What Rust Creates: The Seed Kernel
┌─────────────────────────────────────────────────────────────────┐
│ INSTALLED BY BOOTSTRAP (1 binary) │
├─────────────────────────────────────────────────────────────────┤
│ rustc (compiler) │
│ - Executable binary for your architecture │
│ - Pre-compiled, ready to use │
│ - Can compile Rust source code │
└────────────────┬────────────────────────────────────────────────┘
│
│ (Now we have a compiler!)
│
├─────────────────────────────────────────────┐
│ │
↓ ↓
┌──────────────────────────┐ ┌──────────────────────────┐
│ Installed with rustc │ │ Standard Library Included │
│ (comes in same download) │ │ (Pre-compiled for arch) │
├──────────────────────────┤ ├──────────────────────────┤
│ - cargo (package mgr) │ │ ~/.rustup/lib/rustlib/ │
│ - rustfmt (formatter) │ │ └── aarch64-apple-darwin │
│ - clippy (linter) │ │ ├── libstd.rlib │
│ - rust-gdb (debugger) │ │ ├── libcore.rlib │
│ │ │ └── ... (250 files) │
└────────────┬─────────────┘ └──────────────┬───────────┘
│ │
└─────────────────────┬───────────────────┘
│
(Now we can compile Rust source!)
│
┌──────────────────────────────┼──────────────────────────────┐
│ │ │
↓ ↓ ↓
┌──────────────┐ ┌──────────────────┐ ┌────────────────┐
│ Compile NuShell │ │ Compile Just │ │ Compile our app │
│ cargo install nu │ │ cargo install just │ │ cargo build │
│ │ │ │ │ │
│ ⏱️ 1-2 minutes │ │ ⏱️ 1-2 minutes │ │ ⏱️ 5-10 min │
│ │ │ │ │ │
│ ✅ nu shell │ │ ✅ just runner │ │ ✅ All binaries │
└──────────────┘ └──────────────────┘ └────────────────┘
Diagram 4: Architecture Detection Flow
┌───────────────────────────────────────────────────────────────┐
│ $ curl https://sh.rustup.rs | bash │
│ │
│ → Installer script executes on YOUR system │
│ → Must detect YOUR architecture │
└───────────────┬───────────────────────────────────────────────┘
│
↓
┌───────────────────────┐
│ uname -s │
│ (Detect OS) │
└────────┬──────────────┘
│
┌────────────┼────────────┐
│ │ │
↓ ↓ ↓
Darwin Linux Windows
│ │ │
└────┬───────┴────┬───────┘
│ │
(apple-darwin) (unknown-linux-gnu or pc-windows-msvc)
│ │
└────┬───────┘
│
↓
┌───────────────────────┐
│ uname -m │
│ (Detect Architecture) │
└────────┬──────────────┘
│
┌────────────┼────────────────┬────────────┐
│ │ │ │
↓ ↓ ↓ ↓
arm64 x86_64 aarch64 armv7l
│ │ │ │
│ │ │ │
(aarch64) (x86_64) (aarch64) (armv7l)
│ │ │ │
└────────┬───┴──────────────┬──┴────────┬───┘
│ │ │
COMBINE COMBINE COMBINE
│ │ │
↓ ↓ ↓
aarch64-apple-darwin x86_64-... aarch64-...
│ │ │
└──────────┬───────┴────┬──────┘
│ │
TARGET TRIPLE (determines which
pre-compiled to download)
│ │
↓ ↓
┌──────────────────────────────────────────────┐
│ Download from: │
│ https://static.rust-lang.org/dist/ │
│ │
│ Examples: │
│ - rust-1.91.1-aarch64-apple-darwin.tar.gz │
│ - rust-1.91.1-x86_64-unknown-linux-gnu.tar.gz│
│ - rust-1.91.1-x86_64-pc-windows-msvc.tar.gz │
│ etc. │
└──────────────────────────────────────────────┘
Diagram 5: Supported Architectures
Rust supports 50+ architectures
┌───────────────────────┬───────────────────────┐
│ │ │
┌───┴──┐ ┌──────┴────┐ ┌────────┴──┐
│ macOS │ │ Linux │ │ Windows │
└───┬──┘ └──────┬────┘ └────────┬──┘
│ │ │
┌────┴────┐ ┌───────┴──────┐ ┌──────┴─────┐
│ │ │ │ │ │
↓ ↓ ↓ ↓ ↓ ↓
aarch64 x86_64 aarch64 x86_64 aarch64 x86_64
(ARM) (Intel) (ARM) (Intel) (ARM) (Intel)
│ │ │ │ │ │
│ │ ┌────┴─────┐ │ │ │
│ │ │ │ │ │ │
↓ ↓ ↓ ↓ ↓ ↓ ↓
(Apple) (Intel) (GNU) (musl) (MSVC) (GNU) (MSVC)
│ │ │ │ │ │ │
└──┬───┬───┘ │ │ │ └──┬───┬──────┘
│ │ │ │ │ │ │
↓ ↓ ↓ ↓ ↓ ↓ ↓
✅ ✅ ✅ ✅ ✅ ✅ ✅
aarch64 x86_64 aarch64 aarch64 x86_64 aarch64 x86_64
-apple- -apple- -unknown -unknown -pc- -pc- -pc-
darwin darwin -linux- -linux- windows- windows windows
-gnu -musl -msvc -msvc -msvc
PLUS: Android, FreeBSD, iOS, tvOS, WebAssembly, RISC-V,
PowerPC, MIPS, s390x, SPARC, and more...
Diagram 6: Binary Size Comparison
┌────────────────────────────────────────────────────────────────┐
│ Installation Approach Size Comparison │
└────────────────────────────────────────────────────────────────┘
APPROACH 1: Bootstrap (Actual Rust Strategy) ✅
┌─────────────────────────────────────────────────┐
│ Download from https://sh.rustup.rs │
│ └─ 50KB bash script │
│ │
│ Download from https://static.rust-lang.org/ │
│ └─ 100-150MB pre-compiled Rust │
│ │
│ TOTAL: ~150MB (what you actually download) │
│ TIME: ~45 seconds │
└─────────────────────────────────────────────────┘
APPROACH 2: Pre-Compile Everything (Hypothetical) ❌
┌─────────────────────────────────────────────────┐
│ Pre-compile for all 50+ architectures: │
│ │
│ ├─ rustc for aarch64-apple-darwin (100MB) │
│ ├─ rustc for x86_64-apple-darwin (120MB) │
│ ├─ rustc for aarch64-unknown-linux (100MB) │
│ ├─ rustc for x86_64-unknown-linux (120MB) │
│ ├─ ...other 45+ architectures... │
│ │ │
│ ├─ nu shell for 50+ architectures (~500MB) │
│ ├─ just binary for 50+ architectures (~200MB) │
│ └─ syntaxis for 50+ (1000MB+) │
│ │
│ TOTAL: 5GB+ (bloated repository) │
│ TIME: Hours to compile & upload │
│ MAINTENANCE: Rebuild every Rust release │
└─────────────────────────────────────────────────┘
APPROACH 3: Compile Everything from Source ❌
┌─────────────────────────────────────────────────┐
│ Download Rust source code: 2GB │
│ Download dependency sources: 1GB │
│ │
│ Compile on your machine: │
│ ├─ Rust: 3 hours │
│ ├─ NuShell: 1.5 hours │
│ ├─ Just: 5 minutes │
│ └─ syntaxis: 10 minutes │
│ │
│ TOTAL TIME: ~4.5 hours │
│ STORAGE NEEDED: 10GB+ during compilation │
│ COMPLEXITY: Requires C compiler + toolchain │
└─────────────────────────────────────────────────┘
WINNER: Approach 1 (Bootstrap) 🏆
Diagram 7: Installation Process: Before and After
┌────────────────────────────────────────────────────────────────┐
│ BEFORE: What You Have Installed │
├────────────────────────────────────────────────────────────────┤
│ │
│ Every macOS/Linux System Includes: │
│ ├─ bash shell ✅ (100+ years old, universal) │
│ ├─ curl ✅ (for downloading files) │
│ ├─ tar/gzip ✅ (for extracting archives) │
│ ├─ uname ✅ (for detecting OS/arch) │
│ ├─ chmod ✅ (for changing permissions) │
│ ├─ git ✅ (for cloning repositories) │
│ └─ Standard POSIX tools ✅ │
│ │
│ ❌ NOT Installed (bootstrap solves this): │
│ ├─ rustc (Rust compiler) │
│ ├─ cargo (package manager) │
│ ├─ rustup (toolchain manager) │
│ ├─ nu (NuShell) │
│ ├─ just (task runner) │
│ └─ Our syntaxis │
│ │
└────────────────────────────────────────────────────────────────┘
│
$ curl ... | bash
│
↓
┌────────────────────────────────────────────────────────────────┐
│ AFTER: What Gets Installed │
├────────────────────────────────────────────────────────────────┤
│ │
│ Phase 1: Bootstrap (rustup) - PRE-COMPILED │
│ ├─ rustc (binary for your arch) ✅ │
│ ├─ cargo (with rustc download) ✅ │
│ ├─ rustup toolchain manager ✅ │
│ └─ Standard library (pre-compiled) ✅ │
│ │
│ Phase 2-7: Compiling with rustc (FROM SOURCE) │
│ ├─ nu shell (compiled from Rust source) ✅ │
│ ├─ just (compiled from Rust source) ✅ │
│ ├─ Our project (compiled from Rust source) ✅ │
│ ├─ Workspace CLI (compiled binary) ✅ │
│ ├─ Workspace TUI (compiled binary) ✅ │
│ ├─ Workspace Dashboard (WASM compiled) ✅ │
│ └─ Configuration files deployed ✅ │
│ │
│ You now have: │
│ ├─ Full Rust toolchain (~2GB) │
│ ├─ Compiled workspace binaries (~500MB) │
│ ├─ Configuration in ~/.config/core/ │
│ └─ Everything working and ready to use! 🎉 │
│ │
└────────────────────────────────────────────────────────────────┘
Diagram 8: Compilation Flow During Installation
Our install.sh orchestrates this process:
┌──────────────────────────────────────────────────────────────┐
│ Phase 1: System Detection │
│ ├─ Detect OS (Darwin, Linux, Windows/MSYS) │
│ ├─ Detect architecture (arm64, x86_64, aarch64) │
│ └─ Calculate target triple (aarch64-apple-darwin) │
└────────┬───────────────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────────────┐
│ Phase 2: Install Rust (Bootstrap) │
│ ├─ curl https://sh.rustup.rs | bash │
│ │ (Downloads pre-compiled rustc, cargo, stdlib) │
│ ├─ Installs to ~/.rustup/toolchains/ │
│ └─ Adds to PATH │
│ │
│ rustc --version → rustc 1.91.1 │
│ cargo --version → cargo 1.91.1 │
│ ✅ Now we can compile Rust! │
└────────┬───────────────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────────────┐
│ Phase 3: Install NuShell │
│ ├─ cargo install nu --locked │
│ │ (Compiles nu from Rust source) │
│ ├─ Installs to ~/.cargo/bin/nu │
│ └─ Adds to PATH │
│ │
│ ✅ Now we have a better shell! │
└────────┬───────────────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────────────┐
│ Phase 4: Install Just │
│ ├─ cargo install just --locked │
│ │ (Compiles just from Rust source) │
│ ├─ Installs to ~/.cargo/bin/just │
│ └─ Adds to PATH │
│ │
│ ✅ Now we have a task runner! │
└────────┬───────────────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────────────┐
│ Phase 5: Clone Repository │
│ ├─ git clone https://github.com/.../syntaxis │
│ └─ cd syntaxis │
│ │
│ ✅ Project source code ready │
└────────┬───────────────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────────────┐
│ Phase 6: Build Project │
│ ├─ cargo build --release --workspace │
│ │ │
│ │ Compiling: │
│ │ ├─ syntaxis-core (Rust library) │
│ │ ├─ syntaxis-cli (Rust binary) │
│ │ ├─ syntaxis-tui (Rust TUI binary) │
│ │ ├─ syntaxis-dashboard (Leptos WASM) │
│ │ ├─ Other 8+ crates... │
│ │ │
│ │ Installs binaries to target/release/ │
│ │ │
│ └─ Optimized Release Build (-O3, LTO, strip) │
│ │
│ ✅ All binaries compiled & ready │
└────────┬───────────────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────────────┐
│ Phase 7: Deploy Configuration │
│ ├─ Create ~/.config/core/ │
│ ├─ Deploy syntaxis-api.toml │
│ ├─ Deploy feature configurations (11 files) │
│ └─ Set permissions (600 for secrets) │
│ │
│ ✅ Configuration deployed & secured │
└────────┬───────────────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────────────┐
│ Phase 8: Verify Installation │
│ ├─ Run integration tests │
│ ├─ Verify binaries exist │
│ ├─ Test commands work │
│ └─ Check configuration validity │
│ │
│ ✅ Everything verified & working! │
└──────────────────────────────────────────────────────────────┘
Total Time: ~15-30 minutes (depending on your internet)
User Experience: One command, everything works!
Diagram 9: Why NOT to Pre-Compile
The Maintenance Nightmare of Pre-Compiling Everything:
Hypothetical: We pre-compile all binaries for all architectures
Every week, Rust releases a new version:
├─ Rust 1.91 → Rust 1.92
├─ Rust 1.92 → Rust 1.93
└─ New releases continuously
For each release, we'd need to:
1. Compile Rust for 50 architectures (50 machines/hours)
2. Compile NuShell for 50 architectures (50 machines/hours)
3. Compile Just for 50 architectures (50 machines/hours)
4. Compile syntaxis for 50 architectures (50 hours)
5. Run tests on each architecture
6. Upload 5GB+ of binaries to CDN
7. Update documentation
8. Handle bug reports (different for each arch)
WEEKLY MAINTENANCE COST: $1000s in compute + storage + bandwidth
vs. Bootstrap approach:
1. Rely on Rust Foundation to do steps 1-6
2. Every new Rust version automatically available
3. Users always get latest stable
4. Zero maintenance on our end
COST: Free (leverage Rust's infrastructure)
Diagram 10: The Complete Dependency Graph
Start: curl https://install.sh | bash
install.sh (bash script)
│
├─ Requires: bash, curl, tar, gzip
│ └─ ✅ Already on all systems
│
└─ Downloads: https://sh.rustup.rs (50KB bash)
│
├─ Requires: bash, curl, tar, gzip, sha256sum
│ └─ ✅ Already on all systems
│
└─ Downloads: https://static.rust-lang.org/dist/ (100MB)
│
├─ Pre-compiled rustc for your architecture
├─ Pre-compiled cargo
├─ Pre-compiled rustfmt, clippy, etc.
└─ Pre-compiled standard library
│
└─ Now rustc is available!
│
├─ Can compile Rust source
├─ Can run cargo
└─ Dependency graph multiplies:
│
├─ cargo install nu
│ └─ Compiles NuShell from Rust source
│
├─ cargo install just
│ └─ Compiles Just from Rust source
│
├─ cargo build --release (syntaxis)
│ └─ Compiles 14+ crates
│
├─ Scripts deployment
│ └─ Configuration files
│
└─ Verification
└─ Everything works!
EVERYTHING DEPENDS ON: rustc (one pre-compiled binary)
The rest is built by: rustc compiling Rust source
Diagram 11: The Bootstrap Kernel Concept
Think of it like a real "kernel" - the minimal core:
Traditional Operating System Bootstrap:
CPU powers on
│
├─ BIOS/UEFI (firmware on chip)
└─ Minimal code to load bootloader
│
└─ Bootloader loads kernel
│
└─ Kernel loads drivers & userspace
│
└─ Entire OS available
Rust Bootstrap Process:
You run: curl https://sh.rustup.rs | bash
│
├─ BIOS equivalent: bash shell (already on system)
└─ Bootloader equivalent: sh.rustup.rs (50KB script)
│
└─ Kernel equivalent: Pre-compiled rustc
│
└─ Drivers equivalent: Standard library
│
└─ Userspace equivalent: Compiled applications
├─ NuShell
├─ Just
└─ syntaxis
THE KERNEL: rustc (the bootstrap binary)
EVERYTHING ELSE: Built from source using the kernel
Diagram 12: How Targets Are Selected in Cargo
When you run: cargo build
1. Cargo wakes up and asks: "What target am I building for?"
cargo reads:
├─ Cargo.toml (no explicit target = use default)
├─ .cargo/config.toml (no override = use default)
├─ Environment variables (RUSTFLAGS, CARGO_BUILD_TARGET, etc.)
└─ Default = Current system (host platform)
2. To find the "current system":
cargo calls: rustc --print host-tuple
rustc responds:
├─ On M1 Mac: aarch64-apple-darwin
├─ On Intel Mac: x86_64-apple-darwin
├─ On Linux ARM: aarch64-unknown-linux-gnu
└─ On Linux Intel: x86_64-unknown-linux-gnu
3. Cargo then uses this triple to:
├─ Select pre-compiled standard library
│ From: ~/.rustup/toolchains/stable-<triple>/lib/rustlib/<triple>/lib/
│
├─ Select linker for that architecture
│
├─ Create output directory
│ target/<triple>/release/
│
└─ Pass architecture hints to LLVM compiler
4. Result:
Binaries in target/<triple>/release/ are built for that
exact architecture and ready to run on that system.
┌─────────────────────────────────────────────────────────────┐
│ KEY INSIGHT: │
│ │
│ No manual config needed! │
│ Cargo automatically detects and builds for your system. │
│ │
│ For cross-compilation (build for different arch): │
│ $ cargo build --target x86_64-unknown-linux-gnu │
│ $ cargo build --target aarch64-unknown-linux-gnu │
│ │
│ But for NATIVE compilation: Everything automatic! │
└─────────────────────────────────────────────────────────────┘
Summary: The Visual Story
- The Problem: Rust is written in Rust (circular dependency)
- The Solution: Pre-compile one binary (rustc)
- The Bootstrap: Simple bash script downloads pre-compiled rustc
- The Kernel: rustc becomes the seed for building everything else
- The Result: Complex ecosystem builds from one simple bootstrap
This is why Rust's installation process is so elegant and works across 50+ architectures without needing anything pre-compiled in your repository.