prvng_platform/crates/ncl-sync/README.md

ncl-sync

Nickel configuration sync daemon for the provisioning platform.

Watches workspace NCL source files, compiles them to JSON via nickel export, and maintains a shared file-based cache used by all Nu processes and the nu_plugin_nickel plugin.

---

What Problem It Solves

Every prvng command that reads configuration calls nickel export --format json — 124 call sites across the Nu codebase, each taking 2–5 seconds. With ncl-sync, those exports happen once (proactively, in the background) and subsequent calls read from a JSON file in ~3 ms.

Command	Without ncl-sync	With warm cache
prvng component list	3–7 s	~1.5 s
prvng workflow list	3–5 s	~1.5 s
prvng deploy	15–30 s	~3–5 s
prvng component show (multi-export)	12–30 s	~1.5 s

The 1.5 s floor is Nu module parse startup — a separate problem.

---

Architecture

Workspace-local cache (single writer)

ncl-sync daemon (--workspace /ws/libre-daoshi)
  │
  ├── notify watcher → NCL file changed
  │       ↓
  ├── nickel export --format json $path --import-path ...
  │       ↓
  └── write /ws/libre-daoshi/.ncl-cache/<key>.json  ← atomic (tmp → rename)

nu_plugin_nickel (nickel-eval command)
  │
  ├── compute_cache_key(file_content, sorted_imports, "json")
  │       ↓
  ├── resolve cache dir:
  │     1. $NCL_CACHE_DIR                           (explicit override)
  │     2. walk up from CWD → <workspace>/.ncl-cache/   (workspace-local)
  │     3. ~/.cache/provisioning/config-cache/      (global fallback)
  │       ↓
  ├── does <cache_dir>/<key>.json exist?
  │   ├── yes → open file, return Nu record  (~3 ms)
  │   └── no  → nickel export, write to cache, return Nu record  (~100 ms)
  │
  └── Nu script receives typed record — no | from json needed

Nu processes (after C1/C2/C3 migration)
  │
  └── ncl-eval $path [$ws $prov]  →  nickel-eval plugin  →  cache lookup

Cache location policy

The cache lives inside the workspace by default. Three resolution levels, in order:

Priority	Source	When
1	$NCL_CACHE_DIR env var	Explicit override (CI, tests, custom setups)
2	<workspace>/.ncl-cache/	Workspace is active (primary case)
3	~/.cache/provisioning/config-cache/	No workspace — shouldn't happen in normal use

Why workspace-local:

• Cache is a derived artifact of the workspace — deleting the workspace deletes the cache
• No cross-workspace pollution when the same NCL filename exists in two workspaces with different import paths
• Per-workspace disk usage is easy to measure
• .ncl-cache/ is gitignored (derived, regenerable)

Three components stay aligned on the resolution order:

• ncl-sync/src/manifest.rs::resolve_cache_dir()
• nu_plugin_nickel/src/helpers.rs::get_cache_dir()
• lib_provisioning/config/cache/core.nu::get-cache-base-dir

Key derivation (shared between daemon and plugin)

Both ncl-sync and nu_plugin_nickel compute the cache key identically:

key = SHA256(file_content + sorted_import_paths.join(":") + format)

• Content-based: same file at a different path → same key. Move/rename don't invalidate the cache.
• Import-path-aware: same file with different import paths → different key. Correct when schemas resolve differently.
• Sort-stable: --import-path [$a $b] and --import-path [$b $a] produce the same key.

Verify parity between the two implementations:

nu tests/cache/test_key_parity.nu

Manifest

The daemon maintains <cache_dir>/manifest.json — a map of canonical_path → { cache_key, source_mtime, import_paths, cached_at }. Used to detect stale entries on warm-up and after file changes. Each workspace has its own manifest at <workspace>/.ncl-cache/manifest.json.

Sync requests

After a mutating operation, Nu writes .sync-<pid>.json (atomic rename from .sync-<pid>.tmp) to the cache dir. The daemon drains these sidecars every 500 ms:

# Nu side (lib_provisioning/config/cache/core.nu)
write-sync-request [{ path: $file, import_paths: [$ws $prov] }]

---

CLI

ncl-sync <SUBCOMMAND>

Subcommands:
  daemon      Start background daemon (watch + warm + serve sync requests)
  warm        One-shot: export all NCL in workspace, then exit
  invalidate  Remove a single file from cache
  key         Print cache key for (file, import_paths, format)
  stats       Print cache statistics

daemon

ncl-sync daemon [--workspace PATH] [--idle-timeout SECS]

• Reads config from PROVISIONING_CONFIG_DIR/ncl-sync.ncl (via platform-config)
• Cache dir: <workspace>/.ncl-cache/ (or $NCL_CACHE_DIR if set)
• Writes PID to <cache_dir>/ncl-sync.pid
• Auto-shuts down after idle_timeout_secs (default 600) with no file events or sync requests
• Env override: NCL_SYNC_LOG=debug ncl-sync daemon for verbose output

warm

ncl-sync warm /path/to/workspace

Exports all .ncl files under the workspace path, up to warm_concurrency parallel processes. Updates manifest. Exits when done.

# Warm libre-daoshi before a work session
ncl-sync warm ~/workspaces/libre-daoshi

invalidate

ncl-sync invalidate /path/to/file.ncl [--workspace PATH]

Removes the manifest entry and the corresponding .json cache file. --workspace resolves the cache location; if omitted, walks up from CWD looking for a workspace root.

key

ncl-sync key /path/to/file.ncl [--import-path PATH]...

Prints the SHA256 key. Use to verify daemon–plugin parity:

sync_key=$(ncl-sync key settings.ncl --import-path $ws --import-path $prov)
nu_key=$(nu -c "use nu_plugin_nickel; nickel-cache-key settings.ncl --import-path [$ws $prov]")
[ "$sync_key" = "$nu_key" ] && echo "parity OK" || echo "PARITY MISMATCH"

stats

ncl-sync stats [--workspace PATH]
# entries:   47
# stale:     0
# cache_dir: /Users/me/workspaces/libre-daoshi/.ncl-cache/
# size:      2.3 MB

--workspace selects which workspace's cache to inspect; if omitted, walks up from CWD.

---

Configuration

Config file: provisioning/platform/config/ncl-sync.ncl

Deploy to PROVISIONING_CONFIG_DIR (macOS: ~/Library/Application Support/provisioning/platform/):

cp provisioning/platform/config/ncl-sync.ncl \
   ~/Library/Application\ Support/provisioning/platform/ncl-sync.ncl

Field	Default	Description
cache_dir	<workspace>/.ncl-cache	Override cache location (absolute path; bypasses workspace resolution)
idle_timeout_secs	600	Daemon auto-shutdown after N seconds idle
sync_poll_interval_ms	500	How often to check for .sync-*.json sidecars
warm_concurrency	4	Max parallel nickel export during warm-up
extra_import_paths	[]	Additional import paths beyond workspace + $PROVISIONING

Env overrides: NCL_CACHE_DIR (path), NCL_SYNC_IDLE_TIMEOUT, NCL_SYNC_CONCURRENCY.

---

Integration with nu_plugin_nickel

The daemon and plugin are complementary:

Concern	ncl-sync	nu_plugin_nickel
Cache writer	✅ primary	✅ on miss
Cache reader	—	✅ always
File watching	✅	—
Proactive warm-up	✅	—
Nu value conversion	—	✅
Fallback on no daemon	—	✅ (runs nickel directly)

Without ncl-sync running: nickel-eval still works — it runs nickel export on miss and caches the result itself. Performance degrades to ~100 ms per call (first invocation per file per session).

With ncl-sync running: nickel-eval hits the daemon-written cache. First call of the session is already ~3 ms because the daemon warmed the cache on prvng platform start.

---

Platform Lifecycle

ncl-sync is managed by the platform service layer:

# service-manager.nu
prvng platform start   # → ncl-sync-start → nohup ncl-sync daemon &
prvng platform stop    # → ncl-sync-stop  → kill $(cat ncl-sync.pid)
prvng platform status  # → ncl-sync-status → { running, pid }

To start manually:

ncl-sync daemon --workspace ~/workspaces/libre-daoshi &

---

Build & Install

cd provisioning/platform
cargo build --release --package ncl-sync
install -m 0755 target/release/provisioning-ncl-sync ~/.local/bin/provisioning-ncl-sync

---

Design Constraints

• No platform service dependencies: no NATS, no SurrealDB, no orchestrator. The daemon configures those services — it cannot depend on them.
• Single writer: only ncl-sync writes .json cache files. Nu processes write only .sync-*.json sidecar signals.
• Cache miss is safe: if the daemon is down, nickel-eval falls back to direct nickel export. No hard dependency.
• No HTTP/socket: reads happen via the filesystem. Zero latency overhead, zero coupling.

See ADR-022 for full design rationale.