prvng_platform/crates/ncl-sync/README.md

# ncl-sync

Nickel configuration sync daemon for the provisioning platform.

Watches workspace NCL source files, compiles them to JSON via `nickel export`, and maintains a shared file-based cache used by all Nu processes and the `nu_plugin_nickel` plugin.

---

## What Problem It Solves

Every `prvng` command that reads configuration calls `nickel export --format json` — 124 call sites across the Nu codebase, each taking 2–5 seconds. With ncl-sync, those exports happen once (proactively, in the background) and subsequent calls read from a JSON file in ~3 ms.

| Command | Without ncl-sync | With warm cache |
|---------|-----------------|-----------------|
| `prvng component list` | 3–7 s | ~1.5 s |
| `prvng workflow list` | 3–5 s | ~1.5 s |
| `prvng deploy` | 15–30 s | ~3–5 s |
| `prvng component show` (multi-export) | 12–30 s | ~1.5 s |

The 1.5 s floor is Nu module parse startup — a separate problem.

---

## Architecture

### Workspace-local cache (single writer)

```
ncl-sync daemon (--workspace /ws/libre-daoshi)
  │
  ├── notify watcher → NCL file changed
  │       ↓
  ├── nickel export --format json $path --import-path ...
  │       ↓
  └── write /ws/libre-daoshi/.ncl-cache/<key>.json  ← atomic (tmp → rename)

nu_plugin_nickel (nickel-eval command)
  │
  ├── compute_cache_key(file_content, sorted_imports, "json")
  │       ↓
  ├── resolve cache dir:
  │     1. $NCL_CACHE_DIR                           (explicit override)
  │     2. walk up from CWD → <workspace>/.ncl-cache/   (workspace-local)
  │     3. ~/.cache/provisioning/config-cache/      (global fallback)
  │       ↓
  ├── does <cache_dir>/<key>.json exist?
  │   ├── yes → open file, return Nu record  (~3 ms)
  │   └── no  → nickel export, write to cache, return Nu record  (~100 ms)
  │
  └── Nu script receives typed record — no | from json needed

Nu processes (after C1/C2/C3 migration)
  │
  └── ncl-eval $path [$ws $prov]  →  nickel-eval plugin  →  cache lookup
```

### Cache location policy

The cache lives **inside the workspace** by default. Three resolution levels, in order:

| Priority | Source | When |
|----------|--------|------|
| 1 | `$NCL_CACHE_DIR` env var | Explicit override (CI, tests, custom setups) |
| 2 | `<workspace>/.ncl-cache/` | Workspace is active (primary case) |
| 3 | `~/.cache/provisioning/config-cache/` | No workspace — shouldn't happen in normal use |

**Why workspace-local:**
- Cache is a derived artifact of the workspace — deleting the workspace deletes the cache
- No cross-workspace pollution when the same NCL filename exists in two workspaces with different import paths
- Per-workspace disk usage is easy to measure
- `.ncl-cache/` is gitignored (derived, regenerable)

**Three components stay aligned on the resolution order:**
- `ncl-sync/src/manifest.rs::resolve_cache_dir()`
- `nu_plugin_nickel/src/helpers.rs::get_cache_dir()`
- `lib_provisioning/config/cache/core.nu::get-cache-base-dir`

### Key derivation (shared between daemon and plugin)

Both ncl-sync and `nu_plugin_nickel` compute the cache key identically:

```
key = SHA256(file_content + sorted_import_paths.join(":") + format)
```

- **Content-based**: same file at a different path → same key. Move/rename don't invalidate the cache.
- **Import-path-aware**: same file with different import paths → different key. Correct when schemas resolve differently.
- **Sort-stable**: `--import-path [$a $b]` and `--import-path [$b $a]` produce the same key.

Verify parity between the two implementations:

```bash
nu tests/cache/test_key_parity.nu
```

### Manifest

The daemon maintains `<cache_dir>/manifest.json` — a map of `canonical_path → { cache_key, source_mtime, import_paths, cached_at }`. Used to detect stale entries on warm-up and after file changes. Each workspace has its own manifest at `<workspace>/.ncl-cache/manifest.json`.

### Sync requests

After a mutating operation, Nu writes `.sync-<pid>.json` (atomic rename from `.sync-<pid>.tmp`) to the cache dir. The daemon drains these sidecars every 500 ms:

```nushell
# Nu side (lib_provisioning/config/cache/core.nu)
write-sync-request [{ path: $file, import_paths: [$ws $prov] }]
```

---

## CLI

```
ncl-sync <SUBCOMMAND>

Subcommands:
  daemon      Start background daemon (watch + warm + serve sync requests)
  warm        One-shot: export all NCL in workspace, then exit
  invalidate  Remove a single file from cache
  key         Print cache key for (file, import_paths, format)
  stats       Print cache statistics
```

### daemon

```bash
ncl-sync daemon [--workspace PATH] [--idle-timeout SECS]
```

- Reads config from `PROVISIONING_CONFIG_DIR/ncl-sync.ncl` (via `platform-config`)
- Cache dir: `<workspace>/.ncl-cache/` (or `$NCL_CACHE_DIR` if set)
- Writes PID to `<cache_dir>/ncl-sync.pid`
- Auto-shuts down after `idle_timeout_secs` (default 600) with no file events or sync requests
- Env override: `NCL_SYNC_LOG=debug ncl-sync daemon` for verbose output

### warm

```bash
ncl-sync warm /path/to/workspace
```

Exports all `.ncl` files under the workspace path, up to `warm_concurrency` parallel processes. Updates manifest. Exits when done.

```bash
# Warm libre-daoshi before a work session
ncl-sync warm ~/workspaces/libre-daoshi
```

### invalidate

```bash
ncl-sync invalidate /path/to/file.ncl [--workspace PATH]
```

Removes the manifest entry and the corresponding `.json` cache file. `--workspace` resolves the cache location; if omitted, walks up from CWD looking for a workspace root.

### key

```bash
ncl-sync key /path/to/file.ncl [--import-path PATH]...
```

Prints the SHA256 key. Use to verify daemon–plugin parity:

```bash
sync_key=$(ncl-sync key settings.ncl --import-path $ws --import-path $prov)
nu_key=$(nu -c "use nu_plugin_nickel; nickel-cache-key settings.ncl --import-path [$ws $prov]")
[ "$sync_key" = "$nu_key" ] && echo "parity OK" || echo "PARITY MISMATCH"
```

### stats

```bash
ncl-sync stats [--workspace PATH]
# entries:   47
# stale:     0
# cache_dir: /Users/me/workspaces/libre-daoshi/.ncl-cache/
# size:      2.3 MB
```

`--workspace` selects which workspace's cache to inspect; if omitted, walks up from CWD.

---

## Configuration

Config file: `provisioning/platform/config/ncl-sync.ncl`

Deploy to `PROVISIONING_CONFIG_DIR` (macOS: `~/Library/Application Support/provisioning/platform/`):

```bash
cp provisioning/platform/config/ncl-sync.ncl \
   ~/Library/Application\ Support/provisioning/platform/ncl-sync.ncl
```

| Field | Default | Description |
|-------|---------|-------------|
| `cache_dir` | `<workspace>/.ncl-cache` | Override cache location (absolute path; bypasses workspace resolution) |
| `idle_timeout_secs` | `600` | Daemon auto-shutdown after N seconds idle |
| `sync_poll_interval_ms` | `500` | How often to check for `.sync-*.json` sidecars |
| `warm_concurrency` | `4` | Max parallel `nickel export` during warm-up |
| `extra_import_paths` | `[]` | Additional import paths beyond workspace + `$PROVISIONING` |

Env overrides: `NCL_CACHE_DIR` (path), `NCL_SYNC_IDLE_TIMEOUT`, `NCL_SYNC_CONCURRENCY`.

---

## Integration with nu_plugin_nickel

The daemon and plugin are complementary:

| Concern | ncl-sync | nu_plugin_nickel |
|---------|----------|-----------------|
| Cache writer | ✅ primary | ✅ on miss |
| Cache reader | — | ✅ always |
| File watching | ✅ | — |
| Proactive warm-up | ✅ | — |
| Nu value conversion | — | ✅ |
| Fallback on no daemon | — | ✅ (runs nickel directly) |

**Without ncl-sync running**: `nickel-eval` still works — it runs `nickel export` on miss and caches the result itself. Performance degrades to ~100 ms per call (first invocation per file per session).

**With ncl-sync running**: `nickel-eval` hits the daemon-written cache. First call of the session is already ~3 ms because the daemon warmed the cache on `prvng platform start`.

---

## Platform Lifecycle

ncl-sync is managed by the platform service layer:

```nushell
# service-manager.nu
prvng platform start   # → ncl-sync-start → nohup ncl-sync daemon &
prvng platform stop    # → ncl-sync-stop  → kill $(cat ncl-sync.pid)
prvng platform status  # → ncl-sync-status → { running, pid }
```

To start manually:

```bash
ncl-sync daemon --workspace ~/workspaces/libre-daoshi &
```

---

## Build & Install

```bash
cd provisioning/platform
cargo build --release --package ncl-sync
install -m 0755 target/release/provisioning-ncl-sync ~/.local/bin/provisioning-ncl-sync
```

---

## Design Constraints

- **No platform service dependencies**: no NATS, no SurrealDB, no orchestrator. The daemon configures those services — it cannot depend on them.
- **Single writer**: only ncl-sync writes `.json` cache files. Nu processes write only `.sync-*.json` sidecar signals.
- **Cache miss is safe**: if the daemon is down, `nickel-eval` falls back to direct `nickel export`. No hard dependency.
- **No HTTP/socket**: reads happen via the filesystem. Zero latency overhead, zero coupling.

See [ADR-022](../../adrs/adr-022-ncl-sync-daemon.ncl) for full design rationale.