263 lines
9 KiB
Markdown
263 lines
9 KiB
Markdown
# ncl-sync
|
||
|
||
Nickel configuration sync daemon for the provisioning platform.
|
||
|
||
Watches workspace NCL source files, compiles them to JSON via `nickel export`, and maintains a shared file-based cache used by all Nu processes and the `nu_plugin_nickel` plugin.
|
||
|
||
---
|
||
|
||
## What Problem It Solves
|
||
|
||
Every `prvng` command that reads configuration calls `nickel export --format json` — 124 call sites across the Nu codebase, each taking 2–5 seconds. With ncl-sync, those exports happen once (proactively, in the background) and subsequent calls read from a JSON file in ~3 ms.
|
||
|
||
| Command | Without ncl-sync | With warm cache |
|
||
|---------|-----------------|-----------------|
|
||
| `prvng component list` | 3–7 s | ~1.5 s |
|
||
| `prvng workflow list` | 3–5 s | ~1.5 s |
|
||
| `prvng deploy` | 15–30 s | ~3–5 s |
|
||
| `prvng component show` (multi-export) | 12–30 s | ~1.5 s |
|
||
|
||
The 1.5 s floor is Nu module parse startup — a separate problem.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
### Workspace-local cache (single writer)
|
||
|
||
```
|
||
ncl-sync daemon (--workspace /ws/libre-daoshi)
|
||
│
|
||
├── notify watcher → NCL file changed
|
||
│ ↓
|
||
├── nickel export --format json $path --import-path ...
|
||
│ ↓
|
||
└── write /ws/libre-daoshi/.ncl-cache/<key>.json ← atomic (tmp → rename)
|
||
|
||
nu_plugin_nickel (nickel-eval command)
|
||
│
|
||
├── compute_cache_key(file_content, sorted_imports, "json")
|
||
│ ↓
|
||
├── resolve cache dir:
|
||
│ 1. $NCL_CACHE_DIR (explicit override)
|
||
│ 2. walk up from CWD → <workspace>/.ncl-cache/ (workspace-local)
|
||
│ 3. ~/.cache/provisioning/config-cache/ (global fallback)
|
||
│ ↓
|
||
├── does <cache_dir>/<key>.json exist?
|
||
│ ├── yes → open file, return Nu record (~3 ms)
|
||
│ └── no → nickel export, write to cache, return Nu record (~100 ms)
|
||
│
|
||
└── Nu script receives typed record — no | from json needed
|
||
|
||
Nu processes (after C1/C2/C3 migration)
|
||
│
|
||
└── ncl-eval $path [$ws $prov] → nickel-eval plugin → cache lookup
|
||
```
|
||
|
||
### Cache location policy
|
||
|
||
The cache lives **inside the workspace** by default. Three resolution levels, in order:
|
||
|
||
| Priority | Source | When |
|
||
|----------|--------|------|
|
||
| 1 | `$NCL_CACHE_DIR` env var | Explicit override (CI, tests, custom setups) |
|
||
| 2 | `<workspace>/.ncl-cache/` | Workspace is active (primary case) |
|
||
| 3 | `~/.cache/provisioning/config-cache/` | No workspace — shouldn't happen in normal use |
|
||
|
||
**Why workspace-local:**
|
||
- Cache is a derived artifact of the workspace — deleting the workspace deletes the cache
|
||
- No cross-workspace pollution when the same NCL filename exists in two workspaces with different import paths
|
||
- Per-workspace disk usage is easy to measure
|
||
- `.ncl-cache/` is gitignored (derived, regenerable)
|
||
|
||
**Three components stay aligned on the resolution order:**
|
||
- `ncl-sync/src/manifest.rs::resolve_cache_dir()`
|
||
- `nu_plugin_nickel/src/helpers.rs::get_cache_dir()`
|
||
- `lib_provisioning/config/cache/core.nu::get-cache-base-dir`
|
||
|
||
### Key derivation (shared between daemon and plugin)
|
||
|
||
Both ncl-sync and `nu_plugin_nickel` compute the cache key identically:
|
||
|
||
```
|
||
key = SHA256(file_content + sorted_import_paths.join(":") + format)
|
||
```
|
||
|
||
- **Content-based**: same file at a different path → same key. Move/rename don't invalidate the cache.
|
||
- **Import-path-aware**: same file with different import paths → different key. Correct when schemas resolve differently.
|
||
- **Sort-stable**: `--import-path [$a $b]` and `--import-path [$b $a]` produce the same key.
|
||
|
||
Verify parity between the two implementations:
|
||
|
||
```bash
|
||
nu tests/cache/test_key_parity.nu
|
||
```
|
||
|
||
### Manifest
|
||
|
||
The daemon maintains `<cache_dir>/manifest.json` — a map of `canonical_path → { cache_key, source_mtime, import_paths, cached_at }`. Used to detect stale entries on warm-up and after file changes. Each workspace has its own manifest at `<workspace>/.ncl-cache/manifest.json`.
|
||
|
||
### Sync requests
|
||
|
||
After a mutating operation, Nu writes `.sync-<pid>.json` (atomic rename from `.sync-<pid>.tmp`) to the cache dir. The daemon drains these sidecars every 500 ms:
|
||
|
||
```nushell
|
||
# Nu side (lib_provisioning/config/cache/core.nu)
|
||
write-sync-request [{ path: $file, import_paths: [$ws $prov] }]
|
||
```
|
||
|
||
---
|
||
|
||
## CLI
|
||
|
||
```
|
||
ncl-sync <SUBCOMMAND>
|
||
|
||
Subcommands:
|
||
daemon Start background daemon (watch + warm + serve sync requests)
|
||
warm One-shot: export all NCL in workspace, then exit
|
||
invalidate Remove a single file from cache
|
||
key Print cache key for (file, import_paths, format)
|
||
stats Print cache statistics
|
||
```
|
||
|
||
### daemon
|
||
|
||
```bash
|
||
ncl-sync daemon [--workspace PATH] [--idle-timeout SECS]
|
||
```
|
||
|
||
- Reads config from `PROVISIONING_CONFIG_DIR/ncl-sync.ncl` (via `platform-config`)
|
||
- Cache dir: `<workspace>/.ncl-cache/` (or `$NCL_CACHE_DIR` if set)
|
||
- Writes PID to `<cache_dir>/ncl-sync.pid`
|
||
- Auto-shuts down after `idle_timeout_secs` (default 600) with no file events or sync requests
|
||
- Env override: `NCL_SYNC_LOG=debug ncl-sync daemon` for verbose output
|
||
|
||
### warm
|
||
|
||
```bash
|
||
ncl-sync warm /path/to/workspace
|
||
```
|
||
|
||
Exports all `.ncl` files under the workspace path, up to `warm_concurrency` parallel processes. Updates manifest. Exits when done.
|
||
|
||
```bash
|
||
# Warm libre-daoshi before a work session
|
||
ncl-sync warm ~/workspaces/libre-daoshi
|
||
```
|
||
|
||
### invalidate
|
||
|
||
```bash
|
||
ncl-sync invalidate /path/to/file.ncl [--workspace PATH]
|
||
```
|
||
|
||
Removes the manifest entry and the corresponding `.json` cache file. `--workspace` resolves the cache location; if omitted, walks up from CWD looking for a workspace root.
|
||
|
||
### key
|
||
|
||
```bash
|
||
ncl-sync key /path/to/file.ncl [--import-path PATH]...
|
||
```
|
||
|
||
Prints the SHA256 key. Use to verify daemon–plugin parity:
|
||
|
||
```bash
|
||
sync_key=$(ncl-sync key settings.ncl --import-path $ws --import-path $prov)
|
||
nu_key=$(nu -c "use nu_plugin_nickel; nickel-cache-key settings.ncl --import-path [$ws $prov]")
|
||
[ "$sync_key" = "$nu_key" ] && echo "parity OK" || echo "PARITY MISMATCH"
|
||
```
|
||
|
||
### stats
|
||
|
||
```bash
|
||
ncl-sync stats [--workspace PATH]
|
||
# entries: 47
|
||
# stale: 0
|
||
# cache_dir: /Users/me/workspaces/libre-daoshi/.ncl-cache/
|
||
# size: 2.3 MB
|
||
```
|
||
|
||
`--workspace` selects which workspace's cache to inspect; if omitted, walks up from CWD.
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
Config file: `provisioning/platform/config/ncl-sync.ncl`
|
||
|
||
Deploy to `PROVISIONING_CONFIG_DIR` (macOS: `~/Library/Application Support/provisioning/platform/`):
|
||
|
||
```bash
|
||
cp provisioning/platform/config/ncl-sync.ncl \
|
||
~/Library/Application\ Support/provisioning/platform/ncl-sync.ncl
|
||
```
|
||
|
||
| Field | Default | Description |
|
||
|-------|---------|-------------|
|
||
| `cache_dir` | `<workspace>/.ncl-cache` | Override cache location (absolute path; bypasses workspace resolution) |
|
||
| `idle_timeout_secs` | `600` | Daemon auto-shutdown after N seconds idle |
|
||
| `sync_poll_interval_ms` | `500` | How often to check for `.sync-*.json` sidecars |
|
||
| `warm_concurrency` | `4` | Max parallel `nickel export` during warm-up |
|
||
| `extra_import_paths` | `[]` | Additional import paths beyond workspace + `$PROVISIONING` |
|
||
|
||
Env overrides: `NCL_CACHE_DIR` (path), `NCL_SYNC_IDLE_TIMEOUT`, `NCL_SYNC_CONCURRENCY`.
|
||
|
||
---
|
||
|
||
## Integration with nu_plugin_nickel
|
||
|
||
The daemon and plugin are complementary:
|
||
|
||
| Concern | ncl-sync | nu_plugin_nickel |
|
||
|---------|----------|-----------------|
|
||
| Cache writer | ✅ primary | ✅ on miss |
|
||
| Cache reader | — | ✅ always |
|
||
| File watching | ✅ | — |
|
||
| Proactive warm-up | ✅ | — |
|
||
| Nu value conversion | — | ✅ |
|
||
| Fallback on no daemon | — | ✅ (runs nickel directly) |
|
||
|
||
**Without ncl-sync running**: `nickel-eval` still works — it runs `nickel export` on miss and caches the result itself. Performance degrades to ~100 ms per call (first invocation per file per session).
|
||
|
||
**With ncl-sync running**: `nickel-eval` hits the daemon-written cache. First call of the session is already ~3 ms because the daemon warmed the cache on `prvng platform start`.
|
||
|
||
---
|
||
|
||
## Platform Lifecycle
|
||
|
||
ncl-sync is managed by the platform service layer:
|
||
|
||
```nushell
|
||
# service-manager.nu
|
||
prvng platform start # → ncl-sync-start → nohup ncl-sync daemon &
|
||
prvng platform stop # → ncl-sync-stop → kill $(cat ncl-sync.pid)
|
||
prvng platform status # → ncl-sync-status → { running, pid }
|
||
```
|
||
|
||
To start manually:
|
||
|
||
```bash
|
||
ncl-sync daemon --workspace ~/workspaces/libre-daoshi &
|
||
```
|
||
|
||
---
|
||
|
||
## Build & Install
|
||
|
||
```bash
|
||
cd provisioning/platform
|
||
cargo build --release --package ncl-sync
|
||
install -m 0755 target/release/provisioning-ncl-sync ~/.local/bin/provisioning-ncl-sync
|
||
```
|
||
|
||
---
|
||
|
||
## Design Constraints
|
||
|
||
- **No platform service dependencies**: no NATS, no SurrealDB, no orchestrator. The daemon configures those services — it cannot depend on them.
|
||
- **Single writer**: only ncl-sync writes `.json` cache files. Nu processes write only `.sync-*.json` sidecar signals.
|
||
- **Cache miss is safe**: if the daemon is down, `nickel-eval` falls back to direct `nickel export`. No hard dependency.
|
||
- **No HTTP/socket**: reads happen via the filesystem. Zero latency overhead, zero coupling.
|
||
|
||
See [ADR-022](../../adrs/adr-022-ncl-sync-daemon.ncl) for full design rationale.
|