Vapora/docs/adrs/0034-autonomous-scheduling.md

ADR-0034: Autonomous Cron Scheduling — Timezone Support and Distributed Fire-Lock

Status: Implemented Date: 2026-02-26 Deciders: VAPORA Team Technical Story: vapora-workflow-engine scheduler fired cron jobs only in UTC and had no protection against double-fires in multi-instance deployments.

---

Decision

Extend the autonomous scheduling subsystem with two independent hardening layers:

1. Timezone-aware scheduling (chrono-tz) — cron expressions evaluated in any IANA timezone, stored per-schedule, validated at API and config-load boundaries.
2. Distributed fire-lock — SurrealDB conditional UPDATE ... WHERE locked_by IS NONE OR locked_at < $expiry provides atomic, TTL-backed mutual exclusion across instances without additional infrastructure.

---

Context

Gaps Addressed

Gap	Consequence
UTC-only cron evaluation	"0 9 * * *" fires at 09:00 UTC regardless of business timezone; scheduled reports or maintenance windows drift by the UTC offset
No distributed coordination	Two vapora-workflow-engine instances reading the same scheduled_workflows table both fire the same schedule at the same tick

Why These Approaches

chrono-tz over manual UTC-offset arithmetic:

• Compile-time exhaustive enum of all IANA timezone names — invalid names are rejected at parse time.
• The cron crate's Schedule::upcoming(tz) / Schedule::after(&dt_in_tz) are generic over any TimeZone, so timezone-awareness requires no special-casing in iteration logic: pass DateTime<chrono_tz::Tz> instead of DateTime<Utc>, convert output with .with_timezone(&Utc).
• DST transitions handled automatically by chrono-tz — no application code needed.

SurrealDB conditional UPDATE over external distributed lock (Redis, etcd):

• No additional infrastructure dependency.
• SurrealDB applies document-level write locking; UPDATE record WHERE condition is atomic — two concurrent instances race on the same document and only one succeeds (non-empty return array = lock acquired).
• 120-second TTL enforced in application code: locked_at < $expiry in the WHERE clause auto-expires a lock from a crashed instance within two scheduler ticks.

---

Implementation

New Fields

scheduled_workflows table gains three columns (migration 011):

Field	Type	Purpose
timezone	option<string>	IANA identifier ("America/New_York") or NONE for UTC
locked_by	option<string>	UUID of the instance holding the current fire-lock
locked_at	option<datetime>	When the lock was acquired; used for TTL expiry

Lock Protocol

Tick N fires schedule S:
  try_acquire_fire_lock(id, instance_id, now)
    → UPDATE ... WHERE locked_by IS NONE OR locked_at < (now - 120s)
    → returns true (non-empty) or false (empty)
  if false: log + inc schedules_skipped, return
  fire_with_lock(S, now)         ← actual workflow start
  release_fire_lock(id, instance_id)
    → UPDATE ... WHERE locked_by = instance_id
    → own-instance guard prevents stale release

Lock release is always attempted even on fire_with_lock error; a warn! is emitted if release fails (TTL provides fallback).

Timezone-Aware Cron Evaluation

compute_fire_times_tz(schedule, last, now, catch_up, tz):
  match tz.parse::<chrono_tz::Tz>():
    Some(tz) → schedule.after(&last.with_timezone(&tz))
                        .take_while(|t| t.with_timezone(&Utc) <= now)
                        .map(|t| t.with_timezone(&Utc))
    None     → schedule.after(&last)     ← UTC

Parsing an unknown/invalid timezone string silently falls back to UTC — avoids a hard error at runtime if a previously valid TZ identifier is removed from the chrono-tz database in a future upgrade.

API Surface Changes

PUT /api/v1/schedules/:id and PATCH /api/v1/schedules/:id accept and return timezone: Option<String>. Timezone is validated at the API boundary using validate_timezone() (returns 400 InvalidInput for unknown identifiers). Config-file [schedule] blocks also accept timezone and are validated at startup (fail-fast, same as cron).

---

Consequences

Positive

• Schedules expressed in business-local time — no mental UTC arithmetic for operators.
• Multi-instance deployments safe by default; no external lock service required.
• ScheduledWorkflow.timezone is nullable/optional — all existing schedules without the field default to UTC with no migration required.

Negative / Trade-offs

• chrono-tz adds ~2 MB of IANA timezone data to the binary (compile-time embedded).
• Distributed lock TTL of 120 s means a worst-case window of one double-fire per 120 s if the winning instance crashes between acquiring the lock and calling update_after_fire. Acceptable given the schedule_runs audit log makes duplicates visible.
• No multi-PATCH for timezone clearance: passing timezone: null in JSON is treated as absent (#[serde(default)]). Clearing timezone (revert to UTC) requires a full PUT.