Vantage is the embedded, server-rendered dashboard bundled with autumn-harvest-plugin. It mounts at the path configured by HarvestPlugin::api (e.g. /api/harvest/ui) and requires no external assets or CDN — all CSS is inlined. The workflow list and detail pages load one same-origin script for live refresh (see Live refresh). No other page loads a script.

Pages

Workflow list — /workflows

Displays a paginated table of workflow executions across all configured shards.

Filter controls

ControlQuery paramNotes
StatestateMulti-value; defaults to all states
Workflow nameworkflow_nameExact match
Started afterstarted_afterISO-8601 datetime string (e.g. 2026-01-01T00:00:00Z)
Started beforestarted_beforeISO-8601 datetime string
Execution ID searchexec_id_searchPrefix match on the execution UUID
FilterfilterAn expression with AND, OR and parentheses, for example attrs.phase = 'blocked' OR attrs.amount > 1000. The list ignores a bad expression and shows its error below the field. See Filter expressions.

Combine any number of filters in a single URL. Hitting Reset clears all filters and returns to the default view.

Workflow detail — /workflows/{exec_id}

Shows a single workflow execution in full detail.

Metadata card — execution ID, workflow/run IDs, shard, queue, start/complete timestamps, duration, parent execution link, sticky worker, and execution timeout.

Input / Output / Memo / Search attributes — JSON cards rendered below the metadata.

Activity attempts panel — grouped by activity_id; shows the activity function name, attempt count, last status, and last error (if any). Hidden when there are no activity events.

Children panel — lists child workflow executions that have parent_id pointing at this execution. Hidden when there are none.

Signals & Updates panel — shows SignalReceived, UpdateAdmitted, UpdateCompleted, and UpdateFailed events with type, signal/update name, and timestamp. Capped at 20 entries; the heading shows the total when truncated. Hidden when there are none.

Event timeline — paginated at 100 events per page. Shows a human-readable label alongside the raw event type code and timestamp. Pagination controls appear above and below the table for large histories.

Blocked-on panel — for RUNNING workflows, shows pending task queue entries (activities), pending timers, and pending signals. Terminal workflows show "No pending work items." instead.

Operator actions

  • Cancel — POST to /workflows/{exec_id}/cancel with a confirmation dialog. Redirects to the detail page with a flash message.

  • Terminate — POST to /workflows/{exec_id}/terminate with a confirmation dialog. It seals the run as TERMINATED. The button is disabled once the workflow is terminal, and the route requires admin auth. Redirects to the detail page with a flash message.

  • Send signal — Expandable form (collapsed by default). POST to /workflows/{exec_id}/signal with signal_name and an optional JSON payload. Redirects with flash on success.

  • Reset to event N — Expandable form. POST to /workflows/{exec_id}/reset with reset_to_event_id (1-based event number, as shown in the timeline "#" column) and optional reason. Creates a new fork execution. Redirects with flash on success. See Workflow reset.

  • Trigger update — Expandable form. POST to /workflows/{exec_id}/trigger-update with update_name and optional JSON payload. Redirects with flash on success.

    On a rejected submission (malformed JSON payload, non-numeric event number, or the engine call itself failing), all three forms instead render the detail page directly as the POST's own response — no redirect — with the rejected form open, the operator's entered values pre-filled, and the error shown inline next to the field that caused it. A typo does not cost the whole entry. This is deliberately not a redirect: putting a signal or update payload in a redirect's query string would put it in browser history, proxy/server access logs, and the same-origin referrer, and a large payload could push the URL past a typical request-line limit. Because the response is served from one path segment below the canonical /workflows/{exec_id} page, it carries a <base href=".."> element so every relative link and form action on the page keeps resolving correctly. A <base> element only fixes path-relative references, not a bare ?query reference (which inherits the browser's whole current base path, not just its directory) — so the event-timeline pagination links and the "Jump to event" form's action are execution-id-prefixed explicitly, rather than relying on that distinction.

  • Export history — GET link to …/workflows/{exec_id}/history/export for the full JSON event log.

Event timeline collapsible payload — each event row in the history table has a <details> element. Click "view payload" to expand and see the full JSON event data.

Jump to event N — when the history has more than 100 events, a number input labeled "Jump to event:" appears alongside the pagination controls. Enter a 1-based event number and click Go to navigate to the page containing that event.

Status badges include aria-label="Status: {state}" and role="status" for screen-reader accessibility.

Schedules — /schedules

The operator surface for cron/interval schedules: one row per schedule across every shard, with the health, policy and bounded-run state needed to answer "is this schedule healthy, when does it fire next, and how did its last runs end?" without composing API calls by hand.

Every field is read from the existing GET /admin/schedules response fields, and every mutation goes through an already-audited endpoint. The page adds no new backend endpoint, no new WorkflowEvent variant, and no migration.

List columns

ColumnSourceNotes
Schedule IDharvest_schedules.idTruncated; links to the drill-downs
Kind / Targetdag_name / workflow_nameDag or Workflow
Expressionschedule_exprCron or interval:*
TimezonetimezoneUTC is subdued; other zones get a badge
Next Runnext_run_atPlus the jitter-adjusted effective fire time and the jitter window when jitter_secs > 0
Last Runlast_run_at
HealthderivedSee badges below
Overlapoverlap_policyBuffering policies also show buffered <n> (and /<cap> under buffer_all)
Catchupcatchup_policyEffective policy, the window duration, and dropped <n> from the most recent recovery tick
Bounded runsmax_runs / runs_started / end_at / exhausted_reason<remaining> of <max> left · ends <ts> · exhausted: <reason>. max_runs = 0 is the engine's unlimited (every bound check guards on max > 0), so no budget is rendered for it.
Createdcreated_at
Shardfan-outMulti-shard deployments only

Health badges and ordering

A healthy schedule renders one calm Active badge. An unhealthy one renders a badge per condition:

BadgeCondition
Pausedis_paused
Auto-pausedauto_paused_at set — supersedes Paused
Exhausted: <reason>exhausted_at set, or the row is terminal on its live bounds before a tick has stamped the column: runs_started >= max_runs (for max_runs > 0), or the pending slot is at/past the cutoff (next_run_at >= end_at; falling back to now >= end_at only when there is no pending slot). An unstamped exhaustion names its bound (run budget spent / past end_at). The pending slot includes jitter. Rows with a calendar, or with MostRecent or Window catchup, use the raw next_run_at. The scheduler may stop such a row before the badge shows.
Catchup dropped ×Nlast_catchup_dropped > 0

Unhealthy schedules sort above healthy ones; healthy rows keep their existing next_run_at-ascending order among themselves. A "Needs attention" strip at the top of the page counts each bucket, and the Health filter (?health=Unhealthy / ?health=Healthy) narrows the list. An unrecognised value is a 400.

Filters — target, kind, paused, health, shard_id, limit, refresh. All of them round-trip into the bulk pause/resume actions, and both bulk handlers apply the same ScheduleUiFilters::matches predicate the list does — so "Pause all matching (N)" acts on exactly the N rows on screen.

Per-row actions — Pause, Resume, Run now, and Delete, each behind a confirmation dialog and each writing an audit record with source = ui and the matching operation (schedule.pause, schedule.resume, schedule.trigger, schedule.delete). The UI adds no unaudited mutation path.

A row is offered Resume when is_paused or auto_paused_at is set. The scheduler's auto-pause sets auto_paused_at without setting is_paused, so a row can be non-firing with is_paused = false; keying the action on is_paused alone would show an "Auto-paused" badge next to a Pause button and leave no way to restore firing. Resume — per row and in bulk — clears auto_paused_at and resets consecutive_failure_count, exactly as POST /admin/schedules/{id}/resume does, so the next tick does not immediately re-pause the schedule.

The paused filter matches the is_paused column only; use health=Unhealthy to find auto-paused schedules alongside the other non-firing states.

Fire-time preview — /schedules/{id}/preview

Renders the next N fire times from the same computation as GET /admin/schedules/{id}/preview, so the two can never disagree:

  • Scheduled (the raw cron instant), Local (in the schedule's timezone), and Effective (after calendar adjustment and jitter) side by side.
  • Calendar-excluded entries show suppressed with reason skipped:calendar-excluded; rebased entries show deferred:calendar.
  • The jitter window ([scheduled_at, scheduled_at + jitter_secs]) per entry.
  • An advisory Overlap risk flag when skip/buffer_one could drop the firing.
  • Bounded-run truncation is applied, and a preview that comes back empty says why: paused (with the pause reason), auto-paused (#360 — the scheduler's due-list filters on auto_paused_at IS NULL, so such a schedule will not fire even though its is_paused column is false), exhausted (with exhausted_reason), the end_at cutoff, or an expression with no future firings.

?count=N (1–100, default 10) controls how many entries are projected.

Run history — /schedules/{id}/runs

Consumes GET /admin/schedules/{id}/runs. Newest-slot-first rows with nominal_fire_time (— (no slot) for a manual trigger), started/completed timestamps, a terminal state badge, origin (scheduled / backfill / manual_trigger), the first line of a terminal failure, and a link to the execution detail view.

Above the table, the scheduled-run summary counts scheduled-origin runs only, so a backfill storm or an ad-hoc trigger never inflates the failure ratio.

A filter bar exposes limit (1–200), origin, and state. The Next link carries the active filters alongside the cursor, because a keyset cursor is only meaningful under the filters it was computed with.

Cross-shard degradation is always visible, never silent:

Response statusRendering
completeNo banner
partial"Some shards unreachable" banner listing each unavailable shard and its error; the summary is flagged as possibly understated
unavailable"No shard could be reached" banner; no "no runs yet" message, because nothing could be read

A schedule that exists but has never run renders an explicit "No runs yet." card. Paging uses the endpoint's own keyset cursor (?limit=, ?cursor=).

This page is admin-gated, matching GET /admin/schedules/{id}/runs — the one schedule read route the management API gates.

Backfill launcher — /schedules/{id}/backfill

A two-stage flow over POST /admin/schedules/{id}/backfill:

  1. GET renders the window form (start, end, max slots, include-paused). This is a pure read — it dispatches nothing.
  2. Submitting posts stage=preview, which runs the endpoint's dry run and renders the planned slot count, the would-dispatch and would-skip counts, the machine-readable skip reasons, and the first 20 planned fire times.
  3. Only an explicit stage=commit (behind a confirm() dialog) dispatches. A POST that omits stage falls back to the dry run, so a bare form post can never dispatch work.
  4. On success the operator is redirected to that schedule's run history, which renders a flash summarising what was dispatched — including the failure count, which is reported nowhere else.

The submitted window is normalised with SecondsFormat::AutoSi, preserving sub-second precision: an interval: backfill treats from as its first slot, so truncating …00.900Z to …00Z would shift every slot in the plan rather than merely respelling the window.

max_count is capped at the engine's DEFAULT_BACKFILL_MAX_COUNT (1,000). The endpoint treats a supplied max_count as the planning limit that replaces its own default, so an uncapped value from a browser form could make one request enumerate millions of timestamps.

A paused DAG schedule cannot be committed at all (the endpoint rejects it, because backfilled runs would sit QUEUED until the schedule resumes), so the confirmation says so instead of offering a button that can only fail. A rejected submission re-renders the form with the operator's own window still in it.

The dry run is a POST rather than a GET because it writes a harvest_backfill_log row and an audit record — the read path stays side-effect-free. Both stages are audited through the API handler with source = ui, and every guard the endpoint enforces (paused, exhausted, max_active_runs, max_runs, window size) applies unchanged; a rejection is rendered on the form rather than as a bare error page.

A malformed or inverted window is a rendered form error, never a 500.

Not-found vs. indeterminate

All three drill-downs resolve the schedule through the API's own resolve_schedule_with_shard, so an unparseable id is a 400, "checked every expected shard and none had it" is a 404, and "a shard could not be checked" is a 503 — never a 404 claiming a schedule was deleted when a shard was merely unreachable.

load_schedule_by_id_with_shard — the lookup behind the preview and backfill endpoints — delegates to the same resolver, so a schedule on a healthy shard is found even while an earlier shard is down, and a not-found result during an outage is a 503 rather than a 404. The preview additionally passes its already-resolved row to compute_schedule_preview_for, avoiding a second lookup entirely.

Out of scope (tracked elsewhere): creating or editing schedules from the UI, overdue-schedule detection, recent-runs enrichment.

Event label mapping

Vantage translates raw WorkflowEvent type strings to friendlier labels:

Event typeLabel
WorkflowStartedWorkflow started
WorkflowCompletedWorkflow completed
WorkflowFailedWorkflow failed
WorkflowCancelledWorkflow cancelled
WorkflowCancelRequestedCancel requested (deferred)
ActivityScheduledActivity scheduled: {name}
ActivityCompletedActivity completed
ActivityFailedActivity failed: {error} (truncated)
SignalReceivedSignal received: {signal_name}
UpdateAdmittedUpdate admitted
DecisionCommittedDecision committed: build {build_id}, worker {worker_id} (<none> for an empty build, ? for a missing worker id)
TimerStartedTimer started
TimerFiredTimer fired
LocalActivityScheduledLocal activity scheduled
VersionMarkerVersion marker
ContinueAsNewContinue as new
(other)Raw type string

Event fields are extracted from the inner data object of the adjacently-tagged serde envelope {"type": "...", "data": {...}}.

RouteDescription
/Redirects to /workflows
/workflowsWorkflow list
/workflows/{exec_id}Workflow detail
/workersWorker fleet
/schedulesCron/interval schedules
/schedules/{id}/previewNext-fire-time preview for one schedule
/schedules/{id}/runsPer-schedule run history (admin)
/schedules/{id}/backfillBackfill launcher (form → dry run → commit)
/dagsDAG list
/dags/{dag_name}DAG detail
/dlqDead letter queue

Auto-refresh

/workers, /schedules, /dlq, and /dags/{dag_name} (DAG detail) take a refresh query parameter (30 or 60 seconds, or a custom value) that emits a <meta http-equiv="refresh"> tag — the page reloads itself over plain HTTP at that interval. /dags (the DAG list) has no refresh parameter — ?refresh=30 there is silently ignored. The workflow list and detail pages use live refresh instead.

Live refresh

The workflow list and the detail page of a live run update without a reload. They load assets/live.js with defer. The script:

  1. Reads an SSE stream with fetch:
  2. On a frame, fetches the same page as HTML. Then it swaps each region that has the data-live-region attribute. The server stays the only renderer.
  3. Fetches at most once each 2 s (list) or 3 s (detail). A region with the same HTML stays as it is.

A region does not change while an element in it has focus, or while a form field in it holds a changed value. The script tries the region again each 2 s or 3 s, so it updates when the operator leaves it. The filter form of the list and the action result (the flash) are outside each region.

A hidden tab closes its stream. When the tab shows again, the script connects again and fetches the page once. After a reconnect, the script also fetches the page once, so no change from the gap is lost. When a page fetch fails, the script tries again after 10 s.

When the stream is not available, the script fetches the page each 10 s. That occurs for a viewer without admin access, because both streams are admin-gated. It also occurs when no notification URL is set. While the stream is open, the script also fetches the page each 60 s. That covers a connection that stops with no error. It also covers a change that appends no event. The list hears a triage update at once. An open detail page shows it at the next 60 s fetch, because the execution stream sends events only.

The detail page of a run that ended has no live script. When a live run ends, the stream sends stream-end, the script fetches the page one last time, and then it stops. A region in use at that time takes the final state when the operator leaves it. A page rendered after a rejected action has no live script, so the values that the operator typed stay.

A browser without JavaScript gets the same server-rendered pages. The operator reloads the page.

Accessibility

  • Status badges on the detail page include aria-label and role="status".
  • All filter and action forms use standard <form> / <button> elements navigable by keyboard.
  • Every action is a plain HTTP form submission or link navigation. The live-refresh script only reads pages. It does not submit a form.
  • The live status line has no aria-live attribute, so a refresh does not interrupt a screen reader.

Security

  • All user-controlled values are HTML-escaped by Maud before rendering.
  • The dashboard emits no inline <script> element. The workflow list and detail pages load one same-origin script, assets/live.js, served with X-Content-Type-Options: nosniff. The script inserts only HTML that the server rendered and escaped. It inserts nothing from a redirect or from a page with no live regions.
  • Content Security Policy. Some forms use inline onsubmit or onclick handlers for a confirm() prompt, for example Cancel, Pause, Terminate and Reset. A script-src 'self' policy blocks those handlers, so the action then submits with no prompt. A policy for Vantage must allow them, for example with 'unsafe-hashes' and the hash of each handler.
  • The require_harvest_admin middleware (configured via HarvestBuilder) can gate Vantage behind token authentication in production.
  • Cross-site mutation rejection. Every POST Vantage renders a <form> against — every route on this router, plus the two dead-letter bulk routes those forms submit to directly on the sibling management-API router — rejects a request with no same-origin evidence. The DLQ page's per-row buttons post to these same two bulk routes with a single-id filter, so no separate route is needed for them. POST /dead-letters/{id}/replay and POST /dlq/redrive are bodyless API-only operations no Vantage form targets, so they stay outside this guard. Sec-Fetch-Site: same-origin is checked first and, when present, is always decisive; only when it is absent does the request fall back to a matching Origin/Host pair (default ports normalized, X-Forwarded-Host preferred over Host when a reverse proxy reports one). That fallback also requires a proxy-confirmed X-Forwarded-Proto matching Origin's claimed scheme exactly — Origin alone proves only the requesting page's scheme, never this connection's, so an unconfirmed claim is rejected regardless of which scheme it names. Failing all of that, a Content-Type a bare cross-site form cannot send (application/json, chiefly) is the last resort. Checking Sec-Fetch-Site first, ahead of Content-Type, matters: hyperlink auditing (<a ping>) and navigator.sendBeacon can reach a server with a non-CORS-simple content type and no preflight, so content type alone is never proof of safety. GET/HEAD/OPTIONS/TRACE and PUT/PATCH/DELETE are exempt outright, since a bare cross-site form or no-cors fetch cannot send those methods at all. So is the harvest CLI's own X-Harvest-Source: cli header, and a request carrying a verified scoped API token. The rest of the management API — the documented, headerless curl-and-CLI operator surface in docs/runbooks/ — is out of scope; see same_origin.rs for the full rationale.
  • Embedder responsibility. Set SameSite=Lax or stricter on the session cookie. Modern browsers already block a cross-site form POST under Lax. Treat this as a second, independent layer; the guard above does not depend on it.

Operator runbook: Diagnosing a stuck workflow

Use the workflow detail page at /workflows/{exec_id} to investigate a workflow that appears stuck or unresponsive. The following five scenarios cover the most common root causes.

Scenario 1 — Stuck activity retry (activity never completing)

Symptoms: The workflow has been RUNNING for an unexpectedly long time. The Activity attempts panel shows high attempt counts or repeated ActivityFailed events in the event timeline.

Steps:

  1. Open the detail page and check the Activity attempts panel. Look for activities with last_status = ActivityFailed and a non-zero attempt count.
  2. Check the Blocked on panel — if there is a row in the pending activities table with state = BACKOFF, the worker is waiting for the retry delay to elapse.
  3. Expand the ActivityFailed event payload in the Event history timeline using the "view payload" details toggle to read the error message.
  4. If the error is transient (network timeout, downstream unavailable), wait for the next retry attempt. If the error is permanent, use Cancel to halt the workflow and re-investigate.
  5. Check harvest_task_queue directly with GET /api/harvest/admin/queue?workflow_exec_id={exec_id} to confirm the task state.

Scenario 2 — Workflow waiting on a signal that never arrived

Symptoms: The workflow is RUNNING but appears idle. The Blocked on panel shows pending signals, or the Signals & Updates panel shows an expected signal name that has not been received.

Steps:

  1. Open the detail page and check the Blocked on panel. A row in the pending signals table indicates the workflow registered a signal handler that has not been triggered yet.
  2. Check the Signals & Updates panel for any signals already received — the workflow may have received the wrong signal name.
  3. If the upstream system is confirmed to have sent the signal but it did not arrive, use Send signal in the operator actions to manually inject the signal with the expected signal_name and a JSON payload.
  4. Verify via the event timeline that a SignalReceived event was appended after sending.

Scenario 3 — Workflow blocked waiting on a child workflow

Symptoms: The workflow is RUNNING but not making progress. The Children panel shows one or more child executions in RUNNING or FAILED state.

Steps:

  1. Open the detail page and scroll to the Children panel. Click the child execution ID link to open the child detail page.
  2. Diagnose the child using the same runbook scenarios (the child may have its own stuck activity, signal, or replay error).
  3. If the child is permanently stuck and must be abandoned, cancel it via Cancel on its own detail page. The parent workflow will then receive a ChildWorkflowFailed event and may retry or propagate the failure depending on its retry policy.
  4. If the parent needs to be rolled back past the child launch point, use Reset to event N on the parent, specifying the event ID before the ChildWorkflowStarted event.

Scenario 4 — Replay non-determinism (workflow failing on resume)

Symptoms: The workflow repeatedly transitions to FAILED with an error message containing "non-determinism" or "replay mismatch". It may also produce WorkflowFailed events immediately after being retried.

Steps:

  1. Open the detail page and expand the most recent WorkflowFailed event payload in the Event history timeline. The error field will describe which expected event did not match.
  2. Use the WorkflowReplayer testing harness (see testing.rs) to reproduce the replay error locally against an exported history (/workflows/{exec_id}/history/export).
  3. Identify the code change that introduced the non-determinism — common causes are: reordering activity calls, adding unconditional branches, changing timer durations, or removing version gates.
  4. Fix the workflow code (add a ctx.version() gate to branch on the new vs. old code path) and deploy.
  5. Use Reset to event N to reset the execution to a safe boundary before the non-deterministic branch, then allow the workflow to re-execute with the corrected code.

Scenario 5 — Timed-out activity (activity exceeded its start-to-close timeout)

Symptoms: The workflow has an ActivityTimedOut event in the event timeline. The workflow may be retrying or may be stuck in FAILED.

Steps:

  1. Open the detail page and find the ActivityTimedOut event in the Event history timeline. Use "view payload" to read the activity_id and timeout type.
  2. Check the Activity attempts panel — last_status = ActivityTimedOut confirms the activity was not heartbeating within the configured start_to_close or heartbeat_timeout window.
  3. Check the worker fleet at /workers — if workers are shown as Degraded or Unhealthy, the activity tasks may not be getting picked up.
  4. If the timeout was caused by the activity taking too long, consider increasing the start_to_close timeout in the #[activity] annotation and deploying the update.
  5. If the worker is healthy and the activity should complete quickly, the task may have been lost during a worker crash. Cancel the workflow and restart it, or use Reset to event N to roll back past the activity scheduling event.