Saga + Cancellation Semantics and Idempotency Contract

Saga (autumn-harvest/src/saga.rs) composes multi-step distributed transactions with explicit LIFO compensation. This document specifies the interaction between Saga and the workflow cancellation primitive, and the idempotency invariants that compensation activities must satisfy.

Cancellation interaction

Semantic chosen: cancellation does NOT auto-compensate

When an operator calls cancel_workflow_execution, a WorkflowCancelled event is appended to the execution's event history. On the next worker pick-up, the executor replays the workflow function with a WorkflowContext where ctx.is_cancelled() returns true.

The Saga struct never observes this. It holds a plain Vec of pending compensation closures and has no visibility into WorkflowContext state during forward execution. Cancellation therefore does not trigger automatic compensation.

Rationale

This matches Temporal's well-documented model and avoids two classes of surprising behaviour in long-running sagas:

1. Implicit partial-unwind surprise. In a ten-step saga, automatic compensation after step 4 may be worse than no compensation: steps 5–10 never ran, so there is nothing to undo for them. Forcing the author to call compensate_all() explicitly makes the decision visible.
2. Compensation-as-side-effect. Compensation activities can be expensive (refund calls, seat releases, inventory restores). Triggering them silently on every cancellation—including operator test cancels or retry-storm mitigations—creates unpredictable cost.

Recommended pattern

Observe ctx.is_cancelled() in the workflow function and call saga.compensate_all() explicitly:

#[workflow]
async fn checkout(ctx: &WorkflowContext, order: Order) -> Result<(), String> {
    let mut saga = Saga::new(ctx);

    let charge_id = saga
        .step(
            || ctx.execute_activity_raw("charge_payment", &order, "payments"),
            |charge_id| ctx.execute_activity_raw("refund_payment", &charge_id, "payments"),
        )
        .await
        .map_err(|e| e.to_string())?;

    let reservation_id = saga
        .step(
            || ctx.execute_activity_raw("reserve_inventory", &order, "inventory"),
            |rsv_id| ctx.execute_activity_raw("release_reservation", &rsv_id, "inventory"),
        )
        .await
        .map_err(|e| e.to_string())?;

    // ── Check for cancellation before committing ──────────────────────
    if ctx.is_cancelled() {
        saga.compensate_all()
            .await
            .map_err(|e| e.to_string())?;
        return Err(ctx.cancellation_reason().unwrap_or("cancelled").to_string());
    }
    // ─────────────────────────────────────────────────────────────────

    ctx.execute_activity_raw("confirm_order", &(charge_id, reservation_id), "default")
        .await
        .map_err(|e| e.to_string())?;

    Ok(())
}

ctx.check_cancellation()? is a shorthand that returns Err(HarvestError::Cancelled(…)) without running compensations. Use it for fast-exit points where no compensations are registered yet.

Known limitation — durable compensations cannot run after cancellation

Once the WorkflowCancelled event is recorded in history, a compensation that dispatches an activity (ctx.execute_activity_raw(...) inside the compensation closure, as in the example above) cannot actually run: the cancel event has no workflow-command counterpart, so the compensation's dispatch diverges (expected ActivityScheduled(..), got WorkflowCancelled) and the unwind fails with SagaCompensationFailed instead of scheduling the activity. This is a pre-existing engine limitation — verified empirically against pre-#801 builds, where the identical flow fails with the identical error — and is consistent with cancellation being a terminal seal: the engine will not schedule new durable work for a CANCELLED execution.

Consequences for the cancel-and-compensate branch specifically:

• In-memory compensation closures run fine after cancellation (the canonical pattern locked in by saga_compensate_all_on_cancel_pattern).
• A durable compensation in the cancel branch fails the unwind; the workflow observes SagaCompensationFailed, and the harvest.saga.compensation_failed page fires — correctly, since the state is genuinely dangling. Remediation is manual reconciliation (or reset), the same as any durable-compensation failure.
• Durable compensations in the step-failure rollback path (rollback_after, triggered by a forward step's error while the run is still live) are unaffected — the limitation is specific to unwinding after a recorded WorkflowCancelled.

Pinned by known_limitation_durable_compensation_after_recorded_cancel_fails_and_is_counted in tests/integration/saga_tests.rs.

Idempotency contract

Why compensations re-run

Compensation closures are re-registered on every workflow replay. The Saga struct is rebuilt from scratch each time the workflow function executes; it has no persistent identity in harvest_events. If a worker crashes after compensate_all() has started but before it finishes, the next worker will:

1. Replay all forward steps (returning their results from recorded history).
2. Re-register all compensation closures via the same Saga::step() calls.
3. Call compensate_all() again, re-invoking all compensation closures from the top of the LIFO stack — including closures that already ran before the crash.

Consequence: compensation activities must be idempotent.

Good pattern — release by ID

// Safe to call twice: the second call is a no-op when the reservation is
// already released.
|rsv_id: String| ctx.execute_activity_raw(
    "release_reservation",
    &rsv_id,      // ← specific, stable identifier from forward step result
    "inventory",
)

The rsv_id is sourced from the ActivityCompleted event recorded when the forward step ran, so it is the same value on every replay.

Anti-pattern — release most-recent

// Dangerous: the second invocation releases whichever reservation was
// created most recently at that moment — which may belong to a different order.
|_| ctx.execute_activity_raw("release_last_reservation", &(), "inventory")

On a replay after a crash, release_last_reservation would release a reservation that was never part of this saga.

Replay-determinism contract

The compensate closure in Saga::step receives the forward step's T result. When the forward step calls ctx.execute_activity_raw(...), that result is sourced from the recorded ActivityCompleted event on replay rather than re-executing the activity. Do not place non-deterministic or side-effecting logic directly inside the compensation closure body; invoke an activity via ctx.execute_activity_raw(...) instead, so that the compensation itself is durable and replay-safe.

Saga for DAGs — declarative node compensation

A #[dag] node can declare the activity that undoes it. On terminal DAG failure the engine builds a Saga over the nodes that completed successfully and unwinds them for you — no hand-written #[workflow] wrapper, no manual saga.step(…) per node. Everything specified above (idempotency, LIFO order, replay determinism, the observability counters) applies unchanged; this section documents only what is DAG-specific.

Declaring a compensator

Two builder methods, opt-in per node, at most one per node (last call wins):

compensate_named is name-based dispatch, not remote dispatch. The named activity must still be registered with the builder: plugin preflight fails the boot for a DAG compensator that resolves to no registered activity, exactly as it does for a forward node (see Build-time guards). It buys you a name computed at build time — not a reference to an activity that only exists on a remote/polyglot worker.

#[dag]
fn fulfillment(dag: &mut DagBuilder) {
    let reserve = dag.activity(reserve_inventory).compensate(release_inventory);
    let charge  = dag.activity(charge_payment).upstream(&reserve).compensate(refund_payment);
    let _label  = dag.activity(print_label).upstream(&charge).compensate(void_label);
}

One compensator activity may be shared by several nodes — the envelope's dag_compensate field says which node it is undoing.

What triggers the unwind, and what it covers

The unwind runs on the DAG's terminal failure check — the same condition that produces Err("one or more DAG tasks failed"). A node is compensated iff it BOTH reached TaskStatus::Succeeded AND declares a compensator:

That last row is the one non-obvious case: a mapped (.map_activity(…).over(&up)) node whose upstream produced [] settles Succeeded without dispatching anything. Compensating it would undo work that was never done, so the unwind skips any node whose forward pass dispatched nothing.

More generally, a deterministic pre-dispatch rejection — an error the engine raises before it allocates an activity id, pushes a command, or records an event — is reported as an ordinary node Failed rather than escaping the run, so the terminal failure is reached and the unwind still runs. Two cases qualify today: a mapped node whose upstream output is not a JSON array, and an activity input that exceeds the configured payload cap. Both leave no history footprint and no side effect, and the caller-visible error still names the precise cause rather than the generic DAG failure.

Errors that are not in that class keep propagating directly and never trigger an unwind: a replay divergence (unwinding from a diverged cursor would nd-block the run), a cancellation (see above), and transient engine/storage errors (the workflow task is retried, so the run is not terminal).

Sizing note. The compensation envelope embeds the compensated node's whole resolved input and its whole output, so it is necessarily larger than the node's own input. A node that runs close to the activity-input cap can therefore have its compensator rejected by that same cap — surfacing as SagaCompensationFailed rather than a dispatched rollback. Keep compensable nodes' payloads well clear of the cap, or offload large values (register a PayloadStore) so the envelope carries a reference instead.

Order — reverse topological (LIFO)

Compensations are pushed in the DAG's own forward order (levels forward, ascending index within a level) and the Saga unwind pops them LIFO, so an undo never runs before the undo of a node that depended on it. This is deterministic by construction: the push order is a pure function of the DAG's level structure, not of completion timing.

For the diamond a → {b, c} → d where d fails, the push order is a, b, c and the dispatch order is comp_c, comp_b, comp_a.

The compensator envelope

Every compensator receives one fixed shape:

{
  "dag_compensate": "charge_payment",
  "input":  { "…": "the node's resolved forward input" },
  "output": { "charge_id": "ch_abc123" }
}

• dag_compensate — the compensated node's activity_name. One generic compensator can therefore serve N nodes by branching on this field.

• input — the node's resolved forward input, in one of four shapes:

  Node kind	input shape
  Unbound	the { "conf": …, "dag_task": … } wrapper the forward dispatch used
  .input_from(&up)	the raw upstream output, verbatim
  .input_from_all(…) / .input_from_aliased(…)	the keyed object the binding produced (e.g. {"extract": …, "enrich": …}) — not any single upstream's raw output
  Mapped (.map_activity(…).over(&up))	the whole mapped upstream array (node granularity — never one cell's item)

• output — the node's recorded output, sourced from its ActivityCompleted event.

Payload caps apply to the envelope. The envelope embeds the node's whole resolved input and its whole output, so for a mapped node over a large array it can be substantially larger than either. It is dispatched through the ordinary activity lowering, so the issue #252 activity-input cap applies to the envelope, not to the node's original input — a node whose forward input sat just under the cap can still produce an over-cap compensation envelope.

Compensate by ID, read out of output. The same idempotency contract as the rest of this document applies verbatim: compensations re-run wholesale on replay, so release_inventory(rsv-9001) (an id read from output) is safe and release_most_recent_reservation() is not.

Queue inheritance

A compensator is dispatched on the compensated node's queue string, so an undo lands on the same worker pool that performed the forward step. A node with no .queue(…) yields the empty-string queue, which the worker resolves to the compensator activity's own default_queue (falling back to "default") — exactly as an unqueued forward node resolves.

The node's .retry(…) / .start_to_close(…) overrides are deliberately not applied to the compensator: they describe the forward step's failure budget, not the undo's. The compensator activity's own #[activity(…)] defaults apply.

Failure semantics

• A successful unwind still returns the original Err("one or more DAG tasks failed") — compensation is not an outcome change, it is a cleanup.
• A failing compensator does not abort the unwind: every remaining compensation is still attempted (continue-not-abort, the same contract as compensate_all), and the run then surfaces a stringified HarvestError::SagaCompensationFailed carrying both the original DAG error and every compensation error.

Observability — DAG unwinds feed the same counters

A DAG unwind rides the same saga_compensated:{seq} / saga_compensation_failed:{seq} dedup markers and the same harvest.saga.compensated / harvest.saga.compensation_failed counters described in the observability section below, including the starter alerts harvest_saga_compensation_spike and harvest_saga_compensation_failed.

Operator note: those counters now include DAG rollbacks as well as hand-written sagas. The two are distinguished by the workflow label — a unified DAG's shadow WorkflowInfo carries the DAG's own name, so harvest.saga.compensated{workflow="fulfillment"} is a DAG unwind.

Cancellation — a cancelled run does not unwind

Consistent with "cancellation does NOT auto-compensate" above, a cancelled run skips the DAG unwind entirely and returns the original DAG error: zero compensator dispatches, no saga_compensated: marker, no saga metric.

Beyond consistency, this is also load-bearing: a recorded WorkflowCancelled has no workflow-command counterpart, so dispatching a compensator into it would diverge (expected ActivityScheduled(..), got WorkflowCancelled) and nd-block a run the operator has already cancelled.

Accepted edge: a cancel landing mid-unwind terminates the run FAILED without running the remaining compensators. This is the same class as the durable-compensation-after-cancel limitation documented above — the engine will not schedule new durable work for a cancelled execution. The unwind is guaranteed to terminate (never nd-block), but the remaining state is genuinely dangling and needs manual reconciliation (or reset).

Known limitation — compensators are invisible to the DAG topology surfaces

A compensation is recorded as an ordinary activity (that is the whole point: no new WorkflowEvent variant), and it is not a declared DAG node. So it is visible in the raw event history and the execution timeline, but not as a node in:

• the DAG run-graph view (GET /dags/{name}/runs/{id}) — its topology comes from the registered DagDefinition, which carries forward nodes only;
• dag_export / any other definition-derived rendering.

To see whether a run unwound, read its history (or the saga_compensated:{seq} marker) rather than its graph.

Known limitation — a stray signal silences unwind observability

The saga_compensated:{seq} marker match is deliberately conservative about a drained signal frontier: when the recorded history ends in un-awaited signals at the point the unwind begins, the unwind is left uncounted — no marker, and neither saga counter fires.

A DAG consumes no signals of its own (a signal gate consumes exactly the gate's own signal), so any unsolicited signal delivered to a DAG run can put it in that state. The compensations still run, and still replay deterministically — only the observability is lost for that run. Do not send ad-hoc signals to DAG executions.

This is an observability gap only. Anything that must be correct in the presence of a marker-less unwind does not rely on the marker: the retry guard also detects a recorded compensator dispatch, so a marker-less rolled-back run is still correctly refused a retry.

Known limitation — compensation is node-granular

Compensation operates on nodes, never on individual mapped cells (selective/partial compensation is out of scope for this slice). Two concrete consequences for a mapped node:

• A CollectAll mapped node that reaches Succeeded with some failed cells is compensated once, with the full cells array — failed cells included — as the envelope's output. The compensator must decide per cell what actually needs undoing.
• A FailFast mapped node that a single failed cell drove to Failed is not compensated at all, so the side effects of the cells that did succeed before the failure are left uncompensated. If a mapped node's cells commit real side effects, prefer CollectAll plus a cell-aware compensator, or make each cell self-compensating.

Compensation dispatches are excluded from name-keyed history reads

A compensator dispatch is an ordinary ActivityScheduled event carrying the compensator's activity name — a new class of event issue #780 introduced, which every name-keyed history reader must now exclude.

Within one definition version CompensatorNameCollidesWithNode keeps a compensator distinguishable from a forward node. That is a per-definition-version guarantee, though: a later definition may introduce a forward node named after an older definition's compensator. Reading the old compensation dispatch would then report a node that never executed as succeeded/failed, with the compensation's timestamps.

Both readers therefore skip a dispatch whose input is a compensation envelope naming a node dispatched in the same run — the same history-only corroboration the retry guard uses, so it cannot drift across versions in either direction:

• dag_retry::node_outcome — node status (also feeds the run-graph view);
• dag_graph::latest_scheduled — node timing, attempts, and error.

Keeping the exclusion in both is what stops status and the timings from disagreeing about which attempt they describe.

The readers must decode before they filter

The corroboration reads ActivityScheduled.input, a payload-bearing field. Both run-graph readers — the #690 API handler and the Vantage DAG detail page — load raw history (store::load_history_with_timestamps explicitly leaves payload fields undecoded), so on a codec-encrypting or payload-offloading deployment the filter would be handed an opaque envelope, return false, and the exclusion would silently stop working — exactly the defect it exists to close.

Both therefore run an inflate/decode pass before deriving. Three properties keep it cheap and safe:

• Skipped entirely when nothing is opaque. A cheap structural scan of the two fields the derivation reads (ActivityScheduled.input and MarkerRecorded.details) gates the pass, so the default deployment — identity codecs, no payload store — does no extra work and stays on the pre-#780 path.
• Runs after the DB connection is released. Blob inflation is network I/O and must never hold a database pool slot.
• Degrades, never errors. The run graph is a read-only observability surface: on an inflate/decode failure it logs and renders from the raw history (worst case, the narrow cross-version display inaccuracy above) rather than failing the page. The retry endpoint takes the opposite posture and fails closed, because there a hidden envelope permits a double-spend.

No payload is surfaced by either reader — the run graph carries node names, statuses, timings, attempts, and a truncated error (not a payload-bearing field) — so decoding here needs no #608 read-decode opt-in or audit row.

Known limitation — raising a payload cap during an unwind diverges

A deterministic pre-dispatch rejection — today only HarvestError::PayloadTooLarge — is reported as an ordinary node failure so it routes through the terminal check and runs the unwind, instead of ?-escaping past it and stranding an already-succeeded compensable upstream.

Such a rejection leaves no history footprint (no activity id, no command, no event) and is a pure function of recorded state plus live configuration. So it is re-decided on every replay. If the cap is raised while the run is mid-unwind, the previously-rejected node now dispatches, and its ScheduleActivity lands where history records the compensator's — a divergence.

The consequence is a #603 non-determinism block, not a silent partial rollback: the run stays RUNNING and retries with backoff, and recovers as soon as the cap is reverted. It needs a four-way conjunction (PayloadTooLarge + a compensating DAG + a cap change + that change landing inside the unwind's decision-cycle window).

This is the same class as any config-dependent decision that leaves no history footprint. Issue #952 closed the sibling case where such a failure is uncaught and seals the run — a trailing terminal WorkflowFailed is now transparent to in-progress match_* calls, pinned by replayer_tests::early_config_dependent_failure_replays_cleanly — but a rejection the unwind catches and routes around records nothing at all, so it is still re-decided on every replay. Issue #780 enlarges the surface rather than creating it: before the unwind existed, a rejection sealed the run FAILED with no compensation events, so there was nothing for a later dispatch to collide with.

A durable fix means giving the engine's activity-dispatch path a history footprint for deterministic pre-dispatch rejections, so replay reads the decision back instead of re-deciding it. It cannot be done inside the DAG runtime: a level dispatches concurrently through join_all, so a marker pushed from inside a task future has no deterministic position, and one pushed after the join is read too late to gate the dispatch. That is an engine-wide change affecting every workflow, tracked separately.

Operationally: do not change payload caps while compensating DAG runs are in flight. Pinned by dag_compensation_tests::known_limitation_raising_the_cap_mid_unwind_diverges.

Build-time guards

Every misuse below is rejected before a single node runs, rather than surfacing mid-unwind when the state is already dangling:

Plugin preflight additionally flags a compensator that names an unregistered activity (dag '…' references unregistered compensator '…' for task '…'), so a missing compensator is caught before rollout rather than mid-unwind.

Rolling back to a pre-#780 build — drain in-flight unwinds first

Rolling the engine back while a DAG run is mid-unwind silently truncates that unwind. This is NOT the usual nd-block-and-roll-forward rule, so it is worth stating precisely:

• A pre-#780 run_unified_dag returns Err("one or more DAG tasks failed") at the terminal-failure check before consuming anything — there is no unwind step to reach the recorded compensation events.
• saga_compensated:{seq} is an issue #801 marker that the old matcher already knows, so it is not an "unknown marker" that would trip replay either.

The result: the old build seals the run terminally FAILED, with a partial unwind and no error signal at all — the compensations already dispatched stay applied, the ones still pending never run, and nothing nd-blocks to tell you. The dangling state must be reconciled manually (or via reset).

Operator rule: before rolling back past #780, drain in-flight compensating runs (a DAG run in a terminal-failure unwind is short-lived), or roll forward instead. Alert on harvest.saga.compensated going quiet while harvest.workflow.terminal{outcome="failed"} does not.

Runs that never unwind are unaffected: the saga is reached only on the failure branch, so a DAG that succeeds is byte-identical to a pre-#780 build.

Retrying a compensated run (issue #366 interaction)

A DAG run that executed its unwind is not retryable from a failed node. POST /dags/{name}/runs/{id}/retry rejects it with 409 Conflict:

this run already executed its compensation unwind, so its succeeded nodes' side effects were rolled back; retrying would resume on rolled-back state — start a fresh DAG run instead

The reason is structural: retry-from-node deliberately carries over the succeeded upstream nodes — which is exactly the set the unwind just undid. A retry would therefore resume as if those side effects still existed and double-spend the compensation.

Detection uses two independent signals, both read from the run's own recorded history and never from the registered definition:

1. a saga_compensat* marker followed by a started activity dispatch; or
2. a started activity dispatch whose input is a compensation envelope — the reserved {"dag_compensate": …, "input": …, "output": …} shape written only by the unwind — whose dag_compensate names a node that was dispatched in the same run.

Signal 2 is required, not redundant. An unwind at a drained signal frontier records no marker at all, so a marker-only check would leave that fully rolled-back run retryable.

Both signals require the dispatch to have started, not merely been scheduled. An operator cancel landing mid-unwind fails every open task row without appending any event, so a compensator that was dispatched but never claimed sits in history as an ActivityScheduled having rolled back nothing. ActivityStarted is the right bar: the worker appends it in a setup phase before invoking the handler, and a failed append aborts the dispatch, so a body that executed always has one — its absence proves the body never ran. Deliberately not keyed on a terminal event: a worker that started and then crashed mid-body may or may not have landed the rollback, and that ambiguity must resolve to "compensated" (refuse the retry), not to "safe". Neither activity_id nor worker_id is a payload field, so this signal survives erasure too.

The "followed by a dispatch" half of signal 1 is required in the opposite direction. Saga::run_compensations records its saga_compensated:{seq} marker before running the first compensation, so a compensator whose dispatch is rejected pre-dispatch — an oversized {dag_compensate, input, output} envelope against the activity-input cap with no payload offloading configured — leaves a marker in a run that rolled back nothing. Treating the bare marker as proof would 409 a perfectly safe retry and send the operator to a fresh run, re-running an upstream node whose side effect is still live. A marker with no dispatch after it therefore stays retryable.

The check is positional (any ActivityScheduled after the marker index), not name- or payload-keyed, which is what lets it survive payload erasure when a marker exists: an issue #495 tombstone blanks the envelope's contents (and the marker's details) but never removes or reorders events, and a marker's name is not a payload field — so signal 1 still sees the dispatch on a run whose envelopes signal 2 can no longer read. That is why signal 1 is refined rather than dropped. It does not cover an erased run that recorded no marker at all; see erased runs are refused outright. It also cannot go blind to a genuine unwind: a DAG compensator is always dispatched as an activity, the marker always precedes it, and no forward dispatch can follow the marker — the unwind runs only after the level walk has returned.

Signal 2 does not rely on shape alone, because the shape is reachable by accident: a mapped cell or an input_from binding hands a node the raw upstream output, which is arbitrary user data, and an input_from_all alias set could even be named dag_compensate/input/output outright. Shape alone would 409 a perfectly retryable run, including one whose DAG declares no compensators at all.

The corroboration is that the envelope's dag_compensate value must name an activity that was actually dispatched in the same run — precisely the unwind's own guard (dispatched_forward), so a genuine envelope always satisfies it. Completion would be too strict: a CollectAll mapped node whose cells all failed still settles TaskStatus::Succeeded at the DAG level and is genuinely compensated, yet records no ActivityCompleted under its name. An empty mapped fan-out is excluded for free — it dispatches nothing. It is drawn from the history, never the definition, so signal 2 has no registry dependence and survives every way the currently registered definition can drift from the run that produced the history:

• the compensator was renamed;
• the compensator was removed outright;
• a later definition reuses the old compensator's name as a forward node;
• an older definition's forward node shares a name with a compensator the current definition introduces.

Those last two are why no name-keyed check against the current definition is used, in either direction. An earlier revision also accepted a dispatch whose name was a currently-declared compensator; that was dropped as redundant (every unwind this engine produces writes the envelope, and only succeeded nodes are compensated) while being exactly the cross-version false-positive vector. CompensatorNameCollidesWithNode is a per-definition-version guarantee — no compensator shadows a node in the build that declared it — and nothing constrains names across versions.

The residual is a documented false positive in the safe direction: a forward dispatch whose input is exactly the three envelope keys and whose dag_compensate string happens to name a node dispatched in the same run. That marks a retryable run non-retryable (start a fresh run) rather than permitting the double-spend.

Signal 2 reads a payload-bearing field, so the retry endpoint loads history inflated and codec-decoded. Left un-inflated, an oversized compensation envelope stored as a payload-offload reference — or an encrypted codec envelope — would replace the whole input and hide the three compensation keys, silently leaving a marker-less rolled-back run retryable. The resolver cannot recover this itself: an offloaded input is indistinguishable from a large ordinary forward input, so treating one as a compensation signal would reject every legitimate retry of a big-payload compensating DAG.

Inflating is only possible when a payload store is actually registered, and a store is optional — so a reference can outlive the configuration that wrote it. If the store is unregistered (or fails to construct at boot), the inflate pass is skipped entirely and decode_event returns the reference verbatim, because it is not a codec envelope and nothing else rejects it. Signal 2 would then be reading {"_harvest_offload_envelope": 1, …} while believing it had the real payload. Any reference that survives the pass is therefore an error, naming the blob key and the store id so the operator can restore the right configuration. When a store is configured this is unreachable — inflation either resolves the reference or rejects a foreign store_id — so the check only fires on the configuration-removal case.

A DAG run that failed without compensators (and every pre-#780 history) triggers neither signal and stays fully retryable.

Erased runs are refused outright

Both signals can be blinded at once by PII erasure. Erasure tombstones every payload field, so ActivityScheduled.input becomes {"_harvest_erased": true} and signal 2 is gone; and a run that unwound at a drained signal frontier recorded no marker for signal 1 to anchor on. On that intersection a fully rolled-back run would look retryable, and erasure is irreversible — no amount of inflating or decoding recovers it.

The retry endpoint therefore refuses an erased source run up front, with a 409 naming erasure as the blocker. It is an O(1) check on the execution row's own input column (erasure always tombstones it first) — the same guard the issue #612 terminal-query path uses.

That up-front check is a fast path, not the guarantee. It reads the row without a lock, and the handler then releases its database connection to inflate offloaded blobs before opening the fork's transaction — so erase-payloads, which takes its own FOR UPDATE lock, can commit tombstones anywhere in that window and the pre-flight would have already passed on the intact row. The fork therefore rechecks erasure under the same row lock it takes to seal the source, before copying a single event. That serialises the two operations: erasure either committed before the lock, in which case the fork sees it and refuses, or it queues behind the lock and the fork completes on intact events. Every fork path runs this check, not only DAG retry. See Workflow reset.

The refusal is correct independently of compensation: the issue #148 fork carries over the upstream events, whose output fields are now tombstones, so a re-executing downstream node with an input_from binding would be handed {"_harvest_erased": true} as its input. Retry-from-node on an erased run is broken regardless of whether it compensated.

The cost is a safe-direction false positive: an erased run of a DAG that declares no compensators is refused too. That is deliberate — "the current definition declares no compensators" is exactly the cross-version registry signal the rounds above removed, since the definition that produced the history may have declared them.

Divergence inside an unwind still nd-blocks

Saga collects every compensation error — including a HarvestError::NonDeterministic — into compensation_errors, so a genuine code-drift divergence during an unwind surfaces as SagaCompensationFailed. That is the author-visible result; the engine result is different and takes precedence: the divergence also sets the run's non-determinism details, so the worker nd-blocks it under issue #603 rather than sealing it FAILED.

That is the correct outcome — a divergence during an unwind is a deploy problem, not a dangling-state problem, and #603's block is non-terminal and recoverable by rolling the workflow code back. Do not read a SagaCompensationFailed in the logs as "the unwind ran and failed" without checking whether the run is nd-blocked (GET /workflows?nd_blocked=true).

Observability

Compensation is a first-class, alertable signal. Two engine counters are emitted from inside Saga::run_compensations (the single choke point both compensate_all() and automatic step-failure rollback funnel through):

execution.id is never a label (ADR-0001 §7). A saga that never compensates emits nothing; an empty compensate_all() is not a sequence.

Exactly-once across replays. Compensations re-run wholesale on every replay (see the idempotency contract above), so the counters cannot simply fire per call. Each unwind is keyed to durable MarkerRecorded dedup markers reusing the existing event variant — saga_compensated:{seq} recorded at unwind start (persisted in the same command batch as the first compensation's own dispatch, so a crash at any point mid-unwind resumes silent) and saga_compensation_failed:{seq} recorded at failed-unwind end. The counter fires only when its marker is first recorded on the live frontier; every replay observes the marker and stays silent. No new WorkflowEvent variant and no migration — the markers are opaque names on MarkerRecorded, exactly like fan_out:{n} / race:{seq} / patch:{id}.

Backward compatibility. A pre-#801 history (no saga markers) replays untouched and uncounted: the marker matcher's Absent arm never moves the cursor, so the recorded events still match the compensation's own commands — never a divergence, never a retroactive count.

In-Saga emission (not the worker terminal boundary). The failure counter fires even when the workflow author catches SagaCompensationFailed and the run goes on to COMPLETE, and the compensated counter fires for cancel-and-compensate unwinds in runs that never terminally fail — both cases a terminal-boundary emission would structurally miss. It is therefore fully separable from harvest.workflow.terminal{outcome=failed}.

Per-unwind coherence (post-review hardening). The unwind's disposition (counted or not) is resolved once, at unwind start, and the failure counter follows it — so the pair can never disagree about one logical unwind, and the invariant failed ≤ compensated holds per unwind (ratio dashboards never divide by zero or exceed 100%). Two concrete consequences:

• A counted unwind's failure is always counted, including past a trailing un-awaited signal at the failed-unwind end (e.g. a retried cancel webhook ingested at the final unwind cycle's wake) — the failure marker is recorded past the drained signal, replay-consistently.
• The cancel-and-compensate pattern is counted: a trailing WorkflowCancelled has no workflow-command counterpart and never leaves the replay cursor, so the marker matcher treats it as transparent — the unwind of a freshly-cancelled run is the live frontier. In a WorkflowReplayer strict/canary probe the byte-identical history shape (a pre-#801 marker-less terminal cancelled run) is a pure read, not a live unwind: the observe layer suppresses the frontier arms in probe contexts, so old cancelled histories are never counted retroactively and the strict replayer never sees a fresh marker command.
• Cancel + durable compensation (round-3 review): the marker transparency is saga-marker-local — match_activity still (and deliberately) diverges on the unconsumed WorkflowCancelled, so a cancel-branch unwind whose compensations are activity-backed fails at its first dispatch (the pre-existing limitation documented in the cancellation section above; baseline-identical on pre-#801 builds). Such an unwind counts both compensated (it began) and compensation_failed (it failed) — the truthful pair, and precisely the dangling-state page the failure counter exists for. The counted-but- cannot-proceed combination is therefore not a metric bug: the alert that fires is the correct one.

Accepted edges (documented deliberately):

• An unwind entered while unconsumed non-marker, non-cancellation events sit at the replay cursor is conservatively uncounted as a whole (both counters) — for a metrics-only feature, a silent under-count of a rare edge beats any chance of recording a marker at a wrong history position.
• Signal-with-start caveat (inherited from ctx.patched()): an unwind entered at a drained-signal frontier — canonically a signal-with-start run whose unwind begins before the staged signal is awaited — is conservatively uncounted as a whole, deterministically on every replay. The failure counter follows (never failed=1 with compensated=0).
• Crash window (at-least-once), both counters, durable and in-memory unwinds alike: the sample is emitted in-process within the single-decision-cycle gap before its marker's batch commits (start marker: before the unwind's first dispatch batch persists; failure marker: before the post-unwind batch persists). A worker crash inside that gap re-emits on resume. A crash mid-unwind — between compensations, after the first batch committed — is exactly-once. A pure in-memory unwind (zero durable footprint) is the maximal case: a crash-resume anywhere within its single cycle can re-count — the metric mirror of the "compensations re-run wholesale" contract.
• Pause race (at-least-once): an operator pause committing while the unwind's first decision cycle is in flight discards the cycle's pending commands — including the freshly pushed marker — after the sample already fired; the re-derived cycle after resume emits again. Same class as the crash window.
• Forward compatibility: a history recorded by #801+ code does not replay under pre-#801 builds (the marker is unknown to the old matcher); a rollback nd-blocks (non-terminal, recoverable) until rolled forward. Same rule as every marker feature.

Starter alerts harvest_saga_compensation_spike and harvest_saga_compensation_failed ship in docs/alerts/starter-pack-v0.1.0.json with runbook sections in docs/runbooks/harvest-alerts.md.

Test coverage

The integration tests in autumn-harvest/tests/integration/saga_tests.rs lock in these semantics:

The declarative DAG unwind is locked in by autumn-harvest/tests/integration/dag_compensation_tests.rs:

Added by the post-review hardening pass:

Build-time and preflight guards are covered by unit tests in autumn-harvest/src/dag.rs (gate / empty / collision / shared-compensator / last-wins / compensate_named trimming), autumn-harvest/src/builder.rs (classic-DAG and local-activity rejections), autumn-harvest/src/saga.rs (push_compensation registers without running; compensate_all_after carries the caller's original error), and autumn-harvest-plugin/src/preflight.rs (unregistered compensator). A worked example with embedded self-checks lives in autumn-harvest/examples/dag_compensation.rs.

Out of scope

• Saga::with_auto_compensate_on_cancel(true) — separate issue if demand emerges.
• Cross-shard saga semantics — a Saga is scoped to a single workflow execution and therefore a single shard via ExecutionId::shard().
• Saga + Update or saga + child-workflow interaction — separate specs.
• Physical backout in place of a saga, for a workflow whose effects stay in the Harvest Postgres. The R&D spike rnd/contention-adaptive-atomicity.md measures both and sketches a selection rule.