Upgrade check: a verdict for each in-flight run

The upgrade check runs before a deploy. It gives each in-flight run one verdict against a candidate build:

The check covers three failure modes:

The verdicts are conservative. A change that the check cannot place gives review, never migrate. The design and its proofs are in docs/design/DESIGN-1995.md.

1. Write a structure manifest for each build

Run harvest-verify on the source of each build, with the same toolchain:

$ git checkout v1.4.0
$ cargo harvest-verify -p my-workflows --lib --emit-structure old.structure.json
$ git checkout v1.5.0
$ cargo harvest-verify -p my-workflows --lib --emit-structure new.structure.json

The manifest lists each body that a workflow can reach, with a digest, its call sites and the commands it emits. A digest ignores spans, so a line shift does not change it. See the harvest-verify guide.

2. Add the command to the candidate build

The check runs inside the candidate build, because only that build holds its workflow types and codec keys. Add a small binary next to the worker binary. Enable the db and testing features of autumn-harvest for it. Register the same workflows, signals, updates, queries, codecs and offloader as the worker:

use std::sync::Arc;
use autumn_harvest::upgrade_check::{UpgradeCheck, run_command};

#[tokio::main]
async fn main() {
    let check = UpgradeCheck::new()
        .register(workflows![place_order])
        .signals(signals![approve])
        .queries(queries![order_status])
        .with_codecs(Arc::new(my_codecs()));
    // Lossy: `args()` would panic and print a non-UTF-8 argument.
    let args = std::env::args_os()
        .map(|arg| arg.to_string_lossy().into_owned())
        .collect();
    std::process::exit(run_command(check, args).await);
}

The upgrade_check example in autumn-harvest/examples/ is a complete binary.

The replay must run as the candidate worker runs. A workflow can branch on what the worker sets, and a builder-only change leaves both structure manifests the same. So give the check each value the candidate worker sets:

Each default matches the worker default. The replay applies the payload caps to the commands that a run sends next. A run whose next payload is over a candidate cap gets pin, because the candidate worker rejects that payload. With an offloader, the replay also takes the offload threshold, so a payload above it is not held to the cap. Build the history policy from WorkflowHistoryPolicy::default() with the same values as the worker. Use map_replayer for anything else, such as shared state.

3. Run the check

The check reads the in-flight runs once. A run that starts on the baseline build after that read is not in the report. So close the baseline set first: point the build policy of each queue at the candidate build, so every new run starts there. See Build-id routing. Then run the check. Without build routing, stop new starts of the checked workflows until you act on the report.

$ my-upgrade-check \
    --database-url-env SHARD0_URL --database-url-env SHARD1_URL \
    --baseline-structure old.structure.json \
    --baseline-build sha-old456 \
    --candidate-structure new.structure.json

A flag also takes its value as --flag=value. The report goes to stdout, and an error goes to stderr.

A usage error names a plain flag or variable name only. Any other argument shows as <hidden>, so a mistyped URL never reaches stderr. An error that names a file shows <hidden> for a path with other characters than letters, digits and _-./+~ .

The check reads RUNNING and PAUSED runs. It reads inside read-only transactions and writes nothing.

4. Read the findings

A finding holds code names and event indexes only. It never holds a payload.

When a changed helper counts as passed

A change to a helper body gives migrate only when the history proves that the run finished the helper. Each rule must hold in both manifests that have the helper, because the run ran the baseline code:

• The helper is not the workflow body.
• The helper starts at most once: one call site on each body up to the workflow body, and no call site in a loop.
• Only the helper calls the bodies under it, and none of their calls is in a loop.
• Each command the helper can emit has a known name, such as the activity name or the timer id. A call that can park with no command, such as await_condition, has no such name.
• No other body emits a command of the same kind with the same name or an unknown name.
• The run completed each command as often as the helper emits it, and none is still open.
• A decision ran after the last result of those commands. So a PAUSED run that holds a result with no decision after it needs review.

So factor long workflows into helper steps. A change to a step that every run has passed then gives migrate.

5. Act on the verdicts

declare_compat works per build, not per run. So:

• Every run gets migrate. Declare compat from the candidate build to the current build, after the check and never before it. See Build-id routing, Scenario A.
• A run gets pin. Do not declare compat. Keep workers on the current build until those runs end. build_reachability says when the old build is safe to retire.
• A run gets review. Read its findings. Then accept the change for that run, or treat the run like a pin. Declare compat only when each run gets migrate, or a person accepts each review finding.

Trust boundary

The check decodes payloads in memory with the candidate codecs. No history is exported, so no history needs scrubbing. A run verdict holds no payload and no error text. The incomplete list holds shard database errors only. Candidate workflow code runs during the replay, so keep payloads out of its logs.

Known limits

• The check covers activity and side-effect payloads only through replay. Issue #1994 adds schemas for them.
• Value-level drift is invisible to the call graph. A changed static or configuration value can change behavior with no digest change.
• A step name that comes from a function parameter is unknown, so that helper never counts as passed.
• A helper in a crate that harvest-verify does not analyze is a boundary, so the verdict is review.
• MIR text is not a stable API. Write both manifests with the same toolchain. The check refuses two manifests from different toolchains.
• Nothing ties a manifest to the binary under test. Write both from the commits that the two binaries come from.
• The replay timeout cannot stop candidate code that never yields.
• Code in std and in the crates that harvest-verify trusts, such as chrono and serde_json, is not in the call graph. A new version of a trusted crate can change a value with no manifest change. Review a bump of such a crate as you review an activity change.
• The check sees the signals and updates that exist when it reads. One that arrives later, before you declare compat, is not in the report. The build policy stops new starts, not input to existing runs. Pause the senders, or publish candidate schemas so the candidate validates a late payload when it admits it.