Simulation Testing

Autumn ships a deterministic simulation testing (DST) harness: a #[sim_test] attribute that hands your test a seeded [Sim] handle, a paused deterministic executor, and a virtual clock, so your whole app's concurrency — retries, scheduled ticks, background jobs — runs identically on every machine and every run. A failure prints a copy-pasteable line that reproduces it exactly, and a seed-sweep runner (sweep_proptest, see below) lets you drive your own workload against many seeds looking for rare-interleaving bugs a single-seed test would never stumble into. This sweeping is opt-in, not automatic: autumn's own CI runs a built-in sim-sweep job over a few hundred seeds, and sweeps one example app each night (see "The reddit-clone nightly"). It sweeps a toy scenario (a smoke check of the harness) and three framework scenarios: jobs, scheduler and lock across replicas (see "Multiple replicas"). It does not explore your application's interleavings. Wiring sweep_proptest against your own scenario, in your own CI, is how you get that coverage for your app; see "Property-based op-driving + the seed sweep" below. For a complete app sweep to copy, see "The reddit-clone nightly" below: 4,000,000 seeds each night.

┌──────────────────┐
│   #[sim_test]     │  seed → deterministic Sim
└─────────┬─────────┘
          │
  ┌───────▼────────┐   paused, current-thread   ┌──────────────────┐
  │  Sim::advance / │ ─────────────────────────► │  your app's real  │
  │  run_to_idle    │      virtual clock          │  jobs/scheduler/  │
  └───────┬─────────┘                             │  request path     │
          │ panic (always! violated)              └──────────────────┘
  ┌───────▼────────────────────────────┐
  │ AUTUMN_SIM_SEED=0x… cargo test …   │  ← replay line, prints on failure
  └─────────────────────────────────────┘

Why simulation testing

The nastiest production bugs are not logic errors in a single handler — they are interleaving bugs: a retry storm when a downstream stalls, a deadlock between a request and a scheduled tick, a timeout that fires in the wrong order. These bugs need many actors racing at the same instant to trigger, so a real-clock test would need to get impossibly lucky to reproduce one — and even luckier to rerun it to prove a fix. A deterministic simulation makes "many actors racing at the same instant" the cheap, constructible default: pause the clock, drive everything from one seed, and the same instant is trivial to manufacture on purpose.

Quick start

#[sim_test] needs no extra feature flag — it's part of the default autumn-web surface.

use autumn_web::sim::Sim;
use autumn_web::sim_test;

#[sim_test]
async fn deterministic(mut sim: Sim) {
    // The seed comes from `AUTUMN_SIM_SEED` (hex `0x..` or decimal), default 0.
    // `sim`'s own clock and RNG (`sim.rng()`) are pure functions of this seed.
    // An app mounted with `sim.build(...)` gets the virtual clock and an
    // entropy source seeded from it, so its ids replay too.
    assert_eq!(sim.seed, 0);
}

Reproduce a failing run by copying the replay line printed on panic:

AUTUMN_SIM_SEED=0x9f3a cargo test -p my-crate deterministic

Mounting a real app

use autumn_web::sim::Sim;
use autumn_web::sim_test;
use autumn_web::test::TestApp;

#[sim_test]
async fn app_boots_under_the_sim(mut sim: Sim) {
    sim.build(
        TestApp::new()
            .routes(routes![index])
            .jobs(jobs![send_receipt])
            .tasks(tasks![nightly_digest]),
    );

    let response = sim.client().get("/").send().await;
    response.assert_ok();
}

sim.build mounts an [autumn_web::app::AppBuilder]-configured app on the sim's paused runtime. It wires in the virtual clock, an entropy source seeded from sim.seed (see "Deterministic identifiers" below), and the fault-injection hooks below if you configured any. Register #[job]s with TestApp::jobs and #[scheduled] tasks with TestApp::tasks, or through a plugin, exactly as on AppBuilder.

Virtual time

sim.advance(std::time::Duration::from_secs(24 * 3600)).await; // 24h, zero wall-clock sleep
sim.run_to_idle().await; // drain everything the advance released

• [Sim::advance] steps the injected [autumn_web::time::Clock] and tokio's paused timer wheel together, so a #[job]'s exponential backoff or a #[scheduled] tick fires the instant virtual time crosses its deadline — with zero real waiting. sim.build starts the app's scheduled tasks, so an every = "1m" task first ticks one virtual minute after build, and a cron task at its next occurrence after the sim epoch.
• [Sim::run_to_idle] drains everything already-ready — the job worker, due scheduler ticks, durable repository commit hooks — until the runtime is quiescent. It does not fast-forward to a future timer; pair it with advance for "jump to the next interesting instant, then settle what fired." The drain runs 1024 rounds, then stops at 64 quiet rounds in a row. If it sees no such quiet run within 2048 rounds (for example, a job that enqueues itself again), run_to_idle panics with a sim drain stall message and the seed. Sim::try_run_to_idle returns the SimStall instead.
• [Sim::advance_to] / [Sim::advance_to_local] jump to a specific (optionally timezone-zoned, DST-aware) instant instead of a raw duration — useful for business-calendar / SLA tests that must cross a spring-forward or fall-back boundary deterministically.

Deterministic identifiers

Framework-minted IDs (job IDs, request IDs, idempotency keys, session tokens) draw from an injected [autumn_web::entropy::Entropy] source. Reach a seeded one from a handler via the [autumn_web::entropy::Rng] extractor, or from the sim directly via sim.rng().

Sim::build seeds the mounted app's entropy from sim.seed, so a job's retry jitter, a minted UUID and anything else reading state.entropy() replay from AUTUMN_SIM_SEED with no extra call. The default is the same source as with_entropy(SeededEntropy::new(sim.seed)). To use a different source, pass your own; an explicit with_entropy always wins:

use autumn_web::entropy::SeededEntropy;

sim.build(
    TestApp::new()
        .routes(routes![index])
        .with_entropy(SeededEntropy::new(0xfeed)),
);

An app mounted again after [Sim::kill] (a restart) gets a new stream derived from the seed, so the restarted process does not repeat the crashed one's ids and the run still replays.

Before 0.8.0 the default was the OS entropy source, and a test had to pass with_entropy(SeededEntropy::new(sim.seed)) itself. That call still works and now changes nothing.

Fault injection

Chaos injects faults probabilistically — you give it a rate and the seed decides which operations get hit:

use autumn_web::sim::Chaos;

sim.chaos(
    Chaos::default()
        .db_transient_errors(0.05)     // 5% of connection checkouts fail
        .job_duplicate_delivery(0.10)  // occasional at-least-once double-delivery
        .clock_skew(std::time::Duration::from_millis(250)),
);

Every fault decision is drawn from the seed, so enabling chaos never breaks reproducibility — the same seed replays the same fault schedule. See [autumn_web::sim::Chaos] for the full catalog, including SMTP transport faults. Two related tools sit outside Chaos: sim::llm's seeded LLM stub for agent retry paths, and [Sim::kill] / [Sim::restart] for crash-recovery proofs. A kill drops the app between two drains, after its transactions commit, not in the middle of one. When you want to name a fault rather than sample one — "fail the 3rd database checkout" — reach for FaultPlan in the next subsection instead.

Authored fault scenarios (FaultPlan)

Chaos is a rate: "5% of checkouts fail", with the seed choosing which ones. That is the right shape for a sweep hunting rare interleavings, and the wrong shape for a regression test — because the sentence a post-mortem actually produces is "the third connection checkout failed while the second send_invoice execution was retrying", and no probability reproduces that on purpose. [autumn_web::sim::FaultPlan] lets you author that scenario directly, by ordinal rather than by rate, and hands back a serializable record of what happened that a test can assert on and CI can replay byte-for-byte.

use autumn_web::sim::{FaultPlan, Sim};
use autumn_web::sim_test;
use autumn_web::test::TestApp;

#[sim_test]
async fn the_third_checkout_and_the_second_invoice_fail(mut sim: Sim) {
    let plan = FaultPlan::from_seed(sim.seed)
        .fail_db_checkout(3)           // 3rd checkout on any pool (ordinals are 1-based)
        .fail_job("send_invoice", 2);  // 2nd execution of that job by name

    sim.build(
        TestApp::new()
            .routes(routes![checkout])
            .plugin(InvoiceJobPlugin)
            .with_fault_plan(plan),
    );

    for _ in 0..5 {
        sim.client().post("/checkout").send().await;
    }
    sim.run_to_idle().await; // drains the job worker through the fault seam

    let outcome = sim.client().fault_outcome().await;

    assert_eq!(outcome.fired.len(), 2);
    assert_eq!(outcome.fired[0].ordinal, 3);            // the checkout that failed
    assert_eq!(outcome.server_errors[0].status, 503);   // captured through reporting
    assert_eq!(outcome.final_state.db_checkouts, 5);    // every checkout, fired or not

    // Canonical, byte-identical on every replay of this seed — commit it as a
    // fixture and this scenario becomes a CI regression test.
    assert_eq!(outcome.to_json_string(), include_str!("fixtures/invoice_scenario.json").trim_end());
}

Ordinals are 1-based, and 0 matches nothing (the same convention as Chaos::smtp_faults). fail_db_checkout(n) and fail_job_execution(n) count across every pool and every job name; fail_db_checkout_on("replica", 2) and fail_job("send_invoice", 2) count on that target's own counter. Duplicate entries collapse, so a plan is a set — plan.planned() returns the whole schedule sorted and plan.describe() prints it one fault per line, both before you run anything. A plan is pure data: clone it onto two apps and you get two independent ledgers.

Faults can be confined to a window of virtual time, driven by the injected clock rather than wall time:

let plan = FaultPlan::from_seed(sim.seed)
    .fail_job_execution(2)
    .only_between(Duration::from_secs(5), Duration::from_secs(10));

The window is half-open (from <= elapsed < to) and elapsed is measured on the app's injected monotonic clock since the app started — so sim.advance moves it and no real time ever does. Outside the window the effect still consumes its ordinal, and the near-miss is recorded in outcome.suppressed rather than vanishing: a scenario that stopped firing because your timings shifted is visible in the outcome instead of silently passing.

The seed earns its keep in the random_* lane, which spreads faults across a range instead of naming each one:

let plan = FaultPlan::from_seed(0x5EED).random_db_checkout_faults(2, 1..=8);

That picks 2 distinct checkout ordinals from 1..=8 using the plan's seed, and resolves them into explicit entries at builder-call time — so planned() still describes the schedule completely, a different seed picks different checkouts, and a plan built only from explicit fail_* calls draws no entropy at all.

What the outcome record holds

to_json_string() is canonical (declaration-order fields, no maps, no floats), fingerprint() is an FNV-1a 64 over that string, and FaultOutcome::from_json_str round-trips it. TestClient::fault_outcome() is async because autumn dispatches error reports on a detached task: it settles those with bounded cooperative yields before snapshotting, and never advances the virtual clock while doing so. TestClient::fault_ledger() returns the same handle for an un-settled outcome() snapshot, and None when no plan was attached.

The determinism contract

The byte-identical-replay guarantee holds inside a specific box, and build enforces the parts it can:

• A paused, current-thread runtime — #[sim_test], or #[tokio::test(start_paused = true)]. On a multi-threaded runtime concurrent executions race for their ordinals and the ordering is not reproducible.
• The injected clock. Window checks and every at / elapsed_ms read the app's ClockSource, so no wall-clock read leaks into the record.
• Seeded entropy. Attaching a plan defaults the app's entropy to SeededEntropy derived from the plan's seed, so retry jitter and minted IDs replay from it too. Unlike the rest of the sim this one is automatic — an explicit with_entropy still wins.
• One job worker (jobs.workers = 1), reporting at sample_rate = 1.0, and failure capture off (the default). build asserts all three when a plan is attached, rather than letting a second worker, a sampled-out 5xx, or a capsule write that reporting awaits before any reporter runs quietly break replay.

What it does not cover

• Only two effect classes: database connection checkout and job execution. Mail, outbound HTTP and channels have no FaultPlan lane — for SMTP use Chaos::smtp_faults, which already takes an explicit 1-based schedule.
• Test-only. FaultPlan attaches to a TestApp. For a running staging app, use [fault_injection]. It is refused in prod by default.
• perform_enqueued_jobs bypasses it. TestClient::perform_enqueued_jobs invokes handlers directly and never crosses the intercept_execute seam, so job faults never fire under it. Drain with sim.run_to_idle() — plus sim.advance to cross a retry backoff — instead.

autumn/tests/integration/sim_fault_plan.rs carries the worked example, in the same before/after shape as the retry-storm bug above: a charge_card job and a plan that fails its first execution. With max_attempts = 1 the charge is never recorded and the scenario's resilience assertion fails; with max_attempts = 3 the retry lands it exactly once and the same plan passes — flip that one attribute back to reproduce the failure. See issue #1680 for the design.

always! / sometimes!

use autumn_web::{always, sometimes};

always!(total_balance == 0, "ledger must stay zero-sum (seed={:#x})", sim.seed);
sometimes!(response.status() == 409, "a transfer was rejected (insufficient funds)");

• always! is a hard invariant — it panics the instant its condition is false, exactly like assert!, but the panic flows through #[sim_test]'s replay-line printing.
• sometimes! is a reachability target — it records whether the labeled condition was ever true. A single run doesn't fail just because a sometimes! was never satisfied (call [autumn_web::sim::assert_all_sometimes_satisfied] for an explicit single-run check); the seed-sweep runner below fails the sweep if a label was observed but never satisfied by any seed in the range — so a green sweep is provably non-vacuous, never accidentally testing nothing.

Property-based op-driving + the seed sweep

Behind the sim-testing feature (dep:proptest), sim::op and sim::sweep add a proptest-driven workload generator and a batch seed runner:

use autumn_web::sim::{Sim, sweep_proptest, SweepOutcome};

let strategy = proptest::collection::vec(any::<Op>(), 1..32);
match sweep_proptest(0..1000, &strategy, |sim, ops| apply_ops(sim, ops)) {
    SweepOutcome::Passed { seeds_run } => println!("{seeds_run} seeds, non-vacuous"),
    SweepOutcome::Failed { failure, .. } => panic!("{failure}"), // shrunk to a minimal op-sequence
    SweepOutcome::Vacuous { unsatisfied, .. } => panic!("never satisfied: {unsatisfied:?}"),
    SweepOutcome::Empty => panic!("swept zero seeds"),
}

sweep_proptest runs [Sim::run_proptest] sequentially across every seed in the range, stopping at the first failure and reporting its seed plus a proptest-shrunk minimal op-sequence.

This is a library function you call from your own [[bin]] or test, driven against your own Strategy/body for your own app's properties — nothing runs it for you automatically. Autumn's own repository ships one example of wiring it up: autumn/src/bin/sim_sweep.rs, a small [[bin]] that sweeps a fixed, deliberately-correct toy account scenario as a smoke check that the sweep mechanism itself works (sim_sweep_driver's DoD test proves it catches a real invariant break using a deliberately-buggy variant of the same toy scenario). Autumn's own CI runs that binary as its own job (Sim sweep, structured like the loom job, 512 seeds) on every push and PR. To get this coverage for your own app, write a [[bin]] following the same shape against your own scenario and wire it into your own CI.

Built with --features sqlite, the binary also sweeps three framework scenarios. Each one mounts two or three replicas on one database and injects seeded faults:

The first seeds also run twice, and their traces must match (see "Same-seed trace check").

The sweep found one contract limit on its first run: each replica reaps SQLite scheduler leases with its own clock. If one clock is ahead by more than scheduler.lease_ttl_secs, that replica reaps a live lease, and a second replica runs the tick again. Processes on one host use one clock. Only processes with their own clocks can have this skew. The scenario keeps its skew below the TTL. See docs/guide/scheduled-multi-replica.md.

AUTUMN_SIM_SEEDS=1000 cargo run -p autumn-web --release --features "sim-testing,sqlite" --bin sim-sweep

• AUTUMN_SIM_SEED_START sets the first seed (default 0).
• AUTUMN_SIM_SCENARIOS selects scenarios, for example jobs,lock (default: all that the build has).
• AUTUMN_SIM_TRACE_CHECKS sets how many of the first seeds run twice (default 16).

On a failure the binary prints a replay command for only the failing seed and scenario, for example AUTUMN_SIM_SCENARIOS=lock AUTUMN_SIM_SEED_START=300 AUTUMN_SIM_SEEDS=1 cargo run ….

Catching a deadlock

A deadlock parks every task with no timer that could wake one, so by default a deadlocked #[sim_test] hangs. Set AUTUMN_SIM_LIVENESS_BUDGET_SECS to arm a watchdog: a virtual-time timeout around the test body. Because the paused runtime advances straight to the next timer when every task is parked, a deadlock reaches the watchdog in microseconds of real time, and the test fails with a sim liveness panic and its replay line.

AUTUMN_SIM_LIVENESS_BUDGET_SECS=31536000 cargo test -p my-crate sim_ -- --test-threads=1

The watchdog is off by default because of how the paused clock works: it also advances while a task waits on something outside the runtime, such as a lock that a test on another thread holds, or real I/O. So arm it only where sim tests run one at a time (--test-threads=1) and do no real I/O. A busy loop that never parks keeps the runtime from advancing, so the watchdog does not detect it; an always! invariant is the tool for that. Autumn's own CI arms the watchdog on its single-threaded sim_ step.

Crash at any await

crash_at(index, op) runs op and drops it at its index-th suspension point (an await that returns Pending). A re-poll with no new wake does not count, so the index is the same inside Sim::interleave. The work before that point stays done. The work after it never runs. Then call sim.kill() and sim.restart(app) to model the process dying there.

use autumn_web::sim::{crash_at, CrashOutcome};

let point = sim.crash_point().unwrap(); // seeded
let outcome = crash_at(point.await_index, sim.client().post("/transfer").send()).await;
if outcome.is_crashed() {
    sim.crash_and_restart(app_on_the_same_db());
    sim.run_to_idle().await; // recovery runs here
}

To test every crash point, increase index from 0 until the outcome is CrashOutcome::Completed.

Interleaving shuffler

Tokio polls ready tasks in a fixed order. The shuffler changes that order from the seed, so a sweep explores task interleavings, not only timer and fault orders.

let client = sim.client();
let ops = vec![client.post("/reserve").send(), client.post("/reserve").send()];
let responses = sim.interleave(ops).await; // seeded poll order
let handle = sim.spawn(async { /* … */ }); // seeded yields

• Sim::interleave polls the futures in a seeded order each round, and can hold one back for a round. A request's handler runs inside its future, so handler awaits are reordered too.
• Sim::spawn spawns a task that can yield before a poll, which moves it behind the other ready tasks.

The same seed replays the same interleaving. Tasks that the framework spawns itself (job workers, scheduled loops) keep tokio's order.

Simulated network

SimNet replaces the real network for outbound calls through http_client::Client:

use autumn_web::sim::SimNet;

let net = SimNet::new()
    .host("payments", payments_router()) // an in-process axum::Router
    .latency(Duration::from_millis(5), Duration::from_millis(80))
    .drop_rate(0.1);
sim.net(net.clone());
sim.build(TestApp::new().routes(routes![checkout]));

net.partition("payments"); // calls to http://payments/… fail at once
net.heal("payments");
let log = net.events(); // one NetEvent per attempt

• Latency is seeded, in virtual time. A drop fails the attempt after its latency. The client's retry policy applies: attempts, jittered backoff, Retry-After on 429 and 503, 502-504 retries and request_timeout.
• A host with no router falls back to the app's http_mocks. A host with neither is an error. No call reaches the real network.
• The same seed records the same events().

Only a Client built from the app state (the Client extractor or Client::from_state) uses the network, and only after sim.net(..). It needs the http-client feature. Like http mocks, a sim call skips the process-global circuit breaker, the SSRF checks, pin_to and redirect following.

Multiple replicas

A Sim can run several apps on one clock. Each replica is its own app: its own state, job runtime and scheduled tasks. Give each replica its own pool on one SQLite database, so they share data as a fleet does:

use autumn_web::sim::substrate::SqliteSubstrate;
use autumn_web::sim::{Replica, Sim};

#[sim_test]
async fn two_replicas(mut sim: Sim) {
    let substrate = SqliteSubstrate::new().unwrap();
    for spec in [Replica::named("a"), Replica::named("b").clock_behind(Duration::from_secs(2))] {
        let name = spec.name().to_owned();
        let pool = substrate.replica_pool(&sim.db_link(&name)).unwrap();
        sim.mount_replica(spec, TestApp::new().config(config(&name)).with_db(pool).jobs(jobs![work]));
    }
    sim.run_for(Duration::from_secs(60)).await;
    sim.replica("a").get("/status").send().await.assert_ok();
}

• sim.replica(name) returns the replica's client. sim.replica_names() lists the live replicas.
• sim.kill_replica(name) stops the replica's tasks and drops the app, as a process crash does. A job handler that was running stops and does not settle its claim. sim.restart_replica(name, app) mounts it again with the same clock.
• With sim.net(..), each replica is also a SimNet host under its name: http://b/ reaches the replica b. A killed replica is unreachable.
• Process-wide state is not per replica. The cache, the event bus and the global job client belong to the replica mounted last. Enqueue through the replica's own JobClient extension, not job::enqueue.

Advance time with run_for

Use sim.run_for(duration) with replicas on SQLite. Tokio moves its paused clock to the next timer only when all tasks wait and no blocking work is queued, such as a SQLite query. So database work ends before time moves on, and a seed replays the same order. sim.advance and sim.run_to_idle move or drain immediately. They do this also while a query runs, so they do not fix the order of database work.

#[sim_test] runs on sim::runtime(): a paused current-thread runtime with one blocking thread. A blocking task that waits for a second blocking task therefore never ends. Do not nest blocking work in a sim test.

Clocks per replica

Each replica reads its own clock: the sim's elapsed time, scaled by a drift rate, plus an offset.

The drift changes what the replica's clock reads, not its tokio timers: a replica that sleeps 60 s wakes 60 s of sim time later.

Database faults per replica

sim.db_link(name) is the fault switch between one replica and the shared database. The replica's pool must come from substrate.replica_pool(&sim.db_link(name)).

The faults hit only this replica. The write faults use TEMP triggers on the replica's own connection: RAISE(ABORT) before the write, RAISE(FAIL) after it. So only writes fault, not reads. Inside an explicit transaction, the caller rolls back on the error, so an ambiguous fault acts as a mid-query error. Each link draws from its own seeded stream.

Regression tests for the P0 bugs

autumn/tests/sim_replicas.rs holds the two-replica regressions for #3050:

• sim_cron_tick_runs_once_when_a_replica_clock_steps_forward and sim_fixed_delay_tick_runs_once_across_staggered_replicas. With the lease held only for its TTL (the code before the fix) both fail: the late replica runs the tick again.
• sim_long_job_runs_once_across_two_replicas: a 6 s job with a 2 s visibility timeout. Before the claim heartbeat it ran five times. Now it runs once.
• sim_lock_session_loss_past_the_lease_lets_the_peer_in: the SQLite Lock has no loss signal, so a session loss longer than the lease lets a second holder in. The test proves the harness finds that overlap.

Same-seed trace check

sim::trace::capture(future) records every framework tracing event that a run logs, down to TRACE, with its sim time. Run one seed twice and compare:

let (_, first) = autumn_web::sim::trace::capture(scenario(Sim::from_seed(7))).await;
let (_, second) = autumn_web::sim::trace::capture(scenario(Sim::from_seed(7))).await;
assert_eq!(first.diff(&second), None);

A difference is nondeterminism in the app, the framework or the harness. Use capture_targets(&["autumn_web", "my_app"], ..) to record your own crate too. tracing caches callsite interest for the whole process, so a test on another thread can hide events. Run trace checks in their own test binary.

Fuzzing the shared scenario

Autumn's sim-sweep bin and its sim_ops cargo-fuzz target drive one Op vocabulary. Random seeds and coverage-guided search then look at the same state space. An op sequence moves between them: encode a shrunk sweep failure as a fuzz input, and decode a fuzz crash into ops for a test.

cargo +nightly fuzz run sim_ops

The built-in scenario is correct by design, so both are smoke checks of the harness. For your own app, use the same shape: one Op type, a proptest strategy for the sweep, and a byte decoder for a fuzz target.

Worked example: a real retry-storm bug

The harness's value isn't hypothetical — it caught a genuine bug in autumn's own job runtime. The local job runtime's retry backoff computed a pure exponential delay, initial_backoff_ms * 2^(attempt - 1), with no jitter. That delay depends only on a job's configuration, not its identity — so when several jobs in the same queue fail at the same instant (a downstream dependency blips and takes every in-flight job down with it), every one of them computes the identical delay and retries at the identical instant: a synchronized "thundering herd" that immediately re-floods the dependency it just backed off from.

A real-clock integration test can't reproduce this on purpose — it would need N real jobs to fail within the same millisecond of wall time, a coincidence no ordinary test schedule can force. Under the sim's virtual clock it's the opposite: "N jobs fail at the same instant" is the deterministic, trivial condition, because the paused runtime never lets real time elapse between enqueuing them.

#[sim_test]
async fn retries_are_not_synchronized_under_load(mut sim: Sim) {
    // The retry jitter reads `state.entropy()`, which `sim.build` seeds
    // from `sim.seed` (see "Deterministic identifiers" above).
    sim.build(TestApp::new().plugin(StormProbeJobPlugin));

    for id in 0..STORM_SIZE {
        StormProbeJob::enqueue(StormArgs { id }).await.unwrap();
    }
    sim.run_to_idle().await;                    // every job fails, "at once"

    // Step through the backoff window in bounded checkpoints, recording
    // which checkpoint each retry lands in as it drains. A single big
    // `sim.advance(Duration::from_millis(1_500))` looks tempting here, but
    // both the injected `ClockSource` and Tokio's own paused clock jump
    // straight to their target instant in one step *before* any timer
    // fires — so every retry woken by that one big advance would read the
    // *same* post-advance `now()` regardless of its individual delay,
    // silently hiding a real spread (or a real herd) behind a single
    // observed instant. Checkpointing avoids that: each drain only sees
    // the retries whose delay fell inside that specific step.
    for step_ms in [550, 100, 100, 100, 100, 550] {
        sim.advance(Duration::from_millis(step_ms)).await;
        checkpoint += 1;
        sim.run_to_idle().await;                // records `checkpoint` on each retry that lands here
    }

    let distinct_checkpoints = /* … collected from the drained retries … */;
    always!(
        distinct_checkpoints > 1,
        "all retries landed in the same checkpoint of the backoff window — a thundering herd"
    );
}

Before the fix, this always! fired on every seed: every retry landed in the same checkpoint. The fix draws a full-jitter delay, a random value in [0, min(cap, base_delay)], from the framework's injected Entropy seam. The herd spreads out with real OS entropy in production, and replays bit for bit under a fixed sim seed. Issue #3054 moved the helper to autumn_web::backoff and applied it to every job backend and to the HTTP client. See autumn/tests/integration/sim_retry_storm.rs and autumn/tests/integration/sim_retry_storm_http.rs for the tests, and job_retry_delay_ms in autumn/src/job.rs for the job side.

A worked example in an app

examples/reddit-clone/tests/sim_hot_rank.rs is the smallest complete shape of an application #[sim_test]: it mounts a route on the sim's paused runtime, walks 48 virtual hours in checkpoints, and asserts the app's hot-rank decay curve through the ordinary [Clock] extractor rather than around it. It uses always! for the hard invariants and sometimes! for reachability, and it arranges two deliberately-separated input bands so that assert_all_sometimes_satisfied holds at every seed — the pattern to copy when you want a single-run non-vacuity check rather than a sweep.

The reddit-clone nightly

examples/reddit-clone/tests/sim_nightly.rs is a seed sweep over the whole reddit-clone app. The sim-nightly.yml workflow runs it each night.

What one seed does

Each seed mounts the app on a new sim runtime and runs these phases:

An in-memory store is the database of all apps of one seed. It lives longer than a crashed app. Each step that is a database round trip in the real app yields a seeded number of times, so the races of the real app can occur. The checks include:

• Subscribers never get an older vote score after a newer one, per process.
• The score of a post is the sum of its votes.
• Ranks are finite, in order, and fall as a post ages.
• Each post is ranked, has its verdict and has its webhook at the end, after the faults, the partition and the crash.

The guards

• The planted bug. sim_nightly_catches_unordered_publish runs the vote rule without PublishOrder. The sweep must fail, and the failing seed must fail again alone.
• The trace check. The first AUTUMN_SIM_TRACE_CHECKS seeds of each process (2 by default, 8 in the nightly) run twice. The two traces must be the same.
• Not vacuous. Each sometimes! label that a seed reaches must be true in at least one seed of the same process. A window of fewer than 32 seeds, such as a one-seed replay, skips this check. A failure also prints a second replay that runs the earlier seeds of the process too.
• No leak. Each 1,000 seeds, the sweep reads the memory of its process. If the process grew by more than 256 MiB since the first 1,000 seeds, the sweep fails with a memory-leak message. Without this check, a leak fills the runner, and the runner stops with no report.
• The API surface gate. dst_api_surface_is_fully_exercised reads the public fn items of the sim files that do not need sqlite. The nightly must call each one, or EXCLUDED lists it with a reason. The check is textual: it shows that a call exists, not what the call asserts. The SQLite parts (Sim::db_link, DbLink, the substrate, the fleet scenarios) are left out. reddit-clone cannot enable sqlite, because backend_select! then compiles the whole app for SQLite. The sim-sweep job covers them.

Cost and scale

One process runs one seed at a time: the framework keeps the job runtime and the event bus per process. A release build runs about 45 seeds/s on one core, and about 150 seeds/s with 4 processes on a 4-vCPU runner. So 4,000,000 seeds take about 7.5 runner-hours: about 25 minutes on 20 runners. Each night sweeps a new window (start = UTC day number × seeds), so each night tests new seeds.

# The default: 8 seeds and the guard tests. CI runs this on each PR, in debug.
cargo test -p reddit-clone --release --test sim_nightly
# One shard of a large window, with a JSON report.
AUTUMN_SIM_SEED_START=0 AUTUMN_SIM_SEEDS=4000000 AUTUMN_SIM_SHARD=3/64 \
AUTUMN_SIM_REPORT=report.json \
  cargo test -p reddit-clone --release --test sim_nightly -- --nocapture sim_nightly_sweep

A failure prints the seed and a replay command for only that seed.

What it found

On its first sweep the nightly found a bug in reddit-clone. A replica clock that was more than 2 hours slow saw a new post "from the future". The rank formula score / (age + 2)^1.5 then gave NaN. Postgres raised an error on the same POWER in the scheduled rank UPDATE. Thus one such post stopped the rank update for all posts. calculate_hot_rank and its SQL now keep the age at 0 or more.

What's virtualized (and what isn't)

Keeping your own code deterministic

Tokio's paused runtime virtualizes tokio::time::Instant — not std::time::Instant. So a handler or job that measures how long something took with std::time::Instant::now() reads the real machine clock even inside a #[sim_test], and two runs of the same seed disagree. Read time through the seams instead:

Each Sim has its own elapsed-time timeline, and the system clock is another one. ambient_monotonic() readings are comparable only within one timeline. ambient_instant() joins the timelines one after the other: a Sim starts its instants at the latest instant already given out in the process. So state that outlives one Sim sees the next one's instants as later, and its TTLs and idle ages run on. After a nested Sim ends, the outer one's instants go on from where the nested one stopped, so on one thread an ambient instant never goes back. Instants from code outside the sim, or from a Sim on another thread, are not ordered this way: do not share state that stores ambient instants (a cache, a presence map) with those.

Three more rules hold for code that a sim drives:

• Use biased; in tokio::select!. Without it, tokio selects the branch order at random. A stable build cannot set the seed for this order. Two branches that are ready at once then run in a different order per run.
• Do not use the database's random(). Draw from state.entropy() and pass the value in.
• Do not drop a SQLite query in flight. diesel-async panics when it drops one on a current-thread runtime, such as the sim's. A select! drops the futures it owns when another branch wins. Pin the query outside the select! and poll it by reference. If another branch wins, spawn the query. The job worker still drops a handler on a lost claim or a timeout, so a handler query in flight at that time panics in a sim.

The sim-sweep found all three in the framework's own job, scheduler, lock and heartbeat code; they now follow these rules.

If you write a custom impl ClockSource whose now() is virtual, you must also override monotonic(). The trait ships a default body that reads the real process-monotonic clock — that keeps every pre-existing implementation compiling and behaving exactly as before, but it means a virtual clock that forgets to override it silently reports real elapsed time.

The framework holds itself to the same rule on the modules listed in scripts/check-determinism-gate.sh, where a clippy deny-lint enforces it; see CONTRIBUTING.md's "Determinism seam gate" for the enforced subset and the parts of the framework that are not on the seam yet.

A sim-only green run proves your orchestration, timing, ordering, and identity logic — it is not a substitute for testcontainer integration tests against real Postgres, and it does not simulate SQL-dialect or isolation-level behavior. See issue #1797 for the full design rationale and determinism boundary.