Autumn provides first-class ad-hoc background jobs for request-triggered async work.
Define a job
use autumn_web::prelude::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct WelcomeEmailArgs {
pub user_id: i64,
}
#[job(name = "send_welcome_email", max_attempts = 6, backoff_ms = 500)]
async fn send_welcome_email(state: AppState, args: WelcomeEmailArgs) -> AutumnResult<()> {
// perform async side effect
Ok(())
}
Register jobs
autumn_web::app()
.routes(routes![signup])
.jobs(jobs![send_welcome_email])
.run()
.await;
Enqueue from handlers
SendWelcomeEmailJob::enqueue(WelcomeEmailArgs { user_id: 42 }).await?;
Delayed and scheduled jobs
Sometimes you want a job to run once, at a future time — "email a signup
reminder in 24h", "expire this cart in 30 minutes", "publish at 9am", "retry
this external call in 5 minutes". Use enqueue_in (relative delay) or
enqueue_at (absolute instant) instead of enqueue:
use std::time::Duration;
// Run once, 24 hours from now.
SendReminderJob::enqueue_in(ReminderArgs { user_id: 42 }, Duration::from_secs(24 * 60 * 60)).await?;
// Run once, at an absolute UTC instant (an `autumn_web::time::Timestamp`).
let when = autumn_web::time::ambient_now() + autumn_web::time::SignedDuration::from_hours(2);
PublishPostJob::enqueue_at(PublishArgs { post_id: 7 }, when).await?;
The same free functions exist on the job module
(autumn_web::job::enqueue_in(name, payload, delay) /
enqueue_at(name, payload, when)), mirroring enqueue.
A delayed job is recorded immediately but is not delivered to a worker until
its due time passes. Once due, it runs through the normal path — the same
max_attempts / initial_backoff_ms retry/backoff and dead-letter semantics
apply unchanged. An enqueue_at time in the past runs immediately.
Transactional delayed enqueue
Delayed enqueue composes with the transactional variants, so a job is invisible to workers until both the row commits and the due time passes:
use scoped_futures::ScopedFutureExt;
// Crash-safe on Postgres: the future run time is written inside your tx.
db.tx(move |conn| async move {
let cart = carts::create(new_cart, conn).await?;
autumn_web::job::enqueue_in_on_conn(
"expire_cart",
ExpireArgs { cart_id: cart.id },
Duration::from_secs(30 * 60),
conn,
).await?;
Ok(cart)
}.scope_boxed()).await?;
// Process-local after-commit defer (not crash-safe), absolute or relative:
autumn_web::job::enqueue_in_after_commit("send_reminder", args, Duration::from_secs(3600)).await?;
autumn_web::job::enqueue_at_after_commit("publish_post", args, when).await?;
Durability
| Backend | Pending delay survives restart? | How |
|---|---|---|
postgres | Yes (crash-safe) | future run_at column; claim query skips it until due |
redis | Yes (crash-safe) | :delayed ZSET scored by due-time; promoted to the queue when due |
local | No (local-safe only) | in-process timer; a pending delay is lost on restart, consistent with other in-process caveats |
Pick the right tool
| Need | Use |
|---|---|
| Recurring work on a cron / fixed interval | #[scheduled] |
| One-shot "run once, later" timer | delayed #[job] (enqueue_in / enqueue_at) |
| Durable multi-step orchestration, long-horizon timers, history | Autumn Harvest (autumn-harvest-plugin) |
#[scheduled] is for repeating tasks; it does not do one-shot future work.
Autumn Harvest is for durable workflows with history and stronger orchestration
semantics — heavier than a one-shot timer. Delayed #[job] fills the gap
between "now" and "durable workflow".
A job is one unit of work: a retry runs the handler again from the top, so
make it idempotent. The durable backends (postgres, redis, sqlite)
deliver a persisted job at least once. The local backend and
enqueue_after_commit can lose one on a crash or restart (see
Backend selection and
enqueue_after_commit). If the framework would have to
remember where inside the work it got to — per-step checkpoints, a durable
sleep mid-handler, signals, compensation, "run B after A" dependencies — that is
a workflow, and Autumn core deliberately leaves it to Harvest
(ADR-0016). A job may still
enqueue a follow-up job from its own body. When an app write and a workflow
start must commit together, use Harvest's transactional outbox
(enqueue_workflow_start_outbox) on the same connection, the way
enqueue_in_tx works
for jobs.
Admin dashboard
Delayed jobs appear in a distinct Scheduled list on GET /admin/jobs
showing each job's due time, and can be canceled before they run. (A job
that has already become due / started running cannot be canceled.)
Backend selection (autumn.toml)
[jobs]
backend = "local" # local | postgres | redis | sqlite
workers = 2
max_attempts = 5
initial_backoff_ms = 250
max_backoff_ms = 3600000 # cap on the jittered retry backoff (default: 1 h)
default_timeout_ms = 0 # 0 = no limit
[jobs.postgres]
# Reuses the configured [database] pool. No extra URL needed.
visibility_timeout_ms = 30000 # default: 30 000 ms
[jobs.redis]
url = "redis://127.0.0.1/"
key_prefix = "autumn:jobs"
visibility_timeout_ms = 30000
dead_letter_limit = 10000 # default: 10 000; 0 = unbounded
[jobs.sqlite]
# Reuses the configured [database] pool. No extra URL needed.
visibility_timeout_ms = 30000 # default: 30 000 ms
poll_interval_ms = 250 # default: 250 ms
| Backend | Durable | Shared by | Extra infra |
|---|---|---|---|
local | No | Nothing (in-process) | None |
postgres | Yes | Every replica (SKIP LOCKED) | DB only — no Redis |
redis | Yes | Every replica | Redis |
sqlite | Yes | Every process on the host | DB only — no Redis |
local: in-process channel, zero configuration. Jobs are lost on restart. Fine for development or single-process demos.postgres: Postgres-backed queue that reuses your existing[database]pool. Jobs survive restarts and are claimed atomically across replicas viaSELECT … FOR UPDATE SKIP LOCKED. Requires thepostgresfeature (on by default) and anautumn migraterun before the first worker starts.redis: Durable, Redis-backed queue for multi-replica workers. Higher throughput ceiling thanpostgresbut adds Redis as an infrastructure dependency.sqlite: durable queue in the app's own SQLite file, for the single-host SQLite tier. Requires thesqlitecargo feature. The runtime creates the table itself, so no migration is needed. See SQLite delivery semantics.
Web and worker process roles
The same binary can run in one of three process roles, so you can scale the
HTTP tier separately from the background-work tier without touching app code. No
handler, #[job], or #[scheduled] definition changes — only config or a flag.
| Role | Serves user routes | Runs workers + scheduler | Enqueues jobs |
|---|---|---|---|
combined (default) | Yes | Yes | Yes |
web | Yes | No | Yes |
worker | No (probes/actuator only) | Yes | Yes |
combinedis the default and preserves today's single-process behavior: it serves HTTP, drains#[job]workers, and runs the#[scheduled]cron. Existing apps andautumn devneed zero changes.webserves HTTP and still enqueues jobs, but runs no workers and no scheduler, so background work never competes with request handling on a web replica.workerruns the workers and the cron scheduler against the shared durable backend and does not serve user routes — but it still binds the HTTP listener to expose only the liveness/readiness probes (/live,/ready,/startup,/health) and the actuator (/actuator/*, including/actuator/jobs), so an orchestrator can supervise it.
Selecting a role
Three equivalent mechanisms:
# Environment variable — the usual container knob.
AUTUMN_ROLE=web # or: worker | combined | a [roles.<name>] preset
# autumn.toml — top-level key.
role = "worker" # or: web | combined
# CLI flag on the production server command.
autumn serve --role worker # or: --role web
The env var and the TOML key accept the same values. Case is not important.
The aliases are all, web_and_worker and server_and_worker (combined),
server and http (web), and jobs and worker_only (worker).
A value can also name a [roles.<name>] preset. See
Named roles.
An unknown value stops the boot. It does not fall back to combined. The
error names the value, lists the accepted values and lists the declared
presets:
configuration error: AUTUMN_ROLE: unknown process role "workers" (accepted: combined, all, web_and_worker, server_and_worker, web, server, http, worker, jobs, worker_only), and no [roles.workers] is declared (no roles are declared)
An empty AUTUMN_ROLE= is unset: the role key applies, or combined.
autumn doctor shows the same error on its process_role_backend row.
Role capabilities
A process resolves its role into a capability set once at boot. Each framework subsystem reads its own capability:
| Capability | Subsystem | combined | web | worker |
|---|---|---|---|---|
serves_http | User routes | Yes | Yes | No |
job_workers | #[job] worker loops | Yes | No | Yes |
scheduler | #[scheduled] tasks | Yes | No | Yes |
outbox_relay | Outbox relay | Yes | No | Yes |
commit_hooks | Durable commit-hook workers (app pool and each shard) | Yes | No | Yes |
fleet_relay | SQLite fleet sweeper: open closed databases, relay, backfill | Yes | No | Yes |
restore_check | Fleet replication restore check | Yes | No | Yes |
retention | Retention sweeps (#[repository(retention(...))], [retention]) | Yes | No | Yes |
maintenance | Plugin and app housekeeping background tasks | Yes | No | Yes |
Read the set with state.capabilities(). Read one field, or ask about one
subsystem with state.capabilities().runs(Capability::Scheduler).
GET /actuator/info shows the set under role. For a worker:
"role": {
"name": "worker",
"base": "worker",
"preset": false,
"capabilities": {
"serves_http": false,
"job_workers": true,
"scheduler": true,
"outbox_relay": true,
"commit_hooks": true,
"fleet_relay": true,
"restore_check": true,
"retention": true,
"maintenance": true
}
}
serves_http() and runs_workers() stay on ProcessRole and Role. Framework code
does not use runs_workers() as a gate: a test fails on a new gate in
autumn/src/app.rs, so a new subsystem must name its capability.
Background tasks for plugins and apps
The framework gates each subsystem in the capability table
on its own capability. A loop that a plugin or an app starts itself, with
tokio::spawn in on_startup, runs on every replica. Do not do that.
Declare the loop. The framework then places it:
use autumn_web::Capability;
app.background_task("app.cleanup", Capability::Maintenance, |state, shutdown| async move {
loop {
tokio::select! {
() = shutdown.cancelled() => return Ok(()),
() = tokio::time::sleep(std::time::Duration::from_secs(60)) => reap(&state).await?,
}
}
})
- The framework starts the task after the startup hooks, only on a role that
runs its capability.
maintenanceruns oncombinedandworker, and not onweb. - The task gets a child of the server shutdown token. The graceful drain
waits for the tasks for at most half of the remaining
server.shutdown_timeout_secs. If a task does not stop in that time, the framework aborts it. The fleet close and theon_shutdownhooks keep the other half. - The framework does not restart a task that returns or panics.
/actuator/tasksshows its status. autumn task(one-off tasks) starts no background tasks.
Use the capability that matches the work: job_workers for a queue
consumer, scheduler for a periodic loop, maintenance for housekeeping.
Plugin capabilities
A plugin can declare its own capability. Its parent, a built-in capability, gives the default:
use autumn_web::Capability;
use autumn_web::background::PluginCapability;
pub const WORKER: PluginCapability = PluginCapability::new("harvest.worker", Capability::JobWorkers);
app.background_task("harvest.dispatch", WORKER, |state, shutdown| async move { /* ... */ Ok(()) })
A named role can set it:
[roles.cron]
base = "worker"
pin = [] # no job workers: harvest.worker is off
capabilities = { "harvest.worker" = true } # turn it on for this role
- A plugin capability name is
<plugin>.<name>. - A role cannot turn on a plugin capability that its base does not run, as
for the built-in keys.
trueis valid only when the base runs the parent. - A built-in name in
capabilitiesfails the load. Use its top-level key. trueabove the base fails the boot, on every declared preset.- A key that no plugin declares fails the boot of that role. On another
preset, the boot logs a warning: that binary may not link the plugin.
autumn graphprints the same warnings. - Code that is not a task reads the same value with
state.role().runs_plugin(WORKER). Declare such a capability withapp.plugin_capability(WORKER).
Queues and explicit switches
app.background(BackgroundTask) is the full form:
use autumn_web::background::BackgroundTask;
app.background(
BackgroundTask::new("harvest.dispatch", WORKER, dispatch)
.queues(["harvest"]) // the role must drain one
.explicit(config.worker_enabled), // Option<bool> from the plugin config
)
- Queues. The task runs only on a role that runs job workers and drains
one of its queues. The role
pin(orjobs.pin) applies, as for#[job]queues.autumn jobs manifestlists the queues, so theautumn doctorcoverage check includes them. A pin that names only a task queue is not an error. - Explicit switch. The plugin config wins.
Some(true)runs the task on every role. The boot logs a warning on a role that does not place it.Some(false)never runs the task, and its queues leave the manifest.Nonelets the role decide. - The boot logs a warning for a task that no role of the config runs.
Where to see them
-
GET /actuator/taskshasbackground_tasks, a map from task name to:Key Value capabilityThe capability name, built-in or plugin parentThe built-in parent of a plugin capability queuesThe queues of the task explicitThe explicit switch, when set registered_byThe plugin that registered the task. Not set for the app locationfile:lineof the registrationrolesThe roles of the loaded config that run the task runs_hereWhether this process runs the task statusrunning,not_placed,disabled,stopped,exited,failed,panickedorabortedlast_errorThe error or panic message /actuator/tasksis mounted only with[actuator] sensitive = true. -
autumn graphhas onebackground_tasknode for each task, with the same roles. -
rolesuses the loaded config, with its env overrides. For example, a process withAUTUMN_JOBS__PIN=criticalgives each built-in role that pin.
Tasks of the in-repo plugins
| Plugin | Task | Capability | Queues | Default roles |
|---|---|---|---|---|
autumn-media-plugin | media.room_reaper (db store) | media.maintenance | combined, worker | |
autumn-media-plugin | media.room_reaper (memory store) | serves_http | combined, web | |
autumn-media-plugin | media.recording_retention | media.retention | media (or .queue(...)) | combined, worker |
autumn-billing | billing.dunning_rearm (re-arms once per process) | billing.dunning | billing | combined, worker |
autumn-plugin-reference | reference.heartbeat | maintenance | combined, worker |
The memory room store is in the memory of the process that serves the room
routes, so its reaper runs there. The media guide gives the reason for each
row. The job queues of autumn-media-plugin, autumn-billing and
autumn-search come from their JobInfo lists, so autumn jobs manifest
and the role pin include them.
A test in xtask (plugin_spawns) fails on a spawn that a plugin crate's
on_startup argument reaches, directly or through a function of the same
crate. Request-scoped spawns go on its allow-list.
state.role() returns a &Role (exported at autumn_web::Role).
role.name() is the role name, for example "critical" or "worker".
role.base() is the built-in ProcessRole. state.capabilities() returns its
RoleCapabilities, one field for each row of the capability table above.
Named roles
Declare a tier once in autumn.toml as a [roles.<name>] preset.
Each Deployment or Fly process then sets only AUTUMN_ROLE=<name>.
autumn release init --target kubernetes and --target fly write one
workload per preset (see Kubernetes),
and [[deploy.groups]] sets a role per fleet host group (see
Fleet deploys):
[roles.critical]
base = "worker" # combined | web | worker: the safety contract
pin = ["critical"] # replaces jobs.pin
workers = 8 # replaces jobs.workers
scheduler = false # capability overrides; unset = base default
outbox_relay = false
[roles.cron]
base = "worker"
pin = [] # drains no queue: no job workers
scheduler = true
[roles.marketing]
base = "combined"
Select a preset with AUTUMN_ROLE=critical, role = "critical" or
autumn serve --role critical.
| Key | Value | Unset |
|---|---|---|
base | combined, web or worker. Required. | — |
pin | Queues this role drains. [] = no job workers. | jobs.pin |
workers | Concurrent worker loops. 1 or more. | jobs.workers |
job_workers, scheduler, outbox_relay, commit_hooks, fleet_relay, restore_check, retention, maintenance | true or false | the base default |
capabilities | Plugin capability values, for example { "harvest.worker" = false }. See Plugin capabilities. | the parent value |
mount | Route groups this role serves. Needs a web or combined base. See Route groups. | every group |
Rules:
- The base is the ceiling. An override can only turn a capability off.
scheduler = trueon awebbase fails the load. Usebase = "combined"and turn the other capabilities off. - The base decides HTTP.
serves_httpis not a key. Aworker-based preset serves only the probes and the actuator. - The guards read the base. A preset over
weborworkerneeds a durable jobs backend, as the built-in roles do. On thesqlitejobs backend, it also needs a file-backed database. Acombinedpreset with no job workers counts asweb: it enqueues jobs and does not drain them. See Split roles require a durable backend. - The built-in names are reserved.
[roles.worker],[roles.jobs]and every other built-in spelling fail the load. - A name starts with a lowercase letter and has only
a-z,0-9,_and-, with at most 63 characters. Case inAUTUMN_ROLEis not important. - A preset holds only role keys. An unknown key fails the load, so a preset cannot become a second config profile.
- A preset
pinwins overjobs.pinfrom every source, includingAUTUMN_JOBS__PINandautumn serve --pin. The boot logs a warning when it replaces a different pin. [roles]belongs to the framework. Do not use a top-level[roles]table for app data.pinwith no job workers,pin = []withjob_workers = true,workerswith no job workers, andworkers = 0fail the load.- An undeclared name fails the load. The error lists the declared presets.
The name is visible in these places:
state.role().name()in handlers,on_startuphooks, plugins andstate_initializer.GET /actuator/info(role.name,role.base,role.preset) andGET /actuator/jobs(role.name,role.base,role.pin).- A
role="<name>"label on every framework metric family in/actuator/prometheus. A built-in role adds no label, so its series do not change. - The default scheduler
replica_id:<name>-<host>. An explicitscheduler.replica_idstays as it is.
autumn doctor --strict derives [jobs.fleet] tiers from the presets when
tiers is not set. See Zero-coverage guard.
Split roles require a durable backend
A split web/worker topology requires a durable jobs backend
(jobs.backend = "postgres", "redis", or "sqlite"). The default local
backend is an in-process, in-memory queue: a web replica would enqueue into
its own memory, where no separate worker replica can ever drain it.
sqlite qualifies because the queue is a table both processes open — but only
while they run on the same host, which a SQLite deployment does by
definition, and only against a file-backed database. An in-memory target
(sqlite::memory:, file::memory:, ?mode=memory) is private to each process,
so a split role on one is refused at boot and flagged by
autumn doctor --strict.
Autumn rejects this combination at startup — a web or worker role on the
local backend exits with a clear error rather than silently dropping work —
and autumn doctor --strict reports it as a Fail. combined on local is
always fine, because one process both enqueues and drains.
[jobs]
backend = "postgres" # or "redis" / "sqlite"; required once role is "web" or "worker"
Graceful drain on workers
A worker-role process joins the same shutdown sequence as a web replica. On
SIGTERM it stops claiming new jobs, gives in-flight jobs the configured drain
window (server.shutdown_timeout_secs / server.prestop_grace_secs) to finish
or be released back to the queue, then exits cleanly. Its /ready probe flips
to 503 during the drain, so an orchestrator supervises a rolling worker deploy
exactly as it would a web deploy. See
Rolling Deploy Lifecycle for the
full phase ordering.
Where job execution shows up
/actuator/jobs and the admin dashboard attribute each job's execution to the
process that actually ran it. In a split topology that is always a worker
process — web replicas only enqueue, so handler execution, in-flight counts,
and last-error data surface on the worker tier.
Postgres delivery semantics
The Postgres backend provides at-least-once delivery. Each job is a row in
the autumn_jobs table. Workers claim a row atomically with
UPDATE … WHERE id IN (SELECT … FOR UPDATE SKIP LOCKED), which prevents any
two replicas from claiming the same job simultaneously.
Workers poll. There is no LISTEN/NOTIFY wake-up. A busy worker claims the
next job at once. An idle worker polls again after 200ms. A new job can wait up
to 200ms, also when the same process enqueued it. The 200ms interval is fixed.
A claimed job's status is set to running with a claimed_at timestamp and a
claimed_by worker id. While the job runs, the worker renews claimed_at (see
Claim leases and timeouts). A maintenance loop
running inside each worker process requeues jobs whose claimed_at is older
than jobs.postgres.visibility_timeout_ms. That happens only when the worker
stopped renewing, for example after a crash. Recovered stale claims consume
another attempt and record a last_error explaining the visibility timeout.
If a job exhausts max_attempts, its status is set to failed; it is no longer
retried.
Because the backend provides at-least-once delivery, handlers must be idempotent. If the worker crashes after a side effect and before the ack, the job runs again. Use an idempotency key for each external side effect: the job id, a domain aggregate id, or a provider idempotency token.
Redis delivery semantics
The Redis backend provides at-least-once delivery. A job is written as a
durable record, queued by id, atomically claimed into an in-flight set, and
acked only after the handler returns Ok(()).
If a worker crashes after claiming a job, the record remains in Redis. Another
worker requeues the stale claim after jobs.redis.visibility_timeout_ms. A
live worker renews its claim, so this does not happen to a slow job. Claim
deadlines use the Redis server clock (TIME). A worker with a skewed clock
does not see a live claim as expired. The scripts need Redis 3.2 or later.
Recovered stale claims consume another attempt and retain a last_error
explaining the visibility timeout. If the job has exhausted max_attempts, it
is moved to the dead-letter list instead of being requeued.
Because Redis uses at-least-once delivery, handlers must be idempotent. If the worker crashes after a side effect and before the ack, the job runs again. Use an idempotency key for each external side effect: the job id, a domain aggregate id, or a provider idempotency token.
Redis dead-letter retention
The Redis backend keeps dead letters in a list, newest first. Each entry also has a per-id record that the dashboard uses to retry or discard the job.
jobs.redis.dead_letter_limit sets the maximum length of that list. The
default is 10 000. Before, the limit was 1 000 and you could not change it.
Each dead letter uses Redis memory two times: one list entry and one per-id
record. Thus the new default can use 10 times more memory.
When the worker adds a dead letter and the list becomes longer than the limit,
the worker removes the oldest entries and their per-id records. One dead letter
removes at most 1 000 entries, so one script call does not block Redis for
long. If you decrease the limit, the list becomes shorter over the next dead
letters. Set the limit to 0 to keep all dead letters. Then you must remove
old entries yourself, because Redis memory increases with each dead letter.
A trim is never silent:
- The worker logs a
warnevent withtrimmed,dead_letter_limitanddead_key. - The worker adds the count to
autumn_jobs_dead_letter_trimmed_totalon/actuator/prometheus. The counter is per process.
Alert on increase(autumn_jobs_dead_letter_trimmed_total[1h]) > 0. After a
trim, you cannot replay the removed jobs.
The dashboard shows the newest 1 000 dead letters. Older entries stay in Redis, and you can retry or discard them by id.
Dead-letter retention by backend
| Backend | Retention of dead letters |
|---|---|
redis | The newest jobs.redis.dead_letter_limit entries (default 10 000; 0 = all). |
postgres | All failed rows. No count limit. If you set retention.job_history, the sweep deletes rows older than that window. If you do not set it, the rows stay. |
sqlite | All failed rows. No count limit. If you set retention.job_history, the runtime deletes rows older than that window. If you do not set it, the rows stay. |
local | In memory only. The process loses them at restart. The dashboard keeps the newest 1 000 finished jobs. |
SQLite delivery semantics
The SQLite backend provides at-least-once delivery with the same row
lifecycle as Postgres. Each job is a row in an autumn_jobs table in the app's
own database file, so work survives a restart with no Redis and no Postgres.
SQLite serializes writers, so a worker claims a row with a single statement —
UPDATE … WHERE id = (SELECT … ORDER BY run_at LIMIT 1) RETURNING … — which is
the single-host analog of FOR UPDATE SKIP LOCKED. Two workers can never claim
one row.
A claimed row is running with a claimed_at timestamp and a claimed_by
worker id. The worker renews claimed_at while the job runs. A maintenance loop
re-enqueues rows whose claimed_at is older than
jobs.sqlite.visibility_timeout_ms, at start and on an interval, so a crash
mid-job loses nothing. A recovered claim consumes another attempt and records a
last_error. A job that exhausts max_attempts becomes failed and is not
retried.
There is no LISTEN/NOTIFY. An enqueue in the same process wakes a worker
directly; work another process enqueued is seen within
jobs.sqlite.poll_interval_ms (default 250ms). Lower it for latency, raise it
to cut idle wakeups.
The runtime prunes terminal rows. autumn db retention also sweeps them. Set retention.job_history to bound the table; leave it
unset and history is kept forever, exactly as on Postgres.
The runtime creates the table and its indexes at start, because framework migrations are Postgres SQL. Nothing to run by hand.
Because delivery is at-least-once, handlers must be idempotent — the same rule as every other durable backend.
Claim leases and timeouts
A durable worker (postgres, redis, sqlite) holds a lease on each job it
runs. A heartbeat renews the lease every third of the visibility timeout. Thus,
while renewals succeed, a long job does not run again on a second worker. The
minimum visibility timeout is 30ms. A worker uses 30ms if the configured value
is shorter, and logs a warning.
- Crash. The heartbeat stops with the process. After the visibility timeout, another worker recovers the job and runs it again.
- Lost lease. If a renewal finds that the worker no longer holds the claim, the worker stops the handler. It does not ack, retry, or fail the job. The worker that holds the claim now owns the job.
- Failed renewal. If a renewal fails (for example, a database error), the worker logs the error and tries again. The handler continues. If no renewal succeeds for two thirds of the visibility timeout, the worker stops the handler as for a lost lease. Another worker can recover the job soon after.
Warning: with no timeout, a hung handler keeps its claim until the process stops, because the heartbeat continues to renew it. Set a timeout on a job that can hang.
Set a timeout to stop a handler that hangs:
#[job(timeout = "30s", max_attempts = 3)]
async fn export_report(state: AppState, args: ExportArgs) -> AutumnResult<()> {
// ...
Ok(())
}
timeout accepts ms, s, m, h, and d units, for example "500ms"
or "1m 30s". A job with no timeout uses jobs.default_timeout_ms. 0, the
default, means no limit. A run that exceeds its timeout fails with
job timed out after <n>ms and retries like any other failure. The worker is
free for the next job at once. The timeout applies on every backend, local
too. The timeout stops a handler only at an .await. A handler that blocks its
thread does not stop.
For a tracked job, the timeout also covers the mark_running call to the
tracking store. If the store stalls, the run fails with the timeout error. The
normal retry rules apply. Each terminal complete or fail call has its own
cap of 5 seconds. This includes the fail call for a cancelled job. When the cap is reached, the worker logs a warning and continues.
The tracked record then expires through its TTL.
When the worker stops a run (lost lease or timeout), it drops the handler future. Work that the handler spawned continues, so check the signal there:
#[job(timeout = "10m")]
async fn crunch(state: AppState, args: CrunchArgs) -> AutumnResult<()> {
let ctx = autumn_web::job::JobContext::current();
let _ = tokio::task::spawn_blocking(move || {
for chunk in args.chunks() {
if ctx.is_cancelled() {
return; // lease lost or timeout: stop here
}
process(chunk);
}
})
.await;
Ok(())
}
JobContext::lease_lost()istrueafter the worker lost its claim.JobContext::is_cancelled()istrueafter a lost lease or a timeout.JobContext::cancelled()waits for either.
An operator cancel from the dashboard does not set these. It applies only to a job that did not start.
ADR 0016 records the design.
Retry/backoff and dead letters
- Jobs retry with exponential backoff and full jitter: the wait is a random
value in
[0, min(max_backoff_ms, initial_backoff_ms * 2^(attempt-1))]. Jobs that fail together do not retry together. All four backends use the same rule. Under aSim, the jitter replays from the seed. max_backoff_mscaps the wait. Default: 3 600 000 (1 hour). Set it in[jobs]or withAUTUMN_JOBS__MAX_BACKOFF_MS.- A durable backend recovers a claim after its visibility timeout (the worker crashed or the handler hung). The recovered job uses the same jittered wait, so claims that expire together do not all run again at once.
- Retries stop at
max_attempts(job-level override or config default). - Exhausted jobs are dead-lettered.
- Redis retries are scheduled in Redis before the worker moves on, so a crash during the backoff window does not drop the job.
Cost-aware deferral
Mark a job #[job(deferrable)] to let it wait while the cost signal is above
[cost] defer_threshold. The job runs when the signal falls. The runtime never
drops it. It uses no attempt while it waits. Only the local backend defers
jobs. Other backends run the job and log a warning at boot. See
Request Cost and Carbon-Aware Deferral.
Job priorities
By default every job drains from a single FIFO queue, so a flood of low-value work (analytics rollups, thumbnails, bulk re-indexing) can sit ahead of latency-sensitive work like password-reset emails or payment-webhook fan-out. Named queues fix this head-of-line blocking: route each job to a queue, and let workers drain queues in priority order.
Tag a job's queue with queue = "...". Jobs with no queue land on the
"default" queue, so apps that don't opt in behave exactly as before.
#[job(queue = "critical", max_attempts = 5)]
async fn send_password_reset(state: AppState, args: ResetArgs) -> AutumnResult<()> { … }
#[job(queue = "low")]
async fn rebuild_search_index(state: AppState, args: IndexArgs) -> AutumnResult<()> { … }
// No queue → the "default" queue.
#[job]
async fn send_receipt(state: AppState, args: ReceiptArgs) -> AutumnResult<()> { … }
Configure the worker drain order in autumn.toml. Two forms:
# Strict priority — workers always empty higher queues before lower ones.
# A single `critical` job jumps ahead of a 1,000-job `low` backlog.
[jobs]
queues = ["critical", "default", "low"]
# Weighted — fair draining that never starves a lower queue. Over a sustained
# mixed load each queue is served in proportion to its weight (here roughly
# 4 : 2 : 1), so `low` always makes forward progress even while `critical` has work.
[jobs.queues]
critical = 4
default = 2
low = 1
- Strict (
queues = [...]) is the simple case: highest priority first, and a worker only pulls a lower queue when every higher queue is empty. - Weighted (
[jobs.queues]table) avoids starvation under sustained load: it uses smooth weighted round-robin, so each queue is the first choice in proportion to its weight over each cycle.
Routing is honored end-to-end on every backend (local, Redis, Postgres): the
queue is preserved through retries/backoff, dead-lettering, delayed enqueues, and
enqueue_after_commit. The actuator/admin job view shows each job's queue.
If a job declares a queue that is not in the configured drain list, that is
a loud, documented condition — it is logged at startup (WARN) and the queue is
appended at lowest priority so the job still drains instead of silently stalling.
Add the queue to [jobs] queues to control its priority.
Role-based scaling — running a separately scaled worker tier that drains all configured queues — is covered in Web and worker process roles.
Per-queue worker pools, caps, and pinning
By default every worker process draws from one shared pool of jobs.workers
slots and drains all configured queues. Three config-only knobs (no application
code changes) let you carve that pool up per queue and dedicate worker processes
to specific queues.
Dedicated capacity and per-queue caps
Extend the weighted [jobs.queues] table so a queue's value is a table instead
of a bare weight. Two per-queue controls are available:
reserved = N— dedicated slots that no other queue may ever consume. A flood on another queue can never starve this one:Nof the process's worker slots are always available to it.concurrency = N— a hard cap: this queue may occupy at mostNof the process's worker slots at once, so a bulk queue can never take more than its configured share.
[jobs]
workers = 8
[jobs.queues]
# `critical` keeps 2 slots reserved for it at all times — a slow flood on
# `bulk` can never delay a password-reset email.
critical = { weight = 4, reserved = 2 }
# `bulk` is capped at 4 of the 8 slots, so re-indexing never monopolizes the
# process.
bulk = { weight = 1, concurrency = 4 }
# A bare integer is still just a weight (no cap, no reservation).
default = 2
The bare-integer form (critical = 4) and the strict array form
(queues = ["critical", "default", "low"]) keep working unchanged; a queue with
neither reserved nor concurrency behaves exactly as before. Slots are
accounted per process: total capacity is always jobs.workers, and the
reserved/cap rules only redistribute how those slots are shared between queues.
The accounting is enforced on every backend (local, Redis, Postgres): before a worker claims a job it restricts itself to the queues that currently have a free slot, so a queue at its cap — or whose only free slots are reserved for another queue — is skipped rather than blocked behind.
Pinning a worker tier to a subset of queues
Combine per-queue pools with process roles to
dedicate an entire worker tier to a subset of queues. Set jobs.pin (or the
AUTUMN_JOBS__PIN environment variable, comma-separated) to the queues that
process should claim:
# A worker replica dedicated to latency-sensitive work.
# `role` is a TOP-LEVEL key (AutumnConfig.role) and must appear before any
# `[table]` header — equivalently, set `AUTUMN_ROLE=worker` in the environment.
role = "worker"
[jobs]
backend = "redis"
pin = ["critical"]
# Same thing via env — handy for a separate deployment of the same image.
AUTUMN_JOBS__PIN=bulk,default
# Or as a flag, when one image is started under several roles from a script.
# `--pin` is repeatable and comma-separated, and sets `AUTUMN_JOBS__PIN` for the
# app process; `autumn serve restart` restores it.
autumn serve --role worker --pin critical
autumn serve --role worker --pin bulk --pin default
A pinned process never claims jobs from queues outside its subset, on both
the Postgres and Redis backends. Weighted/strict ordering is preserved within
the pinned subset. An empty/unset pin (the default) keeps today's behavior:
the process drains every configured queue from the single shared pool.
Zero-coverage guard
If pinning leaves a configured queue with no worker coverage anywhere —
e.g. you pin every worker to critical but still enqueue to bulk — those jobs
would silently accumulate. Autumn diagnoses this loudly:
The runtime startup warning is the authoritative check: a worker process
that pins to a subset logs a WARN at startup naming the configured queues it
does not cover (and an ERROR if the pin matches no queue on its effective
schedule at all, so the process would claim nothing). This runs inside the
app, against the job registry and the real effective schedule, so it knows both
the [jobs.queues] config and any queues declared solely via
#[job(queue = "…")] — and diagnoses a genuinely uncovered queue loudly at
boot.
autumn doctor reports coverage as a jobs_queue_coverage check. What that
check can prove — report only, or fail the run — depends on how much of the
deployment you have declared.
Without a declared topology it only reports. Doctor is config-only and
per-process: it sees one process, so it cannot know what sibling worker
tiers drain — a multi-tier deployment where each pinned tier omits queues
covered by other tiers (one process AUTUMN_JOBS__PIN=critical, another
AUTUMN_JOBS__PIN=bulk,default) is perfectly legitimate. It also inspects only
[jobs.queues] (plus the implicit default queue), so it cannot see a queue
declared solely via #[job(queue = "…")], which the runtime appends to the
effective schedule and drains. Any hard-fail asserted from that vantage point
would false-positive on a valid deployment, so the check passes and prints what
this tier claims, which configured queues it does not claim, and any pinned
queues absent from [jobs.queues].
Declare the fleet topology and doctor can hard-fail. Give doctor every
worker tier's pin under [jobs.fleet] and it reasons about the union of what
the fleet drains instead of one process, which makes a zero-coverage gap
provable:
[jobs.queues]
critical = { weight = 4, reserved = 2 }
bulk = { weight = 1, concurrency = 4 }
default = 2
[jobs.fleet]
# One entry per worker tier, each holding that tier's `jobs.pin`. This must list
# EVERY tier you actually run — see the warning below.
# An empty entry (`[]`) is an unpinned tier that drains every queue.
tiers = [["critical"], ["bulk", "default", "thumbnails"]]
# Optional — teach doctor about queues declared in code but absent from
# `[jobs.queues]` (here, a `#[job(queue = "thumbnails")]`), so they count toward
# coverage too. List them inline…
declared_queues = ["thumbnails"]
…or, better, point at a manifest the app emits, which is the ground-truth set
rather than a list you maintain by hand. Generate it with
autumn jobs manifest <path> (it runs your app binary to dump the queues it
really drains) and reference it instead of declared_queues — when both are
set, manifest wins and declared_queues is never read:
[jobs.fleet]
tiers = [["critical"], ["bulk", "default", "thumbnails"]]
manifest = "target/jobs-manifest.toml"
With a topology declared, autumn doctor fails — exit 1, in every mode, not
only under --strict — when some queue that must be drained is claimed by no
tier anywhere, naming the uncovered queues. (--strict escalates warnings; a
failed check is fatal regardless.)
manifest and declared_queues can only add to the set of queues that must
be covered, so an incomplete or missing manifest never manufactures a failure —
it just leaves doctor blind to the queues it did not see.
tiersmust be exhaustive. It is the one input where under-declaring is unsafe: doctor can only reason about the tiers you list, so omitting a tier that really runs — especially an unpinned one, which would make coverage total — reports a gap that does not exist. Treat[jobs.fleet] tiersas the checked-in description of the whole fleet, and update it when you add or remove a worker tier.
Or declare the tiers once as [roles.*]. When tiers is not set,
doctor derives it from the named roles. Each preset that runs
job workers is one tier: its pin, else jobs.pin. A preset with no job
workers (pin = [], job_workers = false, or a web base) is not a tier.
The check detail says derived from [roles.*]. An explicit tiers wins.
tiers = [] is the same as no tiers.
A gap in derived tiers is a warning, not a failure, because a tier that
runs as a plain built-in role can drain the gap. autumn doctor --strict
turns the warning into a failure.
[roles.critical]
base = "worker"
pin = ["critical"]
[roles.bulk]
base = "worker"
pin = ["bulk", "default", "thumbnails"]
The derived tiers list only the presets. A tier that runs as a built-in role
(for example a plain AUTUMN_ROLE=worker Deployment) is not in the list.
Declare it as a preset, or set tiers.
[jobs.fleet] is purely declarative. A running process never acts on it — it
describes the other tiers, which no single process can observe — so it changes
no runtime behavior, and an app that declares nothing keeps today's behavior.
Absent that declaration, ensuring every queue is drained by some tier remains the operator's responsibility: make sure some process (an unpinned worker tier, or one pinned to those queues) covers every queue you enqueue to.
Observability
The actuator jobs endpoint (<actuator-prefix>/jobs) reports per-queue queue
depth and the age of the oldest still-waiting job under a queues key, in
addition to the existing per-job-type gauges under jobs:
{
"jobs": { "send_password_reset": { "queued": 0, "in_flight": 1, "...": 0 } },
"queues": { "critical": { "depth": 3, "oldest_waiting_age_ms": 1200 } },
"role": { "name": "critical", "base": "worker", "pin": ["critical"] }
}
The role key names the process role. pin is there only when a
named role sets it.
On the durable backends (Postgres/Redis) these gauges are not per-process
approximations: a periodic survey of the durable store wholesale-replaces this
process's local enqueue marks each tick, so an enqueue-only web
replica reports the true shared backlog, and a queue absent from the latest
survey resets to depth 0 rather than leaking a stale one. On the in-process
local backend there is only one process, so the local marks are the whole
picture.
Still out of scope (separate follow-up): per-job-instance dynamic priority at enqueue time.
Uniqueness and concurrency limits
#[job] can declare dedup and in-flight caps directly, so double-submits and
bursts cannot duplicate side effects or overwhelm downstream systems — no
hand-rolled advisory locks in job bodies.
// At most one identical sync in flight: a burst of N identical enqueues
// runs exactly once. The key defaults to a stable hash of the full args.
#[job(unique)]
async fn sync_search_index(state: AppState, args: SyncArgs) -> AutumnResult<()> { … }
// Key by selected args fields, and cap simultaneous executions per account.
#[job(unique_by = "account_id", concurrency = 1, concurrency_key = "account_id")]
async fn recalculate_account(state: AppState, args: RecalcArgs) -> AutumnResult<()> { … }
// Debounce: coalesce repeat enqueues for 60s from the first enqueue,
// even after the job completed.
#[job(unique_for_ms = 60_000)]
async fn rebuild_report(state: AppState, args: ReportArgs) -> AutumnResult<()> { … }
Attributes:
| Attribute | Meaning |
|---|---|
unique | Dedupe on a stable hash of the full args payload. |
unique_by = "a, b" | Dedupe on the listed args fields (implies unique). |
unique_window = "running" | Default: key held while the job is pending or running; released when it settles. |
unique_window = "pending" | Key released when execution starts, so a new instance may queue while one runs. |
unique_for_ms = N | TTL window: key held for N ms from enqueue (and while in flight on Postgres), even past completion. Mutually exclusive with unique_window. |
concurrency = N | At most N simultaneously-executing jobs of this type. |
concurrency_key = "field" | Scope the limit per distinct value of this args field. |
Semantics:
- A coalesced enqueue is a no-op
Ok(()); it is counted astotal_deduplicatedin/actuator/jobsand recorded with thededuplicatedjob-admin status. - Jobs over the concurrency cap wait (they stay enqueued/parked and run when a slot frees) — they are never dropped. They stay on the admin dashboard's enqueued tab; Redis also marks the row "waiting on a concurrency slot".
- Keys and slots are released on success, terminal failure, and worker crash: Postgres ties them to row status recovered by the visibility timeout; Redis settles them in the claim-validated transition and stale-recovery scripts, with a TTL backstop on lock keys.
- Enforcement is distributed-safe across replicas on the durable
backends: Postgres uses a partial unique index plus
ON CONFLICT DO NOTHINGfor dedup and (only when a limited job is registered) a transaction-scoped advisory lock around claims; Redis usesSET NX PXlocks and atomic Lua claim/settle scripts. - With neither attribute set, behavior is unchanged: no dedup and unbounded per-type concurrency.
- Retries keep a
running-window key held (the job is still in flight) and re-acquire apending-window key while waiting out the backoff; the concurrency slot is released during the backoff either way. - After a pending-window job's first execution attempt, dedup is best
effort: the key is released when execution starts (that is the window's
contract), so a duplicate accepted while the job runs legitimately holds
the key, and a retry or crash-recovered attempt then waits as pending
without it. Workloads that must never overlap should use the default
runningwindow, which holds the key until the job settles. - Operator actions respect uniqueness: canceling an enqueued job (including
one parked behind a concurrency slot) releases its key immediately, and
retrying a failed unique job re-takes the key — or fails with a clear
conflict error when an equivalent job is already pending or running. A
retried
pending-window job gets its key back at once, so an equal enqueue stays deduplicated while the retry waits. - On Redis, pending/running unique locks carry a 24-hour crash backstop TTL that is refreshed every time the job is claimed, retried, or recovered, so only a job left completely untouched for a full day can lose its lock.
- The Postgres backend needs the additive
autumn migratemigration that adds the nullableunique_key/unique_window/concurrency_key/concurrency_limitcolumns; rows and jobs without them behave as before.
Tracked jobs and progress polling
Plain enqueue is fire-and-forget — there's no handle, no progress, and
nowhere for the caller to check "is it done yet?". enqueue_tracked fixes
that: it returns a handle carrying a public, unguessable token, distinct
from the internal job id, that the browser can poll at a built-in status
route while the job reports progress from the inside.
Enqueue a tracked job
let handle = ExportOrdersJob::enqueue_tracked(ExportArgs { account_id: 42 }).await?;
// handle.token is the raw, unguessable token — deliver it to the caller.
// handle.status_path() is "/_autumn/jobs/{token}".
By default the token is an anonymous capability: anyone holding it can
poll the status. To bind status access to the caller's session/user instead,
use enqueue_tracked_for with an owner derived from the current session:
use autumn_web::job::TrackedJobOwner;
let owner = TrackedJobOwner::from_session(&session, &state).await;
let handle = ExportOrdersJob::enqueue_tracked_for(args, owner).await?;
A request whose session doesn't match the bound owner gets the identical
404 an unknown token would — the route is never an existence/ownership
oracle.
enqueue_tracked/enqueue_tracked_for wrap your Args in an internal
envelope under a reserved top-level field named __autumn_tracked. Don't
give a job's Args struct a field with that exact name — enqueue/
enqueue_in/enqueue_at and their on_conn variants reject a payload
shaped that way with a 400 rather than risk it being misread as a tracked
envelope.
Report progress from inside the handler
Add a third JobContext argument to a #[job] handler to opt into
progress reporting; the two-argument form keeps working unchanged:
#[job(name = "export_orders")]
async fn export_orders(
state: AppState,
args: ExportArgs,
ctx: JobContext,
) -> AutumnResult<()> {
ctx.set_progress(0, Some("Starting export")).await?;
// ... do the work, reporting progress as it goes ...
ctx.set_progress(50, Some("Rows 2500/5000")).await?;
// On success, the JSON result is whatever the caller wants back —
// e.g. a link to the finished file.
ctx.set_result(serde_json::json!({ "download_url": "/blob/orders-42.csv" }));
Ok(())
}
If the handler returns Err, the job retries as usual (max_attempts,
backoff); only the final failed attempt (or a panic, which always
dead-letters) settles the tracked record to failed. Call
ctx.set_user_error("...") before returning Err to control the message
shown to the caller — otherwise a generic "The job failed." is recorded (the
raw error is never leaked to the tracked-status response).
Poll the status
GET /_autumn/jobs/{token} (mounted automatically; disable with
jobs.tracking.route_enabled = false) is content-negotiated:
-
API clients (no
Accept: text/html, noHX-Request) get JSON:Json{"status": "running", "progress": 50, "message": "Rows 2500/5000", "result": null, "error": null} -
htmx requests (
HX-Request: true) or a browserAccept: text/htmlget a self-polling fragment. While the job is pending/running, the fragment carrieshx-get={path} hx-trigger="every 2s" hx-swap="outerHTML", so it keeps re-fetching and replacing itself with zero app-authored JS. Once the job reaches a terminal state, the fragment drops everyhx-*attribute — htmx has nothing left to poll — and renders either a download link (when the result carries adownload_url) or the failure message.
Embed the poll target directly in a page:
html! {
div hx-get=(handle.status_path()) hx-trigger="load" hx-swap="outerHTML" {
"Starting export…"
}
}
Result store TTL and backends
Progress/result records expire jobs.tracking.ttl_secs after their last
write (default 86400, 24h). The record store follows whichever job
backend is configured — local and redis use an in-memory or Redis-backed
store respectively, postgres uses the autumn_job_tracking table (see
Migration notes), and sqlite uses the same table in the
app's own database file — so a tracked job's status composes with the backend
an app already runs, with no extra setup. Expired records are invisible to
reads/writes immediately on every store; each also actually frees the expired
record so long-running processes don't accumulate one dead entry per tracked
job forever: the in-memory store sweeps them out opportunistically (amortized
across create calls), the Postgres and SQLite background sweeps run every 5
minutes to DELETE expired rows, and Redis expires keys natively via EX.
The Postgres store uses the database clock only. It sets and checks
expires_at with NOW(). Replica clocks can differ, but this does not change
jobs.tracking.ttl_secs. The store clamps a TTL above 100 years to 100 years.
Async CSV export, end to end
The synchronous GET /{plural}/export.csv admin route runs inline on the
request thread — fine for small tables, but a 50k-row export blocks the
worker and risks tripping a proxy idle timeout. A tracked job moves that
work off the request thread.
autumn_web::data::csv is behind the non-default csv feature:
autumn-web = { version = "0.11", features = ["csv"] }
use autumn_web::data::csv::export_csv;
use autumn_web::prelude::*;
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ExportOrdersArgs {
pub account_id: i64,
}
#[job(name = "export_orders")]
async fn export_orders(
state: AppState,
args: ExportOrdersArgs,
ctx: JobContext,
) -> AutumnResult<()> {
let repo = OrderRepository::from_state(&state);
let orders = repo.for_account(args.account_id).await?;
let total = orders.len();
let mut buffer = Vec::new();
for (i, chunk) in orders.chunks(500).enumerate() {
export_csv(chunk.iter().cloned(), &mut buffer)?;
let done = ((i + 1) * 500).min(total);
ctx.set_progress(
u8::try_from(done * 100 / total.max(1)).unwrap_or(100),
Some(&format!("Rows {done}/{total}")),
)
.await?;
}
let url = state.storage().put("exports/orders.csv", buffer).await?;
ctx.set_result(serde_json::json!({ "download_url": url }));
Ok(())
}
// Kick it off from a request handler and hand back a poll target — the
// initiating request returns immediately instead of blocking on the export.
#[post("/orders/export")]
async fn start_export(state: AppState, session: Session) -> AutumnResult<Markup> {
let owner = TrackedJobOwner::from_session(&session, &state).await;
let handle = ExportOrdersJob::enqueue_tracked_for(
ExportOrdersArgs { account_id: 1 },
owner,
)
.await?;
Ok(html! {
div hx-get=(handle.status_path()) hx-trigger="load" hx-swap="outerHTML" {
"Export starting…"
}
})
}
The browser gets a progress bar within milliseconds and a download link the moment the job finishes — no hand-written status table, token, or polling endpoint anywhere in app code.
Tracked status follows the queue ack
On the sqlite, postgres and redis backends, a worker writes the
terminal tracked status (succeeded or failed) only after the queue
accepts its acknowledgement. The queue checks that the worker still owns the
claim.
Delivery is at-least-once. An attempt can run past its visibility timeout while a second worker claims the same job. The queue rejects the ack of the first attempt, so Autumn does not write its status. The second attempt settles the record.
If the ack fails with an error, the record stays running. Autumn recovers
the job and runs it again. The next attempt settles the record. If the worker
crashes after the ack and before the status write, the record stays
running until its TTL ends. If a failed ack was for the last attempt, the
queue dead-letters the job and the record ends failed, even when the handler
succeeded. The local backend has no claim, so it writes
the status at once.
The queue row can be retried as soon as the ack lands. An operator retry in the short time before the status write can reset the record first. The late write then overwrites that reset. This needs a compare-and-swap token on the tracking write, which Autumn does not have yet.
To see the durable queue state, use the admin dashboard
(Observability) or JobAdminBackend.
Known limitation: stale progress writes on the Redis/Postgres stores
mark_running/set_progress intentionally no-op once a tracked record is
already terminal, so a stray write from an abandoned attempt can't overwrite
a legitimate final result — but on the Redis and Postgres tracking stores
that guard is evaluated against the value read at the start of that write
(read the record, decide, write it back), not atomically at write time. If a
worker's claim times out and it keeps running past that point while a
replacement worker claims and completes the same job, the original worker's
next progress write can land after the replacement's terminal write and
briefly clobber it, because its in-memory guard was evaluated against the
running record it read before the replacement settled. The record
self-corrects on the next terminal write (the replacement's own retry
path settles it, or TTL expiry clears it), so this is a transient display
glitch, not a lost result — the queue's own durable state (visible via the
admin dashboard) is never affected. Closing this fully means moving the
terminal-status guard into the durable write itself (a Lua script for
Redis, a conditional UPDATE for Postgres — the same compare-and-swap
approach JobTrackingStore::reset_for_retry already uses to make an
operator retry race-safe); tracked as a follow-up rather than folded into
this change.
Known limitation: a JobInterceptor erroring after next still settles the tracked record as failed
enqueue_tracked/enqueue_tracked_for treat any Err from the enqueue call
as "never delivered" and settle the tracked record to failed. If an app
installs a JobInterceptor whose intercept_enqueue
successfully awaits next (so the job was delivered to the backend) and
then its own post-next logic returns an error — e.g. an audit-log call
that fails — that error is indistinguishable, from this call site alone,
from a delivery failure: the tracked record settles to failed and the
caller sees an error even though the job will still run. A caller that
retries on that error risks enqueueing a duplicate. Closing this requires
JobClient::enqueue_with_outcome to expose whether the backend write
actually happened on its error path too (it currently tracks this
internally via a started flag but only uses it to select the Ok
variant), which is a signature change shared by every enqueue path — plain
enqueue() has the identical gap for the same reason — not just the
tracked ones; tracked as a follow-up. In the meantime, avoid returning an
error from intercept_enqueue after next has resolved successfully.
Known limitation: the built-in job-status route can collide with a user route
GET /_autumn/jobs/{token} mounts by default (jobs.tracking.route_enabled = true) for every app that registers jobs, merged after user routes are
already mounted. If an app happens to define its own route at that exact
path, try_build_router_inner panics on the resulting Axum overlapping-route
error at startup rather than surfacing a typed, recoverable collision error
(the framework's other reserved-but-default-on routes — the mail
unsubscribe endpoint, mail previews — have the same gap). /_autumn/ is a
framework-reserved path prefix, so this should not come up in practice;
set jobs.tracking.route_enabled = false if a conflict is ever hit.
Building a general framework-route-vs-user-route collision preflight
(mirroring the existing OpenAPI/MCP-vs-everything check) is a broader
router-building change than this PR's scope; tracked as a follow-up.
Observability
Mount autumn-admin-plugin to get the built-in operator dashboard at
GET /admin/jobs (or the plugin prefix you choose). It lists enqueued, running,
recently completed, and failed jobs with retry/discard/cancel actions. See the
Operating Background Jobs guide for dashboard
setup, action semantics, and bounded refresh behavior.
GET /actuator/jobs returns per-job:
queuedin_flightblocked_on_concurrencytotal_successestotal_failuresdead_letterstotal_deduplicatedlast_error
For Redis deployments these counters are process-local operational telemetry, not a strongly consistent Redis aggregate. They remain useful for seeing queued, in-flight, success, retry/failure, and dead-letter activity observed by the replica serving the actuator request.
Migration notes
When using jobs.backend = "sqlite", the runtime creates its own autumn_jobs
table at start; no migration step is needed.
When using jobs.backend = "local" or jobs.backend = "redis", no SQL migration
is required.
When using jobs.backend = "postgres", the autumn_jobs table must exist before
workers start. Run your app migrations as a one-shot autumn migrate job before
scaling web and worker replicas:
autumn migrate # creates autumn_jobs, autumn_job_tracking, your domain tables, etc.
The migration is bundled with the framework and is applied automatically by
autumn migrate as long as the db feature is enabled. enqueue_tracked
works the same way regardless of jobs.backend — the framework migration
also creates autumn_job_tracking, the table the Postgres-backed tracking
store uses (see Tracked jobs and progress polling).
A later migration adds its version column. Run autumn migrate before you
deploy a new binary.
Transactional enqueue
When a job must be coordinated with a database write, choose the API based on which guarantee you need:
enqueue_after_commitprevents jobs for rolled-back data on any backend, but the post-commit callback is process-local and can be lost if the process exits after commit.enqueue_in_tx/enqueue_on_connon the Postgres backend write the job row in the same transaction as the domain row, which is the crash-safe handoff.
enqueue_after_commit — any backend
autumn_web::job::enqueue_after_commit registers the enqueue as an
after-commit callback inside the surrounding db.tx block. The job is only
dispatched if the transaction commits. Works with every job backend.
This is not crash-safe delivery. If the process exits after the transaction
commits but before the callback runs, no job may be recorded. Use this for
rollback coordination across backends, not as a durable outbox substitute.
For a crash-safe enqueue on the Redis or SQLite backend, use
outbox.enqueue_job(conn, "name", &args) in the transaction. The relay
enqueues the job after commit. The local backend still loses the job in a
crash. See Transactional Outbox and Inbox.
use autumn_web::prelude::*;
use scoped_futures::ScopedFutureExt;
async fn create_order(mut db: Db) -> AutumnResult<()> {
db.tx(|conn| async move {
// ... INSERT order ...
// Enqueued only after INSERT commits; dropped if the tx rolls back.
// For crash-safe Postgres handoff, use enqueue_in_tx instead.
autumn_web::job::enqueue_after_commit("ship_order", &args).await?;
Ok::<_, AutumnError>(())
}.scope_boxed())
.await
}
enqueue_in_tx / enqueue_on_conn — Postgres backend only
On the Postgres backend the job row can live in the same transaction as
the domain row. Both commit or roll back together, avoiding the post-commit
process crash window at the cost of being limited to the postgres backend.
use autumn_web::prelude::*;
use scoped_futures::ScopedFutureExt;
async fn create_order(mut db: Db) -> AutumnResult<()> {
db.tx(|conn| async move {
// ... INSERT order using conn ...
// Job row written into the same transaction.
autumn_web::job::enqueue_in_tx("ship_order", &args, conn).await?;
Ok::<_, AutumnError>(())
}.scope_boxed())
.await
}
See Transactions -> after_commit for a full comparison of the two strategies and guidance on when to use each.
For cloud-native rollout run the migration job first, then start web and workers.