SQLite Database Fleet

One SQLite database per tenant, or per routing slot, behind the same sharding extractors a Postgres-sharded app uses. Design and research: ADR 0019.

SQLite has one writer per database. Put every tenant in one file and every tenant's writes queue behind each other. Give each tenant its own file and the writer is per tenant. A tenant's data is also one file: export it, restore it or delete it with a file operation.

When to use it

Configuration

The fleet needs the sqlite feature and a file-backed sqlite: control database. Sessions, jobs, flags and other framework state stay on the control database. Tenant data goes to the fleet.

[database]
url = "sqlite:///var/lib/app/control.db"

[database.fleet]
mode = "tenant"                 # or "slot"
root = "/var/lib/app/fleet"     # on the host's local volume
# path = "{bucket}/{tenant}.db" # default for mode = "tenant"
max_open = 256                  # soft limit on open databases (LRU past this)
# min_residency = "2s"          # LRU does not close a database used this recently
pool_size = 2                   # connections per open database
idle_close_secs = 300           # close a database idle this long
# create_on_demand = false      # default: false for tenants, true for slots
# restore_missing = false       # see "Replication" below
# replica_retention_after_delete = "0s"  # see "Deleting a database's replica"
# restore_check_sample = 1      # see "Restore check"
# replication_workers = 4       # see "Replication lag"

# [database.fleet.ownership]    # see "Failover: slot ownership" below
# owner = "host-a"
# slots = ["0-8191"]

The control database must live outside root: otherwise a tenant or slot path could resolve to it, so config validation (and boot, after resolving symlinks) refuses that layout. An app with a custom DatabasePoolProvider must name its control database with DatabaseTopology::with_migration_url, or a fleet refuses to boot: it cannot otherwise tell where that pool points.

Every key has an AUTUMN_DATABASE__FLEET__* override. With no TOML section, setting both AUTUMN_DATABASE__FLEET__MODE and AUTUMN_DATABASE__FLEET__ROOT creates one.

Path templates

path is relative to root and uses three placeholders:

A tenant fleet must use {tenant}; a slot fleet must use {slot}. The file name ends in an extension such as .db, so SQLite's -wal/-shm sidecars are never mistaken for databases. A bucket is 64 contiguous slots, so a host that owns slots 0–8191 owns buckets 000–127: moving a cell is moving directories.

Tenant ids

A tenant id becomes a file name, so a tenant fleet accepts only [a-z0-9_-], starting with a letter or digit, up to 128 bytes, and never a Windows device name (con, nul, …). Anything else answers 400 and creates nothing. Uppercase is refused rather than folded: on a case-insensitive file system Acme and acme would share a file. If your ids are UUIDs, send them lowercase.

A slot fleet takes any routing key: the key never names a file. So an app that moves from tenant to slot mode loses this id check. Check ids where they enter the app, for example with TenantDbId::parse (autumn_web::fleet_layout).

Handlers

Nothing changes. The fleet is the app's shard set:

#[post("/notes/{body}")]
async fn create(mut db: ShardedDb, Path(body): Path<String>) -> AutumnResult<String> {
    diesel::sql_query("INSERT INTO notes (body) VALUES (?)")
        .bind::<Text, _>(&body)
        .execute(&mut *db)
        .await?;
    Ok(db.shard().to_owned()) // "tenant:acme"
}

#[autumn_web::repository(Note, tenant_scoped, sharded)]
pub trait NoteRepository {}

#[get("/notes")]
async fn list(repo: SqliteNoteRepository) -> AutumnResult<Json<Vec<Note>>> {
    Ok(Json(repo.find_all().await?))
}

#[repository] generates PgNoteRepository. A SQLite build also names it SqliteNoteRepository. The two names are one type. The generators still write the Pg name.

In a module with #[model] or #[repository], use autumn_web::db::prelude::*, not diesel::prelude::*. Diesel's prelude also imports the blocking RunQueryDsl, and the generated code then fails with E0034 ("multiple applicable items in scope").

The routing key is the tenant id, resolved exactly as for ShardedDb (ShardKeyOverride, then the tenancy middleware, then [tenancy] extraction). In a tenant fleet, tenant_scoped is belt and braces. In a slot fleet several tenants share a file, so tenant_scoped is what keeps them apart.

CrossShard<R>, Shards::each_shard and across_tenants() fan out over every database on disk, opening each. Use them on admin paths, not hot paths. For analytics, query the lake instead.

When a handler holds a Db past database.slow_query_threshold, Autumn logs database connection held past slow_query_threshold with held_ms. The time covers all work in the hold, not one statement. For a fan-out, take LazyDb. Call checkout() after the fan-out.

Provisioning, deletion, backup

A tenant fleet does not create a database for an unknown tenant: any client could otherwise fill the disk with ids. Provision on signup:

#[post("/signup/{tenant}")]
async fn signup(shards: Shards, Path(tenant): Path<String>) -> AutumnResult<StatusCode> {
    let fleet = shards.fleet().expect("fleet configured");
    fleet.provision(&fleet.key_for(&tenant)?).await?; // 409 when it exists
    Ok(StatusCode::CREATED)
}

Import a backup

fleet.import(&key, &src) copies a database file into the fleet as key, for example a file that fleet.backup wrote before a delete:

fleet.backup(&key, &archive).await?;
fleet.delete(&key).await?;
// Later: undo the delete.
fleet.import(&key, &archive).await?;

The import does not change src. It copies src with VACUUM INTO into a private staging file. Then it does these steps:

1. It runs PRAGMA integrity_check.
2. It reads the identity row. The row must record key. A file with no identity row (a file that no fleet made) is refused.
3. It applies the pending migrations. With ownership, it sets the epoch of the copy to this host's epoch. Without ownership, it removes the epoch.
4. With replication, it purges the tombstone and the old replica of a deleted key, as provision does. It does this only after steps 1 to 3 pass. Thus a refused file keeps the replica.
5. It publishes the file in one step. It never replaces a file.

A refused source answers 422 (FleetError::InvalidSource), and a key that has a file answers 409. Nothing is published then. fleet.import_file does the same steps, but does not open the database.

Import only files that you trust: the copy keeps the schema, the triggers and the migration records of src. On Windows, the import opens src read-write, so SQLite can apply a -wal or a hot journal to it.

FleetLifecycle::restore_missing is a lower-level hook. Each request-path open of a missing file asks it, under the path lock, whatever [database.fleet] restore_missing says. A background open does not ask. With replication, an open of a deleted key does not ask. Use import for a local backup.

Operating the fleet

autumn fleet runs these operations from the command line. Use it when the app is stopped and when it runs. It reads [database.fleet] the same way the app does. delete, import, restore and lake also read [replication]. The sources are autumn.toml, the [profile.<p>] overlay, .env.<profile> and AUTUMN_* overrides. Run it from the app directory. --profile <p> selects a profile.

The fleet API is in the SQLite build only. Install that build of the CLI:

cargo install autumn-cli --no-default-features --features sqlite

autumn fleet list --open
autumn fleet migrate --concurrency 4
autumn fleet backup tenant:acme /backups/acme.db
autumn fleet import tenant:acme /backups/acme.db
autumn fleet delete tenant:acme --force
autumn fleet restore tenant:acme --timestamp 2026-10-01T12:00:00Z --force

• A key is a name as autumn fleet list prints it: tenant:<id> or slot:<nnnnn>. tenant/<id> and slot/<n> are also accepted.
• No command creates the fleet root. Each command stops when database.fleet.root does not exist, so a wrong directory does not give an empty fleet. lake does not read the root, so it runs without one. For a new volume, create the root directory first.
• Only restore and import create a database file. The other commands do not.
• import applies migrations/ and the framework tables to the copy before it publishes it. Like migrate, it stops when the copy records a migration that is not in these sets. With [replication] on, it needs the replica destination and the control database, to purge the old replica of a deleted database. With [database.fleet.ownership] on, it stops: this command holds no slot lease. Call DatabaseFleet::import in the app that owns the slot.
• Like boot, each command validates [database] first and stops when the control database resolves inside the fleet root.
• list --open reads each file and changes no data. It exits non-zero when it cannot read a file.
• migrate reads the app's migrations/ directory. It does not apply plugin migrations. For those, run the app once with AUTUMN_MIGRATE=1. If an app migration has the same version as a framework migration, migrate stops: only boot can record that pair correctly. migrate also stops when a database records a migration that is not in migrations/ or the framework sets (a plugin migration, for example). It checks each database again inside the migration's write lock, so a file replaced after the first check is not migrated. After you add a plugin, run the app once with AUTUMN_MIGRATE=1 before you use fleet migrate: until then, no database records the plugin's migrations, so migrate cannot see them.
• A failure on one database does not stop the others. migrate lists each failure and exits non-zero.
• With [replication] on, a migration of a database that the app has open can make the replicator start a new generation. AUTUMN_MIGRATE=1 does the same.
• backup never replaces or follows a file at dest. On Unix, only its owner can read the copy, and backup stops when another user could change a directory on the way to dest: each one must belong to root or to you, and must not be writable by others unless it has the sticky bit (as /tmp has). On Windows, the copy gets the ACL of the dest directory: put backups in a directory that only you can read.
• delete and restore destroy data, so they need --force on every profile. When another process has the file open (an app that serves the tenant), they stop with "open in another process" and change nothing. Wait until the app closes the database (idle_close_secs), or stop the app.
• restore verifies the replica chain before it replaces the file, as autumn db replica restore does.
• With [replication] on, delete also removes the replica, or keeps it for replica_retention_after_delete, and prints which. It needs the replica destination and the control database. If the purge fails, the command prints the error and a tombstone records it: the app tries the purge again. See Deleting a database's replica.
• restore also brings back a deleted database whose replica is still there. It drops the tombstone, so it needs the control database.
• On Windows, another process can open the database while delete or restore removes files (see Limits). The command then stops with "open in another process" or "created ... during the restore". The replica keeps the data. Stop the app, then run the command again.

Migrations

Every fleet database gets the app's migrations and the shard framework tables (version history, commit-hook queue, derivations). The control-plane tables (tokens, jobs, sessions) stay on the control database.

• A new database is migrated when it is created.
• An existing database is migrated when it opens, if this boot applies migrations (database.auto_migrate, auto_migrate_in_production).
• Otherwise opening one with pending migrations answers 503 until you run the app once with AUTUMN_MIGRATE=1, which migrates the control database and then every fleet database. autumn fleet migrate migrates the fleet databases only (see Operating the fleet).

Each database's run is one BEGIN IMMEDIATE transaction: atomic, and safe when two processes race. During a rollout the fleet is mixed, so write migrations expand-then-contract.

Control-only and tenant-only sets

AppBuilder::migrations sends a set to the control database and to every fleet database. To keep a set on one side, use migrations_for:

use autumn_web::migrate::{EmbeddedMigrations, MigrationScope, embed_migrations};

const CONTROL: EmbeddedMigrations = embed_migrations!("migrations-control");
const TENANT: EmbeddedMigrations = embed_migrations!("migrations");

autumn_web::app()
    .migrations_for(MigrationScope::Control, CONTROL) // control database only
    .migrations_for(MigrationScope::Shards, TENANT)   // each fleet database only

Boot and AUTUMN_MIGRATE=1 obey the scope. autumn migrate and autumn fleet migrate read migrations/ and do not know a scope. Keep the tenant set in migrations/ and the control set in its own directory. Then autumn fleet migrate is correct for the fleet. Do not run autumn migrate on this app: it applies migrations/ to the control database. Use AUTUMN_MIGRATE=1.

Opening and closing

• The first request for a database opens it (about half a millisecond for a small schema). Concurrent first requests share that one open.
• A file that records another key's name (it was copied or renamed by hand) is refused with 500 rather than served.
• max_open is a soft limit. Past it, the least recently used idle database closes. The sweep does not close a database that has a connection checked out, that a request holds (a ShardedDb, a sharded repository), or that was used within min_residency (default 2s). An open never waits for room. Thus a burst across many keys opens more databases than max_open. In one measurement, 1,000 tenants at about 1,300 requests per second gave 594 open databases with max_open = 16. The count decreased to 16 about 12 seconds after the burst.
• A lower min_residency keeps the count nearer to max_open during a burst. A request can then reopen a database that it used a moment before.
• A new database is built and migrated in a private staging file, then published in one step: it is absent or complete, never half made. When two processes on one volume create the same database, both end up on one file.
• Budget: pool_size connections and about 2 × pool_size + 1 file descriptors (database and -wal per connection, one -shm) per open database. Size the open-file limit (ulimit -n) for the peak open count under a burst, not for max_open. The default limit on macOS is 256: 594 open databases go past it.
• When connections reach database.max_connections_warn_threshold (default 100), the boot warning gives the open files at max_open. The formula is 2 × connections + databases. It counts the control database.
• On Unix, boot also warns when those files reach 80 % of the open-file limit (ulimit -n), at any connection count. The rest is for sockets and logs. database.max_connections_warn_threshold = 0 silences both warnings.
• Metrics label every fleet database shard=fleet; the database name goes on spans and logs only, so a tenant fleet cannot explode metric cardinality.

Background work: the sweeper

Outbox rows, commit hooks and derivation backfills work on fleet databases. A database does not have to be open for its work to run: the fleet sweeper visits it.

#[post("/orders/{id}")]
async fn place(mut db: ShardedDb, outbox: Outbox, Path(id): Path<i64>) -> AutumnResult<()> {
    autumn_web::db::scoped_transaction(&mut *db, |conn| async move {
        outbox.write(conn, &format!("order:{id}"), "order.placed", &id).await?;
        Ok(())
    }.scope_boxed()).await
}

• The write also adds a row for the database to _autumn_fleet_pending on the control database, in the same transaction. A commit-hook enqueue does the same. If the control database cannot take the row, the write fails.
• A worker process (role = "worker" or "combined") reads that table each second. It opens each listed database, sends its outbox, drains its hooks and runs its derivation backfill. Then it closes the database again, if nobody else used it.
• The sweeper never opens a database past max_open. With no room, it waits for the next pass. It never closes a database that a request holds, and it never restores a deleted database from its replica.
• An outbox handler for a tenant fleet runs in the tenant's scope (CURRENT_TENANT), so a tenant_scoped repository finds that tenant.
• A web process does not open databases for the sweeper. It drains the hooks of the databases it has open. A worker sends their outbox.
• A new or changed #[derivation] is backfilled in every database: the first worker pass after the deploy marks each file. When a database opens first, it runs up to 8 backfill batches before its first request, and the sweeper does the rest. During a rolling deploy, only the new binary runs backfills. After a rollback to older derivations, run a resweep.
• Write outbox rows in a transaction. Do not enqueue on a fleet database while you hold a write transaction on the control database: the row on the control database then cannot be written, and the write fails.

/actuator/health reports fleet:sweeper: backlog (databases with work), oldest_pending_secs, and last_pass (visited, cleared, deferred, failed, handled). A backlog that grows, or an old oldest_pending_secs, means the sweeper cannot keep up: a handler fails, the fleet is full, or no worker process runs.

Replication: object storage between volumes

A cloud host attaches one volume. The volume can't be the unit of durability or of moving data, so object storage is. With [replication] on, each open fleet database ships its WAL to the same destination as the control database, under <prefix>/<profile>/fleet/tenant/<id> or .../fleet/slot/<nnnnn>:

• fleet databases run with wal_autocheckpoint = 0, and their replicator is the only checkpointer (a fleet without replication keeps SQLite's own);
• closing a database ships everything it committed before the file closes. If the destination is down, the replicator stays open and keeps shipping until it catches up (closing in the health details);
• fleet.restore(&key, at) rebuilds one database, point in time included;
• restore_missing = true restores a database whose file is missing before opening it. A host with an empty volume takes over a tenant or a slot range by serving it. Turn on slot ownership with it: without ownership, nothing stops the previous owner, and the two copies diverge. Boot logs a warning for restore_missing without ownership. This setting is only for a missing file. With ownership on, a host also restores a stale file (older than the replica) without restore_missing. See step 5 of the failover procedure.

Replication is asynchronous. A host acknowledges a write when the write is on its own volume. The next pass ships it to the replica. So a failover to another volume can lose an acknowledged write. See What a failover costs.

Processes can share a replicated fleet root. Only one process replicates a database. It holds the lock file <db>.replicator.lock. The other processes open the database and retry the lock on each tick. When the owner exits or dies, the OS frees the lock and another process takes over with a new generation. Use a local file system: advisory locks are not reliable on network file systems.

/actuator/health reports replication:fleet with the number of databases replicating, the number this process does not own (unowned), the worst lag and which database has it.

Replication lag

The [replication] rpo_secs bound holds while one pass over the open databases fits in sync_interval_secs. A pass ticks each open database once. It starts one interval after the last pass started, or at once if the last pass took longer. replication_workers threads (default 4, 1 to 64) ship databases at the same time. The database that shipped longest ago goes first.

Many busy databases in one process make a pass long. Monitor these values in replication:fleet:

• worst_lag_seconds: the longest time since a database was last caught up. A takeover can lose this much of that database. The indicator is DOWN above three times rpo_secs.
• last_pass_ms: the time of the last pass. If it is near sync_interval_secs × 1000, increase replication_workers, or put fewer busy databases on each host.
• ship_workers: the threads that ship.

unowned counts open databases that this process does not replicate. Usually another process on the same root holds their lock. A database that closes and opens again during a pass can show as unowned for some milliseconds. The next pass claims it.

Deleting a database's replica

fleet.delete(&key) and autumn fleet delete remove the replica too:

1. The delete writes a tombstone row to _autumn_fleet_tombstones on the control database. If it cannot write the row, the delete fails (503) and removes nothing.
2. The delete removes the files.
3. The delete writes a marker object, <prefix>/tombstone, in the replica.
4. The delete purges the replica, then drops the row. If replica_retention_after_delete is set, the delete keeps the replica until the retention ends.

The purge sends one delete request for each object, and the delete call waits for it. On a versioned bucket, a delete request only adds a delete marker: add a lifecycle rule that removes old versions.

If the purge fails, the local delete is not rolled back. The row stays. A task tries the purge again every minute. After each failure, it waits longer, up to 1 hour. If the marker write failed, the task writes the marker again. replication:fleet shows the last pass under tombstones (purged, failed, waiting, failing, last_error). A failed purge does not turn the indicator Down.

A tombstone stops the restore of a deleted database:

• restore_missing does not restore a key that has a row.
• On a host with its own control database, the marker stops the restore. The marker stops each generation that is not newer than the delete.
• fleet.restore(&key, at) and autumn fleet restore ignore the tombstone. They take the row first, and drop the marker after the restore. If the restore fails, the row goes back.

Use the retention as an undo window:

[database.fleet]
replica_retention_after_delete = "30d"  # default "0s": purge at once

Use a unit: a bare number is milliseconds. A value under one minute is refused.

Rules:

• During the retention, a create on demand (create_on_demand) of the key answers 410. With restore_missing, a host with its own control database also answers 410: it reads the retention from the marker.
• A create on demand also answers 410 while the row is pending, with any retention. A pending row has no cutoff yet. The task confirms the row when it is older than 10 minutes, the file is not on disk, this host owns the slot, and the replica can be read. Until then, only provision makes the database again.
• provision makes a new database: it purges the old replica first and drops the row. This ends the undo window. If the purge fails, provision fails with 503: try again.
• A purge removes each generation that is not newer than the delete. A newer generation is a new database with the same key, and it stays. The delete time is the later of the app clock and the newest generation, so a host clock that is ahead does not keep the deleted data.
• A host purges only a key whose slot it owns, and only when the file is not on its disk. A row next to a live file is old: the task drops it.

Restore check

A replica that does not restore is not a backup. Every [replication] verify_interval_secs (default 6 hours), each worker-role process restores restore_check_sample random databases (default 1) into scratch files under <root>/.fleet-locks/restore-check/. The first check runs one interval after start. For each database, the check runs PRAGMA integrity_check and reads the identity row, then removes the file. A failed check is tried once more. Scratch files older than one day (from a process that stopped during a check) are removed. Each check needs free space for the largest sampled database on the fleet volume.

replication:fleet shows the result under restore_check (checked_at, passed, unreplicated, failed). A failed check turns the indicator Down until a check passes. A database with no replica yet counts as unreplicated, not as a failure. Set restore_check_sample = 0 or verify_interval_secs = 0 to turn the check off. fleet.check_replica_sample(n) runs one check now.

Query the lake

The lake runs SQL across tenants on their replicas in object storage, not on the volume (ADR 0032). It never opens a file under the fleet root. It runs on a host with no volume.

let snapshot = fleet.lake()?.materialize().await?;
let rows = snapshot
    .query("SELECT _autumn_tenant, COUNT(*), SUM(total) FROM orders GROUP BY 1")
    .await?;

• materialize() restores each live tenant and copies its tables into one scratch SQLite file. Each table gets the column _autumn_tenant.
• query(sql) runs one SELECT, WITH or VALUES statement on a read-only connection. FleetLake::query(sql) does the load, the query and the cleanup in one call.
• Drop the snapshot to remove its scratch directory. persist(path) keeps a copy for sqlite3, DuckDB and other tools.
• snapshot.tenants() gives the generation and effective time of each tenant. snapshot.skipped() gives each tenant that is not loaded and why.

On a host with no fleet, build the lake from the destination: FleetLake::new(destination, "<prefix>/<profile>").

Rules:

• Consistency. Each tenant is consistent at its own snapshot. The lake has no atomic snapshot across tenants. as_of(t) does not make one common instant either.
• Deleted tenants. A tombstone row, or a marker that covers the newest generation, skips the tenant. A lake with no tombstone source refuses to load unless you call markers_only().
• Failures. A tenant that fails fails the load, so a total is never silently short.
• Schema drift. Columns union by name. A column that a tenant does not have is NULL. Values keep their storage class. Lake columns have no type affinity and use the BINARY collation, so write COLLATE NOCASE in the query where the source column had it.
• Disk. The lake needs about the size of the loaded tables, plus one restored file per worker. Use with_tables to load less.
• Trust. The SQL is operator SQL. Do not pass request input to it.

autumn fleet lake report.sql --out report.csv
autumn fleet lake report.sql --timestamp 2026-10-01T00:00:00Z --keep lake.db

The CLI uses the control database when its file is on the host. Else pass --markers-only. It writes CSV: NULL is empty, a blob is hex. Slot fleets are not supported.

Failover: slot ownership

With [database.fleet.ownership], a host serves only the slot ranges that it owns. Each claim of a range gets an epoch. An epoch is a number that only increases. The epoch fences the old owner:

• the old owner stops opening the range's databases;
• its replicator stops;
• a restore never selects its data instead of the new owner's data.

[database.fleet]
restore_missing = true

[database.fleet.ownership]
owner = "host-a"     # one id per volume
slots = ["0-8191"]   # default: every slot, as one range
lease_secs = 30      # default; 3 to 3600

The lease also sets the takeover time after a crash. See What a failover costs before you change it.

The overrides are AUTUMN_DATABASE__FLEET__OWNERSHIP__OWNER, ..._SLOTS (comma-separated) and ..._LEASE_SECS. The processes on one volume (a web and a worker role) use the same owner and the same slots. A tenant fleet uses the slot of the tenant id, so the same ranges work in both modes and in the cell router.

How it works

The ownership table is _autumn_fleet_slot_owners on the control database. All hosts of the fleet must use the same table. The control database is a local SQLite file, so the built-in table fences only the processes of one host. For hosts on different machines, implement SlotOwnershipStore (in autumn_web::db::fleet_ownership) on a store that all hosts reach. Then start a FleetOwnership with DatabaseFleet::start_ownership (it takes the volume lock, then claims), and start DatabaseFleet::spawn_ownership before the first request.

The failover procedure

1. Give every host the same [replication] destination and the same ownership table. Set restore_missing = true and a unique owner per volume.
2. Planned move. Stop the old host. It releases its ranges at shutdown. Start the new host with the same slots. It claims the ranges at once and restores each database on its first request.
3. Unplanned failover. The old host stopped, or the network isolates it. Start the new host with the same slots. The new host claims the ranges when the old lease ends: after at most lease_secs, plus one renew interval and the clock skew. Then it serves at a new epoch. The first request to each database restores it from the replica, so the first requests after a takeover are slow. A host that claims many databases at once can use all its CPU for restores. Limit the number of requests that open databases at the same time. Or open the claimed databases in the background before traffic arrives. A takeover loses the commits that the old host did not ship. See What a failover costs.
4. If the old host is still alive, it stops serving when its local lease ends, and returns 421. When it can reach the table again, its renew fails. It claims the range again only when the new owner lets it go, at a newer epoch. Until then, fleet:ownership shows the range as waiting.
5. Fail back. When the old host gets a range back, its files on disk are older than the replica. Each file records the epoch it was last opened under. On the first open at the newer epoch, the host compares that epoch with the newest epoch at the replica. If the replica's is newer, the host restores the file from the replica before it serves it. This works without restore_missing. Without [replication] there is nothing to compare with: give the old host an empty volume.
6. Move part of a range. Change slots on the owner, for example from "0-8191" to "0-4095", and restart it. A clean shutdown releases the old range, so the owner claims "0-4095" at once at a new epoch. Another host can then claim "4096-8191". If the old range was not released, each claim waits until its lease ends.

/actuator/health reports fleet:ownership. The details are the owner, the held ranges with their epoch and whether the host serves them now, and the ranges that are waiting, lost or failed. The status is Down when the last claim or renew could not reach the table, or when a held range stopped serving because a renew did not come back in time.

What a failover costs

Replication is asynchronous (see Replication). A failover costs data and time:

• A long lease gives a long outage after a crash. A short lease allows less clock skew (see Clocks).
• A crashed host loses all the writes that it did not ship.
• A paused host (SIGSTOP, a frozen VM) loses the unshipped writes of each database that the new owner opened during the pause. When the paused host continues, it ships its other writes. The new owner restores them later.
• A fail-back does not recover the lost writes. If the new owner shipped a database, the old host restores the replica over its stale file (step 5). If not, the old host keeps its own file.

Clocks

The lease in the table uses the wall clock of the host that writes it. The holder stops on its own monotonic clock, lease_secs / 3 before the lease ends. The clock skew between hosts plus the longest request must be less than lease_secs / 3 (10 s by default). Otherwise two hosts can serve one slot for a short time. The epoch still protects the replicas: a restore selects the newest epoch, and the old replicator stops.

If the old owner accepts writes after the new owner claims the range, those writes are lost. They are in the old epoch's generation, and no restore selects that generation.

Do not restore the control database to an earlier time while ownership is on. If its epoch counter goes back, the next claim after a fence still goes above the epoch that the replicator found.

Limits

• Shard-local jobs and the job dashboard work on the control database, not on fleet databases.
• Each open database runs its own commit-hook worker, in every process role. Before a database closes (idle, max_open, shutdown), its worker finishes the hook it runs and acks it, for up to 10 seconds. A hook that does not finish in that time can run again later. Hooks stay at-least-once: write them to be idempotent.
• An outbox row written outside a transaction commits before the sweeper's row. If that row cannot be written, the call returns an error for a committed message: do not retry it. The next open of the database writes the row.
• A replicated database takes a fresh base snapshot each time it reopens. Size max_open so busy tenants stay open.
• On Windows, delete and restore close the guard connection before they remove files. If another process opens the database in that gap, delete answers 409. restore also answers 409 if that process creates the file before restore publishes.

With autumn-harvest-sqlite

autumn-harvest-sqlite and the sqlite feature of autumn-web use the same libsqlite3-sys, so the two link one SQLite. SqliteRuntime and its drive future are Send: one tokio task can own the runtime.

See also

• Horizontal Sharding: Postgres shards, slots, routers
• SQLite in Production: the single-database tier
• Cell and Shuffle-Shard Isolation: cells and the cell router