Cache Stampede Protection

When a hot cache key expires, every concurrent request misses at once and recomputes the same value — one expiry turns into a database load spike. get_or_compute / get_or_compute_with add read-through fills to Autumn's cache abstraction that coalesce those concurrent misses, so a hot-key expiry degrades to one recompute instead of a thundering herd.

Why it matters

Autumn's cache already exposes get/insert/invalidate, but nothing coalesces concurrent misses: in-process callers race each other to refill the same key, and across replicas N processes each refill the same Redis key. The read-through API fixes both:

• In-process: concurrent misses for the same key run the fill once per replica; every other caller awaits that one fill and shares its result.
• Cross-replica (opt-in, Redis): a distributed fill lock ensures at most one replica fills a hot key at a time, and/or stale-while-revalidate serves the last-known-good value while one replica refreshes in the background.

Quick start

use autumn_web::cache::{get_or_compute, Cache, CacheFillError};
use std::sync::Arc;
use std::time::Duration;

async fn cached_bookmark_count(
    cache: &Arc<dyn Cache>,
) -> Result<i64, CacheFillError<sqlx::Error>> {
    get_or_compute(cache, "bookmark_count", Some(Duration::from_secs(30)), || async {
        count_bookmarks_in_db().await
    })
    .await
}

Concurrent callers that coalesce onto someone else's in-flight fill can see CacheFillError::FillFailed (not just CacheFillError::Fill) if that fill failed — don't assume into_fill() always returns Some.

On a hit, the cached value returns immediately. On a miss, the first caller runs the closure; every concurrent caller for the same key awaits that one fill instead of recomputing it themselves.

For cross-replica protection, use get_or_compute_with with a Redis-backed cache:

use autumn_web::cache::{get_or_compute_with, GetOrComputeOptions};
use std::time::Duration;

let opts = GetOrComputeOptions::new()
    .ttl(Duration::from_secs(30))
    .distributed_fill_lock(true);           // opt-in: requires a backend that
                                             // implements Cache::try_acquire_fill_lock
                                             // (RedisCache does)

let count: i64 = get_or_compute_with(&cache, "bookmark_count", opts, || async {
    count_bookmarks_in_db().await
})
.await?;

In-process vs distributed protection

Per-replica divergence note: the in-process registry only ever sees requests on its own replica. With the default in-process Moka cache and no distributed lock, N replicas each independently coalesce their own concurrent misses — so a hot-key expiry still produces up to N fills (one per replica), just not N × (requests per replica). This mirrors the existing divergence between per-replica Moka caches described in the cloud-native guide: if you need a single fill cluster-wide, enable the Redis backend and the distributed fill lock.

When the lock is contended, the losing replica polls the cache (starting at lock_poll_interval, default 50ms, doubling on each attempt up to a 1s ceiling so sustained contention doesn't hammer the backend at a fixed rate) for the winner's value and also re-attempts the lock (picking up a crashed holder once its lock TTL expires). If neither happens within lock_wait_timeout (default 5s), the replica gives up waiting and fills locally — bounded extra work, never unavailability.

Stale-while-revalidate

GetOrComputeOptions::stale_while_revalidate(grace) trades a bounded window of staleness for zero-latency reads at expiry:

let opts = GetOrComputeOptions::new()
    .ttl(Duration::from_secs(30))
    .stale_while_revalidate(Duration::from_secs(300));

• While the value is fresh (within ttl), reads return it immediately.
• Once stale (past ttl but within ttl + grace), reads still return the last-known value immediately, and kick off at most one background refresh per replica (through the same process-local single-flight registry — concurrent stale reads on the same replica never start a second refresh). This is a per-process guarantee, not cluster-wide: with only stale_while_revalidate enabled, N replicas can each independently start their own refresh for the same stale key at expiry (N fills, not 1). Add .distributed_fill_lock(true) for the cluster-wide "at most one refresh fleet-wide" guarantee — the background refresh honors the distributed lock exactly like a cold-miss fill.
• Once past ttl + grace, the key is treated as a cold miss again.

Trade-offs: callers may observe a value up to grace old. The fill closure passed to get_or_compute_with must be 'static (it may run on a background task rather than the calling task) — capture owned handles (Arc-clone your pool) rather than borrows. On the in-process Moka backend, per-entry physical expiry isn't available (Moka's TTL is per-cache-instance); freshness is tracked in a stored envelope instead, so correctness doesn't depend on Moka evicting the entry. Omitting .ttl(...) (leaving it at its None default) combined with SWR means the value is fresh forever — it never goes stale and never triggers a background refresh, matching ttl's own "no expiry" semantics. A process-wide cap (64) limits how many background refreshes can run concurrently across all keys, so a burst of simultaneously expiring keys can't spawn unbounded background work; a key that misses the cap simply retries the refresh on its next stale read.

Failure semantics

A failing fill never poisons the key:

• Nothing is written to the cache on error.
• The caller that ran the fill gets a typed CacheFillError::Fill(e).
• Every coalesced waiter gets CacheFillError::FillFailed(message) — the leader's error rendered via Display (the error type isn't required to be Clone).
• The in-flight entry is removed before waiters are notified, so the next caller retries the fill from scratch.
• If a leader is cancelled or panics mid-fill, the in-flight entry is removed (RAII), and any waiters see their channel close and re-contend for leadership instead of hanging forever.
• The distributed lock's lock_ttl bounds the damage from a filler that crashes while holding it: the lock self-clears and another replica takes over.

De-synchronizing mass expiry

Keys written together in a batch (a bulk warmup, a scheduled refresh) tend to expire together too, turning a single expiry into a stampede across many keys at once. jittered_ttl spreads them out:

use autumn_web::cache::jittered_ttl;
use std::time::Duration;

// Each key's TTL is uniformly randomized within ±20% of 5 minutes.
let ttl = jittered_ttl(Duration::from_secs(300), 0.2);

Metrics

The read-through API updates process-wide counters, visible on /actuator/metrics (under "cache") and /actuator/prometheus:

Falsifiable success metric: K concurrent requests hitting an expired key should record 1 fill and K−1 coalesced waits. With the Redis distributed fill lock enabled across replicas, total fills across all replicas should be 1.

See also

• Data caching and #[cached] — function-level memoization and the Redis backend
• Maud fragment caching — caching rendered view fragments
• Plugin Metrics Sources — the MetricsSource extension point for app- and plugin-contributed metrics