Staging Fault Injection

Add latency and errors to a staging app. Use it to test timeouts, retries, fallbacks and alerts before a real outage occurs. You do not need a service mesh or Toxiproxy.

Warning: do not use fault injection in production. The framework refuses it in the prod profile unless you set allow_in_production = true.

Declare faults

Add a [fault_injection] section and one [[fault_injection.faults]] table for each fault:

[fault_injection]
enabled = true

# 5% of /api requests fail with 503 before the handler runs.
[[fault_injection.faults]]
routes = ["/api/*"]
target = "route"
kind = "error"
rate = 0.05
status = 503

# 10% of database checkouts on /orders wait 300 ms.
[[fault_injection.faults]]
routes = ["/orders*"]
target = "database"
kind = "latency"
rate = 0.1
latency_ms = 300

You can declare at most 64 faults. The total latency for one request or one dependency call is at most 300 seconds.

To arm the faults for one deployment, set AUTUMN_FAULT_INJECTION__ENABLED=true. You must declare the faults in autumn.toml.

Targets

In an app with pre-rendered (SSG/ISG) pages, both fault layers also go outside the static cache, so a cached page can get a route fault too. There, a route fault occurs before rate limiting and load shedding. On a dynamic route, the injected latency counts against the request timeout of that route. A cached page waits at most server.timeouts.request_timeout_ms; at that limit the request fails with 503, as a timeout does. A request gets its route faults one time.

A dependency fault applies only in a request that matches routes. Work that runs outside a request (jobs, the scheduler, spawned tasks) gets no faults. All dependency waits of a request share its request timeout: the timeout of its route, else server.timeouts.request_timeout_ms. A wait that reaches it fails, as a timeout does. A wait before the handler (the Redis session load) uses part of the handler's timeout.

A request in which a fault fired writes no failure capsule (failure capsules): its replay runs without the fault.

The random decisions use their own stream, seeded at boot from the app entropy source. A test with SeededEntropy gets the same faults on each run. A failure capsule records none of these draws, so its replay does not need the fault config.

Safety rules

• Refused in prod. In the prod profile, config validation fails, and the app does not start. The check ignores case, and an unset profile counts as prod. To allow it, set allow_in_production = true under [profile.prod.fault_injection]. Do not put it in the base section: then the environment variable alone arms the faults in prod.
• Probes are exempt. The health, liveness, readiness and startup paths, and the actuator paths, never get a fault.
• Stop condition. See Stop condition.
• Audit. Each arm and disarm writes an audit event and a warn log on the autumn.fault_injection target. See Audit.

Stop condition

The framework counts the requests that match a fault, and the bad results among them, in a window. A result is bad when:

• the status is 5xx (an injected error, a real error, or a timeout);
• an injected error fired, at any status;
• the request stops before it completes, and a fault fired in it.

The faults disarm when the burn rate is more than max_burn_rate:

[fault_injection.stop]
objective = 99.0      # percent, as in [[slo]]
max_burn_rate = 14.4  # the fast-page burn rate
window_secs = 60
min_requests = 20

With these defaults, the faults disarm when more than 14.4% of the matched requests in a 60-second window are bad, and the window has 20 or more requests. The burn rule is the same as the one in SLOs as Code. Validation refuses values that let 100% of the requests fail.

Injected latency is bad only when it causes a timeout or a dropped request. Set a request timeout in staging, so that slow requests count.

The faults stay disarmed until an operator calls arm.

The handle

Get the handle from the app state:

use autumn_web::fault_injection::FaultInjection;

if let Some(faults) = state.extension::<FaultInjection>() {
    faults.disarm("[email protected]", "game day over").await;
    faults.arm("[email protected]").await;
    let snapshot = faults.snapshot();
    tracing::info!(armed = snapshot.armed, injected = snapshot.injected);
}

The handle is absent when the section is disabled or refused.

Audit

Each arm and disarm writes an AuditEvent to the installed audit sinks:

The event has the actor and the reason, profile, allow_in_production and sequence metadata. The sequence gives the toggle order. Two toggles in the same moment can reach a sink in the other order, so sort by sequence. Install a sink with AppBuilder::with_audit_sink. See Audit Logging. With no sink, the framework writes a warn at boot, and the events go to the log only.

Tests

For deterministic tests, use FaultPlan (see Simulation Testing). Use [fault_injection] only in a running staging app.