This guide covers serving your Autumn app over HTTPS. There are three ways to do it, and which one you want depends on where TLS is terminated:

  • Direct in-process TLS — the app terminates HTTPS itself on its own host:port using a certificate and key you supply. Best for a single host where you already have a certificate (from certbot, a corporate CA, or a cloud cert vendor).
  • Automatic ACME (Let's Encrypt) — the app obtains and renews its own certificate over HTTP-01, with no static cert on disk and no proxy. Best for a single host that should get and keep a valid public certificate automatically.
  • Reverse-proxy / platform termination — TLS is terminated in front of the app (kamal-proxy, nginx, a cloud load balancer) and the app serves plain HTTP behind it. Best for the managed autumn deploy flow and multi-replica deployments, and still the recommendation when a proxy already fronts your fleet.

Quick decision. Own cert, single host → Direct TLS. Want auto-issued certs, single host, no proxy → ACME. Tenants bringing their own hostnames → Tenant custom domains. Using autumn deploy, a proxy, or multiple replicas → Reverse-proxy termination.

Whichever way the server's certificate is provisioned, you can additionally verify the client's — see Mutual TLS.

Direct TLS and ACME are both off by default and each gated behind an off-by-default cargo feature (tls and acme), so a default build never links the TLS stack. The reverse-proxy path needs neither feature — the app just serves HTTP.


Direct in-process TLS ([server.tls])

With the tls cargo feature enabled and a [server.tls] section configured, the app terminates HTTPS itself on the same host:port — no sidecar proxy.

Enable the feature in your app's Cargo.toml:

TOML
[features]
tls = ["autumn-web/tls"]

Then point the app at your PEM certificate and key:

TOML
[server]
host = "0.0.0.0"
port = 443

[server.tls]
cert_path = "/etc/letsencrypt/live/app.example.com/fullchain.pem"
key_path = "/etc/letsencrypt/live/app.example.com/privkey.pem"
# reload_interval_secs = 60      # certs hot-reload by polling file mtimes (default)
# handshake_timeout_secs = 10    # TLS handshake timeout (default)

The [features] block above only defines the forwarding feature — because tls is off by default you must also build/run with it enabled, otherwise the TLS stack is never linked:

  • cargo run --features tls (or cargo build --release --features tls) — uses the tls = ["autumn-web/tls"] forwarding feature declared above.
  • Or turn it on directly without the forwarding feature: cargo run --features autumn-web/tls.
  • For a CLI-built single binary: autumn build --embed --features tls — the --features flag is forwarded to cargo through every build phase.

Configuring [server.tls] without the tls feature compiled in is a fail-fast boot error, not a silent fallback. AppBuilder::run validates the wiring before binding anything and exits non-zero with:

[server.tls] is configured but this binary was built without the tlsfeature; rebuild with--features tls, or remove [server.tls] to serve plain HTTP

so a build that forgot the feature can never quietly serve plain HTTP on a port you expect to be HTTPS.

Fields:

FieldDefaultMeaning
cert_pathrequiredPEM certificate chain (leaf first, then intermediates).
key_pathrequiredPEM private key for the leaf.
reload_interval_secs60How often the cert/key file mtimes are polled for a hot reload.
handshake_timeout_secs10Maximum time allowed for a TLS handshake.

Once it is running you should get a valid HTTPS response with no proxy in front:

Shell
curl https://app.example.com/health   # -> {"status":"ok", ...}

Fail-fast startup validation

The certificate is validated at startup, so a broken cert stops the boot instead of silently serving an unusable listener. Startup fails fast on:

  • a missing or unreadable cert_path / key_path file,
  • an unparseable or empty PEM,
  • a private key that does not match the certificate leaf,
  • an expired or not-yet-valid leaf or intermediate.

Hot reload (renewals without a restart)

The app polls the cert and key file mtimes every reload_interval_secs (default 60s) and swaps in the new material when either file changes. A certificate renewal — from certbot, an ACME client, or any tool that rewrites the PEM files in place — is picked up without a restart and without dropping the site.

The autumn doctor TLS check

autumn doctor gains a tls check that inspects the configured certificate: it fails on a missing, invalid, or expired certificate and warns when the leaf expires within 30 days — so an approaching expiry surfaces in CI or a pre-deploy check rather than at the moment the cert lapses.

What behaves the same under TLS

Turning on [server.tls] changes the transport, not the app. The framework probes (/health, /live, /ready, /startup), /actuator/health, the inbound request timeout ([server.timeouts]), SSE streams, wss:// WebSockets, and graceful shutdown of an in-flight request all behave exactly as they do over plain HTTP — including that an SSE body still streams incrementally rather than being buffered, and that a stream is still exempt from the request deadline.

That parity is enforced, not asserted: autumn/tests/integration/tls_app_surface.rs serves the same router twice — once over the TLS listener, once over plain TCP — and requires each probe response to match on status, body, and content type; it also drives a wss:// echo, an SSE stream, a timed-out handler, and a shutdown drain over TLS on every CI run.

Two things do differ, both by construction:

  • [server.tls] cannot be combined with server.unix_socket. Direct TLS terminates on host:port; configuring both is a startup error.
  • In-place upgrades (SIGUSR2 handoff) do not apply to a TLS build. A handoff passes the listening socket to a successor process, and a successor that terminates TLS cannot adopt a plaintext listener whose clients are mid-connection. So a TLS build never offers its socket for handoff (the signal logs in-place upgrade refused and the running app carries on), and a TLS build handed an inherited socket exits at startup rather than failing every connection. Restart the process to deploy a new build.

Serving HTTPS from the release image

autumn release init generates a Dockerfile whose builder runs a bare cargo build --release (or autumn build --embed with --embed). Neither passes --features, so the tls feature has to be a default feature of your app for the image to link the TLS stack:

TOML
[features]
default = ["flash", "tls"]
tls = ["autumn-web/tls"]

Mount the certificate and key into the container and point [server.tls] at them with the environment (no autumn.toml edit needed — the runtime materializes the section from these vars), and re-point the image's own HEALTHCHECK at https://:

Shell
docker run -d \
  -v /etc/letsencrypt:/etc/letsencrypt:ro \
  -e AUTUMN_SERVER__TLS__CERT_PATH=/etc/letsencrypt/live/app.example.com/fullchain.pem \
  -e AUTUMN_SERVER__TLS__KEY_PATH=/etc/letsencrypt/live/app.example.com/privkey.pem \
  -e AUTUMN_HEALTHCHECK_URL=https://localhost:3000/startup \
  -e AUTUMN_HEALTHCHECK_INSECURE=1 \
  -p 443:3000 \
  my-app

Mount the whole /etc/letsencrypt tree, at the same path, not just the live/<domain> directory. certbot's live/*.pem are relative symlinks into ../../archive/<domain>/, so a bind mount of live/<domain> alone leaves every target outside the mount: the container resolves them under /etc/archive, finds nothing, and the app fails fast on an unreadable certificate instead of serving HTTPS. Mounting the tree at the identical path keeps the symlinks valid and lets a renewal — which writes a new archive/ file and re-points the symlink — be picked up by the poller, since the mtime it stats is the symlink's target.

With a certificate that is not symlinked (a corporate or vendor PEM), any directory works:

Shell
docker run -d \
  -v /srv/tls:/etc/autumn/tls:ro \
  -e AUTUMN_SERVER__TLS__CERT_PATH=/etc/autumn/tls/fullchain.pem \
  -e AUTUMN_SERVER__TLS__KEY_PATH=/etc/autumn/tls/privkey.pem \
  -e AUTUMN_HEALTHCHECK_URL=https://localhost:3000/startup \
  -e AUTUMN_HEALTHCHECK_INSECURE=1 \
  -p 443:3000 \
  my-app

Both health-check variables matter. AUTUMN_HEALTHCHECK_URL re-points the generated HEALTHCHECK, which defaults to http://localhost:3000/startup: a plain-HTTP probe against an HTTPS listener marks the container unhealthy forever — and in the generated docker-compose.yml, anything waiting on condition: service_healthy then never starts.

AUTUMN_HEALTHCHECK_INSECURE=1 lets that probe skip certificate verification. It is needed because the probe is a loopback call to the container's own listener while your certificate is issued to the app's public hostname, so it can never validate as localhost. Set it only with a loopback URL — it applies to whatever URL you configure, and unset (the default) the probe always verifies. The opt-in is deliberate rather than inferred from the URL: user@, #fragment, and lookalike hostnames all make a URL read as loopback while curl resolves it somewhere else, so a scheme/host parser in the probe would quietly stop verifying certificates for a remote endpoint.

Mount the key read-only and keep it 0600 on the host, owned by a user the container's autumn user (uid 10001) can read. (certbot's archive/ is 0700 root by default, so either run the container as a user that can read it or relax the group permission for the app's user — a certificate the app cannot open is a fail-fast boot error naming the path, not a silent fallback.) Because the certificate lives on a bind mount, a host-side certbot renew rewrites the files the container polls — the hot reload works exactly as it does outside a container, with no image rebuild and no restart.

CI exercises this path on every change to the deployment scaffold: the https-target job in .github/workflows/release-image-boot.yml builds the generated image with tls on, boots it with a self-signed test certificate, and requires an HTTPS /health + /actuator/health 200 (validated with --cacert), that plain HTTP on the same port does not answer, and that the container's own HEALTHCHECK reaches healthy.

Renewing with certbot

certbot pairs cleanly with direct TLS because it writes renewed certificates back to the same paths, and the hot-reload picks them up.

  1. Obtain a certificate. The standalone authenticator answers the HTTP-01 challenge on port 80, so run it while nothing else holds that port:

    Shell
    sudo certbot certonly --standalone -d app.example.com
    

    certbot writes the live certificate to /etc/letsencrypt/live/app.example.com/fullchain.pem and the key to .../privkey.pem — exactly the paths used in the [server.tls] sample above.

  2. Renewal is automatic. certbot installs a systemd timer (or cron job) that runs certbot renew twice daily and rewrites fullchain.pem / privkey.pem in place when a certificate is within its renewal window. Because the app polls the file mtimes, the renewed certificate is served on the next poll with no restart and no dropped requests — you do not need a --deploy-hook to reload the app.

    Shell
    sudo certbot renew --dry-run   # verify the renewal path works
    

If you would rather not run certbot at all, the app can obtain and renew its own certificate — see Automatic ACME certificates below.

Local development certificates (mkcert)

For https://localhost in development, mkcert generates a certificate signed by a locally-trusted CA, so your browser accepts it without warnings.

  1. Install the local CA once (adds it to your system/browser trust store):

    Shell
    mkcert -install
    
  2. Generate a certificate for your dev hostnames:

    Shell
    mkcert localhost 127.0.0.1 ::1
    # writes ./localhost+2.pem and ./localhost+2-key.pem
    
  3. Point a dev config (or a [profile.dev] override) at the generated files:

    TOML
    [server.tls]
    cert_path = "localhost+2.pem"
    key_path = "localhost+2-key.pem"
    

You can now load https://localhost:<port> with a trusted certificate. Keep the generated *.pem files out of version control — they are per-developer and the key is a secret.


Automatic ACME certificates ([server.tls.acme])

With the acme cargo feature, the app provisions and renews its own TLS certificate from an ACME certificate authority (Let's Encrypt by default) over the HTTP-01 challenge — no static certificate on disk and no reverse proxy. It builds on the tls listener: the issued certificate hot-swaps into the same reloadable resolver [server.tls] uses.

Enable the feature:

TOML
[features]
acme = ["autumn-web/acme"]

The happy path is ≤10 lines of config:

TOML
[server]
host = "0.0.0.0"
port = 443

[server.tls.acme]
domains = ["app.example.com"]
contact_email = "[email protected]"
directory = "production"          # omit for Let's Encrypt STAGING (see below)

As with the tls feature, ACME is off by default, so build and run with the acme feature enabled (it turns on tls transitively):

  • cargo run --features acme — uses the acme = ["autumn-web/acme"] forwarding feature declared above (or cargo run --features autumn-web/acme without it).
  • autumn build --embed --features acme for a CLI-built single binary.

Configuring [server.tls.acme] without the acme feature compiled in is a fail-fast boot error, exactly like the tls guard above — AppBuilder::run exits non-zero with:

[server.tls.acme] is configured but this binary was built without the acmefeature; rebuild with--features acme, or configure a static cert_path/key_path instead

On first boot the app answers the ACME HTTP-01 challenge on http_challenge_port (default 80), obtains a certificate for domains, and starts serving HTTPS. That same :80 listener also redirects plain HTTP to HTTPS, so visitors who hit http:// are upgraded automatically.

Fields:

FieldDefaultMeaning
domainsrequiredOne or more non-wildcard domains to issue for. Each entry is used verbatim as the certificate's SAN and as the ACME order's DNS identifier, so an entry with leading or trailing whitespace is rejected at startup rather than requested as-is.
contact_emailrequiredContact address registered with the ACME account (a quoted string).
directoryLet's Encrypt stagingACME directory. Built-in endpoints are the bare strings "staging" (default) and "production". A private CA / Pebble uses the inline table { custom = { url = "https://your-ca.example/dir" } } — a bare URL string is not accepted (see below).
cache_dirconfig/acmeWhere the account key and issued certificate are cached (a quoted path).
http_challenge_port80Port the HTTP-01 challenge (and HTTP→HTTPS redirect) listens on.
renew_before_days30Renew this many days before expiry (a whole number, unquoted, and < 90).
ca_root_pathunsetPEM root that signs the ACME directory's own HTTPS certificate. Only needed for a private CA / Pebble; see below.

Staging is the default — switch to production deliberately. When directory is unset the app uses the Let's Encrypt staging environment, which issues certificates from an untrusted test CA (browsers will warn) but has very generous rate limits. Staging-first is intentional: Let's Encrypt's production environment enforces strict issuance rate limits, and a misconfiguration loop (wrong domain, port 80 unreachable, DNS not pointed yet) can burn your production quota for a week. Validate end-to-end against staging, confirm the challenge succeeds, then set directory = "production" to get a publicly-trusted certificate. The renew_before_days window (default 30) keeps renewals well ahead of the 90-day certificate lifetime so a transient failure has many days of retries before anything expires.

Pointing at a private CA (e.g. Pebble). directory is an enum: the built-in endpoints are the bare strings directory = "staging" and directory = "production", but a custom directory must be given as an inline table naming the custom variant:

TOML
[server.tls.acme]
domains = ["app.example.com"]
contact_email = "[email protected]"
directory = { custom = { url = "https://pebble.test/dir" } }

A bare URL string (directory = "https://pebble.test/dir") is not a valid value and makes the config fail to load at startup — use the inline-table form.

A custom directory almost always also needs ca_root_path. The ACME client speaks HTTPS to the directory and, by default, verifies it against the platform trust store — the host's own installed CA certificates. That is right for Let's Encrypt (both its staging and production API endpoints carry publicly-trusted certificates) and wrong for a private CA or a Pebble test server, whose API certificate chains to a root the host does not know. Unless you have installed that root system-wide, the TLS handshake to the directory fails and every order dies before an authorization is even created:

TOML
[server.tls.acme]
domains = ["app.example.com"]
contact_email = "[email protected]"
directory = { custom = { url = "https://pebble.test/dir" } }
ca_root_path = "config/pebble-root.pem"

ca_root_path replaces the client's trust anchors for the ACME control plane only — it has no bearing on which certificates browsers accept from your site. autumn doctor grades it (acme_ca_root) and fails on a path that is missing or holds no certificate, since that state can only ever produce failed orders.

How issuance and renewal work

  • Provisioning is over HTTP-01: the :80 listener serves the challenge token, the CA validates it, and the issued certificate is cached under cache_dir (default config/acme) and swapped into the live resolver.
  • Renewal runs on a coordinator loop that wakes hourly and renews any certificate within renew_before_days of expiry; the refreshed certificate hot-swaps into the live resolver with no restart. Leader election only serializes which instance orders a certificate — it does not make ACME fleet-safe (see the caveat below).
  • Mutual exclusion. ACME and static cert_path / key_path are mutually exclusive — configure exactly one. Set [server.tls.acme] to auto-issue, or [server.tls] with cert_path / key_path to serve your own certificate.

Single-process / single-host only. This in-process ACME flow keeps the HTTP-01 challenge token map in the process and the issued certificate in a local on-disk cache (cache_dir). Behind a load balancer, or with more than one replica, the CA's :80 challenge can be routed to a replica that lacks the token (issuance and renewal 404), and non-leader replicas cannot adopt a certificate renewed on another instance from that non-shared store. Leader election only decides which instance orders a certificate; it does not make multi-replica ACME work — the app logs a loud startup warning when a distributed scheduler backend is configured. For multi-replica or clustered deployments, terminate TLS at a shared reverse proxy / load balancer, or use a single dedicated TLS-terminating instance.

DNS-01 removes half of this: its challenge record lives in your zone rather than in one replica's memory, so the :80 routing problem goes away and the CA never connects to your host at all. It does not distribute certificates. The store is still local disk, so replicas that did not win the renewal lease never receive the issued certificate and keep serving the self-signed placeholder — the startup warning still fires under DNS-01, naming the certificate store. To run ACME across replicas at all, cache_dir must be on storage every replica shares.

Scope

HTTP-01 as described above is a single-host path: the challenge token lives in the process, so behind a load balancer the CA's :80 probe can land on a replica that never published it. For a wildcard certificate — or for a multi-replica deployment — use DNS-01, next.


Wildcard certificates via DNS-01 ([server.tls.acme.dns])

If you run subdomain-per-tenant (tenant1.myapp.com, tenant2.myapp.com, …), HTTP-01 is the wrong shape: it needs one issuance per hostname, so tenant N's first request waits on a certificate order, and Let's Encrypt's rate limits become a ceiling on how fast you can onboard. A wildcard certificate for *.myapp.com covers every tenant that exists and every tenant that ever will — but no CA will validate a wildcard over HTTP-01. It requires DNS-01: proving you control the zone by publishing a _acme-challenge TXT record.

Add a [server.tls.acme.dns] section naming your DNS provider, and list the wildcard in domains:

TOML
[server.tls.acme]
domains = ["myapp.com", "*.myapp.com"]
contact_email = "[email protected]"
directory = "production"

[server.tls.acme.dns]
provider = "cloudflare"

That is the whole configuration. Everything else — issuance on first boot, renewal before expiry with no restart, persistence across restarts, the staging directory, the health indicator — is the same lifecycle the HTTP-01 path above already has; only the challenge answer changes.

Onboarding a tenant after that costs zero certificate work: no issuance, no restart, no config change. tenant42.myapp.com serves valid HTTPS from the moment it resolves to your host.

Supported providers

providerCredential fieldsNotes
cloudflareapi_tokenA scoped API token with Zone:Read and DNS:Edit on the zone — Zone:Read because the zone id is discovered from the record name.
route53access_key_id, secret_access_key, optionally session_token, hosted_zone_id, regionNeeds route53:ChangeResourceRecordSets and route53:ListResourceRecordSets on the zone (plus route53:ListHostedZonesByName unless you set hosted_zone_id).
execnone — the hook authenticates itselfThe escape hatch: any other provider. See below.

Where the credential lives

Never in autumn.toml. The section has no field that could hold a token, and it rejects unknown keys — so an api_token = "..." written into the config file is a startup error naming the key, not a plaintext secret nobody notices.

Put it in the encrypted credentials store:

Console
$ autumn credentials edit
Console
$ autumn credentials edit --env prod
TOML
# config/credentials/prod.toml.enc (decrypted for editing)
[acme_dns]
api_token = "your-cloudflare-token"

Pass the profile you will run under. AUTUMN_ENV=production resolves to the prod profile, so the server reads config/credentials/prod.toml.enc — while a bare autumn credentials edit writes development.toml.enc. A credentials file that does not exist loads as an empty store, so the mismatch surfaces as "no Cloudflare API token found" rather than as a missing file.

The table name is the credential key from [server.tls.acme.dns], which defaults to acme_dns. For Route 53 the same table carries the AWS fields:

TOML
[acme_dns]
access_key_id = "AKIA..."
secret_access_key = "..."
# optional; skips the hosted-zone lookup
hosted_zone_id = "Z0123456789ABCDEFGHIJ"

The store is decrypted with AUTUMN_MASTER_KEY (or config/master.key) exactly like every other credential — see the credentials guide.

If you inject secrets through the environment instead, these variables override the store, field for field:

VariableField
AUTUMN_ACME_DNS_API_TOKENapi_token
AUTUMN_ACME_DNS_ACCESS_KEY_IDaccess_key_id
AUTUMN_ACME_DNS_SECRET_ACCESS_KEYsecret_access_key
AUTUMN_ACME_DNS_SESSION_TOKENsession_token
AUTUMN_ACME_DNS_HOSTED_ZONE_IDhosted_zone_id
AUTUMN_ACME_DNS_REGIONregion

Tokens never reach logs, error messages, or /actuator output: the secret type they are held in renders as <redacted>, and provider errors carry the API's own message and status, never the request headers.

The escape hatch: provider = "exec"

Any provider autumn does not ship support for — RFC 2136 dynamic updates, a registrar's CLI, a webhook shim — is reachable through a hook program:

TOML
[server.tls.acme.dns]
provider = "exec"
command = ["/usr/local/bin/acme-dns-hook"]

Autumn runs it twice per challenge record, appending three arguments:

Console
$ /usr/local/bin/acme-dns-hook present _acme-challenge.myapp.com <txt-value>
$ /usr/local/bin/acme-dns-hook cleanup _acme-challenge.myapp.com <txt-value>

Exactly those three arguments are appended — no -- marker, so $1 is the action. (A hook that forwards "$@" on to another CLI should insert its own -- there, since a challenge value is base64url and starts with - about once in sixty-four.)

Exit 0 means the record was written (or removed); anything else fails the order, with the hook's stderr quoted in the message. command is an argv array run without a shell, so the record value is never interpreted as shell syntax. A hook that has not finished within 120 seconds is killed, and its output is read through a bounded buffer, so a hook that never stops writing cannot grow the server.

Credentials. The hook inherits autumn's environment — that is how an nsupdate or vendor-CLI hook is expected to authenticate, and autumn passes it no secret of its own. Because that stderr is published (in logs, in the operator alert, on /actuator/health), autumn scrubs it: every value the environment holds under a credential-shaped name (*TOKEN*, *SECRET*, *KEY*, *PASSWORD*, *CREDENTIAL*, *AUTH*) is replaced with <redacted>, as are the DNS credentials autumn itself holds. That is a backstop, not a guarantee — a credential in a variable named something like MY_THING is not recognisable as one — so a hook still should not trace its own secrets to stderr.

An RFC 2136 hook is a five-line script:

Shell
#!/bin/sh
# $1 = present|cleanup, $2 = fqdn, $3 = value
[ "$1" = present ] && ACTION="add $2. 60 TXT \"$3\"" || ACTION="delete $2. TXT \"$3\""
printf 'server ns1.myapp.com\nupdate %s\nsend\n' "$ACTION" | nsupdate -k /etc/autumn/tsig.key

Waiting for propagation

After writing the records, autumn waits until they are visible before telling the CA to validate — signalling early is how a DNS-01 authorization gets burnt.

The probe goes to the zone's authoritative nameservers, not to the resolvers you configure. That is deliberate, and it is the difference between a feature that works and one that fails every renewal: a probe sent to a public recursive resolver the instant the provider's API returns arrives before the record is live on the zone's own servers, so the resolver answers NXDOMAIN and caches that negatively for the zone's SOA minimum — 900s on Route 53, 1800s on Cloudflare, both longer than the propagation budget. Every later probe would then read the cached "not there". So resolvers is used to discover the zone's nameservers (an NS lookup, then an A lookup per server), and the propagation probe is sent to those directly. When discovery fails — a split-horizon setup, a resolver that will not answer NS — autumn falls back to probing resolvers directly and logs a warning.

Discovery runs once per distinct challenge name, so an order spanning several domains probes each zone through its own nameservers. This matters: a nameserver answers only for the zones it is authoritative for, so probing myapp.io's challenge record at myapp.com's nameservers can never see the record, and the wait would time out on an order that is in fact correct. An apex + wildcard order challenges a single name, so the common case still costs one discovery. Fallback is per name too — one domain with a broken delegation does not pull the others onto recursive resolvers.

[server.tls.acme.dns] fields:

FieldDefaultMeaning
providerrequiredcloudflare, route53, or exec.
credentialacme_dnsKey in the encrypted credentials store holding the provider credential. A key name, never a token.
propagation_timeout_secs300Give up if the records are still not visible after this long. Capped at 3600.
poll_interval_secs5Gap between propagation probes.
resolvers["1.1.1.1:53", "8.8.8.8:53"]Resolvers used to discover the zone's nameservers, and probed directly if that fails. Each entry is an IP (port 53 implied) or IP:port; hostnames are rejected.
command—The exec hook's argv array. Required for exec, rejected for the others.

A timeout names the exact record, the value that never appeared, and the resolver that never saw it:

DNS-01 propagation timed out after 300s: the TXT record _acme-challenge.myapp.com still does not carry LPJNul-w… at resolver 1.1.1.1:53 (the name does not exist yet). Check that the record was written to the zone that actually serves this name and that its NS delegation is live, then raise [server.tls.acme.dns] propagation_timeout_secs if the provider is simply slow.

Challenge records are removed after the order finishes — including when it fails, so a retrying deployment does not fill the zone with dead _acme-challenge entries.

Two records, one name

An order for myapp.com and *.myapp.com produces two authorizations whose TXT records share the name _acme-challenge.myapp.com but carry different values. Both must be live at validation time. Every provider here appends a value and deletes by (name, value), never "replace the record set" — worth knowing if you write your own exec hook, because nsupdate delete <name> TXT without a value would remove its sibling and fail the order.

Route 53 is the exception that proves the rule: its API has no "append", only ChangeResourceRecordSets, which replaces the whole set. Autumn therefore read-modify-writes — and applies both values in a single change, because Route 53 may keep serving the pre-change values until the first change reaches INSYNC, so a second read-modify-write could read the old set and write back only its own value, dropping the first. The same read-modify-write preserves the set's existing TTL: the name can already hold another ACME client's challenge values, so autumn's own 60-second TTL is written only when it is creating the set — and Route 53 rejects a DELETE whose TTL differs from the live set, so cleanup has to send the TTL that is actually there. If your exec hook talks to an API shaped like that, batch the same way and preserve what you did not set, or make the hook idempotent under a stale read.

What DNS-01 does not need

  • Inbound port 80. The CA never connects to your host for DNS-01. Autumn still tries to bind http_challenge_port for the HTTP→HTTPS redirect, but a bind failure is only a warning under DNS-01 — the app starts and serves HTTPS without it, so a container with no CAP_NET_BIND_SERVICE is a supported deployment. autumn doctor grades an unreachable :80 the same way.
  • A record for the wildcard. *.myapp.com has no address record of its own; the tenant subdomains point at your host (a wildcard A/AAAA record, or one per tenant). Creating those is your one-time job — autumn only writes the ephemeral challenge TXT records.

autumn doctor for DNS-01

Three checks cover the failure classes DNS-01 adds:

CheckGrades
acme_dns_credentialThe provider credential is readable and carries the fields the provider needs. Fail when missing — without it every issuance and renewal fails.
acme_dns_propagation(--online) Public DNS can answer for _acme-challenge.<domain> at all. Fail on SERVFAIL/timeout — a broken delegation defeats a correctly-written record. Warn on leftover records from an interrupted run.
acme_tenancy_domain[tenancy] base_domain's subdomains are actually covered by [server.tls.acme] domains. Fail otherwise — every tenant host would serve a name mismatch.

Failure surfaces through health and alerts

A failed DNS-01 issuance or renewal — an expired provider token, a propagation timeout, a provider outage — is recorded on the acme health indicator (with challenge: "dns-01" and the provider name in its details, never a credential) and raises the operator-alert scheduled_task_failure condition for acme-renewal. Because renewal starts renew_before_days (default 30) ahead of expiry, that alert fires with weeks of validity left, and clears automatically on the first successful renewal.


Tenant custom domains ([server.tls.acme.custom_domains])

Wildcard certificates cover tenant42.myapp.com. A B2B customer who wants app.clientco.com needs a certificate of their own. Turn the feature on and autumn does the whole journey: it hands each tenant exact DNS instructions, independently confirms the hostname points here and that the tenant controls it, orders one certificate per hostname, serves it by SNI, routes requests on that Host to the owning tenant, and renews it. There is no per-domain configuration — a thousand tenants use the same twenty lines.

TOML
[server.tls.acme]
domains        = ["myapp.com", "*.myapp.com"]
contact_email  = "[email protected]"
directory      = "production"

[server.tls.acme.custom_domains]
enabled          = true
ingress_hostname = "ingress.myapp.com"   # CNAME target for tenant subdomains
ingress_ipv4     = ["203.0.113.10"]      # A records, for tenant APEX domains
# ingress_ipv6   = ["2001:db8::10"]      # AAAA records
# store_dir                  = "config/acme/domains"  # the registry (0600 files)
# max_domains                = 1000
# cert_cache_size            = 256       # certificates held in memory at once
# issuance_per_domain_per_day = 5
# issuance_global_per_hour    = 50
# failure_backoff_secs        = 300      # doubles per consecutive failure
# max_failure_backoff_secs    = 86400
# poll_interval_secs          = 60
# resolvers = ["1.1.1.1:53", "8.8.8.8:53"]  # read ownership TXT records

Each tick checks pending domains 16 at a time, for at most half of poll_interval_secs. A check that takes more than 20 seconds fails and backs off. Domains the tick did not reach wait for the next tick. Issuance and renewal then start on time, even when one tenant's DNS is slow.

A renewal DNS check that takes more than 10 seconds is treated as a DNS error. Autumn renews the certificate anyway.

With the default 60-second interval, verification gets 30 seconds per tick. Each round of 16 checks takes at most 20 seconds. With an interval under 40 seconds, the limit for one check is half of the verification time. A sweep of 1,000 new domains can therefore take several ticks.

ingress_hostname is what tenants CNAME at. An apex domain (clientco.com) cannot carry a CNAME, so it needs ingress_ipv4 / ingress_ipv6 instead; set both kinds if you accept both.

Autumn finds an apex with the Public Suffix List (ADR-0026), so clientco.co.uk is an apex and app.clientco.co.uk is not. A TLD that is not on the list uses the two-label rule. autumn doctor warns when a registered domain gets no DNS instructions.

The hourly budget survives a restart. Autumn keeps a ledger of recent orders in store_dir (attempts.ledger, 0600). An order for an offboarded domain still counts. One process owns the ledger. Replicas do not share it.

resolvers are the recursive resolvers autumn asks for each domain's ownership TXT record. The server must reach them on UDP/53.

The tenant journey

Your app owns the screens; autumn owns the state. Register a hostname when a tenant asks for it, show them what the framework says to publish, and render the status.

Autumn refuses a hostname the deployment already serves — anything under [server.tls.acme] domains or [tenancy] base_domain — so one tenant cannot connect another's subdomain. Per-tenant quotas are yours: max_domains is deployment-wide, so cap registrations per tenant in your own code if one tenant must not be able to spend the whole budget.

Rust
use autumn_web::custom_domain::{CustomDomainPruner, CustomDomainRegistry, DnsInstructions};

let registry = CustomDomainRegistry::from_state(&state).expect("custom domains are enabled");
let ingress = config.server.tls.as_ref()
    .and_then(|t| t.acme.as_ref())
    .and_then(|a| a.custom_domains.as_ref())
    .map(|cd| cd.ingress())
    .unwrap_or_default();

// 1. Connect. Your app decides who may — plan, entitlement, per-tenant quota.
let domain = registry.register("app.clientco.com", &tenant_id, now_unix).await?;

// 2. Show the tenant exactly what to publish. Fields are tab-separated.
let instructions = DnsInstructions::for_domain(&domain, &ingress)?;
println!("{}", instructions.render());
// app.clientco.com                     CNAME  ingress.myapp.com
// _autumn-challenge.app.clientco.com   TXT    <this registration's token>

// 3. Render status, including why it is stuck.
for d in registry.list_for_tenant(&tenant_id) {
    println!("{} — {} {}", d.hostname, d.status, d.failure_reason.unwrap_or_default());
}

// 4. Offboard. Routing, serving and renewal stop AND the certificate is deleted.
let domains = <dyn CustomDomainPruner>::from_state(&state).expect("custom domains are enabled");
domains.offboard_domain("app.clientco.com").await?;
domains.offboard_tenant_domains(&tenant_id).await?;   // every domain the tenant owns

registry.remove forgets the registration only — routing and serving stop, but the certificate stays in the ACME store. Offboard through CustomDomainPruner so the private key goes too.

Ownership proof: the TXT token

A hostname that points here proves nothing on its own. A tenant who leaves does not delete their DNS records, so their hostname still points here, and without more proof the next tenant to register it would get a certificate for it. So each registration gets a random token, and the tenant publishes it as a TXT record at _autumn-challenge.<hostname>. A domain verifies only when its address record points here and that TXT record carries this registration's token. Each registration mints a new token, so a record a previous tenant left behind proves nothing.

The TXT record may also be a CNAME to a name that carries the token. It is needed until the domain is active. Renewals check only the address record.

Domains connected before this rule are handled at the first start after the upgrade:

  • An active domain is grandfathered. It proved control under the old rule, so it keeps serving and renewing with no token.
  • Any other domain gets a token and goes back to pending_dns, with a failure_reason that asks for the TXT record. Show the tenant the DNS instructions again. Its registered_at_unix restarts, so the custom_domains retention window does not delete it before the tenant can publish.

The states are pending_dns → verified → issuing → active. There is no separate failed state: a domain that fails carries a failure_reason and a next_attempt_unix, and one that already reached active stays active while it retries — a failed renewal never takes a live tenant offline.

Once a domain is active, requests carrying that Host resolve to its tenant. The usual [tenancy] base_domain rejection does not apply to a registered domain, and subdomain tenancy is unchanged for every other host.

Connected domains and the host allow-list

[security.trusted_hosts] hosts is mandatory in production, and a tenant hostname is never in it — a thousand tenants must not mean a thousand configuration lines. So the ingress layer asks the registry about each host its static rules do not match:

  • An active domain is trusted, and the request reaches your handler.
  • A domain that is still pending_dns, verified or issuing is not. It gets 400 Invalid Host header, the same answer SNI gives at the handshake.
  • A hostname nobody registered gets the same 400.

Connect or offboard a domain and the answer changes at once. Do not set hosts = ["*"] for this: that turns host validation off for the whole deployment.

Abuse posture toward the CA

Custom domains mean tenant-supplied hostnames drive certificate orders, so three gates stand between a hostname and Let's Encrypt:

  1. Registration. Only a hostname your app registered is known at all. An SNI hostname nobody registered is refused at the TLS handshake, and no code path from there reaches the ACME provider.
  2. Verification. No order is created until autumn independently resolves the hostname and finds it pointing at this deployment, and finds this registration's token in its _autumn-challenge TXT record. A domain whose DNS points elsewhere, or whose TXT record is missing or stale, sits at pending_dns with the reason in its failure_reason and costs the CA nothing.
  3. Budget. At the defaults, at most 5 orders per domain per day and 50 across the whole deployment per hour — comfortably inside Let's Encrypt's 300-new-orders-per-account-per-3-hours limit. Failures back off exponentially from 5 minutes to a day, so a permanently broken domain converges to roughly one attempt a day rather than one a tick.

Tenant certificates are ordered over HTTP-01 even when your own certificate uses DNS-01: the record lives in the tenant's zone, where autumn holds no credential. That is the same condition verification already proved, so :80 must reach this deployment for tenant domains to issue.

Every tenant order uses the same ACME account as your own certificate, so the CA sees one registration however many domains connect.

Scale and restarts

The registry is one 0600 JSON file per domain, loaded into an in-memory index before the listener binds — a restart routes and serves every connected domain on the first request. Certificates load incrementally: at most cert_cache_size are held in memory, and a handshake for a colder domain reads its certificate back from the store and caches it. Nothing requires every certificate to be resident, so cert_cache_size is a memory knob, not a correctness one.

A stored pair that rots while its certificate is still cached is not re-read until the cache stops holding it: the repair pass re-reads and re-parses the pair on the first tick after an eviction (or a restart), and a pair that will not load is treated as missing and re-ordered. So a corrupted-on-disk certificate is detected within one tick of the eviction that exposes it — never earlier, never silently permanent — but also never proactively while the cached copy is still serving.

Custom domains live inside the ACME TLS listener, so they exist only where autumn terminates TLS itself. Behind reverse-proxy termination there is no registry and no Host-based tenant routing — the proxy owns the certificates and your app owns the mapping.

Ordering is leader-elected per domain through the same SchedulerCoordinator the deployment's own renewal uses, so replicas do not race the CA. Everything else about the ACME path is still single-host: the HTTP-01 token map and the certificate store are per-process, so behind a load balancer the CA's validation request usually reaches a replica that never published the token. Run custom domains on a single host.

Offboarding and retention

offboard_domain (or offboard_tenant_domains) stops routing, stops serving, halts renewal, drops the cached certificate and deletes the stored pair. Beyond that, the custom_domains retention dataset prunes two things the app cannot: a registration that never reached DNS verification within its window, and a stored certificate for a hostname no longer registered at all.

TOML
[retention]
custom_domains = "30d"   # drop connections abandoned before DNS was ever published

A registry that cannot be read at boot disables custom-domain connections as well as routing: an index that hydrated nothing cannot tell whether a hostname already belongs to someone, so register refuses with "the custom-domain registry has not loaded" rather than overwriting the durable record of whoever owns it. The deployment's own certificate keeps serving; fix the store and restart.

What doctor checks

CheckWhat it catches
custom_domainsThe section loads, has somewhere for tenants to point, and its cap is not already exceeded. Warn when a registered domain gets no DNS instructions. Fail on anything the server would exit at boot on.
custom_domain_dns(--online) Each registered domain still points here. Fail for an active or verified domain whose DNS has moved away or vanished — it is serving a certificate nobody can reach and will fail its next renewal. Warn for one still pending_dns. Probes the first 25 domains and says so.
custom_domain_txt(--online) Each pending_dns domain's _autumn-challenge TXT record carries its token. Warn when the record is missing, or carries only other values (for example a token from an earlier registration). A separate check from custom_domain_dns, so a wrong address and a missing token read differently.
custom_domain_http01(--online) Port 80 is reachable on every address each configured ingress target resolves to — one member of a load-balancer record set that drops it fails HTTP-01 for whatever share of tenants lands there. Fail when any is unreachable — tenant certificates are always issued over HTTP-01, so a closed port 80 blocks every one of them even under a DNS-01 deployment, where acme_ports rightly calls it optional for the deployment's own certificate.

Failure surfaces through health and alerts

The custom_domains health indicator reports how many domains are registered and active, and names every unhealthy one with its tenant — so an operator reading /actuator/health can act without a registry lookup. It is Down only when a certificate has actually expired: one tenant's pending retry is not a deployment outage. It also reports issuance_budget: global_used and global_limit (orders this hour), global_retry_after_secs (when room returns, null while there is room), per_domain_limit, and domains_at_limit (domains that spent their daily cap). A spent budget keeps the status Up. It means throttled, not broken. A failed order also raises the scheduled_task_failure alert for custom_domain_certificates, naming the domain and tenant, and that alert clears when the last failing domain recovers — or is offboarded, since a domain that no longer exists is not still failing.

A renewal skipped because the domain stopped pointing here is recorded the same way. The certificate keeps serving until it expires, so the domain stays active, but the reason and the alert appear immediately rather than at expiry, when it would already be down: either the tenant restores the record or the domain should be offboarded.


Mutual TLS: verifying client certificates ([server.tls.client_auth])

Everything above secures who the client is talking to. Mutual TLS (mTLS) secures the other half: who is calling. The app requests a certificate during the handshake, verifies it against a CA bundle you supply, and hands the verified identity to your handlers and policies — so an internal API can be locked to known machines without an mTLS-terminating proxy in front.

This is machine identity: service-to-service calls, partner APIs, zero-trust networks. Autumn verifies client certificates; it does not issue them. Use your own CA, Vault PKI, SPIRE, or cert-manager to mint them.

Needs the tls feature. With the [server.tls.client_auth] section absent, the handshake is byte-for-byte the server-only TLS above — nothing changes.

Configuration

TOML
[server.tls]
cert_path = "/etc/autumn/tls/fullchain.pem"
key_path  = "/etc/autumn/tls/privkey.pem"

[server.tls.client_auth]
mode           = "optional"                          # off (default) | optional | required
ca_bundle_path = "/etc/autumn/tls/client-ca.pem"     # one or more PEM CAs
crl_path       = "/etc/autumn/tls/client-ca.crl.pem" # optional revocation list
required_paths = ["/internal/"]                      # routes that demand a certificate
# reload_interval_secs = 60                          # bundle/CRL poll interval

Ten lines, and an internal route is locked to a client CA.

Modes are per listener:

modeHandshakeUse when
off (default)No certificate requested.Server-only TLS.
optionalCertificate requested; a client that presents one must pass verification, a client that presents none still connects.One process serves public routes and mTLS-only routes.
requiredHandshake fails without a valid certificate.The whole listener is machine-to-machine.

optional is not "unverified": a client that offers a certificate must still chain to a configured CA. The option is whether presenting one is mandatory.

A process.role = "worker" replica serves only the probes and the actuator, but it serves them over the same listener, so required_paths applies there too — a prefix covering /actuator/ holds on every role.

Prefixes are taken as written. Startup refuses a prefix that is not canonical, so it does not match the normalized request path. This covers an empty segment (//), a . or .. segment, a {} route template, and an encoded %2F or %2e. Write the resolved prefix.

autumn doctor fails on the same prefixes. It also fails when required_paths, crl_path or reload_interval_secs has the wrong type.

Per-route requirements come from required_paths, matched at segment boundaries — so /internal/ covers /internal/keys but not /internal-tools. A requirement matches the raw request path AND the normalized one, and demands a certificate if either matches: /open/../internal/keys cannot slip in, and /internal/%2e%2e — which normalizes to / but which the router still dispatches to /internal/{id} — cannot slip out.

A request reaching one of these over a connection with no verified certificate is rejected with 403 Forbidden and the standard JSON error envelope:

Json
{
  "type": "https://autumn.dev/problems/forbidden",
  "title": "Forbidden",
  "status": 403,
  "code": "autumn.forbidden",
  "detail": "this route requires a verified client certificate (mTLS)",
  "errors": []
}

403, not 401: there is no credential the client could re-send on this connection, so a WWW-Authenticate challenge would be meaningless.

For a requirement in code rather than config, layer a sub-router:

Rust
use autumn_web::tls::client_auth::RequireClientCertLayer;

let internal = Router::new()
    .route("/keys", get(rotate_keys))
    .layer(RequireClientCertLayer::new());

These compose: a route can be covered by required_paths, by the layer, by the ClientCert extractor below, or by all three.

The verified identity in handlers

Rust
use autumn_web::tls::client_auth::{ClientCert, OptionalClientCert};

// Rejects with 403 when the connection carries no verified certificate.
async fn rotate_keys(ClientCert(client): ClientCert) -> String {
    format!("caller {} (fingerprint {})", client.subject, client.fingerprint)
}

// Never rejects — serves machines and people from one route.
async fn whoami(OptionalClientCert(client): OptionalClientCert) -> String {
    client.map_or_else(|| "anonymous".into(), |c| c.subject.clone())
}

ClientIdentity carries:

FieldExample
subjectO=Acme, OU=Services, CN=svc-orders
issuerO=Acme, CN=Acme Internal Client CA
sans["DNS:svc-orders.internal", "URI:spiffe://acme/svc/orders"]
fingerprintsha256:9f86d0…
serial139C8A13442053B0…
not_after_unix4102444800

common_name() returns the subject's CN and has_san("URI:…") matches a SAN including its kind prefix. Both read the parsed certificate.

Authorize on common_name() and has_san(), never on subject. The DN strings render attributes in certificate order (not the reversed order openssl -nameopt rfc2253 prints) and do not escape attribute values, so a subject whose O happens to contain , CN=svc-payments renders exactly like two real attributes. A CA that copies O from a CSR would then let a caller pick its own identity out of a string search. common_name() cannot be spoofed that way. DNS: SANs compare case-insensitively; every other kind compares exactly.

These compose with session auth on the same router: Auth<T> and RequireAuth read the request, ClientCert reads the connection, and neither consults the other.

Machine identity in a policy

The same identity is on the PolicyContext, alongside the session user and token scopes, so an #[authorize] policy can decide on the calling service:

Rust
impl Policy<ApiKey> for ApiKeyPolicy {
    fn can_update<'a>(&'a self, ctx: &'a PolicyContext, key: &'a ApiKey)
        -> BoxFuture<'a, bool>
    {
        Box::pin(async move {
            // Only the orders service may rotate its own keys.
            ctx.client_has_san("URI:spiffe://acme/svc/orders")
                && key.service == "orders"
        })
    }
}

ctx.has_client_identity() asks whether the connection was verified at all; ctx.client_identity() returns the full record.

The identity is scoped to the task serving the request, like Current::actor() — a policy check moved onto a tokio::spawned task sees None. To exercise such a policy in a unit test, wrap the call in autumn_web::tls::client_auth::with_client_identity(...).

CA rotation without a restart

The bundle and CRL hot-reload on the same mechanism the server certificate uses — a modification-time poll, reload_interval_secs apart. A rotation is two edits, no restart, and no dropped connection:

  1. Overlap. Append the new CA to the bundle, so old and new are both trusted. Re-issue client certificates from the new CA at your leisure.
  2. Retire. Remove the old CA from the bundle.
Console
$ cat old-ca.pem new-ca.pem > /etc/autumn/tls/client-ca.pem   # step 1
$ cp new-ca.pem /etc/autumn/tls/client-ca.pem                 # step 2, later

rustls verifies once, at handshake time, so a connection established under the old bundle keeps serving after step 2 — the swap only affects handshakes that start after it. A bundle that fails to load logs an error and keeps the previous trust store: a bad rotation never takes the listener down.

Revocation

The recommended baseline is short-lived client certificates plus CA rotation, not a revocation list. A certificate that lives hours cannot be usefully revoked, and a CA rotation retires a whole generation at once. Reach for a CRL when a single certificate must be killed before it expires.

crl_path points at a PEM revocation list issued by a CA in the bundle. It hot-reloads like the bundle, so publishing a revocation takes effect within one poll interval, and a connection presenting a revoked certificate is rejected at the handshake — visible in the logs and the metrics below.

Two deliberate choices:

  • Only the presented certificate's revocation status is checked, not the whole chain. A single CRL from the issuing CA cannot speak for intermediates, and chain-depth checking would reject every certificate under an intermediate as "unknown status".
  • A stale CRL keeps working. If nextUpdate has passed, autumn keeps honoring the list rather than failing every handshake — the revocations it names stay enforced. autumn doctor warns so the staleness is not silent.

And one hard rule:

  • Every CA in the bundle needs a CRL. Once crl_path is set, rustls denies handshakes whose revocation status is unknown, so a CRL that names only some of the bundle's CAs would silently refuse the clients of the rest — exactly the availability trap the rotation above walks into if the new CA's CRL is published late. The server fails fast at startup, naming the uncovered CAs, rather than serving half the clients. During a rotation, publish the new CA's CRL (an empty one counts — a CRL with zero revoked certificates) before, or together with, the bundle edit that adds the new CA. A CRL-file change that breaks coverage hot-reloads the same way any bad bundle does: it logs an error and keeps the previous trust store.

    A CRL counts for a CA only if it is signed by that CA's key: a renewed CA that keeps its name under a new key needs a CRL from the new key. If your client certificates chain through an intermediate whose CRL you configure (the bundle holds only the root), the check stands down — revocation is checked against the issuing intermediate, which the bundle cannot show. A root shipped in the bundle alongside its intermediate needs no CRL of its own; the intermediate does.

OCSP and OCSP stapling are not supported. CRL plus short-lived certificates first.

TLS session resumption is off under client auth

A listener with mode other than off disables TLS session resumption — both the session cache and TLS 1.3 tickets. rustls restores a resumed connection's peer certificate from the stored session and does not re-run the verifier, so a client whose CA you rotated out (or whose certificate you just revoked) would otherwise keep reconnecting on a resumed session until it expired — and would keep presenting a verified-looking identity to your handlers. That is the one hole a swappable trust store cannot close by swapping, because the check it swaps is never run.

The cost is a full handshake per connection. For a listener whose purpose is deciding who may connect, that is the right trade. Server-only TLS ([server.tls] without this section) keeps resumption exactly as it was.

Failure diagnostics

A rejected handshake is invisible to the client beyond the standard TLS alert — no reason is ever put on the wire, so a prober learns nothing about the trust store. Operator-side, every rejection is counted and logged with a distinguishing reason:

ReasonMeaning
no_certificateNothing presented on a required listener.
untrusted_caChains to no CA in the bundle.
expiredPast its notAfter.
not_yet_validIts notBefore has not arrived — usually clock skew.
revokedListed in the CRL.
unknown_revocationRevocation status undeterminable.
invalidMalformed, bad signature, or otherwise unacceptable.
Code
WARN rejected an mTLS client certificate reason=untrusted_ca peer=10.0.3.7:51422 suppressed=0

The reason is classified where the handshake fails, so no_certificate — which rustls raises without ever consulting the trust store — is counted and logged like the rest.

Rejections are attacker-triggerable, so the log is rate-limited to one line per second per reason; lines held back are reported in the next one's suppressed field. The counters are exact regardless:

  • tls_client_auth_rejected_total{reason="…"} — handshakes rejected.
  • tls_client_auth_route_rejected_total — requests that reached an mTLS-only route with no verified certificate.

Both surface through the usual metrics/actuator endpoints.

Startup fails fast with the offending path in the message on a missing, unparseable, or empty CA bundle or CRL — the listener never binds trusting nobody. It also fails fast when the CRL does not cover every CA in the bundle (see above), naming the uncovered CAs.

autumn doctor

autumn doctor grades the mTLS surface offline, as the tls_client_auth check:

  • Fail — the bundle or CRL is missing, unparseable, or empty (the same conditions the runtime refuses to boot on), or a CA in the bundle has expired.
  • Warn — the CRL has no entry for every CA in the bundle (the runtime refuses to boot on this too, so --strict fails the run), a CA expires within 30 days, the CRL's nextUpdate has passed, or mode = "optional" with no route requiring a certificate (client auth configured and enforcing nothing).
  • Pass — otherwise, reporting the mode, the CA count, and how many route prefixes demand a certificate.

Posture manifest

A route's mTLS requirement is a dimension of the security posture manifest: autumn routes posture diff raises mtls_requirement_removed (widening, blocks until acknowledged) when a route that demanded a certificate stops demanding one, and mtls_mode_weakened when the listener mode itself drops a rank. So a refactor cannot quietly unlock an internal API.

End to end: a dev client CA in ten minutes

Generate a CA and one client certificate with openssl:

Console
$ openssl req -x509 -newkey rsa:2048 -sha256 -days 3650 -nodes \
    -keyout client-ca.key.pem -out client-ca.pem \
    -subj "/O=Acme/CN=Acme Internal Client CA" \
    -addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
    -addext "keyUsage=critical,keyCertSign,cRLSign"

$ openssl req -newkey rsa:2048 -sha256 -nodes \
    -keyout svc-orders.key.pem -out svc-orders.csr.pem \
    -subj "/O=Acme/OU=Services/CN=svc-orders"

$ cat > svc-orders.ext <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=clientAuth
subjectAltName=DNS:svc-orders.internal,URI:spiffe://acme/svc/orders
EOF

$ openssl x509 -req -in svc-orders.csr.pem \
    -CA client-ca.pem -CAkey client-ca.key.pem -CAcreateserial \
    -days 365 -sha256 -extfile svc-orders.ext -out svc-orders.pem

mkcert -client can mint a client certificate from the mkcert CA if you would rather not run the two commands above; use mkcert for the server certificate either way (see Local development certificates). Point ca_bundle_path at whichever CA signed the client certificate — mkcert's rootCA.pem or the one generated here.

Point the app at the CA, keeping /internal/ mTLS-only:

TOML
[server.tls]
cert_path = "localhost.pem"
key_path  = "localhost-key.pem"

[server.tls.client_auth]
mode           = "optional"
ca_bundle_path = "client-ca.pem"
required_paths = ["/internal/"]

Then smoke-test the whole matrix with curl:

Console
# Valid certificate → 200
$ curl --cacert rootCA.pem --cert svc-orders.pem --key svc-orders.key.pem \
    https://localhost:8443/internal/keys
rotated

# No certificate → 403 with the JSON envelope
$ curl --cacert rootCA.pem https://localhost:8443/internal/keys
{"type":"https://autumn.dev/problems/forbidden","title":"Forbidden","status":403,
 "code":"autumn.forbidden","detail":"this route requires a verified client certificate (mTLS)"}

# Public routes are unaffected
$ curl --cacert rootCA.pem https://localhost:8443/health
ok

# A certificate from another CA → handshake rejected, no HTTP response
$ curl --cacert rootCA.pem --cert other.pem --key other.key.pem \
    https://localhost:8443/internal/keys
curl: (56) OpenSSL SSL_read: ... alert unknown ca

With mode = "required" the second call fails at the handshake instead of returning 403 — the listener never lets an uncertified client speak HTTP at all.

Revoking a certificate

Console
$ openssl ca -config ca.cnf -revoke svc-retired.pem
$ openssl ca -config ca.cnf -gencrl -out client-ca.crl.pem

Add crl_path = "client-ca.crl.pem"; the next poll picks it up, and svc-retired is rejected at the handshake while its siblings keep working.

Not in this slice

  • Proxy-forwarded client-certificate headers (XFCC-style) are never trusted. The identity comes from the handshake this process performed, or there is none.
  • mTLS on outbound calls (http_client) and database connections.
  • Mapping certificates to human user accounts (PIV / smart-card login). Machine identity only.

Terminating TLS at a reverse proxy

If a reverse proxy or platform already fronts your app, terminate TLS there and let the app serve plain HTTP behind it. This is the right choice for the managed autumn deploy flow and for multi-replica deployments behind a shared load balancer.

The push-button autumn deploy path installs kamal-proxy in front of your app. kamal-proxy listens on your configured public HTTP port (server.port) and, by default, 443 — its HTTPS listener is always bound and cannot be disabled, regardless of any app's TLS setting. By default autumn deploy provisions no certificate for your app, so nothing is served over HTTPS until you opt in. You enable TLS termination at the deploy-managed proxy with an opt-in [deploy.tls] table:

TOML
[deploy.tls]
enabled = true
host = "app.example.com"   # public DNS name the certificate is issued for

With [deploy.tls] enabled = true, autumn deploy passes --host <host> --tls on every kamal-proxy route/flip for your app, so kamal-proxy provisions an automatic Let's Encrypt certificate for host on-demand and terminates TLS for it on its always-bound 443 listener. This needs no server.port change — issuance uses TLS-ALPN-01 on the already-bound 443, so it works on both the first deploy and a later redeploy (enabling TLS on an already-deployed app never restarts or reconfigures the shared proxy). With the table absent (the default) the route/flip commands carry no --host/--tls, so the proxy serves your app over plain HTTP only — byte-for-byte the historical behavior.

Setting server.port = 80 is recommended (it is the default) so the proxy also serves plain HTTP on 80 and can offer the HTTP→HTTPS redirect for visitors who hit http://. It is not required for certificate issuance.

An external TLS terminator sharing the same host is not supported. Because kamal-proxy always binds 443 and its HTTPS listener cannot be disabled, you cannot put your own nginx/Caddy/load-balancer TLS terminator on 443 on the same host as the deploy-managed proxy — the two would collide. Terminate TLS at kamal-proxy via [deploy.tls], or run the terminator on a separate host/load balancer in front of the deploy host.

Either way TLS terminates at the proxy, not the app. Do not enable in-process [server.tls]/ACME on a deploy-managed app: autumn deploy binds each app slot to a private loopback HTTP port (the slot systemd unit sets AUTUMN_SERVER__HOST=127.0.0.1), the readiness gate probes it over plain HTTP (curl http://127.0.0.1:{port}/ready), and kamal-proxy routes to a plain 127.0.0.1:{port} target — so putting a TLS listener there would fail both the health checks and the plain-HTTP proxy hop. When TLS is terminated in front, the app needs neither the tls nor the acme cargo feature and no [server.tls] section — the terminating proxy owns the certificate.

In-process [server.tls] and ACME (the sections above) remain the right choice for a self-run / standalone app you start yourself — one that owns its own public host:port, not one deployed via autumn deploy.

The same applies to any terminating proxy (nginx, Caddy, a cloud load balancer): point the proxy's certificate at the public https:// port and forward to the app's HTTP port. For the full autumn deploy walkthrough, fly.io, and container-based deployment, see the deployment guide.