AEO — agent readiness by default

AI agents find, read, and use web sites. Agent Engine Optimization (AEO) helps them. Autumn does most AEO tasks automatically. The checks follow isitagentready.com.

You do one thing: set the site base URL.

[seo]
base_url = "https://example.com"

Autumn then serves the items in the table below.

The router fallback serves each generated document. The fallback runs only when no route matches, so an application route at the same path always wins. With [seo] settings or a SitemapSource, /robots.txt and /sitemap.xml are normal routes; see SEO. Without them, the fallback /sitemap.xml lists / and the pages that declare a seo(title), so the sitemap link in llms.txt always resolves.

Turn parts off

[aeo]
enabled = true          # false turns every AEO part off
markdown = true         # Accept: text/markdown negotiation
markdown_max_bytes = 2097152
link_headers = true
llms_txt = true
site_guide_skill = true
publish_tools = true    # false when a proxy guards /mcp
site_name = "Example Shop"   # default: the OpenAPI title, else the host name
site_description = "Hand-made things."

robots.txt rules for AI

The default robots.txt in production:

# As a condition of accessing this website, you agree to
# ... (the contentsignals.org explanation block) ...

User-agent: *
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
...
User-agent: GPTBot
User-agent: ClaudeBot
...
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no

Sitemap: https://example.com/sitemap.xml
Agentmap: https://example.com/.well-known/ai-catalog.json

The AI crawlers are in the * group. The [seo.robots] additional_rules also apply to them. A bot that additional_rules names (for example User-agent: GPTBot) keeps only your group. When additional_rules opens its own User-agent: * group, Autumn does not name the allowed AI crawlers, so that group binds them too. A Content-Signal in your * rules (before the first User-agent, or in a User-agent: * group) replaces the generated one, in robots.txt and in the Markdown Content-Signal header.

• search: use the content for a search index.
• ai-input: use the content as input to AI answers.
• ai-train: use the content to train AI models.

Change the signals:

[aeo.content_signals]
search = true
ai_input = true
ai_train = false

Block a class of AI crawler:

[aeo.ai_crawlers]
search = "allow"      # OAI-SearchBot, Claude-SearchBot, PerplexityBot
user_fetch = "allow"  # ChatGPT-User, Claude-User, Perplexity-User
training = "disallow" # GPTBot, ClaudeBot, Google-Extended, CCBot, ...

A disallowed class gets its own group with Disallow: / and the same Content-Signal line.

The default robots.txt (no [seo] settings) says Disallow: / only for the dev and test profiles, and when no profile is set. With [seo] settings, every profile except prod and production disallows, unless [seo.robots] allow_all is set.

Markdown for agents

Send Accept: text/markdown to any HTML page:

curl -H 'Accept: text/markdown' https://example.com/about

Autumn converts the page to Markdown:

• It converts <main>, else <body>.
• It drops scripts, styles, navigation, form controls, and hidden elements. A closed <dialog> counts as hidden. A closed <details> keeps its content, because a reader can open it.
• It puts <title> and the description meta tag in YAML front matter.

The response carries these headers:

HEAD gets the same headers as GET, with no body. A streamed page has no known length, and its GET may outgrow markdown_max_bytes, so its HEAD keeps the HTML headers. Browsers get HTML. Each HTML page that Autumn can convert sends Vary: Accept. A cache then keeps the HTML and Markdown copies apart. Some CDNs ignore Vary for HTML; for those, add Accept to the cache key.

Autumn does not convert:

• a page larger than markdown_max_bytes (it stays HTML),
• a streamed page, whose length is unknown, or a body not ready within two seconds,
• a status other than 200,
• a compressed body, a range, or a download,
• a signed response (an RFC 9421 Signature or Signature-Input),
• a page in a charset other than UTF-8 (or US-ASCII).

Call the converter yourself with autumn_web::aeo::markdown::html_to_markdown.

Generated documents

Each document has a strong ETag and answers 304 to a matching If-None-Match. A document never carries a Set-Cookie.

Set [seo] base_url in production. Without it, Autumn builds absolute URLs from the request Host header and sends Cache-Control: no-store, so no shared cache keeps a forged host. Autumn logs a warning in production when base_url is not set.

Static builds

autumn build writes llms.txt, the skills, the ARD manifest, auth.md, the OAuth metadata, and the commerce documents into dist/ when [seo] base_url is set. It always writes /_autumn/webmcp.js, which the page head_tags load; with no MCP server it registers no tools. A file that a static route wrote stays. A static build has no MCP server, so it has no server card. With .openapi(...) it lists the API, and writes the API catalog next to openapi.json. The documents link /openapi.json, where the build writes it, and no Swagger UI. The build does not write the Web Bot Auth directory, because its signature expires. Markdown negotiation and x402 need the server.

A static host serves a file by its extension and sends no CORS headers. So the build also writes dist/_headers, the per-path headers file that Netlify and Cloudflare Pages read. It gives each document its Content-Type, its CORS headers, its Link header and its Cache-Control, and each home page that the build wrote its agent Link header (without the Markdown link: a static host cannot serve the Markdown copy). An app that ships its own _headers keeps it, and the build logs a warning. On another host, set these headers in that host's configuration.

The build writes every file at its route path, pages and documents alike. An app below a path prefix ([seo] base_url = "https://example.com/shop") deploys dist at /shop, as it does the server. The documents' links name /shop/...; the files in dist and the _headers paths do not.

Agent skills

Autumn publishes a site-guide skill. It tells an agent how to read the site, which tools exist, and where the auth rules are. Add your own skills:

use autumn_web::aeo::AgentSkill;

let skill = AgentSkill::parse(include_str!("../skills/refunds/SKILL.md"))
    .expect("valid SKILL.md");
autumn_web::app().agent_skill(skill);

The served file keeps the other front matter fields (license, compatibility, metadata, allowed-tools, ...) as written. A skill named site-guide replaces the generated one.

MCP server card

When you call mount_mcp, Autumn serves the server card at two paths. The card lists the tools. When secure_mcp gates the endpoint, the card does not list the tools and says that auth is required. When a proxy or an app-wide layer guards /mcp, set [aeo] publish_tools = false.

OAuth and auth.md

Autumn does not run an authorization server. It publishes the values you set:

[aeo.oauth]
authorization_servers = ["https://auth.example.com"]
scopes_supported = ["read", "write"]

[aeo.auth_md]
registration_url = "https://example.com/settings/tokens"

When the app itself is the issuer, add the RFC 8414 fields:

[aeo.oauth.authorization_server]
issuer = "https://example.com"
authorization_endpoint = "https://example.com/oauth/authorize"
token_endpoint = "https://example.com/oauth/token"
agent_identity_endpoint = "https://example.com/agent/identity"
agent_identity_types = ["anonymous"]
agent_credential_types = ["api_key"]

The issuer must be this site's origin with no path: an https URL with no query or fragment (RFC 8414). Autumn does not publish metadata for another issuer. An app served below a path prefix ([seo] base_url = "https://example.com/shop") publishes neither OAuth metadata document, nor the 401 pointer to one: discovery (RFC 8414, RFC 9728) asks the host root, which the app does not answer.

Autumn serves /auth.md when [aeo.auth_md] registration_url or OAuth settings give an agent a way to get a credential. [aeo.auth_md] enabled = false turns it off.

/auth.md lists only the methods you configure. It lists oauth2 when there is an authorization server, either authorization_servers or this site as the issuer. It lists api_key in two cases:

• your issuer's agent_credential_types includes api_key;
• registration_url is set and no OAuth is configured.

WebMCP

WebMCP lets an AI agent in the browser call your MCP tools. Add the head tags to your layout:

html! {
    head {
        (autumn_web::aeo::webmcp::head_tags(None, csrf.as_ref().map(CsrfToken::token)))
    }
}

Pass the CSP nonce when your policy uses nonces, and the request's CSRF token (from the Option<CsrfToken> extractor). The token is written as <meta name="csrf-token">, which the htmx CSRF helper reads too. When CSRF protection is on and the MCP endpoint is not exempt, as in the production profile, every call is a checked POST, so without a token the script registers no tools at all. Only when CSRF does not check the endpoint does a page without a token still get the read-only tools. On page load, the script calls document.modelContext.registerTool() for each public tool that is marked safe: readOnlyHint: true (a GET tool), or destructiveHint: false. A write tool gets consequentialHint: true, so the browser can ask the visitor first. The script sends each call to the MCP endpoint with the page cookies and the CSRF token. Auth, CSRF, and rate limits apply to the call.

An app served below a path prefix, such as [seo] base_url = "https://example.com/shop", uses head_tags_at(&autumn_web::aeo::base_path(&config), …). That puts the script and the ARD link under /shop. The script finds the MCP endpoint relative to its own URL, so its calls go to /shop/mcp.

This script calls the public /mcp tools through /mcp. To give the browser agent routes that use the session of the user, see WebMCP. When the app calls mount_webmcp(), this script skips each name that the WebMCP projection registers. Then add both head tags to the layout.

The tags point at AEO documents, so remove them if you set [aeo] enabled = false: the script and the catalog then answer 404.

For a plain HTML form, add the declarative attributes:

html! {
    form action="/search" toolname="search" tooldescription="Search the catalog" {
        input name="q" toolparamdescription="Search words";
    }
}

Web Bot Auth

Web Bot Auth proves that requests from your app's bots come from you.

1. Make an Ed25519 seed: openssl rand 32 | base64 | tr '+/' '-_' | tr -d '='.
2. Put it in an environment variable.
3. Name the variable in autumn.toml:

[aeo.web_bot_auth]
private_key_env = "WEB_BOT_AUTH_KEY"
signature_agent = "https://example.com"   # https; default: [seo] base_url when it has no path

Autumn then serves the public key at /.well-known/http-message-signatures-directory. With [seo] base_url set, Autumn signs that response (@authority and content-digest). Sign an outbound request:

use autumn_web::aeo::web_bot_auth::WebBotAuthSigner;

let signer = WebBotAuthSigner::from_state(&state).expect("key set");
client.get("https://example.org/").sign_web_bot_auth(&signer).send().await?;

With [aeo] enabled = false, Autumn loads no key and from_state returns None. A signed request follows its redirects itself. Each request and retry gets a fresh signature, and so does a redirect to the same origin (scheme, host and port). A redirect to another origin drops the signature, Signature-Input and Signature-Agent, and that hop and every one after it go unsigned. Signature-Agent uses the quoted-string form that Cloudflare verifies, not the draft's dictionary form.

Commerce

x402

Price a route. A request without payment gets 402 and a PAYMENT-REQUIRED header. The facilitator verifies a retry that has PAYMENT-SIGNATURE. For GET and HEAD, the handler runs, then Autumn settles only a 2xx answer (a redirect is not paid). For other methods, Autumn settles first, then runs the handler. A priced route never runs free: when [aeo.x402] is incomplete, the route's amount is invalid, or the build has no http-client feature, it answers 503. A facilitator error answer is a 502. An entry that can never match a request (a method that is not an HTTP method, or a path without a leading /, or with a space, a ? query, or a # fragment) stops the app at startup, and so does a route priced for both x402 and MPP.

CSRF protection (on in the production profile) does not stand in a payment client's way. A request to a priced x402 route skips the CSRF check when it carries no Cookie (it has no session to forge, and gets the 402) or carries a PAYMENT-SIGNATURE header (which a cross-site page cannot send without passing CORS). x402 runs no handler without that header. A browser request with cookies and no payment header still needs its CSRF token.

[aeo.x402]
facilitator_url = "https://x402.org/facilitator"
pay_to = "0xYourWallet"
network = "eip155:8453"    # Base
asset = "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913"  # USDC on Base

[[aeo.paid_routes]]
method = "GET"
path = "/api/reports/{id}"
amount = "10000"           # smallest unit: 0.01 USDC
description = "One report"

If settlement fails, Autumn sends 402 and discards the handler body. If the facilitator does not answer, Autumn sends 502. A paid answer carries Cache-Control: private, no-store. One payment works once per process, however its header is written (key order, encoding, hex case); a second use gets 402. A header that was never settled (a GET whose handler failed or redirected) can be sent again. Autumn remembers a used header until its payment expires, for up to 10,000 unexpired payments at once; past that, a new payment gets 503 until some expire. With several replicas, a client can use a header once on each replica before settlement fails. Make priced handlers idempotent (see idempotency).

The route match ignores one trailing /, a locale prefix (/en/...), and capture names ({rid} matches a {id} route). When the app has its own route for the slashed form, that route keeps its own price: a free /report/ stays free beside a priced /report. Autumn cannot see inside a router added with merge, so there a priced /report also prices a free /report/ (it never serves a priced route free). Give each form its own entry to price them apart.

A priced page in a static (SSG/ISR) build is rendered by its route on each request rather than served from the manifest, so rate limiting, bot protection, load shedding and CORS run before Autumn calls the facilitator. A priced manifest entry that no route renders is still served from the manifest, outside those limits; startup warns about it. A GET entry also prices HEAD, unless a HEAD entry for the same route sets its own price. {name} matches one segment, {*name} the rest. The facilitator URL must use https (http only for localhost), with no query or fragment. Scanners probe GET /api and GET /api/v1, so price one of these to show x402 support.

A priced route that is also an MCP tool is paid the same way: send PAYMENT-SIGNATURE on the POST /mcp request. An unpaid call is a tool error that holds the 402 body, and the /mcp response carries PAYMENT-REQUIRED; a paid one carries PAYMENT-RESPONSE.

MPP

Add mpp_method to a priced route. /openapi.json then carries x-payment-info and a 402 response for that operation (a 402 the operation documents already is kept). x402 skips that route: your handler (or autumn-billing) runs the MPP payment.

[[aeo.paid_routes]]
method = "POST"
path = "/api/render"
amount = "500"
mpp_method = "stripe"
mpp_currency = "usd"

UCP and ACP

Register the documents. UcpProfile::new and AcpDiscovery::new check the required fields.

use autumn_web::aeo::commerce::{AcpDiscovery, UcpProfile};

autumn_web::app()
    .ucp_profile(UcpProfile::new(ucp_json).expect("valid UCP profile"))
    .acp_discovery(AcpDiscovery::new(acp_json).expect("valid ACP document"));

DNS-AID

An app cannot publish DNS records. Autumn writes the records for you to publish. Run the CLI in the project directory:

autumn aeo dns --mcp-path /mcp

It reads [seo] base_url from autumn.toml, with the profile overlay. Pass --base-url to set it on the command line. The same lines come from the library:

use autumn_web::aeo::dns_aid::{DnsAidInput, records};

for line in records(&DnsAidInput::new("https://example.com", Some("/mcp"))) {
    println!("{line}");
}

_mcp._agents.example.com. 3600 IN SVCB 1 example.com. alpn="mcp,h2" port=443 mandatory=alpn,port,key65400 key65400="https://example.com/mcp/server-card"
_index._agents.example.com. 3600 IN SVCB 1 example.com. alpn="h2" port=443
_catalog._agents.example.com. 3600 IN TXT "url=https://example.com/.well-known/ai-catalog.json"

Sign the zone with DNSSEC. Validating resolvers then mark the answers as authenticated.

Check your site

Run the scanner against your deployed site:

curl -s -X POST https://isitagentready.com/api/scan \
  -H 'content-type: application/json' -d '{"url":"https://example.com"}'