Web Bot Auth

WBA · bot authentication · signed agents · HTTP message signatures for bots

Definition

Web Bot Auth is an IETF draft standard under which an automated client (a crawler or an AI agent) cryptographically signs its HTTP requests and the website verifies the signature against a public key from the bot operator's directory.

What Web Bot Auth is

Web Bot Auth is a protocol through which an automated client proves to a website who it is. The bot or agent signs every HTTP request with its private key, and the site verifies the signature against the public key that the bot’s operator publishes in a directory. Cloudflare described the approach in May 2025, and on 23 October 2025 the IETF approved the webbotauth working group. Its scope covers authenticating search crawlers, web archivers, AI crawlers collecting training data and agents that retrieve content on behalf of a user.

Why User-Agent and IP checks fall short

Method	How it works	Weak spot
User-Agent	The bot states its own name	Faked with one line in a script
IP ranges	The address is checked against the operator’s published list	Cloud addresses are shared between services, users go through proxies and VPNs, ranges change
Web Bot Auth	A signature made with the operator’s private key	Only works for bots that take part in the protocol

The last row matters: the protocol helps you trust bots that sign, but it does not detect bots that do not. Those are still filtered by bot protection based on behaviour.

How the signature works

Under the working group’s current draft, a request carries three headers:

Signature-Agent — where the operator’s key directory lives. The directory is served at /.well-known/http-message-signatures-directory in JWKS format.
Signature-Input — what has been signed (at minimum @authority and signature-agent itself), the label tag="web-bot-auth", the key identifier keyid, and the created and expires times. The recommended validity is no more than 24 hours.
Signature — the signature itself, for example using Ed25519.

The site or its CDN fetches the key from the directory, checks the signature and the validity window, and only then treats the request as coming from the bot it claims to be.

Who supports it as of September 2026

Verification: Cloudflare (its verified bots and signed agents programmes), Akamai (since November 2025), AWS WAF (since November 2025, for sites behind Amazon CloudFront).
Signing: ChatGPT agent, Goose, Browserbase, Anchor Browser, Amazon Bedrock AgentCore Browser; Google is testing the protocol for Google-Agent.
Built on top: the Visa Trusted Agent Protocol and Mastercard Agent Pay use Web Bot Auth as their agent authentication layer.

What it gives an online store

The main benefit is being able to tell a shopper’s agent from a price scraper without CAPTCHAs or IP blocklists. Practical steps:

Ask your CDN or WAF whether it verifies signatures and how it labels those requests.
Set a rule: signed agents from known directories get through to the catalogue and basket, unsigned automation goes through the usual protection.
Keep Signature-Agent and keyid in your logs — they show which operators bring in orders.
Do not drop robots.txt: the signature answers who has arrived, the crawling rules answer what they may do.

Frequently asked questions

Why not just check the User-Agent?

The User-Agent is a string the client writes about itself, and any script can put GPTBot or Googlebot in it. IP checks are more reliable but brittle: cloud provider addresses are shared by many services, ranges change, and agents often run on shared browser platforms. A Web Bot Auth signature is tied to the operator's private key — without it, a valid request cannot be built.

Is Web Bot Auth a standard yet?

As of September 2026, no. The IETF webbotauth working group was approved on 23 October 2025, and version -00 of its main document, draft-ietf-webbotauth-httpsig-protocol (with authors from Cloudflare and Google), was published on 1 September 2026. The protocol is already in practical use, though: agents sign their requests and major CDNs and WAFs verify them.

Who signs requests with Web Bot Auth?

In August 2025 Cloudflare listed ChatGPT agent, Goose from Block, Browserbase and Anchor Browser among signed agents. Amazon Bedrock AgentCore Browser has signed requests in preview since October 2025, and Google is experimenting with the protocol for its Google-Agent. The Visa Trusted Agent Protocol and Mastercard Agent Pay rely on Web Bot Auth.

Does Web Bot Auth replace robots.txt?

No, they do different jobs. robots.txt tells bots the crawling rules, while Web Bot Auth lets you check who has actually arrived. They work in sequence: first the site confirms the request came from the bot it claims to be, then it applies its rules to that bot.