AI agent traffic: governance, crawling, attribution

Published 4 August 2026 . Part of catalogue readability for agents.

“We put GPTBot in robots.txt, that’s handled.” It is the sentence you hear most often, and the one that costs most. It rests on a category error: not everything arriving from an AI vendor is a crawler, and robots.txt does not govern the same thing in each case.

This note separates the three families of traffic, and states for each what is actually steerable.

Three families, not one

Training crawlers traverse the web to build corpora. They are recurring, broad, indifferent to any particular user. GPTBot falls here: OpenAI documents it as its training crawler, with a string of the form compatible; GPTBot/1.4; +https://openai.com/gptbot.

Answer indexers feed a generative search surface. OAI-SearchBot is documented by OpenAI for search indexing. Blocking it protects nothing; it simply removes the site from answers. That is a commercial decision dressed as a technical one.

Human-triggered fetches are a different kind of thing altogether. ChatGPT-User covers user-initiated requests, and OpenAI’s documentation notes that robots.txt rules may not apply, precisely because the action comes from a person. A user asking their assistant to open a page is not a robot, and treating them as one degrades a real visit.

OpenAI also documents OAI-AdsBot, dedicated to ad safety validation, neither training nor search. Four tokens, four distinct intents: grouping them under one rule means deciding without knowing what you are deciding.

The Google-Extended case, which is not a crawler

Google documents a distinction many configurations miss. Googlebot, Google-CloudVertexBot and GoogleOther are active crawlers, with real HTTP user-agent strings. Google-Extended is not. The documentation is explicit: “Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.”

In other words, Google-Extended is a control token, not a visitor. It will never appear in access logs. Two concrete consequences:

• searching logs for Google-Extended to measure volume is a dead end, there is nothing to find;
• blocking it at server level, by user-agent filtering, has no effect, since the request arrives under a different identity. Only the robots.txt directive acts, and its effect bears on whether content is used to train Gemini, not on the site’s presence in Google Search.

This is the counter-example that disqualifies the “I block at the firewall, safer than robots.txt” reasoning. Depending on the token, one of the two levers simply does not exist.

What robots.txt does not govern

robots.txt is a declarative convention. It relies on client good faith and on the honesty of the user-agent string, which is trivially forgeable. Three blind spots remain, however good the file:

• agents that ignore the convention, by choice or by design;
• agents running locally in a browser, on the user’s side, which look like human traffic because they nearly are;
• vendors that publish no stable token at all.

The operational conclusion is not to abandon robots.txt, which remains the only standard signal and the only one honoured by the major vendors. It is to stop crediting it with an access-control function it never had. Access control belongs to the network or application layer, with the trade-offs that implies.

A defensible default position

For a merchant who wants to be found and bought from by agents, the default position is not symmetric:

Family	Default position	Reason
Answer indexers	allow	blocking removes the site from answers
Human-triggered agents	allow	this is a real visit
Training crawlers	brand decision	no effect on commercial discoverability
Control tokens	brand decision	acts on use, not on access

That is the configuration this site runs: the robots.txt file explicitly allows the main documented tokens rather than relying on a generic rule.

What is still missing

The ecosystem has no widely deployed mechanism letting a merchant cryptographically verify that an agent is who it claims to be. Until that piece exists, any policy based on user-agent names remains a policy based on the other party’s declaration. That is also what makes measurement hard, a subject covered on the agentic commerce page.

For preparing the catalogue itself, see agent readability and llms.txt.

Sources

• OpenAI, Bots and crawlers
• Google, Common crawlers

---

Back to catalogue readability for agents · All notes · Lire en français