AI Crawler Rules - robots.txt Audit for AI Search

Six lines in this robots.txt do nothing the author intended - for SEO consultants and publishers auditing a client site.

robots.txt
  1  ERROR  No crawler answers to this name - the real token has no hyphen
  4  ERROR  Retired token. Claude-Web and anthropic-ai were replaced by ClaudeBot in 2024
  5  ERROR  Disallow: * is not a valid path value
  7  WARN   Google-Extended is a control token, not a crawler
 10  ERROR  Google stopped honouring noindex in robots.txt on 1 September 2019
 11  ERROR  Sitemap must be an absolute URL including scheme and host.

Those findings come from an eleven-line file whose every line reads like an instruction: User-agent: GPT-Bot, Disallow: *, Noindex: /private, Sitemap: /sitemap.xml. Not one of them is.

The names are wrong

There is no crawler called GPT-Bot. The real token is GPTBot, with no hyphen. Claude-Web was real once and was retired in 2024, when Anthropic consolidated on ClaudeBot. robots.txt ignores unknown user-agent tokens silently: no warning, no error, no log line. The file looks correct and blocks nothing.

Assistants produce these names because the crawler names changed after their training data was collected - which is why you cannot ask a chatbot to write the file that governs chatbots.

Training and citation are two different taps

Blocking a training crawler and blocking a citation crawler look identical in the file, and do opposite things to your business:

User-agent: GPTBot          # training  - your text feeds a future model
User-agent: OAI-SearchBot   # citation  - your link appears inside ChatGPT answers
User-agent: ChatGPT-User    # a reader asked ChatGPT to open your page

GPTBot trains. OAI-SearchBot is how ChatGPT finds and cites you. Anthropic splits the same way: ClaudeBot trains, Claude-SearchBot cites, Claude-User fetches when a person asks.

So someone who pastes "block the AI bots" and disallows the whole family keeps the training exposure they were worried about on the crawlers they missed, and deletes themselves from the surfaces that were sending readers.

Google-Extended does not do what its name suggests

Google-Extended is not a crawler but a control token - you will never see it as a request in your logs. Google documents that it only opts content out of Gemini model training and grounding, that it does not affect inclusion in Google Search, and that it is not a ranking signal. AI Overviews are served from Googlebot. So blocking it does nothing about AI Overviews; the only way out is to block Googlebot, which removes you from Google Search entirely. Applebot-Extended is the same shape.

The wildcard group is not a base layer

User-agent: * is not inherited. A named group replaces it rather than merging with it, so adding a group for GPTBot makes that crawler ignore every rule under the wildcard. And Crawl-delay is not a fix: Googlebot ignores it, though Bing and several AI crawlers honour it.

Check it instead of trusting it

I built the check into the editor: AI Crawler Rules for robots.txt, a VS Code extension that reads the file line by line and says what each line actually controls. It carries nineteen rules, six of them hard errors, with every token taken from the vendors' own crawler documentation. There is a free web version too.

Checking the open file is free and complete: every rule runs, every finding is shown, nothing is hidden or watermarked. A licence is $29 once and adds a different job rather than more of the same one - scanning every robots.txt in a workspace, exporting the report for a client, failing CI when an error appears, and re-checking on save. A freelance technical SEO consultant bills roughly $100-150 an hour, and an AI-crawler robots.txt review is a one-to-two hour job per site.

Open your own robots.txt and search it for a hyphen in GPT-Bot, and for the word Claude-Web.