AI bots do three different jobs
The first kind collects pages to train the models behind an assistant. What it reads may shape what a future model knows, but the companies treat it as a separate matter from whether you appear in answers. GPTBot from OpenAI and ClaudeBot from Anthropic are this kind.
The second kind builds a search index: the store of pages an assistant looks through when it searches the web before answering. For a small business this is the one that counts. Turn it away and you are, at best, harder to find in the answers people read.
The third kind is not really a crawler. It fetches a single page because somebody asked an assistant a question that needed it, like a visitor pasting your web address into ChatGPT to ask whether your café in Aberaeron opens on Sunday mornings. That is a person reading your page through a machine, at the moment they want to know.
A robots.txt file addresses each kind by the name its company publishes, called a user-agent token, and RFC 9309, the standard for the file, tells a crawler to obey the group of rules that names it.
Which bot is which, company by company
OpenAI runs three. GPTBot collects for training. OAI-SearchBot finds the sites shown in ChatGPT’s search features, and OpenAI says a site that refuses it will not appear in those answers, though it can still show up as a plain link. ChatGPT-User is the one that visits when a person asks. OpenAI calls each setting independent of the others, so you can let OAI-SearchBot in and keep GPTBot out.
Anthropic follows the same pattern: ClaudeBot for training, Claude-SearchBot for the index behind Claude’s search results, and Claude-User for pages fetched when somebody asks Claude a question. Anthropic says turning away either of the last two may make your site less visible when people search through Claude.
Perplexity names two. PerplexityBot builds its search index, and Perplexity says it is not used to crawl content for training AI foundation models. Perplexity-User fetches a page when answering somebody’s question calls for it.
Google and Apple control AI use with tokens rather than separate crawlers. Google-Extended is not a crawler of its own: Google’s usual crawlers fetch the pages, and the token decides whether what they collect may train Gemini models and be drawn on for answers in the Gemini apps. Google says it has no effect on your inclusion or ranking in Search, which rests on Googlebot, AI Overviews included. Applebot crawls for search in Spotlight, Siri and Safari, and Applebot-Extended lets you decline the training of Apple’s foundation models while staying in those results.
The lines to write, for each choice
To be read by everything, you need no rules at all. Under RFC 9309 a missing file lets crawlers fetch anything, and a page that no rule matches is allowed. To write the welcome down anyway, use User-agent: * followed by Allow: /.
To stay in answers but out of training, put the four training tokens in one group under a single rule: User-agent: GPTBot, User-agent: ClaudeBot, User-agent: Google-Extended and User-agent: Applebot-Extended, each on its own line, then Disallow: /. The standard lets several user-agent lines share one rule, and the search crawlers and fetchers are untouched. The one cost to weigh is Google’s, because that token also decides whether the Gemini apps can draw on your pages when they answer.
The line not to write is User-agent: * followed by Disallow: /. It turns away every crawler that has no group of its own, Googlebot and the search crawlers among them. It is sometimes left over from building a site out of public view, and if nobody chose it, fix that first.
A named group replaces the general one. A crawler that finds a group naming it follows that group alone and ignores the rules written for every crawler. If you add a group just to welcome OAI-SearchBot, copy into it any rule you still want it to follow, such as keeping out of your basket.
What a robots.txt block does not do
It does not lock anything. RFC 9309 says plainly that its rules are not a form of access authorisation: they are requests, which crawlers that follow the standard agree to honour. Anyone can read the file, too, so listing a folder there to hide it shows people exactly where to look. Anything private belongs behind a password.
It may not stop a fetcher acting for a person. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user started the visit, and Perplexity says Perplexity-User generally ignores them for the same reason. Anthropic is the exception: it says Claude-User lets site owners control which sites these requests reach. Either way, the one who loses out is a customer already asking about you.
It does not reach backwards. Anthropic is explicit that restricting ClaudeBot signals that your future material should be left out of training, which says nothing about what was gathered before. Nor is it instant: OpenAI and Perplexity both say a change can take up to about a day to reach their systems.
Check your own file in ten minutes
Type your web address followed by /robots.txt into a browser. The standard puts the file there, at the top of the site, and gives every host its own, so a shop on a subdomain, or a Welsh site on a .cymru address beside an English one on .wales, has a separate file to check.
A “not found” page is fine, because nothing is refused. A server error is not. RFC 9309 tells a crawler that cannot reach the file because of a server fault to assume the whole site is closed, so a host that fails on that one address can hide a whole site by accident.
A website builder, a plugin or a setting at your host may write or change this file for you. If a line below is there that nobody chose, ask whoever manages the site to take it out.
Or run the free assessment on this site, which reads your robots.txt and names the AI crawlers it finds refused, so you can see whether the file says what you decided.
- No
User-agent: *group withDisallow: /that nobody chose OAI-SearchBot,Claude-SearchBot,PerplexityBot,GooglebotandApplebotleft free to read- Training tokens refused only if refusing them was your decision
- No
Disallowline covering your Welsh pages, such as/cy/ - A file checked on every subdomain and every separate language address
Frequently asked questions
Should I block GPTBot?
Only if you do not want your pages used to train OpenAI’s models. GPTBot is OpenAI’s training crawler, and OpenAI says its settings are independent, so refusing it leaves OAI-SearchBot, the crawler behind ChatGPT search, free to read you. Mix the two up and you lose the answers while the training crawler carries on.
Will blocking AI bots take my site out of Google?
Not if the bot you block is Google-Extended. Google says that token has no effect on whether or where you appear in Search, and AI Overviews are a Search feature whose supporting links come from pages Google has indexed. Blocking Googlebot is another matter: it reads your pages for Search, and so for AI Overviews too.
Does blocking a training crawler remove what it has already collected?
None of the sources listed here says so. Anthropic describes a block as covering your material from now on, not what was taken before. Treat it as a decision about the future.
What about Microsoft Copilot?
Microsoft’s Bing Webmaster Guidelines put the control in a tag on each page rather than in robots.txt: noarchive keeps a page’s content out of Copilot responses. Bingbot is Bing’s own search crawler, so refusing it is a decision about far more than AI.
Do I need a robots.txt file at all?
No. Without one, crawlers that follow the standard may read everything, which is what a business that wants to be found is after. You need the file only when there is something to refuse.