robots.txt generator for AI bots.
The robots.txt generator for AI bots creates a section for your robots.txt that allows or blocks, separately, the bots that come for model training, for AI search and on a user’s request.
How to use
- Choose allow or block for each of the three categories
- Add the address of your sitemap if you like
- Press Generate; add the section to the end of your robots.txt and keep the existing rules
The tool runs entirely in your browser — no data is sent, no cookies are written.
Three categories, three separate decisions
AI companies' bots do not do a single job. Some collect content to train models, some build an index for the company’s AI search, and some fetch a page the moment a user asks a question. A blanket block that lumps all three together means a decision taken out of copyright concerns also switches off search visibility without anyone noticing. The distinction is covered in detail in the guide to AI crawlers and robots.txt.
For a category you allow, the tool writes no separate group. In robots.txt, a bot that finds a group carrying its own name reads only that group and ignores the general `User-agent: *` rules; a separate `Allow: /` group for “allow” could therefore open to that bot the paths you otherwise disallow.
Bots that come on a user’s request
Some bots in this category state in their own documentation that they may not follow robots.txt rules, because a person started the request. Blocking them in robots.txt is a statement of intent, not a hard block.
The training preference signal (2027)
A new signal that states only your training preference in robots.txt has been proposed: `User-Agent: __usage_ai-training`. It is still an IETF draft; Google has announced that it will read it from early 2027 and Microsoft from 1 February 2027, while Anthropic says it already recognises it. The general `User-Agent: *` rule does not cover this signal.
Not on the list
No crawler documentation from the provider itself could be found for Bytespider, xAI or DeepSeek; the list carries only documented bots. CCBot works for an open crawl archive and its documentation does not mention training; if you want to block it, you can add it to your file separately.
Why the file may not be enough
robots.txt is a request, not a lock. A network layer such as Cloudflare can stop a request before it reaches your server; even if your file allows it, the bot may not get in. Once the file is published, you can read your site’s robots.txt bot by bot with the machine reading tool.
The tool sends no request to any site; the section is generated in your browser.