AI crawlers and robots.txt.
robots.txt is the plain-text file that declares which crawler may enter which parts of a site, and the same rules apply to AI crawlers.
The first layer of answer engine visibility is access, and the gate of access is robots.txt. If this layer is broken, everything else — structure, schema, content quality — runs on empty. The first step of answer engine optimization is therefore making sure the crawlers can actually get in.
The good news: on most sites access is already open, because a general allow rule covers all crawlers. The bad news is that this is usually a default rather than a deliberate decision — and defaults can quietly change one day.
Two kinds of agent, two different decisions
The crawlers of AI companies do not do a single job, and knowing the distinction lets you see which decision affects what. There are roughly three categories.
- Training crawlers — collect content for model training. Blocking them affects future model versions, not current visibility.
- Index/search crawlers — crawl for the answer engine’s own index. Blocking them shuts down visibility in that engine directly.
- User-triggered fetchers — fetch on demand when a user asks about a specific page or opens a link. Blocking them stops the user from reading that page through the model.
The critical point: training permission and search permission are NOT THE SAME THING. A publisher who lumps them together and blocks in bulk, acting out of copyright concern, shuts down their own visibility without noticing. The decisions can be made separately, and they should be.
The implementation on this site
In this site’s robots.txt seventeen agents are written out one by one, and all are allowed. The general rule already covered them; two reasons for writing the groups explicitly are on record.
The first is technical: some crawlers, when reading the file, honor only the group written under their own name and skip the general rule. A named block guarantees against that behavior.
The second is documentation: intent in writing means the place is ready should a single bot ever need restricting. One line closes an agent, and the decision stays visible in the file.
- Search side — Googlebot, Bingbot, YandexBot, Applebot.
- Generative side, OpenAI — GPTBot, OAI-SearchBot, ChatGPT-User.
- Generative side, Anthropic — ClaudeBot, Claude-User, Claude-SearchBot.
- Other generative — PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot.
Google-Extended is a special case on the list: it is not a separate agent from Googlebot but a control switch that determines whether content may be used in Google’s generative products. Blocking it does not affect classic search ranking, but it closes use in generative features.
Common mistakes
robots.txt is a simple file, but it fails silently in a few places, and because the consequences are invisible they go unnoticed for months.
Wrong location. The file must sit at the site root; a copy in a subdirectory or on a subdomain is not read. Subdomains need robots.txt files of their own.
Blocked resource files. Blocking CSS and JavaScript files stops the crawler from seeing the page as it is. This has long been advised against for search engines.
Confusing blocking with de-indexing. robots.txt blocks crawling, not indexing. A page that receives links from elsewhere can appear in the index even without being crawled. To remove a page from the index, it must be crawlable and carry an indexing directive of its own.
Forgetting the sitemap line. One of the file’s most practical functions is declaring the sitemap address; without it, discovery depends entirely on links and manual submission.
robots.txt blocks crawling, not indexing — confuse the two and the page is neither crawled nor removed.
Content signals
A recent layer added to robots.txt declares USAGE intent beyond crawling: whether content may enter the search index, whether it may be used in generative answers, and whether it may be used in model training can each be stated separately.
These signals are not a technical block but a declaration of rights — compliance depends on the crawler’s own policy. Stating the intent still has value, especially as the legal side of the copyright debate develops.
One warning: this area is changing fast, and today’s syntax may differ tomorrow. Every line added to the file should be understood; a rule copied but not understood can close visibility by accident.
The real cost of blocking
When deciding to block an agent, you need to see clearly what is being given up. Blocking a training crawler affects the future and does not touch today’s visibility. Blocking a search crawler means becoming invisible in that engine from today on — and when reversed, the effect does not return immediately; re-crawling and re-indexing take time.
That asymmetry shapes the decision: training permission is always reversible; search permission accumulates a cost while closed. When in doubt, keeping search permissions open and reviewing training permissions is the lower-risk start.
There is also a measurement side: the only way to see the effect of a blocking decision is the server logs. If the agent stops coming, the rule works; if it still comes, either the rule is written wrong or that crawler ignores the file. The second case is real and must be known — robots.txt is a request, not a technical barrier.
Auditing: seeing that the file actually works
Assuming that a rule written into robots.txt works is not enough; it must be verified. The simplest verification is making sure the file is actually live — at the root, as plain text, with a 200 response. On a surprising number of sites the file is either missing or served with the wrong content type.
The second verification comes from the server logs: after the rule was written, did the agent’s behavior change? If a blocked agent keeps coming, either the rule is written wrong or that crawler ignores the file. Both are things worth knowing.
The third, and most often skipped: not verifying the file once and leaving it. When the server configuration changes, a security layer is added, or a deploy goes wrong, the file can silently disappear. A regular check — or an automated test — catches it.