murattunalı.
Reading

AI crawlers and robots.txt.

robots.txt is the plain-text file that declares which crawler may enter which parts of a site, and the same rules apply to AI crawlers.

The first layer of answer engine visibility is access, and the gate of access is robots.txt. If this layer is broken, everything else — structure, schema, content quality — runs on empty. The first step of answer engine optimization is therefore making sure the crawlers can actually get in.

The good news: on most sites access is already open, because a general allow rule covers all crawlers. The bad news is that this is usually a default rather than a deliberate decision, and defaults can quietly change one day.

Two kinds of agent, two different decisions

The crawlers of AI companies do not do a single job, and knowing the distinction lets you see which decision affects what. There are roughly three categories.

  • Training crawlers — collect content for model training. Blocking them affects future model versions, not current visibility.
  • Index/search crawlers — crawl for the answer engine’s own index. Blocking them shuts down visibility in that engine directly.
  • User-triggered fetchers — fetch on demand when a user asks about a specific page or opens a link. Blocking them stops the user from reading that page through the model.

The critical point: training permission and search permission are NOT THE SAME THING. A publisher who lumps them together and blocks in bulk, acting out of copyright concern, shuts down their own visibility without noticing. The decisions can be made separately, and they should be.

The implementation on this site

In this site’s robots.txt we list seventeen agents one by one and allow all of them. The general rule already covered them; we have recorded two reasons for writing the groups out explicitly.

The first is technical: some crawlers, when reading the file, honor only the group written under their own name and skip the general rule. A named block guarantees against that behavior.

The second is documentation: intent in writing means the place is ready should a single bot ever need restricting. One line closes an agent, and the decision stays visible in the file.

  • Search side — Googlebot, Bingbot, YandexBot, Applebot.
  • Generative side, OpenAI — GPTBot, OAI-SearchBot, ChatGPT-User.
  • Generative side, Anthropic — ClaudeBot, Claude-User, Claude-SearchBot.
  • Other generative — PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot.

Google-Extended is a special case on the list: it is not a separate agent from Googlebot but a control token. According to Google, blocking it limits the use of your content for training Gemini models (including the models that generate answers in Search) and in Gemini Apps and Vertex AI; it does not change whether your site appears in Search — in classic results, AI Overviews or AI Mode. The way out of Search’s AI features is not robots.txt but the generative AI control in Search Console (AI Overviews guide).

If you use Cloudflare: the gate in front of robots.txt

robots.txt is a request; a network layer such as Cloudflare can stop a request before it reaches your server. So an AI crawler may be unable to reach your site even when your robots.txt allows it. In July 2025 Cloudflare announced that it was moving to block AI crawlers by default, and since July 2026 it has offered three separate categories instead of a single “block AI bots” switch.

  • Search — crawlers that index your content so they can answer questions about it later.
  • Agent — automated activity acting in real time on a person’s behalf: a chat fetching a page, browser-use agents.
  • Training — crawlers that take your content to train or fine-tune a model.

Each category has three options: block on all pages, block only on pages that display ads, or don’t block. Since September 15, 2026, new domains added to Cloudflare get this default: training and agent bots are blocked on pages that display ads, while search stays allowed. Multi-purpose crawlers that work for both search and training can be affected by settings that turn off training.

If Cloudflare’s managed robots.txt setting is on, it prepends its own section to your file: content signals and disallow lines for known training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider). The section starts with the line “# BEGIN Cloudflare Managed content”; open your site’s robots.txt in the browser and search for that line to spot it. But the file itself is still a statement of preference; enforcement happens through the block in Cloudflare’s AI Crawl Control settings.

Three steps are enough to check: open robots.txt and look for the managed section; in the Cloudflare dashboard, check AI Crawl Control for which crawlers are coming and which are blocked; confirm in the server logs whether the agents in question actually arrive. Turning off the search category also turns off visibility in that engine — the asymmetry described below applies here too.

Common mistakes

robots.txt is a simple file, but it fails silently in a few places, and because the consequences are invisible they go unnoticed for months.

Wrong location. The file must sit at the site root; a copy in a subdirectory or on a subdomain is not read. Subdomains need robots.txt files of their own.

Blocked resource files. Blocking CSS and JavaScript files stops the crawler from seeing the page as it is. This has long been advised against for search engines.

Confusing blocking with de-indexing. robots.txt blocks crawling, not indexing. A page that receives links from elsewhere can appear in the index even without being crawled. To remove a page from the index, it must be crawlable and carry an indexing directive of its own.

Forgetting the sitemap line. One of the file’s most practical functions is declaring the sitemap address; without it, discovery depends entirely on links and manual submission.

robots.txt blocks crawling, not indexing — confuse the two and the page is neither crawled nor removed.

Content signals

A recent layer added to robots.txt declares USAGE intent beyond crawling: whether content may enter the search index, whether it may be used in generative answers, and whether it may be used in model training can each be stated separately. A common example is the content signals policy that Cloudflare adds to robots.txt: search, AI input and AI training uses are stated separately.

These signals are not a technical block but a declaration of rights — compliance depends on the crawler’s own policy. Stating the intent still has value, especially as the legal side of the copyright debate develops.

One warning: this area is changing fast, and today’s syntax may differ tomorrow. Every line added to the file should be understood; a rule copied but not understood can close visibility by accident.

The real cost of blocking

When deciding to block an agent, you need to see clearly what is being given up. Blocking a training crawler affects the future and does not touch today’s visibility. Blocking a search crawler means becoming invisible in that engine from today on — and when reversed, the effect does not return immediately; re-crawling and re-indexing take time.

That asymmetry shapes the decision: training permission is always reversible; search permission accumulates a cost while closed. When in doubt, keeping search permissions open and reviewing training permissions is the lower-risk start.

There is also a measurement side: the only way to see the effect of a blocking decision is the server logs. If the agent stops coming, the rule works; if it still comes, either the rule is written wrong or that crawler ignores the file. The second case is real and must be known — robots.txt is a request, not a technical barrier.

Auditing: seeing that the file actually works

Do not assume that a rule you wrote into robots.txt works; verify it. The simplest verification is making sure the file is actually live — at the root, as plain text, with a 200 response. On a surprising number of sites the file is either missing or served with the wrong content type.

Do the second verification in the server logs: after you wrote the rule, did the agent’s behavior change? If a blocked agent keeps coming, either the rule is written wrong or that crawler ignores the file. Both are things worth knowing.

The third, and most often skipped: not verifying the file once and leaving it. When the server configuration changes, a security layer is added, or a deploy goes wrong, the file can silently disappear. A regular check — or an automated test — catches it.

If you want to know how your brand is cited in ChatGPT, Google AI Overviews and Perplexity, our GEO consulting records which questions and engines cite you and builds the prerequisites.

How to use robots.txt generator for AI bots

SOURCES