How far do Turkish websites close their doors to AI bots?

The short version

• Explicit bans are few but rising fast: of 414 live sites, 24 explicitly disallow GPTBot in robots.txt and 27 disallow ClaudeBot. On 7 October these figures were 16 and 18. The number of sites disallowing Google-Extended, the token Google uses for Gemini training, rose from 7 to 16.
• The new bans come from Cloudflare: of the 10 sites that newly disallowed GPTBot in robots.txt within 25 days, in 10 cases the ban comes from the managed section that Cloudflare adds to the file. The section is present on 11 of the sites we measured.
• The server closes more than the file does: 36 sites turn GPTBot away at the server layer; on 12 of them, robots.txt does not block GPTBot at all. In other words, what the site owner writes in the file and what the site does diverge.
• Search bots are less affected: the number of sites explicitly disallowing OAI-SearchBot, the bot behind ChatGPT search, stayed at 8.

Sample and method

The sample is the first 500 domains ending in .tr in the 6 October 2026 version of the Tranco list, which researchers use. The same list is also the sample of the Türkiye Machine Readability Index, so the results can be compared with the 7 October measurement site by site. On 1 November, 414 of the 500 domains returned a live site.

For each site we measured two things separately:

1. The declared policy. We read the robots.txt file and resolved the rule for five AI user-agent tokens: GPTBot (OpenAI’s training crawler), OAI-SearchBot (ChatGPT search), ClaudeBot (Anthropic), PerplexityBot and Google-Extended (the robots.txt token Google uses for Gemini training, not a separate crawler). We also checked whether the file carries Cloudflare’s managed section (# BEGIN Cloudflare Managed content) and a content signals line.
2. The enforced policy. We sent one request to the homepage under each bot name and compared the server’s response with its response to an ordinary browser request and to a random bot name. When the ordinary request and the random bot got through but only the AI bot was turned away, we counted it as an “AI-specific block”. This separated general bot protection from a decision about AI; 3 sites were left out of this count because they turned away every bot name.

Our requests did not hide their identity: next to the bot name we added the address of the page that runs the measurement. We sent at most three requests of our own per site and one probe request per bot; no page was crawled and no content was stored.

Results

• GPTBot · explicit ban in robots.txt 16 → 24 · AI-specific block at the server 28 → 36 · of these, not blocked by robots.txt 15 → 12
• ClaudeBot · explicit ban in robots.txt 18 → 27 · AI-specific block at the server 30 → 37 · of these, not blocked by robots.txt 18 → 14
• PerplexityBot · explicit ban in robots.txt 9 → 9 · AI-specific block at the server 10 → 16 · of these, not blocked by robots.txt 4 → 5
• OAI-SearchBot · explicit ban in robots.txt 8 → 8 · AI-specific block at the server 10 → 14 · of these, not blocked by robots.txt 2 → 2
• Google-Extended · explicit ban in robots.txt 7 → 16 · AI-specific block at the server 1 → 2 · of these, not blocked by robots.txt 0 → 1
• Base: 416 live sites on 7 October, 414 on 1 November.

Explicit bans are rising, but not by hand. Within 25 days, 10 sites newly disallowed GPTBot in their robots.txt; 10 of them have Cloudflare’s managed section in the file. The same holds for ClaudeBot, where 10 of the 11 sites with a new ban have it, and for Google-Extended, with 10 of 10. When this Cloudflare setting is on, disallow lines for known training crawlers are added to the top of the file. In other words, most of the bans come not from site owners editing their robots.txt but from a setting in a network service.

The file and the server diverge. Of the 36 sites that turn GPTBot away at the server, 12 disallow nothing to GPTBot in robots.txt. On these sites the decision lies not in the file but in a rule of the server or of the security layer in front of it. Someone looking at robots.txt may think the site is open to bots; the bot, meanwhile, is turned back at the door. Of the sites that diverged in this way on 7 October, 5 now state the GPTBot ban in robots.txt as well; 2 sites have newly diverged.

Search and training part ways. The number of sites explicitly disallowing OAI-SearchBot, the bot behind ChatGPT search, did not change (8); the number disallowing the training token Google-Extended more than doubled (7 → 16). On all 11 sites that carry the managed section, GPTBot, ClaudeBot and Google-Extended are disallowed; PerplexityBot stays open on 11 of them and OAI-SearchBot on 9. Closing a search bot also closes visibility in that engine’s answers.

By domain extension. An AI-specific block at the server appears on 12 per cent of .com.tr sites, 10 per cent of .gov.tr sites and 3 per cent of .edu.tr sites. The share that explicitly disallows at least one AI bot in robots.txt is 9 per cent for .com.tr, 4 per cent for .edu.tr and 3 per cent for .gov.tr.

The limits of this measurement

• Two measurements show two moments, not a trend; the 25-day difference rests on small numbers.
• The measurement was taken from a single location; network services may decide differently depending on where a request comes from.
• A bot’s real request may look different from our probe (IP address, verification); the server may treat the real bot differently.
• A 403 response is not always a decision about AI; that is why we used the ordinary and random-bot requests as controls.
• We read the presence of the managed section from the text of the file; we cannot see the site’s other settings in the Cloudflare dashboard.

For your own site

Open your robots.txt in the browser and search for the line “# BEGIN Cloudflare Managed content”; if it is there, the rules on AI bots may come from a section you did not write. We explain what is decided where in our guide to AI crawlers and robots.txt.