Türkiye Machine Readability Index.

The home page, robots.txt and llms.txt of the first 500 domains ending in .tr on the Tranco popularity list, measured on the same day with the same method: what is declared to AI crawlers, how the server actually answers them, and which structural signals search engines receive. No verdicts; only counts, the method and the data.

Findings

1. Of the 412 responding sites, 31 (7.5%) turn away at least one AI crawler at the server, while the same address answers an ordinary browser and a bot-shaped control probe.
2. The most common pattern is to turn away only GPTBot and ClaudeBot: 18 of the 31 sites turn these two crawlers away and let OAI-SearchBot and PerplexityBot in. The companies' own documentation describes the first two as crawlers that collect content which may be used to train models, and the other two as crawlers that surface sites in search results.
3. Of the 31 sites that block at the server, 23 send a Cloudflare server header; so do 17 of the 18 sites that turn away only GPTBot and ClaudeBot.
4. On 18 sites, robots.txt allows the crawler that the server turns away: the declaration in the file and the server’s answer diverge. The number of sites that also declare the block in robots.txt is 5.
5. 89 sites (21.6%) publish an llms.txt that follows the llmstxt.org proposal, a text file whose first line is a level-one heading; the share is 28.8% on com.tr, 19.0% on gov.tr and 5.3% on edu.tr. The file’s effect is not what this measurement covers; only its presence and format were counted.
6. The share of sites carrying structured data (JSON-LD) on the home page is 45.4%: 60.3% on com.tr and 20.3% on gov.tr. The lang attribute that declares the home page’s language is missing on 21 of the 79 gov.tr sites.
7. The share of sites that declare the language versions of their pages with hreflang is 19.4%, and 9.5% use x-default; the highest share is on edu.tr: 25.0%.

By domain extension

In each column the denominator is the number of responding sites with that extension (the number in the header). com.tr is the extension for commercial organisations, gov.tr for public bodies and edu.tr for higher education; the last column combines org.tr, net.tr, bel.tr and the remaining .tr extensions.

Per crawler

The declaration is a root-level block written under the crawler’s name in robots.txt. The server response is the crawler receiving a different status code while the ordinary browser and the placebo probe pass. Google-Extended is not a crawler that arrives with its own request identity but a robots.txt token; for it only the declaration is counted. The denominator is the 412 responding sites.

Sites that could not be measured

88 sites could not be measured and are not in the denominator; shares are calculated from responding sites only.

Method

The sample is the first 500 domains ending in .tr in version 26Y29 of the Tranco list (2026-10-06); the list combines the Chrome UX Report, Cloudflare Radar, Cisco Umbrella, Majestic and Farsight rankings. The measurement was taken on 2026-10-07 from a single location.

Each site received at most three requests under our own identity (home page, robots.txt, llms.txt), followed by an ordinary browser probe, a bot-shaped placebo probe and one probe per crawler, all sent to the same address the redirects arrive at. When a crawler receives a different status code while the ordinary browser and the placebo pass, the block was counted as specific to that crawler.

An llms.txt was counted only if its first non-empty line is a level-one heading (the one mandatory rule of the llmstxt.org proposal); this check was made on the same day with one additional request to each site that had the file. A home page without a description and without JSON-LD was also requested once more that day: if the response carried a bot-challenge marker and the challenge title had the same length as the first response’s title, the site was removed from the denominator. A single run measures a moment: CDN decisions can vary by region and time.

Method

Data