The Measurement Observatory.
- sites measured
- 160
- responding
- 129
- measured on
- sample list
- Tranco Q2K34
A periodic measurement of web health across the most visited sites in Türkiye and Europe. No verdicts: only measured values, the method, the date, and the command that reproduces the same numbers.
Türkiye Machine Readability Index
Bot-specific server response
On these sites an ordinary browser and a bot-shaped control probe both receive a response, while the named AI crawler receives a different status code. Because both counter-probes pass, the response depends on the agent identity.
Some sites differentiate: they respond to certain agents but not others. In the table this appears as a difference between the codes within a row.
| Site | Rank | Control | Placebo | GPTBot | OAI-SearchBot | ClaudeBot | PerplexityBot | Google-Extended |
|---|---|---|---|---|---|---|---|---|
| welt.de | 807 | 200 | 200 | 200 | 200 | 200 | 403 | 403 |
| lefigaro.fr | 836 | 200 | 200 | 200 | 200 | 200 | 403 | 403 |
| corriere.it | 971 | 200 | 200 | 403 | 200 | 403 | 200 | 200 |
| wetteronline.de | 1,115 | 200 | 200 | 403 | 403 | 403 | 403 | 200 |
| usercentrics.eu | 1,125 | 200 | 200 | 200 | 200 | 429 | 200 | 200 |
| gazzetta.it | 1,228 | 200 | 200 | 403 | 200 | 403 | 200 | 200 |
| spiegel.de | 1,236 | 200 | 200 | 470 | 200 | 470 | 200 | 200 |
| jutarnji.hr | 1,426 | 200 | 200 | 403 | 200 | 403 | 200 | 200 |
| xvideos.es | 1,495 | 200 | 200 | 403 | 200 | 403 | 403 | 200 |
| t24.com.tr | 7,331 | 200 | 200 | 403 | 200 | 403 | 200 | 200 |
Sites where robots.txt and the server response diverge
The robots.txt file on these sites does not block the named agent; the server nevertheless returns a different status code. robots.txt is a declaration, and the layer that answers is usually a different one.
| Site | Agent | robots.txt | Measurement | Control |
|---|---|---|---|---|
| wetteronline.de | GPTBot | allowed in the catch-all group | 403 | 200 |
| wetteronline.de | OAI-SearchBot | allowed in the catch-all group | 403 | 200 |
| wetteronline.de | ClaudeBot | allowed in the catch-all group | 403 | 200 |
| wetteronline.de | PerplexityBot | allowed in the catch-all group | 403 | 200 |
| usercentrics.eu | ClaudeBot | allowed in the catch-all group | 429 | 200 |
Aggregate measurements
Counted across all responding sites. No individual site is named.
| Measurement | Sites | Share |
|---|---|---|
| carries a valid robots.txt | 117 | 90.7% |
| carries JSON-LD on the home page | 48 | 37.2% |
| carries an hreflang set | 14 | 10.9% |
| sends an HSTS header | 82 | 63.6% |
| serves an llms.txt file | 24 | 18.6% |
| GPTBot — blocked in robots.txt | 27 | 20.9% |
| OAI-SearchBot — blocked in robots.txt | 16 | 12.4% |
| ClaudeBot — blocked in robots.txt | 25 | 19.4% |
| PerplexityBot — blocked in robots.txt | 22 | 17.1% |
| Google-Extended — blocked in robots.txt | 19 | 14.7% |
Data
The full output of each run is published in machine-readable form and is never deleted. Earlier runs stay in place; that is the only way a time series is built.
The data is licensed under CC BY 4.0: free to use, share and adapt as long as the source is credited.
Suggested citation
murattunalı. (2026). The Measurement Observatory [Data set]. https://www.murattunali.com.tr/en/observatory/ · measured on 2026-10-01
- @misc{murattunali_rasathane_2026_10_01,
- title = {The Measurement Observatory},
- author = {{murattunalı.}},
- year = {2026},
- url = {https://
www.murattunali.com.tr /en /observatory /}, - note = {2026-10-01 · CC BY 4.0}
- }