The Measurement Observatory.
- sites measured
- 160
- responding
- 131
- measured on
- sample list
- Tranco 5674N
A periodic measurement of web health across the most visited sites in Türkiye and Europe. No verdicts: only measured values, the method, the date, and the command that reproduces the same numbers.
Bot-specific server response
On these sites an ordinary browser and a bot-shaped control probe both receive a response, while the named AI crawler receives a different status code. Because both counter-probes pass, the response depends on the agent identity.
Some sites differentiate: they respond to certain agents but not others. In the table this appears as a difference between the codes within a row.
| Site | Rank | Control | Placebo | GPTBot | OAI-SearchBot | ClaudeBot | PerplexityBot | Google-Extended |
|---|---|---|---|---|---|---|---|---|
| welt.de | 795 | 200 | 200 | 403 | 403 | 403 | 403 | 403 |
| spiegel.de | 828 | 200 | 200 | 403 | 200 | 200 | 200 | 200 |
| lefigaro.fr | 832 | 200 | 200 | 403 | 403 | 403 | 403 | 403 |
| corriere.it | 967 | 200 | 200 | 403 | 200 | 403 | 200 | 200 |
| franceinfo.fr | 1,106 | 200 | 200 | 403 | 403 | 403 | 403 | 403 |
| wetteronline.de | 1,112 | 200 | 200 | 403 | 403 | 403 | 403 | 200 |
| gazzetta.it | 1,233 | 200 | 200 | 403 | 200 | 403 | 200 | 200 |
| zalando.de | 1,245 | 200 | 200 | 403 | 403 | 403 | 403 | 200 |
| jutarnji.hr | 1,413 | 200 | 200 | 403 | 403 | 403 | 403 | 200 |
| t24.com.tr | 10,403 | 200 | 200 | 403 | 403 | 403 | 403 | 200 |
| hesap.com.tr | 20,084 | 200 | 200 | 403 | 403 | 403 | 403 | 403 |
Sites where robots.txt and the server response diverge
The robots.txt file on these sites does not block the named agent; the server nevertheless returns a different status code. robots.txt is a declaration, and the layer that answers is usually a different one.
| Site | Agent | robots.txt | Measurement | Control |
|---|---|---|---|---|
| wetteronline.de | GPTBot | allowed in the catch-all group | 403 | 200 |
| wetteronline.de | OAI-SearchBot | allowed in the catch-all group | 403 | 200 |
| wetteronline.de | ClaudeBot | allowed in the catch-all group | 403 | 200 |
| wetteronline.de | PerplexityBot | allowed in the catch-all group | 403 | 200 |
| hesap.com.tr | GPTBot | allowed in its own group | 403 | 200 |
| hesap.com.tr | OAI-SearchBot | allowed in its own group | 403 | 200 |
| hesap.com.tr | ClaudeBot | allowed in its own group | 403 | 200 |
| hesap.com.tr | PerplexityBot | allowed in its own group | 403 | 200 |
| hesap.com.tr | Google-Extended | allowed in its own group | 403 | 200 |
Aggregate measurements
Counted across all responding sites. No individual site is named.
| Measurement | Sites | Share |
|---|---|---|
| carries a valid robots.txt | 118 | %90.1 |
| carries JSON-LD on the home page | 47 | %35.9 |
| carries an hreflang set | 13 | %9.9 |
| sends an HSTS header | 83 | %63.4 |
| serves an llms.txt file | 23 | %17.6 |
| GPTBot — blocked in robots.txt | 28 | %21.4 |
| OAI-SearchBot — blocked in robots.txt | 17 | %13.0 |
| ClaudeBot — blocked in robots.txt | 28 | %21.4 |
| PerplexityBot — blocked in robots.txt | 25 | %19.1 |
| Google-Extended — blocked in robots.txt | 22 | %16.8 |
Data
The full output of each run is published in machine-readable form and is never deleted. Earlier runs stay in place; that is the only way a time series is built.
The data may be used freely. Attribution is appreciated, not required.