Top level domains with the highest share of sites blocking AI training bots (100,000+ sites with robots.txt)
TLD
Sites with robots.txt
Block AI training
.top
188,678
21.5%
.id
194,978
12.6%
.shop
170,858
11.4%
.edu
136,560
9.3%
.tr
156,624
8.6%
.net
1,866,156
7.1%
.ai
129,142
6.8%
.tw
128,072
6.6%
.in
389,210
6.5%
.kr
190,759
6.5%
.biz
113,578
6.2%
.org
2,827,689
6%
.online
145,703
5.9%
.ro
185,390
5.6%
.gr
175,510
5.4%
Blocking over time
Share of sites that fully block each crawler, per Common Crawl crawl.
Solid: GPTBot. Dashed and dotted lines: ClaudeBot, CCBot, Google-Extended (see table).
Share of sites fully blocking each crawler, by crawl (%)
Crawl
GPTBot
ClaudeBot
CCBot
Google-Extended
CC-MAIN-2026-12
2.45
2.09
2.01
1.91
CC-MAIN-2026-17
2.76
2.35
2.29
2.18
CC-MAIN-2026-21
3.03
2.63
2.58
2.46
CC-MAIN-2026-25
3.09
2.7
2.65
2.53
CC-MAIN-2026-30
3.37
2.94
2.93
2.77
CC-MAIN-2026-34
3.52
3.07
3.07
2.9
CC-MAIN-2026-39
3.38
2.96
2.96
2.75
Fresh live sample: popular sites
In October 2026 we fetched robots.txt live, as sus.bot, from 20,000 hostnames in the Tranco top 1 million. 6,529 returned a robots.txt. These are popular sites, so they block AI crawlers far more often than the web as a whole.
AI crawlers most blocked in the live sample of popular sites