opencrawlers

Who blocks AI crawlers?

We read every robots.txt in 8 Common Crawl crawls. Here is what 67.9M sites tell AI crawlers.

Most blocked AI crawlers

Top 10 AI crawlers by sites that block them fully (Disallow: /)
CrawlerOperatorTypeSites naming itBlock it fullyShare of naming sites that block
GPTBotOpenAIai training4,002,0472,591,83464.8%
AmazonbotAmazonai training3,516,8532,456,30369.8%
BytespiderByteDanceai training3,396,1882,316,63368.2%
ClaudeBotAnthropicai training3,570,0702,258,96463.3%
CCBotCommon Crawlai training3,367,4812,243,50966.6%
Google-ExtendedGoogleai training3,283,9772,111,81964.3%
meta-externalagentMetaai training3,184,3012,108,75966.2%
applebot-extendedAppleai training3,036,5591,959,28564.5%
chatgpt-userOpenAIai assistant690,278306,52944.4%
facebookbotMetaai training1,295,241295,04522.8%

See all 46 AI crawlers

Blocking by country domain

Top level domains with the highest share of sites blocking AI training bots (100,000+ sites with robots.txt)
TLDSites with robots.txtBlock AI training
.top188,67821.5%
.id194,97812.6%
.shop170,85811.4%
.edu136,5609.3%
.tr156,6248.6%
.net1,866,1567.1%
.ai129,1426.8%
.tw128,0726.6%
.in389,2106.5%
.kr190,7596.5%
.biz113,5786.2%
.org2,827,6896%
.online145,7035.9%
.ro185,3905.6%
.gr175,5105.4%

Blocking over time

Share of sites that fully block each crawler, per Common Crawl crawl.

Line chart: share of sites fully blocking AI crawlers per crawl Exact values are in the table below the chart.
Solid: GPTBot. Dashed and dotted lines: ClaudeBot, CCBot, Google-Extended (see table).
Share of sites fully blocking each crawler, by crawl (%)
CrawlGPTBotClaudeBotCCBotGoogle-Extended
CC-MAIN-2026-122.452.092.011.91
CC-MAIN-2026-172.762.352.292.18
CC-MAIN-2026-213.032.632.582.46
CC-MAIN-2026-253.092.72.652.53
CC-MAIN-2026-303.372.942.932.77
CC-MAIN-2026-343.523.073.072.9
CC-MAIN-2026-393.382.962.962.75

Fresh live sample: popular sites

In October 2026 we fetched robots.txt live, as sus.bot, from 20,000 hostnames in the Tranco top 1 million. 6,529 returned a robots.txt. These are popular sites, so they block AI crawlers far more often than the web as a whole.

AI crawlers most blocked in the live sample of popular sites
CrawlerOperatorSites naming itBlock it fullyShare that block
CCBotCommon Crawl86762772.3%
BytespiderByteDance74360080.8%
GPTBotOpenAI1,00458658.4%
ClaudeBotAnthropic88653760.6%
AmazonbotAmazon61743670.7%
applebot-extendedApple60541668.8%
Google-ExtendedGoogle76340853.5%
meta-externalagentMeta61040165.7%
diffbotDiffbot41935083.5%
anthropic-aiAnthropic53734263.7%

What we collect, and how to change your mind